Sunday 10 a.m.–1 p.m. in
Curriculum for project-based data science classes and their building blocks
Nadia Udler, Adomous Wright, Eli Udler
Description
Machine learning, artificial intelligence and data science are
interdisciplinary subjects and therefore they are difficult to teach. We suggest educational building blocks that help students to understand machine learning software (such as scikit-learn) and create their own artificial intelligence algorithms. These building blocks are theoretical components of global optimization algorithms (derived in [1][1]- more information of these building blocks will be given later ) as well as some other numerical procedures (such as automatic differentiation, see, for example, Python implementations [Autograd] (https://github.com/HIPS/autograd) and [AlgoPy](https://pythonhosted.org/algopy/)).
Based on these building blocks the programming projects
are created that demonstrate certain steps in machine learning algorithms and show how these steps appear in popular modern machine learning methods.
These projects become a foundation of project based data
science courses, such as Data Analysis and Decision Making using Python, Python for Financial Applications, Operations Research models using Python, etc.
Building Blocks
In [Kapl and Prop][1] the theoretical approach for the design of global optimization
methods based on potential theory was introduced. This approach extends
the theory of gradient based optimization for algorithmically defined
functions (or black box functions), where analytical representation of the function is not available or too complicated to work with (say, too hard to compute derivatives). Such situations are very common in real world applications. Based on this theory, the parsimonious set of building blocks
of the optimization algorithms was obtained. The hypothetical algorithm
where all these building blocks are present in their full form is given in [1][1] It is shown that by varying the parameters of the building blocks we obtain the whole universe of optimization methods, some of them we recognize as well known heuristic techniques such as [CMA-ES](https://en.wikipedia.org/wiki/CMA-ES), [Shor r algorithm](https://link.springer.com/article/10.1023%2FA%3A1008739111712), [Nelder and Mead algorithm](https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method) etc. Main building blocks are defined as linear algebra operations: space dilation operator (the space transformation is based on this operator ), Householder transformation as a certain case of space dilation operator, and memory accumulation module (accumulates information from the previous iterations of the algorithms).
Other numerical procedures that algorithms are built upon are automatic differentiation and [natural gradient evolution strategy](https://pdfs.semanticscholar.org/eb2d/7fb3105cd98646943b5dccf799d2bb8b09ed.pdf).
Examples of the projects
Householder transformation in Nelder and Mead algorithm (fmin function in SciPy)
Space dilation operator in Shor r-algorithm and in CMA-ES
Combining automatic differentiation with gradient-based algorithms for
optimization of algorithmically defined functions.
Natural gradient evolution strategy
Coordinate transformation, cootdinate descent algorithm and separable
functions. Invariance of coordinate descent method with respect to
scaling and rotation of the search space.
Memory accumulation module. Comparizon of memory accumulation in Shor
algorithm and genetic algorithm
Multi objective optimization vs constrained optimization
[1]:http://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=at&paperid=4004&option_lang=eng"Investigation of search methods for optimization that use potential theory"