Sunday 10 a.m.–1 p.m. in

Curriculum for project-based data science classes and their building blocks

Nadia Udler, Adomous Wright, Eli Udler

Description

Machine learning, artificial intelligence and data science are interdisciplinary subjects and therefore they are difficult to teach. We suggest educational building blocks that help students to understand machine learning software (such as scikit-learn) and create their own artificial intelligence algorithms. These building blocks are theoretical components of global optimization algorithms (derived in [1][1]- more information of these building blocks will be given later ) as well as some other numerical procedures (such as automatic differentiation, see, for example, Python implementations [Autograd] (https://github.com/HIPS/autograd) and [AlgoPy](https://pythonhosted.org/algopy/)). Based on these building blocks the programming projects are created that demonstrate certain steps in machine learning algorithms and show how these steps appear in popular modern machine learning methods. These projects become a foundation of project based data science courses, such as Data Analysis and Decision Making using Python, Python for Financial Applications, Operations Research models using Python, etc. Building Blocks In [Kapl and Prop][1] the theoretical approach for the design of global optimization methods based on potential theory was introduced. This approach extends the theory of gradient based optimization for algorithmically defined functions (or black box functions), where analytical representation of the function is not available or too complicated to work with (say, too hard to compute derivatives). Such situations are very common in real world applications. Based on this theory, the parsimonious set of building blocks of the optimization algorithms was obtained. The hypothetical algorithm where all these building blocks are present in their full form is given in [1][1] It is shown that by varying the parameters of the building blocks we obtain the whole universe of optimization methods, some of them we recognize as well known heuristic techniques such as [CMA-ES](https://en.wikipedia.org/wiki/CMA-ES), [Shor r algorithm](https://link.springer.com/article/10.1023%2FA%3A1008739111712), [Nelder and Mead algorithm](https://en.wikipedia.org/wiki/Nelder%E2%80%93Mead_method) etc. Main building blocks are defined as linear algebra operations: space dilation operator (the space transformation is based on this operator ), Householder transformation as a certain case of space dilation operator, and memory accumulation module (accumulates information from the previous iterations of the algorithms). Other numerical procedures that algorithms are built upon are automatic differentiation and [natural gradient evolution strategy](https://pdfs.semanticscholar.org/eb2d/7fb3105cd98646943b5dccf799d2bb8b09ed.pdf). Examples of the projects Householder transformation in Nelder and Mead algorithm (fmin function in SciPy) Space dilation operator in Shor r-algorithm and in CMA-ES Combining automatic differentiation with gradient-based algorithms for optimization of algorithmically defined functions. Natural gradient evolution strategy Coordinate transformation, cootdinate descent algorithm and separable functions. Invariance of coordinate descent method with respect to scaling and rotation of the search space. Memory accumulation module. Comparizon of memory accumulation in Shor algorithm and genetic algorithm Multi objective optimization vs constrained optimization [1]:http://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=at&paperid=4004&option_lang=eng"Investigation of search methods for optimization that use potential theory"