Optimization for Machine Learning

Optimization for Machine Learning
Course Slides
  • week1: Introductions and background: [pdf Slides]
  • week2: Continuous optimization formulation and basic concepts; Optimization concepts over multivariate continuous spaces; Convexity and Differentiability; Gradient and Hessian concepts; Examples in Machine Learning [pdf Slides], TD, TP
  • week3: continuous optimizaiton (Gradient descent methods) : Mathetamatical construction of descent methods, Gradient Descent, Descent Directions, Convergence and Convergence rates; Step-size tuning and Line Search; Accelerations [pdf Slides], TD , TP
  • week4: Continuous optimization (Second order methods : Newton methods including Quasi-Newton, secant, IRLS); and mid-term exam [pdf Slides], TD , TP
  • week5: constrained optimizaiton: Equality and Inequality constraints, Duality/Lagrangian, KKT optimality conditions; Examples in ML [pdf Slides], TD , TP
  • week6: Stochastic optimization, Non-convex optimization (Stochastic Gradient, The EM Algorithm); Examples in ML [pdf Slides], TD
TD-TP Exams
  • Mid-Term Exam (cc): week4, November 28th, 2024, 16h15-17h00. [CC]
  • Final Exam (CT): week7, December 19th, 2024. [CT]

Main teaching duties

From 2016 to 2022: Professor of Applied Mathematics, teaching at the maths and computer science departement of Caen University:
  • Director of the Master 2 degree in Statistics and Decision Analytics (SAD), from 2017 - 2019
  • Member of the council of the department of Mathematics and Computer Science
From 2011 to 2016: A/Professor of Computer Science, teaching at the computer science departement of Toulon University:
  • Director of the Bachelor degree in Engineering sciences from 2012 to 2015
  • Responsible of the second year of the Bachelor degree in Engineering sciences
  • Responsible of the tutoring in computer science at the faculty of science at Toulon university
From 2007 to 2011: Instructor and ATER of Computer Science, teaching at the computer science departement of Paris 13.

Responsible and teacher of the following courses from 2016 to 2022 (more than 275 hours in mean of teaching, a year) :
  1. Probabilities, Statistics 3 (Bsc in Maths & Applied Maths)
  2. Data Analysis 1 (Bsc in Maths & Applied Maths)
  3. Data Analysis and Classification (Master 1 Statistics & Master 1 Computer Science)
  4. Statistical Learning (Master 2 Statistics & Master 2 Computer Science)
  5. Time series Analysis (Master 2 Statistics)
Responsible and teacher of the following courses (more than 200 hours of teaching a year) : (2011-2016)
  1. Programming (Language C and Python) (Bsc Computer Science)
  2. Probabilities (Bsc Computer Science)
  3. Inferential Statistics (Master 1 Computer Science)
  4. Statistical Learning and Data Analysis (Master 2 Computer Science)

Master SAAD

Présentation du Master SAAD

Master 2 Statistics & Decision Analytics / Master 2 Computer Science : Probabilistic Learning and Data Analysis (CM/TP)

    [Slides (all)]

    Slides per topic:
  • Review of probability and statistics (in french)[ pdf]
  • Review of parameter estimation - Maximum Likelihood, OLS, Regression... (in french) [pdf]
  • Introductionary concepts, Pattern recognition, Machine Learning [pdf]
  • Supervised Learning: Classification (KNN, Gaussian Discriminant Analyses, Logistic/Softmax regression) [pdf]
  • Unsupervised learning: Clustering, Kmeans, Mixtures, Model Selection [pdf]
  • Unsupervised learning: Topographic Learning (SOM, GTM, GTM through time)[pdf]
  • Unsupervised learning: Sequential Data Modeling, Markov Chains, HMMs, Baum-Welch (EM), Viterbi[pdf]
  • Unsupervised learning: Dimensionality reduction (PCA, Probabilistic PCA, Factor Analysis, EM)[pdf]
    Practical work:
  • Gaussian Discriminant Analysis and Logistic regression (supervised learning)[pdf] [solution]
  • KNN (tp)[pdf]
  • Supervised learning, Tp (LDA-QDA)[pdf] [pdf]
  • Unsupervised learning: Kmeans and xEM clustering[pdf]
  • Kmeans (TP)[pdf]
  • EM-GMM (TP)[pdf]
  • Sequential Data Modeling (HMMs)[pdf] [solution]
  • Principal Component Analysis and Classification (TP, ACP- faces images)[pdf]
  • Regression Mixtures and EM (TP) [pdf]

Master 2 Statistics & Decision Analytics : Time series clustering and segmentation (CM/TD/TP)

[related pdf slides]

Licence 3 Mathematics and applied Mathematics (MIASHS) / Master 1 Stat/CS: Data Analysis (CM/TD/TP)


Licence 3 Mathematics and Applied Mathematics (MIASHS): Probabilités et statistique (CM/TD)


Licence 2 Mathematics : Probabilités et statistique (TD)


Past: at Toulon University

[2011/2012] [2012/2013] [2013/2014] [2014/2015][2015]
  • Master 2 Informatique : Probabilistic Learning and Data Analysis (D33) [PDF] Slides
      • Slides
    • Review of probability and statistics [PDF]
    • Parameter estimation - Maximum Likelihood method, ... [PDF]
    • Pattern recognition an machine learning concepts; Classification (discrimination), ... [PDF]
    • Mixture models, GMM and EM - KmeansClustering - Model-Based Clustering, ... [PDF]
    • SOM, GTM, GTM Through Time [PDF]
    • Markov Chains, HMMs, Baum-Welch (EM), Viterbi [PDF]
    • PCA, PPCA, FA [PDF]
  • Master 2 Informatique : Initiation à la recherche (Initiation to research)
    • Studied Paper: Miin-Shen Yang, Chien-Yo Lai, Chih-Ying Lin. "A robust EM clustering algorithm for Gaussian mixture models", Pattern Recognition , Volume 45, Issue 11, November 2012, Pages 3950–3961 [pdf]
      • Solution: was provided by e-mail (please see your email box)
  • Master 1 Informatique : Eléments de statistique Inférentielle (Elements of inferential statistics) (d13)   [slides pdf]    [Polycopié]
      • Slides (in French) :
    • review of probability and stat, Random vectors, Gaussian vectors, Central Limit thm, etc [PDF]
    • Estimation theory, Estimators properties, Cramér-Rao Lower Bound, estimation methods [PDF]
    • Maximum Likelihood Estimator (MLE) [PDF]
    • Least Squares Estimators (LSE), Linear regression [PDF]
    • Interval estimation, Confidence intervals [PDF]
    • Hypothesis Testing [PDF]
  • Licence 2 Sciences pour l'Ingénieur : Programmation II – Langage C (C programming ) (I41)
    • Polycopié : [pdf]
  • Licence 2 Sciences pour l'Ingénieur : Probabilités discètres (Discrete Probability) (MI48)
    • Polycopié : [pdf]
  • Licence 1 Sciences pour l'Ingénieur : Algorithmique + programmation Python (Algorithmics + Python programming ) (I11 ; I21)

Past at Paris 13 University :

[2007-2011]
  • Temporary Research and Teaching Assistant (ATER) (march 2010 - July 2011)
    • Mathématiques et informatique
    • Introduction Structures de Données Linéaires
    • Algorithmique des Structures de Données Linéaires
    • Programmation Impérative
    • Programmation Orientée Objet (Java)
    • Algorithmique et Arbres
  • Teaching Assistant (Moniteur) (2007-2010)
    • C Programming Langage (64h/year)
    • XHTML (32h/year), Introduction to GTK (32h/year)