Lecture Information
Lectures: MoWe 11:00–11:15, 301-302

Contacts
Prof. Insoon Yang
insoonyang@snu.ac.kr
Office hours: by appointment

Description
This course introduces convex optimization theory and algorithms, with application to machine learning and data science. The first part will provide the basic theory of convex optimization. In the second part, application problems in many disciplines will be discussed. In the third part, basic convex optimization algorithms and acceleration methods will be covered. The last part of this course will introduce advanced algorithms including stochastic subgradient and proximal methods that are useful for solving large-scale problems arising in machine learning and data science.

Prerequisite
Calculus, Linear Algebra, Probability

Evaluation
There will be 2 problem sets, a midterm test, and a final project.
Course grades: Participation (5%); Problem sets (15%); Midterm (30%); Final project (50%).

Policies
Working in groups is encouraged. However, each person must submit his/her own problem sets. Late submissions will not be accepted.

Course References
The course is based on lecture notes.
Text:
• [BV04] S. Boyd, L. Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
• [MDG18] M. W. Mahoney, J. C. Duchi, A. C. Gilbert, The Mathematics of Data, American Mathematical Society, 2018.
References:
• [Nes04] Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, Springer, 2004.
• [BNO03] D. P. Bertsekas, A. Nedic, A. E. Ozdaglar, Convex Analysis and Optimization, Athena Scientific, 2003.
• [Ber15] D. P. Bertsekas, Convex Optimization Algorithms, Athena Scientific, 2015.
• [NW05] J. Nocedal, S. Wright, Numerical Optimization, Springer, 2nd edition, 2006.
• [Ber16] D. P. Bertsekas, Nonlinear Programming, Athena Scientific, 3rd edition, 2016.

Course Schedule
• Part I: Basic theory
– Week 1. Introduction to mathematical optimization
– Week 2. Convex sets
– Week 3. Convex functions
– Week 4. Convex optimization problems
– Week 5. Duality
• Part II: Applications
– Week 6. Approximation and fitting
– Week 7. Statistical estimation
– Week 8. Geometric problems
• Part III: Basic algorithms
– Week 9. Unconstrained minimization and Nesterov’s acceleration
– Week 10. Equality constrained minimization
– Week 11. Interior-point methods
• Part IV: Advanced algorithms for large-scale problems
– Week 12. Subgradient methods
– Week 13. Stochastic subgradient methods
– Week 14. Mirror descent methods
– Week 15. Proximal methods and ADMM

Lecture Information
Lectures: MoWe 9:30–10:45, 301-102

Contacts
Prof. Insoon Yang
insoonyang@snu.ac.kr
Office hours: by appointment

Description
This course aims to introduce rigorous mathematical tools for analyzing and designing feedback control systems. The first part will provide the fundamentals of state-space methods, including stability, reachability, and input/output response. In particular, state and output feedback design methods will be covered. In the second part, frequency domain methods will be discussed with an emphasis on robust performance in the presence of uncertainty. The following analysis and design tools will be introduced: transfer functions, the Nyquist criterion, Bode’s plot, PID control, root-locus and loop shaping.

Prerequisite
Calculus, Linear Algebra, Differential Equations

Evaluation
There will be 2 problem sets, a midterm test, and a final exam.
Course grades: Participation (5%); Problem sets (15%); Midterm (35%); Final exam (45%).

Policies
Working in groups is encouraged. However, each person must submit his/her own problem sets. Late submissions will not be accepted.

Course References
The course is based on lecture notes.
Text:
• [AM08] K. J. Astrom, R. M. Murray, Feedback Systems: An Introduction for Scientists and Engineers, Princeton University Press, 2008.
References:
• [Nise19] Norman S. Nise, Control Systems Engineering, John Wiley & Sons, 8th edition, 2019.
• [Oga09] Katsuhiko Ogata, Modern Control Engineering, Pearson, 5th edition, 2009.
• [Lue79] David G. Luenberger, Introduction to Dynamic Systems, Wiley, 1979.

Course Schedule
• Week 1. Introduction to control systems
• Week 2. Feedback principles
• Week 3. System modeling and examples
• Week 4. Dynamic behavior and stability
• Week 5. Linear systems
• Week 6. State feedback I: Analysis
• Week 7. State feedback II: Design
• Week 8. Output feedback I: Analysis
• Week 9. Output feedback II: Design
• Week 10. Transfer functions
• Week 11. Frequency domain analysis
• Week 12. PID control
• Week 13. Frequency domain design I: Root-locus
• Week 14. Frequency domain design II: Loop shaping
• Week 15. Robust performance

Contacts
Prof. Insoon Yang
insoonyang@snu.ac.kr

Description
This short course introduces a set of deep reinforcement learning algorithms including DQN, DDPG, TRPO (PPO), and SAC. Students will implement and test all the algorithms in several environments in OpenAI Gym. The pros and cons of each method will also be discussed to provide a guideline for selecting an appropriate algorithm to solve a given task.

Course References
The course is based on slides and codes.
References:

Course Schedule
• Lecture 1. Review of Reinforcement Learning
• Lecture 2. Introduction to Deep Reinforcement Learning
• Lecture 3. Deep Q-Networks (DQN)
• Lecture 4. Double DQN
• Lecture 5. Policy Gradient
• Lecture 6. Actor-Critic
• Lecture 7. Deep Deterministic Policy Gradient (DDPG)
• Lecture 8. Challenges of Policy Gradient
• Lecture 9. Trust Region Policy Optimization (TRPO)
• Lecture 10. Maximum Entropy Stochastic Control and Reinforcement Learning
• Lecture 11. Soft Actor-Critic (SAC)

Lecture Information
Lectures: TuTh 9:30–10:45, 301-106

Contacts
Prof. Insoon Yang
insoonyang@snu.ac.kr
Office hours: TuTh 10:45-11:15, 301-708

Description
This course introduces mathematical tools for robotics and autonomous systems, focusing on mobile robots. The first part will provide the fundamentals of mobile robot kinematics. In the second part, robot localization problems will be discussed. In particular, students will learn about probabilistic map-based localization and SLAM. The last part of this course will introduce Markov decision processes (MDP), which is a useful mathematical framework for robot decision-making, planning, control and reinforcement learning.

Prerequisite
Linear algebra, Probability

Evaluation
There will be 2 problem sets, a midterm test, and a final exam.
Course grades: Attendance (5%); Problem sets (15%); Midterm (30%); Final project (50%)

Policies
Working in groups is encouraged. However, each person must submit his/her own problem sets. Late submissions will not be accepted.

Course References
The course is based on lecture notes.
Text:
• R. Siegwart, I. R. Nourbakhsh, D. Scaramuzza, “Introduction to Autonomous Mobile Robots”, MIT Press, 2nd edition, 2011.
• S. Thrun, W. Burgard, D. Fox, “Probabilistic Robotics”, MIT Press, 2006.

Course Schedule
• Week 1. Introduction to autonomous robots
• Week 2. Mobile robot kinematic models and constraints
• Week 3. Mobile robot maneuverability
• Week 4. Mobile robot workspace
• Week 5. Mobile robot kinematic control
• Week 6. Introduction to robot localization
• Week 7. Markov localization
• Week 8. Kalman, EKF localization
• Week 9–10. SLAM (simultaneous localization and mapping)
• Week 11. Introduction to motion planning and control
• Week 12. Markov decision processes (MDP)
• Week 13. Dynamic programming
• Week 14. Partially observable Markov decision processes (POMDP)
• Week 15. POMDP algorithms

Lecture Information
Lectures: TuTh 9:30–10:45, 301-102

Contacts
Prof. Insoon Yang
insoonyang@snu.ac.kr
Office hours: TuTh 10:45-11:15, 301-708

Description
This course introduces optimal control theory mainly for continuous-time problems. Comprehensive and in-depth lectures on Pontryagin’s principle and dynamic programming will be provided with an emphasis on connections between the two. The second part of this course will discuss “Adaptive dynamic programming”, which is useful when a perfect system model is unavailable. In the last part, we study advanced topics in differential games and their application to reachability specification and safe control.

Evaluation
There will be 2 problem sets, a midterm test, and a final project (report and presentation).
Course grades: Attendance (5%); Problem sets (15%); Midterm (30%); Final project (50%)

Policies
Working in groups is encouraged. However, each person must submit his/her own problem sets. Late submissions will not be accepted.

Course References
The course is based on lecture notes.
Text (optional):
• W. H. Fleming, R. W. Rishel, “Deterministic and Stochastic Optimal Control”, Springer-Verlag, 1975.
• Y. Jiang, Z.-P. Jiang, “Robust Adaptive Dynamic Programming”, IEEE Press, 2017.
References:
• L. C. Evans, “An Introduction to Mathematical Optimal Control Theory”, Version 0.2, https://math.berkeley.edu/~evans/control.course.pdf.
• M. Bardi, I. Capuzzo-Dolcetta, “Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations”, Birkhauser, 2008.
• F. Clarke, “Functional Analysis, Calculus of Variations and Optimal Control”, Springer, 2013.
• D. P. Bertsekas, “Dynamic Programming and Optimal Control, Vol. I”, 4th ed., Athena Scientific, 2017.
• D. E. Kirk, “Optimal Control Theory: An Introduction”, Dover, 2004.
• D. Liberzon, “Calculus of Variations and Optimal Control Theory: A Concise Introduction”, Princeton University Press, 2012.

Course Schedule
• Week 1. Introduction to optimal control in continuous time
• Week 2. Basic optimization theory
• Week 3. Pontryagin’s principle I: Proof
• Week 4. Pontryagin’s principle II: Applications
• Week 5. Dynamic programming: Fundamental idea
• Week 6. Dynamic programming: Hamilton-Jacobi-Bellman equations and viscosity solutions
• Week 7. Connections between Pontryagin’s principle and dynamic programming
• Week 8. Infinite-horizon dynamic programming and LQR
• Week 9. Adaptive DP for linear systems I: Policy iteration
• Week 10. Adaptive DP for linear systems II: Policy learning
• Week 11. Adaptive DP for control-affine systems I: Policy iteration
• Week 12. Adaptive DP for control-affine systems II: Policy learning
• Week 13. Adaptive DP for polynomial systems I: SOS optimization
• Week 14. Adaptive DP for polynomial systems II: Policy learning
• Week 15. Introduction to differential games and Hamilton-Jacobi-Isaacs’ equations
• Week 16. Reachability specification and safe control using differential games

Lecture Information
Lectures: TuTh 2–3:15, 302-209-1

Contacts
Prof. Insoon Yang
insoonyang@snu.ac.kr
Office hours: TuTh 3:15-4:15, 301-708

Description
This course aims to introduce stochastic control and reinforcement learning methods. The first part focuses on the fundamental theory of stochastic optimal control. In addition to the Bellman’s optimality condition and the existence of an optimal policy, several solution methods such as value and policy iteration and the linear programming approach are introduced and analyzed. The second part covers several reinforcement learning algorithms including Q-learning, policy gradient and actor-critic, bayesian RL, and their theoretic background (e.g., stochastic approximation and regret bounds). The pros and cons of several algorithms will also be discussed.

Evaluation
There will be 2 problem sets, a midterm test, and a final project (report and presentation).
Course grades: Problem sets (20%); Midterm (30%); Final project (50%)

Policies
Working in groups is encouraged. However, each person must submit his/her own problem sets. Late submissions will not be accepted.

Course References
The course is based on lecture notes.
Text:
• O. Hernandez-Lerma, J. B. Lassarre, “Discrete-Time Markov Control Processes: Basic Optimality Criteria”, Springer, 2012.
• D. P. Bertsekas, J. N. Tsitsiklis, “Neuro-Dynamic Programming”, Athena Scientific, 1996.
References:
• M. L. Puterman, “Markov Decision Processes: Discrete Stochastic Dynamic Programming”, Wiley-Interscience, 2005.
• P. R. Kumar, P. Varaiya, “Stochastic Systems: Estimation, Identification and Adaptive Control”, SIAM, 2015.
• R. S. Sutton, A. G. Barto, “Reinforcement Learning: An Introduction”, MIT Press, 1998.
• D. P. Bertsekas, “Dynamic Programming and Optimal Control: Approximate Dynamic Programming, Vol. II”, 4th ed., Athena Scientific, 2012.
• C. Szepesvari, “Algorithms for Reinforcement Learning”, Morgan and Claypool Publishers, 2010.

Course Schedule
• Week 1. Introduction to sequential decision-making under uncertainty
• Week 2. Preliminaries on probability and measure theory
• Week 3. Finite horizon stochastic optimal control I: Theory
• Week 4. Finite horizon stochastic optimal control II: Algorithm
• Week 5. Infinite horizon discounted stochastic optimal control I: Theory
• Week 6. Infinite horizon discounted stochastic optimal control II: Algorithm (value iteration and policy iteration)
• Week 7. Stochastic approximation
• Week 8. Temporal difference learning
• Week 9. Q-learning
• Week 10. Introduction to approximate dynamic programming
• Week 11. Projected Bellman equation methods: LSTD, LSPE, TD(0)
• Week 12. Approximate Q-learning
• Week 13. State aggregation
• Week 14. Linear programming approach and constraint sampling
• Week 13. Policy gradient methods
• Week 14. Actor-critic methods
• Week 15. Bayesian reinforcement learning and regret bounds