PhD Defense - Alexander Schperberg - Unifying Model-based Optimization and Machine Learning

Abstract: For robots to be completely autonomous, requires advances and the symbiotic relationship between planning, control, and estimation modules. However, formulating these algorithms, especially within an end-to-end architecture, is challenging due to the partial observability of the environment, the need for robustness, and the complexities of non-linear dynamics. In this talk, hybrid solutions are explored to show how model-based optimization and machine learning can be unified to develop planning, control, and estimation frameworks. First, the SABER algorithm is introduced for end-to-end motion planning for multi-agent teams, integrating safety, optimality, and global solutions through stochastic model predictive control, recurrent networks, and reinforcement learning. Secondly, I demonstrate the auto-calibration of controller gains and parameters of an online planner for the objective of legged locomotion – extensions towards other applications are also presented, such as force control for manipulation. Thirdly, we demonstrate OptiState, a state estimation algorithm for legged robots that employs domain knowledge through filtering techniques, optimization, and gated networks along with vision transformers to accommodate for nonlinear errors. Lastly, I motivate my future direction, where Large Language and Vision Language Models can be used to endow robots with intelligence and understanding of their environment, further impacting the future design of end-to-end autonomous systems.

Defense Committee Members:
Professor Veronica Santos, Department of Mechanical and Aerospace Engineering
Professor Stefano Soatto, Department of Computer Science,
Professor Khalid Jawed, Department of Mechanical and Aerospace Engineering

Timestamps:
0:00 Introduction
1:28 Active SLAM
11:44 Auto-Calibration of Parameters
22:02 State Estimation for Legged Robots
30:05 Current Directions - Multi-Modal Robot
33:28 Concept of Understanding, LLM and VLM
42:00 Summary of Contributions
42:48 List of Publications/Patents/Peer-Review/Awards
43:29 Acknowledgements and Conclusion