Robot Perception and Learning

CSIE5117 • Fall 2025 • National Taiwan University

Building a robot to function in the real-world remains challenging, which requires (1) perception, that recognizes the hidden states of the embodied scene, (2) decision making, that generates optimal plans to achieve tasks, and (3) action, that controls robot’s machinery to navigate in/interact with the environment. With decades of efforts, scientists have introduced numerous techniques that enable robots working in a controlled and known environment (e.g. industrial robots). However, these techniques fail in non-controlled and non-structured environments, such as streets/kitchens/homes/offices that are noisy, complex and constantly changing. Moreover, these techniques are incapable of building robots that continually learn new tasks, adapt effectively to new environments, and achieve each user’s individual need.

This course provides a technical overview of decision making and action control in robotics with an emphasis on the development of generalist robots. The course covers: (1) fundamental robot kinematics that discusses geometric relationships between the robot’s body and the end effector, (2) classical control and planning, that provides a theoretical basis, and (3) state-of-the-art robot learning that facilitate adaptive and continual learning of robot tasks.

Note: The perceptual components of robot systems will be discussed in the course of Embodied Vision (CSIE5421).

Course Goals

The goal of this class is to guide you through “what are the essential components for controlling the robot’s motion” and “when/how they are used” in robotic research. Going through each component thoroughly is not our goal.

Upon completion of the course students should know:

  • The fundamental theorem of robot kinematics
  • The classical planning-and-control pipeline in robotics
  • Why learning is important in robotics, and what are the open research questions in robot learning
  • What’re under the hood of robot systems

Upon completion of the course students should be able to:

  • Set up a simulator to kick-start your robotic research
  • Train robot policies with imitation learning
  • Train robot policies with reinforcement learning
  • Implement a diffusion model

Prerequisite Knowledge

Students should have a solid understanding of the following areas:

  • Machine learning: stochastic gradient descent, loss function, optimization, neural network
  • Linear algebra: matrices, vectors, norms, scalar/vector products, orthogonality, singular value decomposition…
  • Probability: expectation, independence, Baye’s Theorem…
  • Computer vision: convolutional neural networks, transformer models, 3D vision geometry
  • Linux system: setting up the required environment, being familiar with bash scripts
  • Python programming: creating python projects, importing required packages, visualizing your results
  • Pytorch programming: creating NNs, setting up training & evaluation pipeline

Prerequisite Hardware

Grading in this course heavily depends on the coding assignments and final project. You should prepare for your own Linux machine with GPUs for the assignments and the final project. Otherwise, you can try cloud platforms for the access of GPUs.

Other Information

  • Lectures: Wednesday 9:10 AM - 12:10 PM
  • Lecture Location: CSIE Building Room 104
  • Discussion: Discord
  • HW submission: Github classroom