Imitation Learning and Learning from Demonstrations

Core Idea

Imitation learning (learning from demonstrations, LfD) trains a robot to mimic expert behavior from demonstrations, avoiding the sample inefficiency of learning from scratch via reinforcement learning. The robot observes an expert (human, simulation, or another robot) and learns to reproduce the expert's actions given similar states. Behavioral cloning applies supervised learning directly: treat demonstrations as (state, action) pairs and train a neural network to predict actions from states. This is simple and fast but suffers from distribution shift: small errors compound over time as the robot deviates from the expert's trajectory, encountering states the expert never demonstrated. Inverse reinforcement learning (IRL) infers the reward function from demonstrations, then solves for a policy that maximizes that reward. This is more robust to distribution shift because the learned reward generalizes. Practical systems often combine imitation (warm-start policy) with reinforcement learning (fine-tuning), achieving rapid learning with safety.

Explainer

Reinforcement learning is powerful but sample-inefficient. A robot learning to grasp from scratch might need thousands of trials, breaking objects and damaging hardware. An alternative: learn from human demonstrations. A human shows the robot how to grasp an apple, retrieve a book, or assemble a plug. Can the robot reproduce this behavior? This is imitation learning, and it cuts sample complexity from millions (RL from scratch) to dozens or hundreds (learning from examples).

Behavioral Cloning: The simplest imitation approach is supervised learning. Collect demonstrations: record (state, action) pairs as the expert performs the task. States are sensor observations (images, joint angles, sensor readings); actions are the commands the expert issued. Train a neural network π_BC(a|s) to predict actions from states using standard supervised regression or classification. For continuous actions (joint velocities), use regression with mean-squared error loss. For discrete actions (grasp or not grasp), use cross-entropy. The trained network is your policy.

Behavioral cloning is fast and simple — essentially learning a regression function. It works well on short, constrained tasks (grasping at a specific location, following a trajectory) where the robot stays close to the demonstrated trajectory. But it has a critical failure mode: distribution shift. If the robot makes a small error early (hand 1cm off target), it enters a state the expert never demonstrated. The network's output for this novel state is undefined by training data — it might continue the demonstrated action regardless of the current situation. The error propagates forward: at the next timestep, the robot is further off, encountering another unseen state. Errors compound. After a few seconds, the robot has diverged catastrophically. This is why behavioral cloning alone rarely works for long-horizon tasks.

Dataset Aggregation (DAgger): To address distribution shift, DAgger iteratively includes the robot's own mistakes in the training set. (1) Train behavioral cloning on initial demonstrations. (2) Run the learned policy and record states where it fails or performs poorly. (3) Ask an expert to label what action should be taken in those error states. (4) Add these (state, action) pairs to the training set. (5) Retrain. Repeat. After several iterations, the training distribution shifts from the expert's trajectory toward the robot's actual on-policy trajectory. The robot learns to handle its own errors. DAgger significantly improves robustness but requires iterative expert annotation, making it slow and expensive for real robots.

Inverse Reinforcement Learning: Rather than cloning actions, infer the expert's underlying reward function, then solve for a policy that maximizes it. The assumption: expert demonstrations are nearly optimal for some unknown reward function. Algorithms like maximum entropy IRL find a reward function R(s,a) that best explains the demonstrations (the expert's trajectory has much higher return than random policies). Once R(s,a) is inferred, solve standard RL to learn a policy π that maximizes E[Σ γ^t R(s_t,a_t)].

Why is this better? The reward function is abstract and generalizes. If the inferred reward is "reach the target efficiently," that objective applies whether the target is moved 10cm to the left or the robot's arm configuration is different. Action-space behavioral cloning doesn't generalize; it learned "in response to this particular image, do this particular action." IRL recovers something deeper — the intent — which transfers.

Trade-offs: IRL is computationally expensive (requires solving an RL problem inside the inference loop). The inferred reward is not unique; multiple rewards could explain the same demonstrations. And IRL still depends on the demonstrations actually being near-optimal for some reward, which may not hold if the human is suboptimal.

Hybrid Approaches (Most Practical): Real-world robotics systems combine multiple methods. Start with behavioral cloning (fast, few demonstrations needed) to get a working policy. Then fine-tune with RL (improving optimality and robustness) or DAgger (handling distribution shift). This gives the speed of imitation with the robustness of RL. Companies like OpenAI and DeepMind use this formula: behavioral cloning for 1-100 demonstrations, then RL fine-tuning for 10,000-100,000 simulated interactions, then real-world validation and adaptation. This is dramatically more sample-efficient than RL alone while avoiding behavioral cloning's brittleness.

When Imitation Learning Excels: Imitation is best for tasks where (1) expert demonstrations are easy to obtain (human can show the robot), (2) the task is primarily reactive (state → action) rather than requiring long-term planning, and (3) performance near expert level is acceptable (don't need superhuman optimization). It's ideal for manipulation (grasping, in-hand manipulation), locomotion (learning walking or jumping gaits), and navigation (following human driving). It's poor for tasks requiring discovering novel solutions beyond the expert's demonstrations or long-horizon planning, where RL excels.