REFLECT: Summarizing Robot Experiences for
FaiLure Explanation and CorrecTion

Zeyi Liu*      Arpit Bahety*      Shuran Song

Columbia University in the City of New York

Conference on Robot Learning (CoRL) 2023

Paper arXiv Video Code Dataset

The ability to detect and analyze failed executions automatically is crucial for an explainable and robust robotic system. Recently, Large Language Models (LLMs) have demonstrated strong common sense reasoning skills on textual inputs. To leverage the power of LLM for robot failure explanation, we propose a framework REFLECT, which converts multi-sensory data into a hierarchical summary of robot past experiences and queries LLM with a progressive failure explanation algorithm. Conditioned on the explanation, a failure correction planner generates an executable plan for the robot to correct the failure and complete the task. By evaluating on a variety of tasks in both simulation and real world, we show that our LLM-based framework is able to generate informative failure explanations that assist successful correction planning.

Technical Summary Video

Real World Results

Task: Boil water in a pot (video has sound)

Failure Explanation

The robot failed because it placed the bowl on the stove burner instead of the pot filled with water, and it didn't move the pot from the sink to the stove burner. Failure timestep: 02:22

Failure Explanation

At 03:00, the robot failed to properly place the pot filled with water on the stove burner. The failure was caused by the robot dropping something on the ground at 02:43, which might have been the pot or its contents. Failure timestep: 02:43

Task: Sauté carrot slice in a saucepan (video has sound)

Failure Explanation

At 01:02, the robot mistakenly picked up a spoon instead of the intended knife, causing the failure in its task to sauté carrots. Failure timestep: 01:02

Failure Explanation

At 01:18, the robot failed to sauté the carrot because it had not successfully sliced the carrot. The failure was caused by the robot dropping the knife at 00:55 while attempting to slice the carrot, leaving it with an empty gripper. Failure timesteps: 00:55, 01:18

Task: Store an apple in the fridge (video has sound)

Failure Explanation

The robot plan is missing the action to close the fridge door after placing the bowl with the apple inside the fridge. Failure timestep: 03:47

Failure Explanation

At 04:13, the robot failed to properly store the apple in the bowl inside the fridge. The failure was caused by the robot placing the white bowl with the apple on top of the fridge instead of inside it. Failure timestep: 04:13

Task: Secure pear and knife

Failure Explanation

The robot plan failed because it placed the knife in the fridge instead of its most suitable container, and the pear was placed in the drawer rather than the fridge. Failure timestep: 02:15

Task: Serve microwaved potato

Failure Explanation

At 05:24, the robot failed to close the microwave door before toggling it on, causing the failure. Failure step: 05:24

Task: Serve coffee on table

Failure Explanation

At 02:27, the robot failed to make coffee because it picked up the pink cup instead of the blue cup that was inside the coffee machine (observed at 02:04). This led to the robot placing the wrong cup (pink cup) on the table, leaving the blue cup with coffee still inside the machine. Failure timesteps: 02:04 & 02:27

Task: Put apple in a bowl

Failure Explanation

At 01:06, the failure occurred because the robot placed the apple on top of the bowl instead of inside it. This was likely caused by the presence of the knife on top of the bowl, obstructing the apple from being placed correctly. Failure timestep: 01:06

Simulation Results

Task: Serve coffee on table



Task: Switch device from laptop to television



Task: Boil water in a pot



Task: Heat potato in microwave

Citation

 
@article{liu2023reflect,
	title={REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction},
	author={Liu, Zeyi and Bahety, Arpit and Song, Shuran},
	journal={arXiv preprint arXiv:2306.15724},
	year={2023}
}

Contact

If you have any questions, please feel free to contact Zeyi Liu.