English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Learning from human demonstrations is an emerging trend for designingintelligent robotic systems. However, previous methods typically regard videosas instructions, simply dividing them into action sequences for roboticrepetition, which poses obstacles to generalization to diverse tasks or objectinstances. In this paper, we propose a different perspective, considering humandemonstration videos not as mere instructions, but as a source of knowledge forrobots. Motivated by this perspective and the remarkable comprehension andgeneralization capabilities exhibited by large language models (LLMs), wepropose DigKnow, a method that DIstills Generalizable KNOWledge with ahierarchical structure. Specifically, DigKnow begins by converting humandemonstration video frames into observation knowledge. This knowledge is thensubjected to analysis to extract human action knowledge and further distilledinto pattern knowledge compassing task and object instances, resulting in theacquisition of generalizable knowledge with a hierarchical structure. Insettings with different tasks or object instances, DigKnow retrieves relevantknowledge for the current task and object instances. Subsequently, theLLM-based planner conducts planning based on the retrieved knowledge, and thepolicy executes actions in line with the plan to achieve the designated task.Utilizing the retrieved knowledge, we validate and rectify planning andexecution outcomes, resulting in a substantial enhancement of the success rate.Experimental results across a range of tasks and scenes demonstrate theeffectiveness of this approach in facilitating real-world robots to accomplishtasks with the knowledge derived from human demonstrations.

Human Demonstrations are Generalizable Knowledge for Robots