PAPERS & MEDIA

2019:

Manipulation Motion Taxonomy and Coding for Robots [arXiv]

A Weighted Functional Object-Oriented Network for Task Planning [arXiv]

  • Supplementary material for this paper can be found here.

    • Long Activity Video Understanding using Functional Object-Oriented Network [.PDF] [arXiv]

  • Supplementary material for this paper can be found here.

    • 2018:

    Functional Object-Oriented Network: Construction & Expansion [.PDF] [arXiv]

  • Fig. 1 - A basic functional unit with three input nodes (in green) and two output nodes (in blue) connected by an intermediary single motion node (in red). Input and output nodes are distinguished by the direction of the edges. [Download]
  • Fig. 2 - An example of how expansion can be used in filling gaps of knowledge. Here we wish to make a salad (node in dark green) using lettuce and other items in the environment (in blue); initially, we only have knowledge on making salads with kale. Using similarity, we understand that kale and lettuce are similar, as they are both leafy vegetables. We create the knowledge of chopping lettuce and adding it to a bowl with other ingredients to make a salad. [Part 1] [Part 2]
  • Fig. 3 - A set of functional units before and after generalization by categories. The node colours indicate which objects and categories are synonymous; the grapefruit object nodes were generalized as citrus and potatoes and onions were generalized as vegetables and root vegetables. Two functional units (0_cut and 1_cut) end up compressed into a single unit. [Part 1] [Part 2]

    • 2016:

    Functional Object-Oriented Network for Manipulation Learning [.PDF] [arXiv]

  • Fig. 1 - A basic functional unit with two input nodes and two output nodes connected by an intermediary single motion node. [Download]
  • Fig. 2 - A FOON subgraph based on an instructional video on making a watermelon-strawberry smoothie. The green solid circles are object nodes and the red solid squares are motion nodes. The object nodes are labeled with object name and their states in parentheses. The motion nodes are labeled with their manipulation motion types. [Download]
  • Fig. 3 - Our current universal FOON that is constructed from 60 videos. [Download]
  • Fig. 4 - The OptiTrack motion capture system with which we collect data for motion learning. The system consists of six motion capture cameras on tripods. Within the blue area on the desk are two objects with reflective markers attached to them. [Download]
  • Fig. 5 - Graph showing the objects found with the ten (10) highest and lowest centrality values. The higher the value, the more important a node is. Objects are also classified as utensils (shown in blue) and ingredients (shown in red). [Download]
  • Fig. 6 - Graph showing the top 10 motions observed in our universal FOON (out of 798 motion instances). [Download]
  • Fig. 7 - Example of a FOON merging two ways of preparing cooked ribs barbecued ribs (node in purple) using available objects (in blue). [Download]
  • Fig. 8 - Task tree showing the steps needed to prepare barbecued ribs (highlighted in purple) using available objects (in blue) to create objects of other states. [Download]
  • Fig. 9 - In degree x and y, the new trajectory meets the constraints well. Without constraints, the rest degrees of the new trajectory equal the mean of the data. The `data traj' (in dark yellow) have been aligned using DTW. [Download]
  • Video demonstrating the annotation of a FOON to its video side-by-side and task tree execution in Unity: [Download].