Media from Papers:

Functional Object-Oriented Network for Manipulation Learning (IROS 2016)

  • Link to publication found here.
  • Fig. 1 - A basic functional unit with two input nodes and two output nodes connected by an intermediary single motion node. [Download]
  • Fig. 2 - A FOON subgraph based on an instructional video on making a watermelon-strawberry smoothie. The green solid circles are object nodes and the red solid squares are motion nodes. The object nodes are labeled with object name and their states in parentheses. The motion nodes are labeled with their manipulation motion types. [Download]
  • Fig. 3 - Our current universal FOON that is constructed from 60 videos. [Download]
  • Fig. 4 - The OptiTrack motion capture system with which we collect data for motion learning. The system consists of six motion capture cameras on tripods. Within the blue area on the desk are two objects with reflective markers attached to them. [Download]
  • Fig. 5 - Graph showing the objects found with the ten (10) highest and lowest centrality values. The higher the value, the more important a node is. Objects are also classified as utensils (shown in blue) and ingredients (shown in red). [Download]
  • Fig. 6 - Graph showing the top 10 motions observed in our universal FOON (out of 798 motion instances). [Download]
  • Fig. 7 - Example of a FOON merging two ways of preparing cooked ribs barbecued ribs (node in purple) using available objects (in blue). [Download]
  • Fig. 8 - Task tree showing the steps needed to prepare barbecued ribs (highlighted in purple) using available objects (in blue) to create objects of other states. [Download]
  • Fig. 9 - In degree x and y, the new trajectory meets the constraints well. Without constraints, the rest degrees of the new trajectory equal the mean of the data. The `data traj' (in dark yellow) have been aligned using DTW. [Download]
  • Video demonstrating the annotation of a FOON to its video side-by-side and task tree execution in Unity: [Download].


    Learning a Functional Object-Oriented Network using Object Similarity (in review)

  • Fig. 1 - A basic functional unit with three input nodes and three output nodes connected by an intermediary single motion node. [Download]
  • Fig. 2 - Our current level 3 universal FOON that is constructed after merging 65 instructional YouTube videos. [Download]
  • Fig. 3 - An example of how expansion helps us to add new units by demonstrating how we can learn to chop pineapples to complete knowledge needed for making pizza (goal node in dark green). There is no link connecting the concepts of "chopped" and "whole" pineapple objects (the latter which exists in the scene). However, chives and pineapples are somewhat similar as dictated by WordNet, so we can create a new functional unit to model the chopping process for pineapples. The new nodes used to link the knowledge is denoted by the orange and yellow nodes. [Before Expansion] [After Expansion]
  • For files and images of the network used, please refer to here.