16-848 Hands: Design and Control for Dexterous Manipulation

Research related to hands has increased dramatically over the past decade. Robot hand innovation may be at an all time high, with new materials and manufacturing techniques promoting an explosion of ideas. Hands have become a priority in virtual reality and telepresence. Even the study of how people use their hands is seeing the growth of new ideas and themes.

With all of this attention on hands, are we close to a breakthrough in dexterity, or are we still missing some things needed for truly dexterous manipulation?

In this course, we will survey robotic hands and learn about the human hand with the goal of pushing the frontiers on hand design and control for dexterous manipulation. We will consider the necessary kinematics and dynamics for dexterity, what sensors are required to carry out dexterous interactions, the importance of reflexes and compliance, the role of machine learning in grasping and manipulation, and the challenge of uncertainty. We will explore state of the art manufacturing and design techniques, including innovations in soft robotics and embedded sensing. We will examine the human hand: its structure, sensing capabilities, human grasp choice and control strategies for inspiration and benchmarking. Students will be asked to present one or two research papers, participate in discussions and short research or design exercises, and carry out a final project.

Instructor: Nancy Pollard

• Email: nsp@cs.cmu.edu

• Office: EDSH 227, Phone x8-1479
• Office hours: send email to schedule an appointment

Piazza: We will use piazza for announcements and class discussion. Our piazza link is https://piazza.com/class/lrhwmvk3t7v1eq/

Syllabus

Week of	Mon	Wed
Jan 15	NO CLASS: Martin Luther King Day	Course Introduction Assignment 0 OUT Slides
Jan 22	Some History of Robot Hands Slides	Developing a Dexterous Soft Hand (Pragna Mannam)
Jan 29	The Human Hand Slides References	Taxonomies, Synergies, and Benchmarks Slides References
Feb 5	Final Project Brainstorming Slides	FFLEX Hand V1 & V2 (Zilin Si)
Feb 12	Third Thumb (Danny Magruder)	Mocap Lab Demo (WeH 1334)
Feb 19	Final Project Pitches	The Palm (Sandhya Sridhar)
Feb 26	Dexterous Soft Hand Design (Dominik Bauer)	Direct Drive Hands (Quanting Xie)
Mar 4	SPRING BREAK	SPRING BREAK
Mar 11	NO CLASS	Hand Tracking and Simulation in VR (Edward Lu)
Mar 18	DexPilot (Sruti Srinidhi)	Hand Mapping for Teleoperation (Madeleine Weaver)
Mar 25	Final Project Checkpoint	Topics on Prosthetic Hands Slides References
Apr 1	GelSight EndoFlex (Jean Udompanyawit)	Hands and Sensing References
Apr 8	Round Table Discussion (Eclipse Day)	Math for Grasping and Manipulation References Slides
Apr 15	Active Surfaces for Manipulation (Jinyun Xu)	Design Optimization References
Apr 22	Learning for Dexterity References	More Learning and Hands in Graphics References
Apr 29	FINAL PRESENTATIONS (1-4PM PH A18B)

Topics of interest for this course include:

• ROBOT HANDS
  We will study existing robot hands and their properties. We'll look at soft hands, underactuated hands, hands utilizing synergies, highly dexterous hands, prosthetic hands, universal grippers, and everything else we can uncover. We will compare these hands along dimensions of strength, weight, sensing capability, ability to securely grasp, ability to manipulate, and aspects which make each hand unique.

• HUMAN HANDS
  We will learn about human hand kinematic and dynamic properties and the internal structure of the hand, including our many sensors. We will consider hand evolution, review what is known about how we control our hands, look at where human hand simulations succeed and fail, and evaluate the human hand along the same guidelines as we evaluate the various robotic hands to uncover obvious gaps and consider their importance.

• HUMAN GRASPING IN THE WILD
  How many grasps? How many manipulation actions?
  Many taxonomies of human grasping have been proposed, and they seem to be getting more detailed and complex with the passage of time, due to evident variety in hand shape, motion, force, stiffness, and intent. However, there are some clear organizing principles and strong similarities across classes of grasps and manipulation actions. We will attempt to identify the few canonical grasp varieties that are most important to human grasping in the wild and understand how to adapt and move between them -- in other words we will attempt to write a complete playbook for everyday grasping and manipulation.

• HAND SHAPE AND JOINT LIMITS
  The shape of a robot finger, and the "negative space" between fingers and palm as the hand closes can strongly influence grasp success. Joint limits are also very important, yet the role of joint limits in optimizing stability and force delivery has been little (not at all?) studied. We will explore both of these aspects of robot hand design.

• COMPLIANCE AND REFLEXES
  I believe two of the reasons for success in human grasping are passive compliance and reflexes, resulting in very fast, carefully shaped responses to contact and collision. We will study what is known about reflexes and recent research results on human finger compliance, look at some real world examples, and see if research relating to series elastic actuators and other compliant design elements aligns with what we think we want in a grasp / manipulation controller.

• UNCERTAINTY
  We will look into the literature on learning, optimization, and planning that is relevant to grasping and manipulation in the cluttered and uncertain real world. One particular question we will consider is how data captured from real world interactions can improve our ability to handle and plan for uncertainty.

• QUALITY AND METRICS
  How do we analyze grasps and what makes for a high quality grasp? There is an established mathematics for grasp analysis, and there are more than a dozen measures out there, but do any of them capture our intuition for what we want a grasp to be? Are any of them good enough to be used as the basis for grasp optimization? Grasping is not just a final result, but a process that may involve some manipulation in order to acquire the object. How should we evaluate a manipulation plan or policy? We will look at various quality measures and objectives that have been posed and stack them up against our intuition for what we really want as an objective.

Grading Information

• Final Project 60%
• In-class paper presentation 30%
• Class Participation 10%

Paper Presentation

You will be required to present one recent research paper during the course of the semester. You will have the entire class period available to you, but you are not required to use all of this time. A typical presentation time may be 45 to 60 minutes. (Less than 30 minutes is too short.)

In your presentatation, please cover the points listed below. In general, plan for about one minute per slide. Make sure the text is easy to read, and replace text with images, plots, videos, and diagrams where it helps with explanation. You will be asked to turn in your presentation material by sending it in an email to me either before or after your presentation.

My goal is that the class can learn, think about, and discuss the material in the paper you choose, and you will be graded based on clarity of the presentation and on providing points that can raise discussion in the class. (Don't worry, I will take into account that different papers are more and less easy to present clearly and more and less easy to discuss!)

Here are the points to cover. You don't have to follow this order, but most papers do. If you have a collection of papers or a survey, your approach will be slightly different.

• Introduce the authors. Who are the authors? What do we know about their lab? What else have they done? (briefly)
• Introduce the paper. What is the goal of the paper and why is it interesting and important? This is sometimes a good place to show a preview of results so we have an idea of what is coming.
• Related work. What has been done in this area already? What background are they building on?
• Specific contribution. What is the specific gap that this paper is trying to close and what is their stated contribution?
• Techniques. How did the authors solve this problem? Tell us whatever is interesting to learn here in the time you have available. If there are parts that were hard to understand on first reading, go to the board or find some way to explain them to us now that you've had a chance to figure them out.
• Results. Show the results and help the audience sort out what is interesting and important (especially if there are endless plots).
• Discussion. In this section you should give your own opinion based on your study of the paper. Be sure to leave plenty of time for this section. Here are some questions to think about. Are the authors trying to solve an interesting problem in the first place? How good a job did they do? Are the results useful? To whom? What might you change or how might you solve this problem in a better way? Where do you expect the field to go from here, i.e., what important problems didn't the authors solve? Don't limit yourself to these questions -- just use them as a starting point to organize your own thoughts about the paper.

Final Project

Overview:
In this assignment, you can work individually or in a group of 2-3 to implement a project you are interested in. This project should be related to some aspect of hands and/or dexterous manipulation. You can choose from the topics covered in our class, or you can find another topic of your choice. Clear all topics with me early on. I can also help you find resources and choose an appropriate scope.

Deadlines:

• BEFORE 02/19: Meet with me about your project ideas.
• 02/19: In-class final project pitches (10 minutes).
• 02/26: Written proposal submission DUE (2-3 pages).
• 03/25: In-class final project checkpoint (10 minutes).
• 04/24: Final project report DUE (6-10 pages).
• Finals Week: Final project presentations (20 minutes).