Skip navigation.

Data-driven Visual Similarity for Cross-domain Image Matching

Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, and Alexei A. Efros

Abstract

The goal of this work is to find visually similar images even if they appear quite different at the raw pixel level. This task is particularly important for matching images across visual domains, such as photos taken over different seasons or lighting conditions, paintings, hand-drawn sketches, etc. We propose a surprisingly simple method that estimates the relative importance of different features in a query image based on the notion of "data-driven uniqueness". We employ standard tools from discriminative object detection in a novel way, yielding a generic approach that does not depend on a particular image representation or a specific visual domain. Our approach shows good performance on a number of difficult cross-domain visual tasks e.g., matching paintings or sketches to real photographs. The method also allows us to demonstrate novel applications such as Internet re-photography, and painting2gps.

Citation

Abhinav Shrivastava, Tomasz Malisiewicz, Abhinav Gupta, and Alexei A. Efros. Data-driven visual similarity for cross-domain image matching. ACM Transaction of Graphics (TOG) (Proceedings of ACM SIGGRAPH ASIA), 30(6), 2011. [BiBTeX]

Links

Fast Simulation of Skeleton-Driven Deformable Body Characters

Junggon Kim and Nancy S. Pollard

Abstract

We propose a fast physically based simulation system for skeleton-driven deformable body characters. Our system can generate realistic motions of self-propelled deformable body characters by considering the two-way interactions among the skeleton, the deformable body, and the environment in the dynamic simulation. It can also compute the passive jiggling behavior of a deformable body driven by a kinematic skeletal motion. We show that a well-coordinated combination of (1) a reduced deformable body model with nonlinear finite elements, (2) a linear-time algorithm for skeleton dynamics, and (3) explicit integration can boost simulation speed to orders of magnitude faster than existing methods, while preserving modeling accuracy as much as possible. Parallel computation on the GPU has also been implemented to obtain an additional speedup for complicated characters. Detailed discussions of our engineering decisions for speed and accuracy of the simulation system are presented in the paper. We tested our approach with a variety of skeleton-driven deformable body characters, and the tested characters were simulated in real-time or near real-time.

Citation

Junggon Kim and Nancy S. Pollard. Fast simulation of skeleton-driven deformable body characters. ACM Transactions on Graphics, 30(5), October 2011. [BiBTeX]

Links

Mid-level smoke control for 2D animation

Alfred Barnat, Zeyang Li, James McCann, and Nancy S. Pollard

Abstract

In this paper we introduce the notion that artists should be able to control fluid simulations by providing examples of expected local fluid behavior (for instance, an artist might specify that magical smoke often forms star shapes). As our idea fits between high-level, global pose control and low-level parameter adjustment, we deem it mid-level control. We make our notion concrete by demonstrating two mid-level controllers providing stylized smoke effects for two-dimensional animations. With these two controllers, we allow the artist to specify both density patterns, or particle motifs, which should emerge frequently within the fluid and global texture motifs to which the fluid should conform. Each controller is responsible for constructing a stylized version of the current fluid state, which we feed-back into a global pose control method. This feedback mechanism allows the smoke to retain fluid-like behavior, while also attaining a stylized appearance suitable to integration with 2D animations. We integrate these mid-level controls with an interactive animation system, in which the user can control and keyframe all animation parameters using an interactive timeline view.

Citation

Alfred Barnat, Zeyang Li, James McCann, and Nancy S. Pollard. Mid-level smoke control for 2d animation. In Proceedings of Graphics Interface 2011, GI '11, pages 25–32. Canadian Human-Computer Communications Society, 2011. [BiBTeX]

Links

A Puppet Interface for Retrieval of Motion Capture Data

Naoki Numaguchi, Atsushi Nakazawa, Takaaki Shiratori, and Jessica K. Hodgins

Abstract

Intuitive and efficient retrieval of motion capture data is essential for effective use of motion capture databases. In this paper, we describe a system that allows the user to retrieve a particular sequence by performing an approximation of the motion with an instrumented puppet. This interface is intuitive because both adults and children have experience playacting with puppets and toys to express particular behaviors or to tell stories with style and emotion. The puppet has 17 degrees of freedom and can therefore represent a variety of motions. We develop a novel similarity metric between puppet and human motion by computing the reconstruction errors of the puppet motion in the latent space of the human motion and those of the human motion in the latent space of the puppet motion. This metric works even for relatively large databases. We conducted a user study of the system and subjects could find the desired motion with reasonable accuracy from a database consisting of everyday, exercise, and acrobatic behaviors.

Citation

Naoki Numaguchi, Atsushi Nakazawa, Takaaki Shiratori, and Jessica K. Hodgins. A puppet interface for retrieval of motion capture data. In Proc. ACM SIGGRAPH/Eurographics Symposium on Computer Animation, August 2011. [BiBTeX]

Links

Content Retargeting Using Parameter-Parallel Facial Layers

Natasha Kholgade, Iain Matthews, and Yaser Sheikh

Abstract

Facial motion retargeting approaches often transfer expressions by establishing correspondences between shared units of motion, such as action units, or spatial correspondences of landmarks between the source actor and target character faces. When the actor and character are structurally dissimilar, shared units of motion or spatial landmarks may not exist, and subtle styles of performance may differ. We present a method to deconstruct the content of an s facial expression into three layers using an additive composition function, transfer the content to parameter-parallel layers for the character, and reconstruct the s expression using the same composition function. Our algorithm uses a parameter-parallel layered model of facial expression for both the actor and character, separating the content of facial expressions into emotion, speech, and eye-blink layers. Facial motion in each layer is embedded in simplicial bases, each of which encodes semantically significant configurations of the face. We show the transfer of facial motion capture and video-based tracking of the eyes and mouth of an actor to a number of faces with dissimilar facial structure and expressive disposition.

Citation

Natasha Kholgade, Iain Matthews, and Yaser Sheikh. Content retargeting using parameter-parallel facial layers. In Proc. ACM SIGGRAPH/Eurographics Symposium on Computer Animation, August 2011. [BiBTeX]

Links

Modeling and animating eye blinks

Laura C. Trutoiu, Elizabeth J. Carter, Iain Matthews, and Jessica K. Hodgins

Abstract

Facial animation often falls short in conveying the nuances present in the facial dynamics of humans. In this article, we investigate the subtleties of the spatial and temporal aspects of eye blinks. Conventional methods for eye blink animation generally employ temporally and spatially symmetric sequences; however, naturally occurring blinks in humans show a pronounced asymmetry on both dimensions. We present an analysis of naturally occurring blinks that was performed by tracking data from high-speed video using active appearance models. Based on this analysis, we generate a set of key-frame parameters that closely match naturally occurring blinks. We compare the perceived naturalness of blinks that are animated based on real data to those created using textbook animation curves. The eye blinks are animated on two characters, a photorealistic model and a cartoon model, to determine the influence of character style. We find that the animated blinks generated from the human data model with fully closing eyelids are consistently perceived as more natural than those created using the various types of blink dynamics proposed in animation textbooks.

Citation

Laura C. Trutoiu, Elizabeth J. Carter, Iain Matthews, and Jessica K. Hodgins. Modeling and animating eye blinks. ACM Transactions on Applied Perception, 8:17:1–17:17, August 2011. [BiBTeX]

Links

Motion Capture from Body-Mounted Cameras

Takaaki Shiratori, Hyun Soo Park, Leonid Sigal, Yaser Sheikh, and Jessica K. Hodgins

Abstract

Motion capture technology generally requires that recordings be performed in a laboratory or closed stage setting with controlled lighting. This restriction precludes the capture of motions that require an outdoor setting or the traversal of large areas. In this paper, we present the theory and practice of using body-mounted cameras to reconstruct the motion of a subject. Outward-looking cameras are attached to the limbs of the subject, and the joint angles and root pose are estimated through non-linear optimization. The optimization objective function incorporates terms for image matching error and temporal continuity of motion. Structure-from-motion is used to estimate the skeleton structure and to provide initialization for the non-linear optimization procedure. Global motion is estimated and drift is controlled by matching the captured set of videos to reference imagery. We show results in settings where capture would be difficult or impossible with traditional motion capture systems, including walking outside and swinging on monkey bars. The quality of the motion reconstruction is evaluated by comparing our results against motion capture data produced by a commercially available optical system.

Citation

Takaaki Shiratori, Hyun Soo Park, Leonid Sigal, Yaser Sheikh, and Jessica K. Hodgins. Motion capture from body-mounted cameras. ACM Transactions on Graphics, 30(4), August 2011. [BiBTeX]

Links

Interactive Region-Based Linear 3D Face Models

J. Rafael Tena, Fernando De la Torre, and Iain Matthews

Abstract

Linear models, particularly those based on principal component analysis (PCA), have been used successfully on a broad range of human face-related applications. Although PCA models achieve high compression, they have not been widely used for animation in a production environment because their bases lack a semantic interpretation. Their parameters are not a natural set for animators to work with. In this paper we present a linear face modelling approach that allows intuitive click-and-drag interaction for animation. Our model is composed of a collection of PCA sub-models that are independently trained but share boundaries. Boundary consistency and user-given constraints are enforced in a soft least mean squares sense to give flexibility to the model while maintaining coherence. Our results also show that the region-based model generalises better than its holistic counterpart when describing previously unseen motion capture data from multiple subjects. The decomposition of the face into several regions, which we determine automatically from training data, gives the user localised manipulation control. This allows the model to be used for face posing and animation in a natural and intuitive style.

Citation

J. Rafael Tena, Fernando De la Torre, and Iain Matthews. Interactive Region-Based Linear 3D Face Models. ACM Transactions on Graphics, 30(4), August 2011. [BiBTeX]

Links

Direct Control of Simulated Non-human Characters

Junggon Kim and Nancy S. Pollard

Abstract

We present an intuitive technique to control simulated self-propelled characters. In our system, a user can control the motions of a character by directly guiding the character to a desired position with mouse drags while a physics simulation determines the character's motion. Based on the user input, the system automatically computes the actuator command that causes the character to follow the user's intention as closely as possible while respecting the underlying physics. This direct control can be more intuitive to use than existing methods, such as controlling character joints to track a given joint trajectory or set of keyframes, especially when physically plausible dynamic motions are desired. We show the power of our method by creating realistic motions of various kinds of characters in our interactive user interface system, including rigid characters, characters with deformable bodies and rigid skeletons, and self locomoting characters whose bodies form closed loops.

Citation

Junggon Kim and Nancy S. Pollard. Direct control of simulated non-human characters. IEEE Computer Graphics and Applications, 31(4):56–65, July 2011. [BiBTeX]

Links