15-463: Computational Photography

Face Morphing

In this assignment you will produce a "morph" animation of your face into another student's face. Each of you will generate two seconds of animation, which we will record onto video for you. The final video will thus be a metamorphosis through each of the faces in the class.

A morph is a simultaneous warp of the image shape and a cross-dissolve of the image colors. The cross-dissolve is the easy part; controlling and doing the warp is the hard part. The warp is controlled by defining a correspondence between the two pictures. The correspondence should map eyes to eyes, mouth to mouth, chin to chin, ears to ears, etc., to get the smoothest transformations possible.

We will take pictures
of everyone in the class. Each student
will then get a starting image ("picture A") and an ending image
("picture B") for the animation sequence. Picture A will be your
face. Picture B will be another student or other primate.
Your pictures will be available here:

http://graphics.cs.cmu.edu/courses/15-463/2010_spring/hw/proj3/faces

Use your photo as Picture A and the next person's photo (in the directory
listing) as Picture B.

You'll morph still
picture A into still picture B and produce 61 frames of animation numbered
0-60, where frame 0 must be identical to picture A and frame 60 must be
identical to picture B. In the video, each frame will be displayed for 1/30 of
a second.
Name your frames `morph??.jpg`
where ?? is 00 to 60.

First, you will need to
define pairs of corresponding points on the two images by hand (the more
points, the better the morph, generally).
The simplest way is probably to use the
`cpselect`
tool or write your own little tool using
`ginput`
and
`plot`
commands (with
`hold on`
and
`hold off`
).
Now, you need to provide a triangulation of these points that will be
used for morphing. You can compute a
triangulation any way you like, or even define it by hand.
A Delaunay triangulation (see `dalaunay` and related functions) is a
good choice since it does not produce overly skinny triangles.
You can compute the Delaunay triangulation on
either of the point sets (but not both -- the triangulation has to be the
same throughout the morph!).
But the
best approach would probably be to compute the triangulation at midway shape
(i.e. mean of the two point sets) to lessen the potential triangle
deformations.

You need to write a function:

morphed_im = morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac);

that
produces a warp between
`im1`
and
`im2`
using point correspondences defined in
`im1_pts`
and
`im2_pts`
(which
are both n-by-2 matrices of (x,y) locations) and the triangulation structure
`tri`
.
The parameters
`warp_frac`
and
`dissolve_frac`
control shape warping and
cross-dissolve, respectively.
In particular, images
`im1`
and
`im2`
are
first warped into an intermediate shape configuration controlled by
`warp_frac`, and
then cross-dissolved according to
`dissolve_frac`.
For interpolation, both
parameters lie in the range [0,1]. They are the only parameters that will vary
from frame to frame in the animation. For your starting frame, they will both
equal 0, and for your ending frame, they will both equal 1.

Given a new intermediate shape, the main task is implementing an affine warp for each triangle in the triangulation from the original images into this new shape. This will involve computing an affine transformation matrix A between two triangles:

A = computeAffine(tri1_pts,tri2_pts)

A set
of these transformation matrices will then need to be used to implement an
inverse warp (as discussed in class) of all pixels. Functions
`tsearch`
and
`interp2`
can come very handy here.
But note that you are not allowed to use Matlab's build-in offerings for computing transformations, (e.g.
`imtransform`, `cp2tform`, `maketform`,
etc).
Note, however, that
`tsearch`
assumes that your triangulation is always Delaunay.
In our case, this might not always be true --
you may start with a Delaunay triangulation, but through the coarse of
the morph it might produce skinny triangles and stop being Delaunay.
David Martin from Boston College
has kindly given access to his versions of
`tsearch` that work on any triangulation: mytsearch.m and mytsearch.c
(compile by typing "`mex mytsearch.c`" in Matlab;
if you get compiler errors try replacing mytsearch.c with this one).

Several fun things are possible with our new morpher.
For example, we can compute the mean face of 15-463 students.
This would involve: 1) computing the average
shape, 2) warping all faces into that shape, and 3) averaging the colors
together.
However, this would also
require a __consistent__ labeling of all the faces.
So, what we will do is ask everyone to label
their own face (and maybe a few others of their own choosing), but do it in a consistent
manner as shown in the following two images: points,
point_labels.
For each face image, you should put on your
website a text file with coordinates of x,y positions, one per line (43 lines
total).
You can read/write such files
with
the
`save -ascii`
and
`load -ascii`
commands.
If everyone does this in a timely manner,
then everyone can use all of the data to compute their mean image.
Show the mean image that you got, as well as
1) your face warped into the average geometry, and 2) the average face warped
into your geometry.
To get you started,
here are the faces and points from
previous years.

I'll copy .txt files into the faces directory as y'all put them up:

http://graphics.cs.cmu.edu/courses/15-463/2010_spring/hw/proj3/faces

- (up to
**20 points**) Use the mean face to produce caricatures, masculitizing and feminizing transforms and any other fun things you can think of. - (up to
**20 points**) Use the class data to compute a PCA basis for the face space. Try performing caricatures and other transformations in the new basis. Compare your results to doing this in normal basis. Are the results better? - (up to
**20 points**) Implement a non-triangle based morphing algorithm. There are a lot out there, meaning this Bell/Whistle is somewhat underdefined. Take it on carefully. I'm, personally, partial to some sort of kernel-weighted regression scheme (i.e. do a weighted least-squares estimate of the transform at each pixel, where the weights on the control points are determined by the distance from the pixel in question -- this will be slower, but will have nice continuity properties). Points will be based both on how good your new scheme looks and how hard it was to actually code up. (But mostly the former.)

To get full credit, you will need to implement a warping algorithm and turn in a video morph. You will also need to compute the mean face and show results of warping your face into the mean face. Doing some Bells & Whistles will earn you extra points. Students in the grad version of the class are expected to do at least 15 points of bells and whistles.

For turn-in, use the `proj3/www` and `proj3/code`
folders, as per the submission
instructions. Remember to start early, or
this
could happen to you!