[SCS dragon logo]

15-463 (15-862): Computational Photography
Computer Science Department
Carnegie Mellon University

INSTRUCTOR: Alexei (Alyosha) Efros (Office hours: Tuesday 4:30-5:30, Smith 225)
TA: Ronit Slyper (Office hours: See our google group home page)
UNIVERSITY UNITS: 12
SEMESTER: Fall 2010
DISCUSSION GROUP:  googlegroups (contact Ronit)
WEB PAGE: http://graphics.cs.cmu.edu/courses/15-463/
LOCATION:
GHC 4301
TIME
: T R 12:00--1:20 PM

COURSE OVERVIEW:
Computational Photography is an emerging new field created by the convergence of computer graphics, computer vision and photography. Its role is to overcome the limitations of the traditional camera by using computational techniques to produce a richer, more vivid, perhaps more perceptually meaningful representation of our visual world.

The aim of this advanced undergraduate course is to study ways in which samples from the real world (images and video) can be used to generate compelling computer graphics imagery. We will learn how to acquire, represent, and render scenes from digitized photographs. Several popular image-based algorithms will be presented, with an emphasis on using these techniques to build practical systems. This hands-on emphasis will be reflected in the programming assignments, in which students will have the opportunity to acquire their own images of indoor and outdoor scenes and develop the image analysis and synthesis tools needed to render and view the scenes on the computer.

TOPICS TO BE COVERED:

  • Cameras, Image Formation
  • Visual Perception
  • Image and Video Processing (filtering, anti-aliasing, pyramids)
  • Image Manipulation (warping, morphing, mosaicing, matting, compositing)
  • Modeling and Synthesis using Lots of Data
  • High Dynamic Range Imaging and Tone Mapping
  • Image-Based Lighting
  • Image-Based Rendering
  • Non-photorealistic Rendering

PREREQUISITES:
Programming experience and familiarity with linear algebra and calculus is assumed.  Some background in computer graphics, computer vision, or image processing is helpful.  This class does not significantly overlap with 15-462 and can be taken concurrently.
Graduate Students: a small number of graduate students will be allowed to take the graduate version of this course (15-862) with the permission of the instructor. Students taking 15-862 will be required to do more substantial assignments as well as a research-level final paper.
Note: if the system doesn’t let you sign up, or puts you on the waitlist, do talk to me.

PROGRAMMING ASSIGNMENTS:

 

·  Project 0: Matlab Worksheets (ongoing)

class results
Class Choice Award: Billy Keyes

·  Project 1: Images of the Russian Empire -- colorizing the Prokudin-Gorskii photo collection
http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/images/3-strip-1.jpghttp://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/images/3-8086-left.jpg
class results
Class Choice Award: Josip Djolonga

·  Project 2: Image Resizing by Seam Carving (for undergrads)
 
·  Project 2g: Gradient Domain Editing (for grad students)
 
class results
Class Choice Award (undergrad): Josip Djolonga
Class Choice Award (grad): Scott Griffith and You Jia

· Project 3: Face morphing and modeling:
http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/images/image1.jpghttp://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/images/imagemc.jpghttp://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/images/image2.jpg
class results

·  Project 4: Stitching Photo Mosaics (all)
http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/images/panorama.jpg
· Project 4g: Single-View Reconstruction (grad students only)
http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/463_files/image060.gif
Class Choice Award (undergrad): Junjie Liang
Class Choice Award (grad): Nick Vandal
class results

Final Projects!

 

TEXT:
Since Computational Photography is such a new discipline, no comprehensive textbook exists for use in the class. Therefore, there is no required text. Various course notes and papers will be made available.  We will also use the following almost-finished textbook (available online) as reference:

            Computer Vision: Algorithms and Applications, Richard Szeliski

There is a number of other fine texts that you can use for general reference:

Computer Vision: The Modern Approach, Forsyth and Ponce
Photography (8th edition), London and Upton, (a great general guide to taking pictures)
Vision Science: Photons to Phenomenology, Stephen Palmer (great book on human visual perception)
Science for the Curious Photographer, Charles Johnson (an fun and easy read)
Digital Image Processing, 2nd edition, Gonzalez and Woods (a good general image processing text)
The Art and Science of Digital Compositing, Ron Brinkmann (everything about compositing)
Multiple View Geometry in Computer Vision, Hartley & Zisserman (a bible on recovering 3D geometry) [on reserve]
The Computer Image, Watt and Policarpo (a nice “vision for graphics” text, somewhat dated)
3D Computer Graphics (3rd Edition), Watt (a good general graphics text)
Fundamentals of Computer Graphics, Peter Shirley (another good general graphics text)
Linear Algebra and its Applications, Gilbert Strang (a truly wonderful book on linear algebra)

CLASS NOTES
The instructor is extremely grateful to a large number of researchers for making their slides available for use in this course.  Steve Seitz and Rick Szeliski have been particularly kind in letting me use their wonderful lecture notes.  In addition, I would like to thank Paul Debevec, Stephen Palmer, Paul Heckbert, David Forsyth, Steve Marschner, Fredo Durand and others, as noted in the slides.  The instructor gladly gives permission to use and modify any of the slides for academic and research purposes. However, please do also acknowledge the original sources where appropriate.

   

CLASS SCHEDULE:

CLASS DATE

TOPICS

Material

Tu Aug 24

Introduction

Th Aug 26

Capturing Light… in man and machine

 

Tu Aug 31

Sampling and Reconstruction
alias

 

Th Sept 2

The Frequency Domain

Tu Sept 7

Point Processing (by Ronit)

Th Sept 9

NO CLASS

Th Sept 14

Continue with Freq. Domain: gradients, frequency perception, compression, morphology

Th Sept 16

Image Blending and Compositing

Slides (ppt

, pdf)

Additional Reading:
Burt and Adelson, A multiresolution spline with application to image mosaics, ACM ToG (1983)
McCann & Pollard, Real-Time Gradient-Domain Painting, SIGGRAPH 2008
Agarwala et al, Interactive Digital Photomontage, SIGGRAPH 2004

Tu Sept 21

Image Warping
    rotated

Slides (ppt

, pdf)

Tu Sept 28

Image Morphing
catwoman

Slides (ppt

, pdf)

Th Sept 30

Data-driven Methods: Faces
BlanzFig3

Slides (ppt

, pdf)
·  Rowland and Ferrett, “Manipulating Facial Appearance through Shape and Color”, CG&A, 1995

·  Additional Reading:

  1. Blanz and Vetter, “A Morphable Model for the Synthesis of 3D Faces”, SIGGRAPH 1999
  2. Cootes, Edwards, and Taylor, “Active Appearance Models”,  ECCV 1998

Tu Oct 5

Data-driven Methods: Video and Texture

 

 

 

Th Oct 7

Data-driven Methods: Features, Histograms, and Image Comparisons
normal-binning

Tu Oct 12

Data-driven Methods: Internet Data

Reading: Hays & Efros, Scene Completion Using Millions of Photographs

Additional Reading:
1.  im2gps
2.  Creating and Exploring a Large Photorealistic Virtual Space

Th Oct 14

The Camera
obscura 

Tu Oct 19

Modeling Light

Slides (ppt

, pdf)

Th Oct 21

Homographies and Mosaics
pan2

 

Sun Oct 24

More Mosaic Madness

·       Slides (ppt

, pdf)

·        Additional Reading: Rick Szeliski, Image Alignment and Stitching, A Tutorial (DRAFT)

Th Oct 28

Automatic Alignment

Tu Nov 2

Multi-perspective Panoramas
IMG_3371

Th Nov 4

Single View Reconstruction
pic4  pic5

·  Slides (ppt, pdf)

·  Reading: Horry et al, “Tour into the Picture”, SIGGRAPH ‘97

Tu Nov 9

More Single View Geometry
fra_small_col     Picture1

·  Slides (ppt, pdf)

·  Additional Reading: Single-view Reconstruction

CAMERAS:
Although it is not required, students are highly encouraged to obtain a digital camera for use in the course (one can get a pretty good camera for under $150). A

METHOD OF EVALUATION:
Grading will be based on a set of programming and written assignments (60%), an exam (20%) and a final project (20%).  For the programming assignments, students will be allowed a total of 5 (five) late days per semester; each additional late day will incur a 20% penalty.

Students taking 15-862 will also be required to submit a conference-style paper describing their final project.

MATLAB:
Students will be encouraged to use Matlab (with the Image Processing Toolkit) as their primary computing platform.  Besides being a great prototyping environment, Matlab is particularly well-suited for working with image data and offers tons of build-in image processing functions.  Here is a link to some useful Matlab resources

PREVIOUS OFFERINGS OF THIS COURSE:
Previous offerings of this course can be found here.

SIMILAR COURSES IN OTHER UNIVERSITIES: