CMU 15-463: Computational Photography


	15-463 (15-862): Computational Photography Computer Science Department Carnegie Mellon University

INSTRUCTOR: Alexei (Alyosha) Efros (Office hours: Th 3:30-4:30pm, Smith 225)
TA: Natasha Kholgade (Office hours: Tu 2-3pm)
UNIVERSITY UNITS: 12
SEMESTER: Fall 2011
WEB PAGE: http://graphics.cs.cmu.edu/courses/15-463/
LOCATION: GHC 4301
TIME: T R 12:00--1:20 PM

COURSE OVERVIEW:
Computational Photography is an emerging new field created by the convergence of computer graphics, computer vision and photography. Its role is to overcome the limitations of the traditional camera by using computational techniques to produce a richer, more vivid, perhaps more perceptually meaningful representation of our visual world.

The aim of this advanced undergraduate course is to study ways in which samples from the real world (images and video) can be used to generate compelling computer graphics imagery. We will learn how to acquire, represent, and render scenes from digitized photographs. Several popular image-based algorithms will be presented, with an emphasis on using these techniques to build practical systems. This hands-on emphasis will be reflected in the programming assignments, in which students will have the opportunity to acquire their own images of indoor and outdoor scenes and develop the image analysis and synthesis tools needed to render and view the scenes on the computer.

TOPICS TO BE COVERED:

Cameras, Image Formation
Visual Perception
Image and Video Processing (filtering, anti-aliasing, pyramids)
Image Manipulation (warping, morphing, mosaicing, matting, compositing)
Modeling and Synthesis using Lots of Data
High Dynamic Range Imaging and Tone Mapping
Image-Based Lighting
Image-Based Rendering
Non-photorealistic Rendering

PREREQUISITES:
Programming experience and familiarity with linear algebra and calculus is assumed. Some background in computer graphics, computer vision, or image processing is helpful. This class does not significantly overlap with 15-462 and can be taken concurrently.
Graduate Students: a small number of graduate students will be allowed to take the graduate version of this course (15-862) with the permission of the instructor. Students taking 15-862 will be required to do more substantial assignments as well as a research-level final paper.
Note: if the system doesn’t let you sign up, or puts you on the waitlist, do talk to me.

PROGRAMMING ASSIGNMENTS:

· Project 1: Images of the Russian Empire – colorizing the Prokudin-Gorskii photo collection

class results
Class Choice Award: Pinar Ozlen (ugrad), Hugh Cover (grad)

· Project 2: Image Resizing by Seam Carving (for undergrads)

· Project 2g: Gradient Domain Editing (for grad students)

class results
Class Choice Award: Ryan Oksenhorn (ugrad); Julia Schwartz (grad)

· Project 3: Face morphing and modeling:

class results
Class Choice Award: Yiling Tay

· Project 4: Home-made Camera Obscura

class results
Class Choice Award: Max Hawkins

· Project 5: Stitching Photo Mosaics

class results
Class Choice Award: Sang Tian (ugrad), Chris Harrisson (grad)

· Final Projects!

class results

TEXT:
There is now a textbook that covers most (if not all) of the topics related to Computational Photography. This will be the primary reference for the course:

Computer Vision: Algorithms and Applications, Richard Szeliski, 2010

There is a number of other fine texts that you can use for general reference:

Photography (8^th edition), London and Upton, (a great general guide to taking pictures)
Vision Science: Photons to Phenomenology, Stephen Palmer (great book on human visual perception)
Computer Vision: The Modern Approach, Forsyth and Ponce
Digital Image Processing, 2nd edition, Gonzalez and Woods (a good general image processing text)
The Art and Science of Digital Compositing, Ron Brinkmann (everything about compositing)
Multiple View Geometry in Computer Vision, Hartley & Zisserman (a bible on recovering 3D geometry) [on reserve]
The Computer Image, Watt and Policarpo (a nice “vision for graphics” text, somewhat dated)
3D Computer Graphics (3rd Edition), Watt (a good general graphics text)
Fundamentals of Computer Graphics, Peter Shirley (another good general graphics text)
Linear Algebra and its Applications, Gilbert Strang (a truly wonderful book on linear algebra)

CLASS NOTES
The instructor is extremely grateful to a large number of researchers for making their slides available for use in this course. Steve Seitz and Rick Szeliski have been particularly kind in letting me use their wonderful lecture notes. In addition, I would like to thank Paul Debevec, Stephen Palmer, Paul Heckbert, David Forsyth, Steve Marschner and others, as noted in the slides. The instructor gladly gives permission to use and modify any of the slides for academic and research purposes. However, please do also acknowledge the original sources where appropriate.

CLASS SCHEDULE:

CLASS DATE	TOPICS	Material
Tu Aug 30	Introduction	Slides (ppt, pdf)
Th Sept 1	Capturing Light… in man and machine	Slides (ppt, pdf)
Tu Spet 6	Sampling and Reconstruction	Slides (ppt pdf) start Szeliski Ch 3
Th Spet 8	Template Matching and Pyramids	continue slides (ppt pdf) continue Szeliski Ch 3
Tu Sept 13	The Frequency Domain	Slides (ppt, pdf) continue Szeliski Ch 3
Th Sept 15	Filtering	Continue slides (ppt, pdf) continue Szeliski Ch 3
Tu Sept 20	Image Blending and Compositing	Slides (ppt, pdf) continue Szeliski Ch.3 + Sec 9.3.4 Additional Reading: Burt and Adelson, Amultiresolution spline with application to image mosaics, ACM ToG (1983) McCann & Pollard, Real-Time Gradient-Domain Painting, SIGGRAPH 2008 Agarwala et al, Interactive Digital Photomontage, SIGGRAPH 2004
Th Sept 22	Point Processing and Image Warping	Slides (ppt, pdf) Slides (ppt, pdf) continue Szeliski Ch 3
Tu Sept 27	Image Morphing	Slides (ppt, pdf) continue Szeliski Ch 3
Th Sept 29	Data-driven Methods: Faces	Slides (ppt, pdf) · Rowland and Ferrett, “Manipulating Facial Appearance through Shape and Color”, CG&A, 1995 · Additional Reading: Blanz and Vetter, “A Morphable Model for the Synthesis of 3D Faces”, SIGGRAPH 1999 Cootes, Edwards, and Taylor, “Active Appearance Models”, ECCV 1998
Tu Oct 4	Data-driven Methods: Video and Texture	Slides (ppt, pdf) Reading: Schold et al, Video Textures, 2000 Reading: Efros & Lreung, Texture Synthesis by Non-parametric Sampling, 1999 Additional Reading: Image Analogies
Th Oct 6	Data-driven Methods: Features, Histograms, and Image Comparisons	Slides (ppt, pdf)
Tu Oct 11	Data-driven Methods: Internet-sized Data	Reading: Hays & Efros, Scene Completion Using Millions of Photographs Additional Reading: 1. im2gps 2. Creating and Exploring a Large Photorealistic Virtual Space 3. ShadowDraw
Th Oct 13	The Camera	Slides (ppt, pdf) Szeliski Ch. 2
Th Oct 20	Modeling Light	Slides (ppt, pdf)
Tu Oct 25	What makes a Great Picture? vs.	Slides (ppt, pdf) Reading: Y. Ke, X. Tang, and F. Jing. The Design of High-Level Features for Photo Quality Assessment. CVPR 2006.
Th Oct 27	Homographies and Mosaics	Slides (ppt, pdf) Szeliski Ch 9
Tu Nov 1	More Mosaic Madness	· Slides (ppt, pdf) · Additional Reading: Rick Szeliski, Image Alignment and Stitching, A Tutorial (DRAFT)
Th Nov 3	Automatic Alignment	Slides (ppt, pdf) Reading: Brown et al, “Multi-Image Matching using Multi-Scale Oriented Patches”, CVPR 2005 Reading: M. Brown and D. G. Lowe. Recognising Panoramas (ICCV2003) (pdf 820kb) Reading: RANSAC
Tu Nov 8	Midterm Review	Slides
Th Nov 10	Midterm
Tu Nov 15	Multi-perspective Panoramas	Slides (ppt, pdf) Reading: Squaring the Circle in Panoramas Reading: Automating joiners Additional Reading: Photographing long scenes with multi-viewpoint panoramas Additional Reading: Scene Collages and Flexible Camera Arrays Flickr:http://flickr.com/groups/joiners/
Th Nov 17	Single View Reconstruction	· Slides (ppt, pdf) · Reading: Horry et al, “Tour into the Picture”, SIGGRAPH ‘97
Tu Nov 22	More Single View Geometry	· Slides (ppt, pdf)
Tu Nov 29	High Dynamic Range Images	Slides (ppt, pdf) Reading: Debevec & Malik, “Recovering High Dynamic Range Radiance Maps from Photographs”, SIGGRAPH 1997
Th Dec 1	Image-based Lighting	Slides (ppt, pdf) Reading: Debevec, Rendering Synthetic Objects in Real Scenes, 1998
Tu Dec 6	Image-based Lighting II	Slides (ppt, pdf) Reading: Debevec, Virtual Cinematography: Relighting Through Computation, IEEE Computer Aug 2006
Th Dec 8	Virtual Double Guest Lecture (Bill Freeman and Steve Seitz)