[SCS dragon logo]

15-463 Rendering and Image Processing
Computer Science Department
Carnegie Mellon University

INSTRUCTOR: Alexei Efros (Office hours: Tu 5:00-6:00, NSH 4207)
TA: James Hays (Office hours: F 4:00-5:00, NSH 4228)
UNIVERSITY UNITS: 12
SEMESTER: Fall 2004
NEWSGROUP: cmu.cs.class.cs463 (read this for important information!)
WEB PAGE: 463.html
LOCATION: Porter Hall A20A
TIME
: TR 10:30--11:50 AM

PREREQUISITES:
The prerequisite will be 15-462 Computer Graphics I (or equivalent undergraduate course) or permission of the instructor.

COURSE OVERVIEW:
Realistic image synthesis is a central goal of computer graphics. Yet, even the best computer-generated feature films like "Final Fantasy" or "Shrek" are a far cry from what we might consider "real". Curiously, the problem here is generally NOT with computer graphics being unable to model the physics of the real world -- the problem is with the real world itself! It's just too complex, too noisy, too rich and vivid to be recreated from scratch by even the most skilled animators. On the other hand, special effects in movies like "The Matrix" demonstrate thrilling possibilities -- graphical models that look and move so realistically that they integrate seamlessly with live action footage. Their trick is exactly that -- instead of recreating the entire world from scratch, they carefully combine synthetic and real data together, letting the real world provide visual complexity while the synthesized world handles controllability.

The aim of this advanced undergraduate graphics course is to study ways in which samples from the real world (mainly images and video) can be used to generate compelling computer graphics imagery. This course will teach how to acquire, represent, and render scenes from digitized photographs. Several image-based approaches will be presented, with an emphasis on how to use these techniques to build practical systems. This hands-on emphasis will be reflected in the programming assignments, in which students will have the opportunity to acquire their own images of indoor and outdoor scenes and develop the image analysis and synthesis tools needed to render and view the scenes on the computer.

TOPICS TO BE COVERED:

  • Image Formation
  • Image and Video Processing (filtering, antialiasing, pyramids)
  • Image Manipulation (warping, morphing, mosaicing, matting, compositing)
  • Data-driven Synthesis
  • Image-Based Rendering
  • Image-Based Lighting
  • Visual Perception
  • High Dynamic Range Imaging and Tone Mapping
  • 3D Data Acquisition
  • Global Illumination
  • Non-photorealistic Rendering

PROGRAMMING ASSIGNMENTS (tentitive):

1. Images of the Russian Empire -- colorizing the Prokudin-Gorskii photo collection:

Results!

Abe Wong
Ben Hollis
Chao-Kuo Lin
Dongryeol Lee
Jackie Kirchhoff
Ken Chu
Miklos Bergou
Matt Pucevitch
Michael Stevens
Suporn Pongnumkul
Ting Ting Wu

2. Image Warping and Photo Mosaics

Results!

Abe Wong
Ben Hollis
Chao-Kuo Lin
Eunjeong Ryu
Jackie Kirchhoff
Ken Chu
Miklos Bergou
Matt Pucevitch
Michael Stevens
Suporn Pongnumkul
Ting Ting Wu

3. Face morphing and warping:

Results!

Abe Wong
Ben Hollis
Chao-Kuo Lin
Eunjeong Ryu
Jackie Kirchhoff
Ken Chu
Miklos Bergou
Matt Pucevitch
Michael Stevens
Suporn Pongnumkul
Ting Ting Wu

 

movie of combined results!

4. Fun with Image Stacks

 

Results!

Abe Wong
Ben Hollis
Chao-Kuo Lin
Eunjeong Ryu
Jackie Kirchhoff
Ken Chu
Miklos Bergou
Matt Pucevitch
Michael Stevens
Suporn Pongnumkul
Ting Ting Wu

 

Final Project

 

Results!

Abe Wong
Ben Hollis
Eunjeong Ryu
Jackie Kirchhoff
Ken Chu
Miklos Bergou
Matt Pucevitch
Michael Stevens
Suporn Pongnumkul
Ting Ting Wu

 

TEXT:
There is no required text. Various course notes and papers will be made available. Furthermore, there are two optional textbooks that you might find helpful. They will be placed on reserve at the Wean Hall library:

Digital Image Processing, 2nd edition, Gonzalez and Woods
Computer Vision: The Modern Approach, Forsyth and Ponce

There is a number of other fine texts that you can use for general reference:

The Computer Image, Watt and Policarpo (a nice “vision for graphics” text, somewhat dated)
3D Computer Graphics (3rd Edition), Watt (a good general graphics text)
Fundamentals of Computer Graphics, Peter Shirley (another good general graphics text)
Realistic Image Synthesis Using Photon Mapping, Henrik Wann Jensen (latest developments in Global Illumination)
Realistic Ray Tracing, 2nd Edition, Peter Shirley, R. Keith Morley (how to write a perfect ray tracer)
Multiple View Geometry in Computer Vision, Hartley & Zisserman (a bible on recovering 3D geometry)

CAMERAS:
Although it is not required, students are highly encouraged to obtain a digital camera for use in the course (one can get a pretty good camera for under $200). A couple cameras with tripods will be available on loan from the instructor.

METHOD OF EVALUATION:
Grading will be based on a set of programming assignments (60%), a midterm (15%) and a final project presentation (25%). There will be no final exam. For the programming assignments, students will be allowed a total of 5 (five) late days; each additional late day will incur a 10% penalty.

COMPUTING FACILITIES:
All students will have access to the graphics cluster in Wean Hall 5336. You will need to setup your Andrew account in the right way before you can longin. Follow these instructions (replacing 462 with 463). A universal memory card reader will be installed so that you can download the data from your Compact Flash, Memory Stick, or other card.

MATLAB:
Students will be encouraged to use Matlab (with the Image Processing Toolkit) as their primary computing platform. Besides being a great prototyping environment, Matlab is particularly well-suited for working with image data and offers tons of build-in image processing functions. Here is a link to some useful Matlab resources.

CLASS SCHEDULE:

CLASS DATE

TOPICS

Material

Tu Aug 31

Introduction

Th Sep 2

Modeling Light

Tu Sep 7

Image Processing I

Th Sep 9

Color Processing

Tu Sep 14

Neighborhood Processing (filtering)

  • Slides I (ppt, pdf) [mostly from Seitz]
  • Slides II (ppt, pdf) [mostly from Forsyth]

Th Sep 16

Fourier Analysis

J.B.J. Fourier (public domain image)

Tu Sep 21

Sampling and Image Pyramids

  • Slides (ppt, pdf)
  • Additional Reading:

       Burt and Adelson, A multiresolution spline with application to image mosaics, ACM ToG (1983)

Th Sep 23

Image Warping and Mosaicing

       Paul Heckbert, Projecting Mappings for Image Warping

       Rick Szeliski, Image Alignment and Stitching, A Tutorial (DRAFT)

Tu Sep 28

Cylindrical Mosaics and Camera Calibration

Th Sep 30

Image Blending and Compositing

Tu Oct 5

Automatic Image Alignment

  • Slides (ppt, pdf)
  • Additional Reading:

Th Oct 7

Image Morphing

  • Slides (ppt, pdf)
  • Additional Reading:

Tu Oct 12

Blob Processing

Th Oct 14

Image Stacks

Tu Oct 19

Homework discussion and Midtern Review

 

Th Oct 21

Midterm

Location: NSH 1507

Tu Oct 19

Background Subtraction (and answers to midterm)

Slides (ppt, pdf)

Tu Oct 26

Modeling with Weather (guest lecture by Srinivasa Narasimhan)

Slides coming…

Assignment #3 OUT!

Tu Nov 2

Faces and PCA

Slides (ppt, pdf)

Additional Reading:

Rowland, Ferrett, Manipulating Facial Appearance through Shape and Color”, CG&A, 1995

Th Nov 4

Texture Synthesis

The image “http://www-2.cs.cmu.edu/~efros/bush_ad.jpg” cannot be displayed, because it contains errors.

Slides (ppt, pdf)

Additional Reading:

Schold, et all, “Video Textures”, SIGGRAPH 2000

Tu Nov 9

Matting

Slides (ppt, pdf)

Additional Reading:

Smith & Blinn, “Blue Screen Matting”, SIGGRAPH 1996

Th Nov 11

Shadow Matting and Flash/No Flash

Slides (ppt, pdf)

Additional Reading:

Petschnigg et all, Digital Photography with Flash and No-Flash Image Pairs, SIGGRAPH 2004

Eisemann and Durand, Flash Photography Enhancement via Intrinsic Relighting, SIGGRAPH 2004

Tu Nov 16

High Dynamic Range Images

Slides (ppt, pdf)

Additional Reading:

Debevec & Malik, “Recovering High Dynamic Range Radiance Maps from Photographs”, SIGGRAPH 1997

Assignment #4 OUT!

Th Nov 18

Image-Based Lighting

Slides (ppt, pdf)

Additional Resources:

http://www.debevec.org

Tu Nov 23

Image-Based Lighting II

Slides (ppt, pdf)

 

Tu Nov 30

Scene Reconstruction from Single View

Slides (ppt, pdf)

Additional Materials:

“Tour into the Picture”, SIGGRAPH ‘97

glTIP software

Th Dec 2

More Single View Geometry™

Slides (ppt, pdf)

Additional Materials:

Single-view Reconstruction

Tu Nov 30

3D from two and more views

 

Slides (ppt, pdf)

 

Tu Nov 30

Wrap-up

Slides (ppt, pdf)

 

 

PREVIOUS OFFERINGS:
Previous offerings of this course can be found here.

Page design courtesy of Doug James