15-463, 15-663, 15-862 Computational Photography, Fall 2018 |

Time: | Mondays, Wednesdays 12:00 PM - 1:20 PM |

Location: | GHC 4102 |

Instructor: | Ioannis (Yannis) Gkioulekas |

Teaching Assistant: | Alankar Kotwal |

Course Description

Computational photography is the convergence of computer graphics, computer vision and imaging. Its role is to overcome the limitations of the traditional camera, by combining imaging and computation to enable new and enhanced ways of capturing, representing, and interacting with the physical world.

This advanced undergraduate course provides a comprehensive overview of the state of the art in computational photography. At the start of the course, we will study modern image processing pipelines, including those encountered on mobile phone and DSLR cameras, and advanced image and video editing algorithms. Then we will proceed to learn about the physical and computational aspects of tasks such as 3D scanning, coded photography, lightfield imaging, time-of-flight imaging, VR/AR displays, and computational light transport. Near the end of the course, we will discuss active research topics, such as creating cameras that capture video at the speed of light, cameras that look around walls, or cameras that can see through tissue.

The course has a strong hands-on component, in the form of seven homework assignments and a final project. In the homework assignments, students will have the opportunity to implement many of the techniques covered in the class, by both acquiring their own images of indoor and outdoor scenes and developing the computational tools needed to extract information from them. For their final projects, students will have the choice to use modern sensors provided by the instructors (lightfield cameras, time-of-flight cameras, depth sensors, structured light systems, etc.).

**Cross-listing:** This class is cross-listed as 15-463 (for undergraduate students), 15-663 (for Master's students), and 15-862 (for PhD students). Please make sure to register for the section of the class that matches your current enrollment status.

Prerequisites

This course requires familarity with linear algebra, calculus, programming, and doing computations with images. In particular, either of the following courses can serve as proof that you satisfy these prerequisites:

- 16-385 Computer Vision, OR
- 16-720 Computer Vision, OR
- 15-462 Computer Graphics, OR
- 18-793 Image and Video Processing.

If you have not taken any of the above courses but want to enroll, please contact the instructor — exceptions will be made on a case-by-case basis. The course does not require prior experience with photography or imaging.

Textbook

Readings will be assigned from the following textbook (available online for free):

- Computer Vision: Algorithms and Applications, by Richard Szeliski.

Additional readings will be assigned from relevant papers. Readings will be posted at the last slide of each lecture.

The following textbooks can also be useful references but are not required:

*Multiple View Geometry in Computer Vision,*by Richard Hartley and Andrew Zisserman.*Computer Vision: A Modern Approach,*by David Forsyth and Jean Ponce.*Foundations of 3D Computer Graphics,*by Steven Gortler.*Digital Image Processing,*by Rafael Gonzalez and Richard Woods.*Photography,*by Barbara London and John Upton.

Evaluation

Your final grade will be made up from:

- Seven homework assignments (70%).
- Final project (25%).
- Class participation (5%).

**Homework assignments:** All homework assignments will have a programming component and a photography component, where students will use a DSLR camera to capture and process their own images. The programming component of all assignments be done in Matlab. The first assignment will serve as a short tutorial in Matlab and DSLR cameras. We have collected a few useful Matlab resources here.

**Late days:** For the homework assignments, students will be allowed a total of six free late days. Any additional late days will each incur a 10% penalty. Each homework may not be submitted more than four days late.

**Collaboration policy:** Students are encouraged to work in groups but each student must submit their own work. This includes: writing your own code, building your own imaging setups, taking your own photographs, and producing your own writeup. If you work as a group, include the names of your collaborators in your writeup. You absolutely should not share or copy code. Additionally, you should not use any external code unless explicitly permitted. Plagiarism is strongly prohibited and will lead to failure of this course.

**Final project:** Details about the final project are available here.

**15-663, 15-862:** Students taking 15-663 or 15-862 will be required to do a more substantial final project, as well as submit a longer paper describing their project.

**Submitting homeworks:** We use Canvas for submitting and grading homeworks.

Cameras

Students are highly encouraged to obtain a digital camera for use in the course. Any digital camera with manual controls should work. A few cameras will be provided by the instructors.

Email, Office Hours, and Discussion

**Email:** Please use [15463] in the title when emailing the teaching staff!

**Office hours:** Teaching staff have regular office hours at the following times.

- Alankar: Tuesdays 5:00 - 7:00 PM, Smith Hall 220.
- Yannis: Fridays 5:00 - 7:00 PM, Smith Hall 225.

Feel free to email us about scheduling additional office hours.

**Discussion:** We use Piazza for class discussion and announcements.

(Tentative) Syllabus

Slides will be updated on this website after each lecture.

Date | Topics | Slides | Assignments |
---|---|---|---|

M, Aug 27 | Introduction | pdf, pptx | |

W, Aug 29 | Digital photography pipeline | pdf, pptx | |

F, Aug 31 | HW1 out | ||

M, Sep 3 | No class (Labor Day) | ||

W, Sep 5 | Pinholes and lenses | pdf, pptx | |

M, Sep 10 | Photographic optics and exposure | pdf, pptx | |

W, Sep 12 | High dynamic range imaging | pdf, pptx | |

F, Sep 14 | HW1 due, HW2 out | ||

M, Sep 17 | Tonemapping and bilateral filtering | pdf, pptx | |

W, Sep 19 | Color | pdf, pptx | |

M, Sep 24 | Image compositing | pdf, pptx | |

W, Sep 26 | No class | ||

F, Sep 28 | HW2 due, HW3 out | ||

M, Oct 1 | Guest lecture: Ravi Teja Mullapudi | ||

W, Oct 3 | Gradient-domain image processing | pdf, pptx | |

M, Oct 8 | Focal stacks and lightfields | pdf, pptx | |

W, Oct 10 | Deconvolution | pdf, pptx | |

F, Oct 12 | Camera models and calibration | pdf, pptx | |

F, Oct 12 | HW3 due, HW4 out | ||

M, Oct 15 | Guest lecture: Anat Levin | pdf1/pdf2, pptx1/pptx2 | |

W, Oct 17 | Two-view geometry | pdf, pptx | |

M, Oct 22 | Radiometry and reflectance | pdf, pptx | |

W, Oct 24 | Photometric stereo | pdf, pptx | |

F, Oct 26 | HW4 due, HW5 out | ||

M, Oct 29 | Light transport matrices | pdf, pptx | |

W, Oct 31 | Computational light transport | pdf1/pdf2, pptx1/pptx2 | |

M, Nov 5 | Stereo and structured light | pdf, pptx | |

W, Nov 7 | Guest lecture: Hanbyul Joo | ||

S, Nov 11 | HW5 due, HW6 out | ||

M, Nov 12 | Time-of-flight imaging | pdf, pptx | |

W, Nov 14 | Non-line-of-sight imaging | pdf, pptx | |

M, Nov 19 | Fourier optics | pdf, pptx | |

W, Nov 21 | No class (Thanksgiving) | ||

M, Nov 26 | Guest lecture: Aswin Sankaranarayanan | pdf, pptx | |

W, Nov 28 | No class (fire alarm) | ||

F, Nov 30 | Monte Carlo rendering 101 | pdf, pptx | HW7 out |

M, Dec 3 | No class | ||

W, Dec 5 | Wrap-up and discussion | pdf, pptx | |

F, Dec 7 | HW6 and HW7 due |

Previous Course Offerings

- 15-463 (15-663, 15-862), Fall 2017 (taught by Ioannis Gkioulekas)
- 15-463 (15-663, 15-862), Fall 2015 (taught by Kris Kitani)

Similar Courses at CMU and Elsewhere

- Computational Imaging and Display (Gordon Wetzstein, Stanford)
- Computational Photography (Derek Hoiem, UIUC)
- Computational Photography (James Hays, Brown)
- Computational Photography (Ramesh Raskar and Jack Tumblin, SIGGRAPH course)
- Digital and Computational Photography (Fredo Durand, MIT)
- Introduction to Computational Photography (Oliver Cossairt, Northwestern)
- Introduction to Visual Computing (Kyros Kutulakos, University of Toronto)
- Visual Computing Systems (Kayvon Fatahalian, CMU)

Special Thanks

These lecture notes have been pieced together from many different people and places. Special thanks TBD.