Course description
Computational photography is the convergence of computer graphics, computer vision, optics, and imaging. Its role is to overcome the limitations of traditional cameras, by combining imaging and computation to enable new and enhanced ways of capturing, representing, and interacting with the physical world.
This course provides an overview of the state of the art in computational photography. At the start of the course, we will study modern image processing pipelines, including those encountered on mobile phone and DSLR cameras, and advanced image and video editing algorithms. Then we will continue to learn about the physical and computational aspects of tasks such as 3D scanning, coded photography, lightfield imaging, time-of-flight imaging, VR/AR displays, and computational light transport. Near the end of the course, we will discuss active research topics, such as creating cameras that capture video at the speed of light, cameras that look around walls, or cameras that can see below skin.
The course has a strong hands-on component, in the form of seven homework assignments and a final project. In the homework assignments, students will have the opportunity to implement many of the techniques covered in the class, by both acquiring their own images of indoor and outdoor scenes and developing the computational tools needed to extract information from them. Example homework includes building end-to-end HDR imaging pipelines and structured light scanners. For their final projects, students will have the choice to use modern sensors and other optical instrumentation provided by the instructor (lightfield cameras, time-of-flight sensors, projectors, laser sources, and so on).
Cross-listing: This is both an advanced undergraduate and introductory graduate course, and it is cross-listed as 15-463 (for undergraduate students), 15-663 (for Master's students), and 15-862 (for PhD students). Please make sure to register for the section of the class that matches your current enrollment status.
Competitions
Photography competitions: The submissions and results for the homework assignment competitions are available in the following pages.
- Homework assignment 1.
- Homework assignment 2.
- Homework assignment 3.
- Homework assignment 4.
- Homework assignment 5.
- Homework assignment 6.
Final project competition: The submissions and results are available in the final project competition page. The submissions are also available at the Carnegie Mellon Computational Imaging YouTube channel.
Prerequisites
This course requires familiarity with linear algebra, calculus, programming, and doing computations with images. In particular, either of the following courses can serve as proof that you satisfy these prerequisites:
- 16-385 Computer Vision, OR
- 16-720 Computer Vision, OR
- 15-462 Computer Graphics, OR
- 18-793 Image and Video Processing.
If you want to enroll but have not taken any of the above courses, please make sure to contact the instructor! We make a lot of exceptions each year, on a case-by-case basis. The course does not require prior experience with photography or imaging.
Textbooks
There is no required textbook. Each lecture and homework assignment includes an extensive list of references (primarily papers) that you will need to consult to successfully complete the course.
The following textbooks can also be useful references in general. All of them are available online from the CMU library:
- Computer Vision: Algorithms and Applications, by Richard Szeliski.
- Computational Imaging Book, by Ayush Bansai, Achuta Kadambi, and Ramesh Raskar.
- Multiple View Geometry in Computer Vision, by Richard Hartley and Andrew Zisserman.
- Computer Vision: A Modern Approach, by David Forsyth and Jean Ponce.
- Foundations of 3D Computer Graphics, by Steven Gortler.
- Digital Image Processing, by Rafael Gonzalez and Richard Woods.
- Photography, by Barbara London and John Upton.
Evaluation
Your final grade will be made up of:
- Six two-week homework assignments (75%).
- Final project (20%).
- Class participation (5%).
Homework assignments: All homework assignments will have a programming component, where students will use Python to implement a computational photography pipeline, and a photography component, where students will use a DSLR camera to capture and process their own images.
Late days: For the homework assignments, students will have a total of five free late days. Any additional late days will each incur a 10% penalty. Please note that we will enforce submission deadlines strictly for the purposes of counting late days. In particular, we will not make any exceptions for reasons such as upload delays, submitting incorrect files, and so on.
Collaboration policy: We encourage students to work in groups, but each student must submit their own work. This includes: writing your own code, building your own imaging setups, taking your own photographs, and producing your own writeup. If you work as a group, include the names of your collaborators in your writeup. You absolutely must not share or copy code, data, or text from any source. Additionally, you must not use any external code or data unless explicitly permitted. These and any other forms of cheating are prohibited and will result in a failing grade for the entire course (not just the assignment you are caught on). If you are uncertain about whether some activity would be cheating, just be cautious and ask the teaching staff before proceeding. Additionally, you must not supply any code or writeups you complete during this course to other students in future instances of the course, or make this material available (e.g., on the web) for use in future instances of the course. You must make sure any online repositories you use for the course are kept private.
Submitting homework: We will use Gradescope, available through Canvas, for submitting and grading homework. When submitting homework, students must follow the homework submission guidelines.
Final project: Towards the end of the semester, each student will work on a project of their own choosing. Final project reports and presentations will be due during the final exam week. Details are available in the final project page.
15-663, 15-862: Students taking 15-663 or 15-862 will need to do a more substantial final project.
Homework assignment competitions: For each homework assignment, the teaching staff will select the student that produced the best result when capturing and processing their own images. The winning student will receive a free photography-related gift relevant to the homework assignment. The competition is separate from the grade for each homework assignment.
Final project competition: A panel of judges comprising computational photography faculty at CMU will select the two best projects. The winning students will receive a free DSLR camera. The competition is separate from the grade for the final project.
Cameras
Students do not need to own a digital camera for this course—though they are welcome to use it if they own one with manual controls. We will provide about 50 camera kits (Nikon D3300/3400/3500 DSLR camera, 18-55 mm zoom lens, tripod) to students that do not have one. Please note that students will be held financially responsible for any camera or other borrowed equipment they fail to return, or return with damaged or missing parts.
Camera tutorial: To make it easier for students to use the camera kits, we have put together a camera tutorial.
Meetings
Lectures: Lectures will take place in GHC 4303. We will make recordings of lectures available on Canvas after each lecture.
Reading group: In addition to regular lectures, the course includes an optional reading group taking place every second Friday (time TBD) in Smith Hall (EDSH) 236 (graphics lounge). During the reading group, we will go over recent papers and advanced topics related to the course.
Email, office hours, and discussion
Email: Please use [15463] in the title when emailing the teaching staff!
Office hours: Teaching staff will have regular office hours at the following times:
- Tuesdays 3:00 - 5:00 pm ET, Gustavo.
- Wednesdays 2:30 - 4:30 pm ET, Yannis.
- Thursdays 2:30 - 4:30 pm ET, Dorian.
All office hours will take place in person in Smith Hall (EDSH) 236 (graphics lounge).
Feel free to email us about scheduling additional office hours. Please do not email the teaching staff with questions about homework or grading. You should post those on Piazza or Slack.
Discussion: We will use Piazza and Slack for course discussion and announcements. Instructions for accessing the course Slack server are available on Piazza.
Interested in research?
We are actively looking for students at all levels (undergraduate, MS, PhD) to help in projects on various aspects of computational photography, imaging, rendering, and graphics in general. If you are interested, please send Yannis an email (or talk to him in person in class).
You should also take a look at the Carnegie Mellon Computational Imaging website and YouTube channel, to find more information about related projects active here at CMU.
Finally, you are welcome to attend the imaging group meetings during the semester. If you want to receive announcements about the group meetings, as well as other emails regarding photography and imaging research, ask Yannis to add you to the imaging group's mailing list.
(Tentative) syllabus
Dates and topics are likely to change during the semester. Slides will be uploaded on this website before each lecture.
| Date | Topics | Homework |
|---|---|---|
| M, Aug 29 | Introduction | |
| W, Aug 31 | Digital photography | |
| F, Sep 02 | HW1 out | |
| M, Sep 05 | No class (Labor Day) | |
| W, Sep 07 | Pinholes and lenses | |
| M, Sep 12 | Photographic optics and aberrations | |
| W, Sep 14 | Exposure and high-dynamic-range imaging | |
| F, Sep 16 | Reading group: lens aberrations | HW1 due, HW2 out |
| M, Sep 19 | Noise | |
| W, Sep 21 | Color | |
| F, Sep 23 | Reading group: catadioptric imaging | |
| M, Sep 26 | Bilateral and edge-aware filtering | |
| W, Sep 28 | Gradient-domain image processing | |
| M, Oct 03 | Focal stacks and depth from (de)focus | HW2 due, HW3 out |
| W, Oct 05 | Lightfields | |
| F, Oct 07 | Reading group: bilateral grid | |
| M, Oct 10 | Deconvolution | |
| W, Oct 12 | Coded photography | |
| M, Oct 17 | No class (Fall break) | |
| W, Oct 19 | No class (Fall break) | |
| M, Oct 24 | Radiometry and reflectance | HW3 due, HW4 out |
| W, Oct 26 | Photometric stereo | |
| F, Oct 28 | Reading group: color | |
| M, Oct 31 | Geometric camera modeling and calibration | Project proposal due |
| W, Nov 02 | Two-view geometry | |
| F, Nov 04 | Reading group: focal flow | |
| M, Nov 07 | Disparity and stereo | HW4 due, HW5 out |
| W, Nov 09 | Time-of-flight imaging | |
| F, Nov 11 | Reading group: reflectometry | |
| M, Nov 14 | Indirect illumination | |
| W, Nov 16 | Light transport matrices | |
| M, Nov 21 | Light transport probing | HW5 due |
| W, Nov 23 | No class (Thanksgiving) | |
| M, Nov 28 | Fourier optics | HW6 out |
| W, Nov 30 | Advanced topics | |
| M, Dec 05 | Advanced topics | |
| W, Dec 07 | Wrap up | |
| F, Dec 09 | Reading group | |
| M, Dec 12 | HW6 due |
Previous course offerings
- 15-463 (15-663, 15-862), Fall 2021 (taught by Ioannis Gkioulekas)
- 15-463 (15-663, 15-862), Fall 2020 (taught by Ioannis Gkioulekas)
- 15-463 (15-663, 15-862), Fall 2019 (taught by Ioannis Gkioulekas)
- 15-463 (15-663, 15-862), Fall 2018 (taught by Ioannis Gkioulekas)
- 15-463 (15-663, 15-862), Fall 2017 (taught by Ioannis Gkioulekas)
- 15-463 (15-663, 15-862), Fall 2015 (taught by Kris Kitani)
Similar courses at CMU and elsewhere
- Computational Imaging and Display (Gordon Wetzstein, Stanford)
- Computational Photography (Derek Hoiem, UIUC)
- Computational Photography (James Hays, Brown)
- Computational Photography (Ramesh Raskar and Jack Tumblin, SIGGRAPH course)
- Digital and Computational Photography (Fredo Durand, MIT)
- Introduction to Computational Photography (Oliver Cossairt, Northwestern)
- Introduction to Visual Computing (Kyros Kutulakos, University of Toronto)
- Visual Computing Systems (Kayvon Fatahalian, CMU)
Acknowledgments
The materials for this course have been pieced together from many different people and places. Special thanks to the following colleagues for sharing their course materials or making them available online (in alphabetical order): Supreeth Achar, Andrew Adams, Amit Agrawal, Michael Brown, Oliver Cossairt, Fredo Durand, Alyosha Efros, Kayvon Fatahalian, Steven Gortler, Mohit Gupta, Sam Hasinoff, James Hays, Hugues Hoppe, Ivo Ihrke, Wojciech Jarosz, Kris Kitani, Kyros Kutulakos, Douglas Lanman, Jaako Lehtinen, Anat Levin, Marc Levoy, Steve Marschner, Srinivasa Narasimhan, Shree Nayar, Ren Ng, Matthew O'Toole, Sylvain Paris, Ravi Ramamoorthi, Ramesh Raskar, Aswin Sankaranarayanan, Robert Sumner, Richard Szeliski, Gavriel Taubin, James Tompkin, Gordon Wetzstein, Todd Zickler. Special thanks to Jiatian (Caroline) Sun for developing the renderer used in some of the homework assignments. Individual slides and homework assignments include their own credits.