15-463, 15-663, 15-862
Computational Photography, Fall 2021
|Time:||Mondays, Wednesdays 11:50 am - 1:10 pm ET|
|Instructor:||Ioannis (Yannis) Gkioulekas|
|Teaching Assistants:||Alice Lai, Jenny Lin|
Computational photography is the convergence of computer graphics, computer vision, optics, and imaging. Its role is to overcome the limitations of traditional cameras, by combining imaging and computation to enable new and enhanced ways of capturing, representing, and interacting with the physical world.
This course provides an overview of the state of the art in computational photography. At the start of the course, we will study modern image processing pipelines, including those encountered on mobile phone and DSLR cameras, and advanced image and video editing algorithms. Then we will continue to learn about the physical and computational aspects of tasks such as 3D scanning, coded photography, lightfield imaging, time-of-flight imaging, VR/AR displays, and computational light transport. Near the end of the course, we will discuss active research topics, such as creating cameras that capture video at the speed of light, cameras that look around walls, or cameras that can see below skin.
The course has a strong hands-on component, in the form of seven homework assignments and a final project. In the homework assignments, students will have the opportunity to implement many of the techniques covered in the class, by both acquiring their own images of indoor and outdoor scenes and developing the computational tools needed to extract information from them. Example homework includes building end-to-end HDR imaging pipelines and structured light scanners. For their final projects, students will have the choice to use modern sensors and other optical instrumentation provided by the instructors (lightfield cameras, time-of-flight sensors, projectors, laser sources, and so on).
Cross-listing: This is both an advanced undergraduate and introductory graduate course, and it is cross-listed as 15-463 (for undergraduate students), 15-663 (for Master's students), and 15-862 (for PhD students). Please make sure to register for the section of the class that matches your current enrollment status.
This course requires familiarity with linear algebra, calculus, programming, and doing computations with images. In particular, either of the following courses can serve as proof that you satisfy these prerequisites:
If you want to enroll but have not taken any of the above courses, please make sure to contact the instructor! We make a lot of exceptions each year, on a case-by-case basis. The course does not require prior experience with photography or imaging.
Readings will be assigned primarily from relevant papers. Readings will be posted at the last slide of each lecture.
Additionally readings may be assigned from the following textbooks, which can also be useful references in general. All of them are available online from the CMU library:
Your final grade will be made up of:
Homework assignments: All homework assignments will have a programming component and a photography component, where students will use a DSLR camera to capture and process their own images. The programming component of all assignments will be done in Python.
Late days: For the homework assignments, students will be allowed a total of five free late days. Any additional late days will each incur a 10% penalty. Please note that submission deadlines will be enforced strictly for the purposes of counting late days. In particular, no exceptions will be made for reasons such as upload delays, submitting incorrect files, and so on.
Collaboration policy: Students are encouraged to work in groups, but each student must submit their own work. This includes: writing your own code, building your own imaging setups, taking your own photographs, and producing your own writeup. If you work as a group, include the names of your collaborators in your writeup. You absolutely must not share or copy code, data, or text from any source. Additionally, you must not use any external code or data unless explicitly permitted. These and any other forms of cheating are strongly prohibited and will result in a failing grade for the entire course (not just the assignment you were caught on). If you have any question about whether some activity would constitute cheating, just be cautious and ask the instructors before proceeding. Additionally, you musy not supply any code or writeups you complete during this course to other students in future instances of the course, or make this material available (e.g., on the web) for use in future instances of the course. You must make sure any online repositories you use for the course are kept private.
Final project: At around the halfway point of the semester, students will propose a final project, to be completed by the end of the semester. Final project presentations will take place during the final exam week, and final project reports will be due before the final grade deadline. Details are available in the final project page.
15-663, 15-862: Students taking 15-663 or 15-862 will be required to do a more substantial final project, as well as submit a longer paper describing their project.
Final project competition: Final projects will be judged by a panel comprising other computational photography faculty at CMU, who will vote on the two best projects. The two winning students will receive a free DSLR camera. Note that you will also receive a separate grade, independently of the results of the competition.
The winning projects for Fall 2021 are:
You can also browse all Fall 2021 final project presentations.
Students are encouraged to obtain a digital camera for use in the course — any digital camera with manual controls should work. However, this is not a requirement, and about 50 camera kits (Nikon D3300/3400/3500 DSLR camera, 18-55 mm zoom lens, tripod) will be provided by the instructors to students that do not have one.
Lectures: Lectures will take place in GHC 4303. All lectures will be recorded, and recordings will be available on Canvas after each lecture.
Email: Please use  in the title when emailing the teaching staff!
Office hours: Teaching staff will have regular office hours at the following times:
All office hours will take place in Smith Hall (EDSH) 236 (graphics lounge).
Feel free to email us about scheduling additional office hours. Please do not email the teaching staff with questions about homework or grading. You should post those on Piazza or Slack.
Discussion: We will use Piazza and Slack for course discussion and announcements. Instructions for accessing the course Slack server are available on Piazza.
We are actively looking for students at all levels (undergraduates, MS, PhD) to help in projects on various aspects of computational photography, imaging, rendering, and graphics in general. If you are interested, please send Yannis an email (or talk to him in person in class).
You should also take a look at the imaging group website, to find more information about related projects active here at CMU.
Finally, you are welcome to attend the imaging group meetings during the semester. If you want to receive announcements about the group meetings, as well as other emails regarding photography and imaging research, ask Yannis to add you to the imaging group's mailing list.
Dates and topics are likely to change during the semester. Slides will be uploaded on this website before each lecture.
|M, Aug 30||Introduction||pptx, pdf|
|W, Sep 01||Digital photography||pptx, pdf|
|F, Sep 03||HW1 out|
|M, Sep 06||No class (Labor Day)|
|W, Sep 08||Pinholes and lenses||pptx, pdf|
|M, Sep 13||Photographic optics and aberrations||pptx, pdf|
|W, Sep 15||Exposure and high-dynamic-range imaging||pptx, pdf|
|F, Sep 17||HW2 out|
|M, Sep 20||Noise||pptx, pdf||HW1 due|
|W, Sep 22||Color||pptx, pdf|
|M, Sep 27||Bilateral and edge-aware filtering||pptx, pdf|
|W, Sep 29||Gradient-domain image processing||pptx, pdf|
|M, Oct 04||Focal stacks and depth from (de)focus||pptx, pdf||HW2 due, HW3 out|
|W, Oct 06||Lightfields||pptx, pdf|
|M, Oct 11||Deconvolution||pptx, pdf|
|W, Oct 13||Coded photography||pptx, pdf|
|M, Oct 18||Radiometry and reflectance||pptx, pdf||HW3 due, HW4 out|
|W, Oct 20||Photometric stereo||pptx, pdf|
|M, Oct 25||Geometric camera modeling and calibration||pptx, pdf||Project proposal due|
|W, Oct 27||Two-view geometry||pptx, pdf|
|M, Nov 01||Disparity and stereo||pptx, pdf||HW4 due, HW5 out|
|W, Nov 03||Time-of-flight imaging||pptx, pdf|
|M, Nov 08||Global illumination||pptx, pdf|
|W, Nov 10||Light transport matrices||pptx, pdf|
|M, Nov 15||Wrap up||pptx, pdf||HW5 due|
|W, Nov 17||Light transport probing||HW5 due, HW6 out|
|M, Nov 22||Fourier optics|
|W, Nov 24||No class (Thanksgiving)|
|F, Nov 26||No class (Thanksgiving)|
|M, Nov 29||Advanced topics|
|W, Dec 01||Advanced topics|
|S, Dec 05||HW6 due|
|Th, Dec 09||Project presentations|
|T, Dec 14||Project report due|
The materials for this course have been pieced together from many different people and places. Special thanks to the following colleagues for sharing their course materials or making them available online (in alphabetical order): Supreeth Achar, Andrew Adams, Amit Agrawal, Michael Brown, Oliver Cossairt, Fredo Durand, Alyosha Efros, Kayvon Fatahalian, Steven Gortler, Mohit Gupta, Sam Hasinoff, James Hays, Hugues Hoppe, Ivo Ihrke, Wojciech Jarosz, Kris Kitani, Kyros Kutulakos, Douglas Lanman, Jaako Lehtinen, Anat Levin, Marc Levoy, Steve Marschner, Srinivasa Narasimhan, Shree Nayar, Ren Ng, Matthew O'Toole, Sylvain Paris, Ravi Ramamoorthi, Ramesh Raskar, Aswin Sankaranarayanan, Robert Sumner, Richard Szeliski, Gavriel Taubin, James Tompkin, Gordon Wetzstein, Todd Zickler. Special thanks to Jiatian (Caroline) Sun for developing the renderer used in some of the homework assignments. Individual slides and homework assignments include their own credits.