CMU 15-769
VISUAL COMPUTING SYSTEMS

Visual computing tasks such as 2D/3D graphics, image processing, and image understanding are important responsibilities of modern computer systems ranging from sensor-rich smart phones to large datacenters. These workloads demand exceptional system efficiency and this course examines the key ideas, techniques, and challenges associated with the design of parallel (and heterogeneous) systems that serve to accelerate visual computing applications. This course is intended for graduate and advanced undergraduate-level students interested in architecting efficient future graphics, image processing, and computer vision platforms and for students seeking to develop scalable algorithms for these platforms.

Basic Info
Mon/Wed 10:30-11:50am
GHC 4303
Instructor: Kayvon Fatahalian
See the course info page for more info on policies and logistics.
Fall 2016 Schedule
Aug 31
History of visual computing, review of multi-core, multi-threading, SIMD, heterogeneity via CPUs/GPUs/ASICs/FPGAs
Part 1: High-Efficiency Image Processing
Sep 7
From raw sensor measurements to an RGB image: demosaicing, correcting aberrations, color space conversions
Sep 9
JPG image compression, auto-focus/auto-exposure, high-dynamic range processing
Sep 12
Balancing parallelism/local/extra work, programming using Halide
Sep 14
Fast bilateral filter and median filters, bilateral grid, optical flow
Sep 26
Contrasting efficiency of GPUs, DSPs, Image Signal Processors, and FGPAs for image processing
Sep 28
Basics of H.264 video stream encoding
Part 2: Trends in Deep Neural Network Authoring and Acceleration
Oct 3
DNN topology, reduction to dense linear algebra, challenges of direct implementation
Oct 5
basics of back-prop, stochastic gradient descent (SGD), memory footprint issues, parallelizing SGD
Oct 10
R-CNN, Fast R-CNN, and then Faster R-CNN
Oct 12
pruning and sparsification techniques, precision reduction, temporal rate reductions
Oct 19
image compression networks, cross-stitch networks, spatial transformer networks, convolutional pose machines
Oct 21
Hardware Accelerators for Deep Neural Network Evaluation (discussion only)
A comparison of the various recent hardware accelerator papers
Oct 23: Exam 1 Released (take home exam)
Part 3: Systems Challenges of 3D Reconstruction
Oct 24-26
City-scale 3D reconstruction, content-based image retrieval
Oct 31
dense 3D reconstruction methods, implicit scene representations (TSDF), KinectFusion, BundleFusion
Part 4: Real-Time 3D Rendering Systems
Nov 2
Graphics pipeline as a machine architecture, basic pipeline semantics/functionality
Nov 7
algorithms and hardware acceleration, depth and color compression
Nov 9
Texture sampling and prefiltering, texture compression, data layout, latency hiding
Nov 14
Molnar sorting taxonomy, scheduling under data amplification, tiled rendering
Nov 16
Deferred shading as a scheduling decision, image-space anti-aliasing
Nov 21
Ray-tracing as an alternative to rasterization, what does modern ray tracing hardware do?
Nov 28
RSL and Cg: Contrasting two different levels of abstraction for shading languages
Part 5: Final Topics
Nov 30
Open research questions on high-performance DSL design
Dec 5
TensorFlow, XLA, and Emerging IRs for Machine Learning (discussion only)
Convergence to tensor-based dataflow abstractions
Dec 7
No Class (Take Home Exam)
Assignments and Projects
Sep 28Assignment 1: The kPhone Camera Pipeline
Nov 8Assignment 2: Kaffe: a Mini-DNN Framework
Dec 16Final Project