Real-Time Computer Graphics

Also known as 15-472/672/772. Mondays, Wednesdays, and Fridays from 10:00-10:50 in Porter Hall A18C Posner Hall 146 during Fall 2024. Taught by Jim McCann. Office hours after class in Smith 229 or by appointment. With TA help from Ajax Shung. Office hours Mondays and Fridays, 11am-Noon in the graphics common area on the second floor of Smith Hall or by appointment.

Real-time computer graphics is about building systems that leverage modern CPUs and GPUs to produce detailed, interactive, immersive, and high-frame-rate imagery. Students will build a state-of-the-art renderer using C++ and the Vulkan API. Topics explored will include efficient data handling strategies; culling and scene traversal; multi-threaded rendering; post-processing, depth of field, screen-space reflections; volumetric rendering; sample distribution, spatial and temporal sharing, and anti-aliasing; stereo view synthesis; physical simulation and collision detection; dynamic lights and shadows; global illumination, accelerated raytracing; dynamic resolution, "AI" upsampling; compute shaders; parallax occlusion mapping; tessellation, displacement; skinning, transform feedback; and debugging, profiling, and accelerating graphics algorithms.

Previous iterations: Spring'24

Principles

This course is based on four principles:

• Do Things for Reasons -- if you don't understand why you are doing something, look deeper. (Corr: if you don't understand why someone else is doing something, ask.)
• No Magic -- (as much as possible) avoid black boxes and needless helper libraries.
• Test and Improve -- make things efficient through real-world testing. (Corr: asymptotic complexity is not the only factor; constants matter in the real world.)
• Go Big -- push systems to their limits.

Work

(Note: subject to revision based on class timing.)

Your grade in this course will be determined by four assignments that cumulatively build a high-performance real-time renderer (60%); a project that adds a state-of-the-art feature to that renderer (30%); and participation in in-class discussions and activities (10%).

Don't Copy and Don't Steal

You will write a lot of code in this class, and I require that you actually type this code. (It is easy to copy-paste faster than you can think.)

You may use code from the internet, but make sure the code is not covered by a license that prohibits copying (copyright violation -- illegal!), be sure to include a comment citing your source (plagiarism -- immoral!), and make sure it passes through your brain on its way to your fingers (e.g., "fix" the coding style, re-work awkward parts, look up and document parts you don't understand).

You may work together, but take extra care to not copy code. Rule of thumb: if you are writing code, you should not be doing it where you can see your classmates' code.

Examples:

• GOOD: re-writing a documentation example in your own coding style (including a comment + URL citing your source).
• BAD: copy-pasting a stackoverflow answer or code snippet from the documentation, then re-working it in place. (You must type the code you use.)
• GOOD: working with a classmate to develop a design, then writing the code separately (and including a comment crediting the discussion in your code).
• GOOD: helping a classmate to debug their code, being inspired by their approach, and -- with your classmate's permission and a citation comment -- using a similar idea in your code.
• BAD: using copilot. (You aren't typing and you aren't thinking.)
• BAD: ChatGPT and similar. (Just bad in general. A copyright [or at least plagiarism] laundry.)
• GOOD: copy-pasting code you already wrote for this class.
• DISCOURAGED (but not forbidden): using autocomplete / codesense.

Resources

This course does not have a textbook; but you will likely find the following web resources helpful:

• C++ Reference.
• Compiler Explorer -- for playing with how code might get optimized.
• MDN's JavaScript reference.
• Node.js documentation.
• Blender Python API.
• Vulkan
  • Vulkan Specification -- latest; has better formatting than 1.3 .
  • Device Database -- check out feature availability on other GPUs.
  • Vulkan Guide -- easier reading than the spec!
• GLSL
  • GL_KHR_vulkan_glsl and GL_EXT_vulkan_glsl_relaxed -- describe language changes when using GLSL to create SPIR-V for Vulkan.
  • GLSL 4.5 Quick Reference Card (starts on Page 9).
  • GLSL 4.5 specification.
• Inspiration
  • Pouët -- a demoscene archive.
  • Shadertoy -- fun fragment shader hacks.
• Advances in Real-Time Rendering -- notes from SIGGRAPH courses.
• GPU Gems -- classic GPU programming books, available free.

See also: occasional "extra readings" links in sections below.

Schedule

Real-Time Graphics

Monday, August 26th

What it means to be real-time and general techniques for getting there.

• M Aug 26
• Course Overview; Core Principles
• Interactive, Real-Time, and Beyond
• Vulkan Intro
• Building the tutorial code
• Javascript, maybe
• A0a For next class -- Vulkan Tutorial Pt1

Vulkan's view of GPUs

Wednesday, August 28th
-Friday, September 13th

How we get pixels onto the screen quickly. From attributes to framebuffers.

• W Aug 28
• Command Buffers
• Render Passes
• Background Shader
• Push Constants
• Pipeline Creation
• A0a For next class -- Vulkan Tutorial Pt2
• F Aug 30
• Lines Shader
• Memory Types
• Memory Transfers
• Vertex Buffers
• A0a For next class -- Finish Vulkan Tutorial
• M Sep 2: No Class / Labor Day
• W Sep 4
• Transform Matrices
• Descriptors
• Uniform Buffers
• F Sep 6
• Objects Shader
• Storage Buffers
• M Sep 9
• Swapchains and synchronization
• Headless mode
• Just initialization things
• W Sep 11
• Vulkan Tutorial Wrap-Up
• Build systems
• F Sep 13
• S72 intro
• Tutorial improvements
• Benchmarking

Data Wrangling

Monday, September 16th
-Friday, September 20th

Getting data into our renderer.

• M Sep 16
• Scene graphs
• Instancing
• A1 statement walkthrough
• A1 Scene Viewer out
• W Sep 18
• Data inflation: subdivision surfaces
• Sub-polygon detail: textures, normals, displacements
• Mesh compression?
• Texture compression
• F Sep 20
• least squares vertex baking(?)
• Creating (and finding) Bottlenecks
• Export scripts

Culling and Occlusion

Monday, September 23rd
-Friday, September 27th

What you don't render can't slow you down.

• M Sep 23
• RAII + Vulkan
• (Jim Travelling)
• W Sep 25
• Bottlenecks
• Frustum culling
• Portal rendering
• Occlusion queries
• Geometry images
• F Sep 27
• A2 introduction

Materials

Monday, September 30th
-Friday, October 4th

Matching the real world, or doing better.

• M Sep 30
• A1 code share
• detail with textures
• antialiasing
• W Oct 2
• physical materials
• approximations and ground truth
• precomputation
• F Oct 4
• procedural materials
• material capture
• A1 Scene Viewer due (end of day)
• A2 Materials out

Color

Monday, October 7th

Primaries, white points, and encodings oh my.

• M Oct 7
• color spaces
• demo: P3 vs sRGB primaries
• demo: HDR

Going Big

Wednesday, October 9th
-Friday, October 18th

Moving beyond what can fit in GPU memory.

• W Oct 9
• Terrains
• LOD streaming
• F Oct 11
• Mega-textures
• M Oct 14: Fall Break
• W Oct 16: Fall Break
• F Oct 18: Fall Break

Direct Lighting

Sunday, October 20th
-Wednesday, October 23rd

The first bounce of lighting is always the brightest.

• U Oct 20
• A2 Materials due (end of day)
• M Oct 21
• sphere, spot, and sun lights
• lambertian contribution from a spherical area light
• the "representative point" approximation
• A3 introduction
• A3 Lights out
• W Oct 23
• light loops
• light sorting

Shadows

Friday, October 25th
-Monday, October 28th

Solving the visibility problem.

• F Oct 25
• shadow maps
• cubic shadow maps
• M Oct 28
• percentage-closer filtering
• soft shadows with PCSS

Real-time Raytracing

Wednesday, October 30th

Just brute-forcing the lighting.

• W Oct 30
• raytracing in Vulkan
• sample sharing

Rendering for VR

Friday, November 1st

The upsides and downsides of virtual reality.

• F Nov 1
• OpenXR APIs
• lens correction
• reprojection
• A3 Lights due (end of day)
• F Final Project topic selection

Post-processing

Monday, November 4th
-Wednesday, November 6th

Fixing the pixels later.

• M Nov 4
• bokeh
• lens flares
• motion blur
• F Final Project out
• W Nov 6
• SSAO
• screen-space reflections

Volume Rendering

Friday, November 8th
-Wednesday, November 13th

Clouds, medical data, and fire.

• F Nov 8
• Nubus3
• M Nov 11
• Nubus3 (continued)
• raymarching smoke and clouds
• screen-space surfacing for liquids
• W Nov 13: No Class: Jim at FMitF PI Meeting

Simulation

Friday, November 15th
-Monday, November 18th

Making things move without keyframes.

• F Nov 15
• Eulerian vs Lagrangian
• shallow-wave equations
• grid-based smoke
• M Nov 18
• particle-based smoke
• particle-based fluid
• particle-based solids

Skinning and Animation

Wednesday, November 20th

Making meshes flexible.

• W Nov 20
• dual-quaternion skinning
• Skinned Mesh Animation

Miscellaneous

Friday, November 22nd
-Friday, December 6th

Topics that didn't fit elsewhere (thematically or temporally) and were interesting enough to bring up.

• F Nov 22
• sparse voxel octrees and sparse voxel directed acyclic graphs
• hierarchical ordered grids
• M Nov 25
• W Nov 27: Thanksgiving Break
• F Nov 29: Thanksgiving Break
• M Dec 2
• Jim travelling
• W Dec 4
• Jim travelling
• F Dec 6
• Jim travelling
• F Final Project due (end of day)