Assignment 1: kPhone Camera Pipeline

Due Wed September 28th, at 11:59pm

In this assignment you will implement a simple image processing pipeline for the data produced by the image sensor of the much-ballyhooed kPhone 769. The kPhone 769 has a camera and your job is to process the data coming off the kPhone 769's sensor to produce the highest quality image you can. In addition to implementing basic image processing of sensor outputs, you are also responsible for controlling the focus of the camera. The goals of the assignment are to get basic experience with two major concepts discussed in the course so far:

  • Implementing image processing algorithms like those we've discussed in class. (Or go further by implementing more advanced algorithms you wish to learn on your own.)
  • Optimizing image processing code using Halide.

Getting Started

Grab the assignment starter code.

git clone ssh://linux.andrew.cmu.edu/afs/cs/academic/class/15769-f16/assignments/asst1.git

The starter code should build on Andrew Linux and OSX.

Scene datasets used in this assignment are located here:

/afs/cs/academic/class/15769-f16/assignments/asst1_scenes    

Linux Build Instructions:

The codebase uses CMake as its cross-platform make utility. We suggest an out-of-source build using cmake, which you can perform via the following steps:

Assuming your top-level source directory is called: SOURCEDIR ...

// 1. Create a build directory anywhere you like
mkdir BUILDDIR

// 2. Enter the build directory
cd BUILDDIR

// 3. Configure the build
cmake PATH_TO_SOURCEDIR

At this point your build should be configured, and you can compile the codebase by typing:

make

Running the starter code:

Now you can run the camera. Just run: (where PATH_TO_SCENES points to your scenes directory)

./bin/camerapipe -noiselevel 2 -scenesdir PATH_TO_SCENES scene3.bin output.bmp

Your camera will "take a picture" and save that picture to output.bmp. The starter code is set up to simply copy the RAW data off the sensor (one byte per pixel) into the red, green, and blue channels of the output image and, as you can see below, the results are quite unsatisfactory. Color is not reproduced, the effects of Bayer filter attenuation are clearly visible, and the image has other defects like, noise, dead pixels that always return white, different pixel gains, and even subtle vignetting near the corners.

Image 1

A zoomed view of the region highlighted in red is shown below.

Image 2

For reference, the original image looked like this. (Note: you are not expected to reconstruct this image perfectly---a significant amount of information is lost in the sensing process)

Image 3

Usage:

  • -noiselevel XXX allows you to adjust the magnitude of noise in the sensor data (0 <= XXX <=4).
  • -focus XXX sets the default focus of the camera to XXX (in units of mm from the camera). This can be useful if you want to manually set the focus to debug your image processing pipeline on the more interesting scenes without first implementing autofocus.
  • -help gives you command line help

Part 1: Image Processing

In the first part of the assignment you need to process the image data to produce an RGB image that, simply put, looks at good as you can make it. The entry point to your code should be CameraPipeline::TakePicture() in CameraPipeline.cpp. This method currently acquires data from the sensor via sensor->ReadSensorData()'. The result is aWidthbyHeightbuffer of 8-bit sensor pixel readings (each pixel measurement is represented as anunsigned char). You must implementCameraPipeline::ProcessShot(), which takes the RAW pixel data stored inbufferand converts it to a RGB imageresult`.

Tips:

  • In this part of the assignment it is likely best to start by working on images the don't require auto-focusing. These include: black.bin (an all black image), gray.bin (a 50% gray image, for which pixels without defects should be [128,128,128]), stripe.bin, and tartan.bin)
  • You may implement this assignment in any way you wish. Certainly, you will have to demosaic the image to recover RGB values. The techniques you employ for handling noise, bad pixels, defect pixels, and vignetting artifacts are up to you.
  • We guarantee that pixel defects (stuck pixels, pixels with extra sensitivity) are static defects that are the same for every photograph taken by the camera. (Hint: What does this suggest a simple way to "calibrate" your camera for static defects?) Note that while the overall noise statistics of the sensor are the same per photograph (and will only change based on the value of -noiselevel), the perturbation of individual pixel values due to noise varies per photograph (that's the nature of noise!)
  • The following is the Bayer filter pattern used on the kPhone's sensor. Pixel (0,0) is the top-left of the image. Image 2

Part 2: Contrast-Detection Autofocus

In the second half of the assigment you need to implement contrast-detection autofocus. Based on analysis of regions of the sensor (notice that sensor->ReadSensorData() can return a crop window of the full sensor), you should design an algorithm that sets the camera's focus via a call to sensor->SetFocus().

As I'm sure you're well aware from your own experiences, it can be very frustrating when camera takes a long time to focus, causing you to miss a great action shot. Therefore, a good implementation of autofocus should try and make its focusing decision quickly by analyzing as few pixels as possible. Although we are not grading based on the performance of your autofocus implementation, it can be fun to design an algorithm that quickly converges to a good solution. The codebase provides autofocus statistics at the end of program execution (how many crop windows requested, how many pixels requested, etc.). The method sensor->GetMinFocalPlane() and sensor->GetMinFocalPlane() are convenient to obtain the range of focal planes the current scene admits.

Tips:

  • We've provided you with a few scenes where it's not immediately clear what the "right answer" is for an autofocus system. You'll have to make some choices and assumptions about what is the best object in the scene to focus on. Please describe your choices in the writeup.
  • Does scene4 cause your autofocus system problems? Why? (Does this situation remind you of experiences with a real camera?)

Part 3: Getting A Bit of Experience Optimizing Code With Halide

In the final part of this project, you are to remiplement your camera image processing pipeline from part 1 using Halide. (Not the autofocus pipeline from part 2, only part 1.) You may also implement a simpler version of the pipeline.

Detailed instructions for downloading and building Halide are located on the Halide site: https://github.com/halide/Halide, however, a quick summary is below:

We recommend building llvm3.8 from source:

svn co https://llvm.org/svn/llvm-project/llvm/branches/release_38 llvm3.8
svn co https://llvm.org/svn/llvm-project/cfe/branches/release_38 llvm3.8/tools/clang
cd llvm3.8
mkdir build
cd build
.....
make -j8

Then build Halide from source:

Setup envvars:

export LLVM_CONFIG=<path to llvm>/build/bin/llvm-config
export CLANG=<path to llvm>/build/bin/clang

Build Halide:

 cd <path to Halide tree>
 make
 make run_tests (optional)

Finally, change the Assignment 1 source code to build the Halide version of your pipeline. Modify the first line in CameraPipeline/CMakeLists.txt to set the halide variable to true. (Or use the CMake utility ccmake PATH_TO_SOURCEDIR to change it.)

You will implement your Halide pipeline in halide_camera_pipe.cpp. For those curious, we've set up the assignment to use a Halide "ahead of time" build, rather than a JIT compile. The build will compile the pipeline you've implemented in halide_camera_pipe.cpp into a library that's linked against by your camera pipeine. This is usually convenient, since it means that you don't have to JIT the Halide pipeline implementation each time you run your program. See the comments in halide_camera_pipe.cpp for more details.

We highly recommend the Halide tutorials available on the Halide web site: http://halide-lang.org/

Grading

This assignment is not graded on wall-clock performance, only image quality. (However, we reserve the right to grudgingly take points off if your autofocus implementation is shockingly brute force or naive.) A reasonable implementation will address mosaicing, pixel defects, and noise (lens vignetting is optional) and produce a good-quality image using the -noiselevel 2 settings. (We don't have a numeric definition of good since there is no precise right answer... it's a photograph, you'll know a good one when you see it.) Of course, we will be more impressed if you are able to robustly handle higher noise settings.

To verify that you have actually implemented the "auto" part of autofocus, we will test your autofocus algorithm on at least one scene that is not given to you in the current /scenes directory. The composition/framing of the grading scenes will be straightforwad. (It will not be a pathological case like scene4).

Handin

Assignment handin directories will be created for you at: /afs/cs/academic/class/15769-f16-users/ANDREWID.

You may need to run aklog cs.cmu.edu prior to copying files over. (Otherwise you may observe a permission denied error.)

Code Handin

Please hand in CameraPipeline.cpp. (We should be able to build and run the code on Andrew Linux machines by dropping your CameraPipeline.cpp directory into a freshly checked out tree. Specifically, my scripts will be looking for these files in the directory (yes, you need to create it):

/afs/cs/academic/class/15769-f16-users/ANDREWID/asst1/

Yes, if you need to include a precomputed image asset with your handin, please also add it to this directory. For simplicity your code can assume this asset is in the local working directory when I run your code.

Writeup Handin

Please include a short writeup describing your implementation of parts 1, 2, and 3.

Specifically address the following:

  • Describe the techniques you employed to address image quality problems of: noise, sensor defects, and vignetting
  • Describe your autofocus algorithm
    • What scene objects did you choose to focus on? Why?
    • Did you have to combat any problems caused by sensor noise?
  • Describe any additional optimizations or features you added to the camera pipeline.