Globally Color Improvement for Prokudin-Gorskii's Photos Collection

By Suporn Pongnumkul

Motivation

Prokudin-Gorskii was an amazing person. His master piece was the color image collection that we worked on for Homework 1. We have almost 2000 negatives of valuable historical images; however manual work to turn them into beautiful images is time consuming. I decided to work on part of the work to make color images from this collection.

Some work has been done manually on the "digichromatography" of these negatives. Therefore, using the original combined negative and the manually combined images from the Library of Congress' website, we can learn a mapping from the original images to the good images. The manually combined images are labelled as "ppmsc" images, and there are 122 of them. Assuming that there is some global color correction done, the mapping will turn out to be a function, and applying this function to some original images, we will get color improved images.

Outline of the Process of Global Color Improving

Details for each step of the Process

1. Registering the Negatives

The registration process for each image is done as followed:
Use the red channel from the ppmsc image, call this R:
For each color channel of the original negative, call it M:
Use the code from Homework 1 to find the best translation between R(static) and M(moved)
Use Affine Tracking software [Jianbo Shi and Carlo Tomasi] to find a good affine parameter from M to match R
Do the affine transformation on M
Crop M to exactly fit R
Put the three channels together to get an aligned image.
The affine transformation is needed here because the ppmsc picture is tilted from the original negative scans from the Library of Congress. [Actually the three color channels of the negative aren't always only translation between each other. Some evidence can be seen in some big combined image, for example, this. Some parts of the image is aligned, while some parts are misaligned. So the affine transformation helps improving the alignment.]

2. Find mapping for each picture

I decided to randomly pick 61 pictures out of 122 to be the trained images. For each image and each color channel, I calculate the mapping from the original picture to the corresponding "ppmsc" image, using the following algorithm.

For each possible pixel values (0 - 255)
Find the pixels in the original image that has the pixel value
Look up all the pixel values of the corresponding pixels in the "ppmsc" image
Find the median of those values
From this algorithm, we learned a mapping (for each channel) of each picture. A sample of the result is shown below. The x-axis is the original pixel values, and the y-axis is the pixel values that are mapped to.

3. Averaging the mappings

After we learned mappings of 61 different images, we average them. And get this result:
The x-axis is the original pixel values, and the y-axis is the pixel values that are mapped to. The black line is identity mapping, which I put there for a refference. By observing the average mapping, it suggests that on average, the red and green channels are almost fixed to the original values, while the blue channel should be reduced a little.

4. Applying the average mapping to some pictures

This step is to take each pixel and change it to the appropriate values by looking up the color mapping we get from step 3. Here is a result of one image.
The change might not be obvious, but looking at it closely, in the original image (left), the blue channel values are higher than they should be. The original picture is too bluish. The mapping reduces the blue part of the image and makes the result (right) picture's color look more natural.

The Problem

This improvement only work for a few images from the collection, namely the pictures that needs correction (reducing) on the blue channel. However, if we look at some color mappings from different pictures:
As we can see, the mappings for each picture aren't uniform. This suggests that some pictures need correction on the red channel, some are already good, some need corrections on more than one channel. This depends on the nature of the picture itself. Hence, there is no global color correction for all images from this correction.

Conclusion:

This method doesn't work as well as we expected, but some extension of this can be done. Future work will be on )for each test image) trying to get the most similar picture in the train-set, and apply the color transformation of that image to our test image.




Hole Fillings

Motivation

Some scratches and dirt on the negative result in blobs of bright red/green/blue on the final image, which is undesirable. Getting rid of this will improve the pictures. The idea behind this is that we have three copies of the same structure (red channel, green channel, blue channel.) In pictures from 100 years ago, the chance that there will be pure red, or green or blue in the picture is very rare. Therefore, each structure should have edges in all color channels. If there is some edge in one color channel, but not the other two, then that is a "false" edge, i.e. some scratch or dirt on that color channel. Working on gradiant domain, we can delete the false edge. Then reconstruct the image.

More Mathematical Details

Since we are looking at structures in the images, we can look at the edges. Therefore, we compute the derivative in x direction and derivative in y direction. Then detect the edges and delete the false edges. (Looking at the picture below, the second red jump is the false edge, because the other two channels don't have edge. So we replace it with the average of the other two channels, to preserve some structures)

Some Results