Scene Completion Using Millions of Photographs

People

• James Hays
• Alexei Efros

Abstract

What can you do with a million images? In this paper we present a new image completion algorithm powered by a huge database of photographs gathered from the Web. The algorithm patches up holes in images by finding similar image regions in the database that are not only seamless but also semantically valid. Our chief insight is that while the space of images is effectively infinite, the space of semantically differentiable scenes is actually not that large. For many image completion tasks we are able to find similar scenes which contain image fragments that will convincingly complete the image. Our algorithm is entirely data-driven, requiring no annotations or labelling by the user. Unlike existing image completion methods, our algorithm can generate a diverse set of image completions and we allow users to select among them. We demonstrate the superiority of our algorithm over existing image completion approaches.

Paper

SIGGRAPH paper. (pdf, 11MB)

Communications of the ACM paper. (pdf, 15MB) with Technical Perspective by Marc Levoy.

Presentation

.zip file (ppt + movies), 90MB

Citation

James Hays, Alexei A. Efros. Scene Completion Using Millions of Photographs. ACM Transactions on Graphics (SIGGRAPH 2007). August 2007, vol. 26, No. 3.
Bibtex

 

 

Data

• Test set, 44MB containing 51 image/mask pairs used for our user study.
• Additional test cases, 22MB. 27 more pairs.
• User study materials, 184MB. Input image/masks, results from our algorithm, results from Criminisi et al., and Matlab code to run and analyze user studies on these sets of images.

Image Downloading Code

Code for downloading Flickr images.

Implementations

I'm aware of two class projects which implemented Scene Completion with smaller data sets. The results are very interesting.

• Implementaiton by Krista Ehinger.
• Implementation by Carlo Furst, Ruben Gerlach, and Moritz Schneeweiss.

I also had my Computational Photography Class implement Scene Completion, with scene matches from the larger im2gps database. The results are significantly better than the original SIGGRAPH project in some cases.

Funding

This research is supported in part by:

• NSF Graduate Research Fellowship to James Hays.
• NSF grant CAREER IIS-0546547.
• NSF grant CCF-0541230.