Asset Pipelines

Whenever the easy-to-edit file format for something is different than the easy-to-use-in-your-game format, it is time to think about an asset pipeline. The key idea of an asset pipeline is that the files you edit (textures, 3D models, level data) are translated by code you don't ship into easy-to-read data formats, possibly through a series ("pipeline") of automated steps.

Thinking in terms of asset pipelines has three main advantages:

• They make game code simpler, because you get to design the data formats.
• They make asset creation easier, because you use whatever tools you want, not whatever tools speak the right format.
• They prevent you from shipping hack-y asset-prep code. You can focus on code that works for the assets you care about, once, rather than making it rock-solid on every system you distribute the game to.

Easy-to-load Formats

The easiest-to-load formats are ones you can copy into memory and immediately use.

std::unique_ptr< uint8_t[] > blob = load_binary_blob("sprite.file");
sprite = reinterpret_cast< Sprite * >(blob.get());
//or:
size_t size = 0;
std::unique_ptr< uint8_t[] > blob2 = load_binary_blob("player.mesh", &size);
glBufferData(GL_ARRAY_BUFFER, size, blob2.get(), GL_STATIC_DRAW);

In practice, you often have structures that aren't quite this nice to load (e.g. contain std::vector's), but you should still strive for this sort of simplicity.

Good format tips:

• Binary -- they should match the data layout your code (or opengl) uses.
• Self-Sizing -- it's much easier to file.read(4, &size) than to do a bunch of seeking to try and find file size. When reading vectors, it's faster to vector.resize() and do one read than to multiple.
• Consider chunks -- use a header which includes a size in each chunk so that you can append many assets together.

Design Tension: Nice C++-standard-library data structures generally can't be loaded from disk in-place; while C-style pointer-based structures can (with "pointer fixup" hacks). You'll need to think which you want to use. I generally use the former -- I find that C++ standard library data structures are much friendlier to work with and the additional loading code is minimal.

Pipeline Tools

Getting your easy-to-edit data into an easy-to-load format often involves little conversion scripts or utility programs.

Python is a great language for writing these scripts, for three reasons:

• struct gives you a way to write structured binary data.
• subprocess gives you a clean way to call external programs.
• Blender supports python scripting. Even if you aren't using blender to author your models, you can do things like use it to load them, bake ambient occlusion to a texture, and save the texture.

But don't be afraid to write small utilities in C++. One bonus is that it's very easy to put data into the structures your game expects. (You can even put these utilities into your main game code, accessed by special command-line options. This is somewhat inelegant but occasionally the most expedient way to do things.)

Makefiles or shell scripts are also useful as a way of stringing together a bunch of commands:

all : \
	../dist/textures/text.png \
	../dist/text.blob \

FONT=36daysag.ttf

../dist/textures/text.png ../dist/text.blob : *.txt ../dist/dump-glyphs Makefile
	../dist/dump-glyphs ../dist/textures/text.png ../dist/text.blob '$(FONT)' *.txt

zh/text.png zh/text.blob : zh/*.txt ../dist/dump-glyphs Makefile
	cd zh && ../../dist/dump-glyphs text.png text.blob '../wqy-microhei.ttc' *.txt

Design Tension: It might seem natural to integrate your asset processing tools into your standard build process. Doing so will avoid asset-out-of-sync errors that might otherwise cause you confusion. On the flip side, making asset processing part of the standard build process commits you to writing asset processing tools that work everywhere you want to build. This can force you to write far more robust code that you would otherwise need to.

Cautionary Note: ImageMagick is really useful for command-line image manipulation. However, ImageMagick also continuously changes command line options, colorspace conversions, etc. So be wary of pipelines that must work on multiple computers and use ImageMagick utilities like convert or montage.