Here is some mail that I've received regarding Libasset. I'm including it here because the discussion will be very educational to anyone who is planning on using the library in a game.

These emails have been posted with permission from the various participants.

The first message is from me, in response to a feature request from Michael Searle

From: Talin <Talin@ACM.org>
Subject: Re: libasset
To: searle@longacre.demon.co.uk

Hey, Mike...

First off, thanks for your interest and comments!

> I just downloaded libasset 0.1 - looks like there's a problem
> with the distribution, as configure needs a scripts
> directory which isn't there. It builds OK without
> configure, make or warnings though, and works fine on
> FreeBSD 4.0. (BTW, what other systems does it run on?)

Hmmm. That's interesting. If you do a mkdir on the scripts directory, and then do an "autoconf --add-missing" it seems to work OK. I'll see what I can do about making the install more friendly. I'm not an autoconf guru by any means.

I don't know yet what other systems libasset runs on, however since it only uses vanilla ANSI C I suspect it will run on any platform that is a) POSIX-compatible, and b) has a zlib port.

> My feature-request - it would be nice if the path could include
> a directory of zipfiles (and internal directories?)
> to make it easier to add extensions. As it is now, an add-on
> package would either have to be referred to by name in the path,
> or unpacked manually before using it. With a directory, all
> packages could be just moved into the resource directory and
> used automatically. Distinguishing such a directory from a
> directory used directly could be a problem, but you could just
> check both cases.

One issue is that there seems to be two different models of how people think of "extensions". The model you are thinking of is similar to that of Photoshop plug-ins, where you generally want to have every possible plug-in loaded into the program. The model I was thinking of is more like DOOM .wad files - you generally don't want to load all possible wad files into a game, but rather you have a "canonical" wad file with perhaps a few variations represented as "deltas" (i.e. diffs/patches) of the main file. So in this case, you would only want to have specific sets of extensions active at any given time out of the set of all possible extensions that you have available.

Note that the model used is going to depend greatly upon the game. Let's take two hypothetical examples:

Example 1: A first-person shooter called "BOOM", in which the goal is to shoot down alien monsters, and where the the game artwork is stored in a zipped asset file. Harry Hacker has decided that he wants to convert the regular vanilla "BOOM" into "South Park BOOM" and wants to replace all of the artwork. So he re-does all of the artwork, and creates a zipfile which is a delta of the canonical BOOM zipfile.

Ben Bitcruncher decides to take it a step further and create a delta of Harry' delta, called "South Park BOOM 2: Kenny's Revenge". Note that applying Ken's delta to a canonical BOOM assetfile, without Harry's delta, is probably nonsensical, or at least will give you only a part of the experience.

Meanwhile Cathy Codewright decides to make "BOOM: The Phantom Menace" by replacing all of the artwork with appropriate (tm) Star Wars characters. Note that running both Cathy and Harry's deltas at the same time is also probably nonsensical - what you'll probably see is only one set of artwork.

This is the primary scenario for which libasset was designed.

Example 2: A civilization-style game called "Barbarism: 2000", where each civilization is stored in a seperate zipfile. All of the artwork, sounds, and behavior data for a particular civilization are stored in a single zipfile, and you can add new civilizations by adding new zipfiles. In a sense, each of the civilizations are true Photoshop-style plug-ins; Each can be added without affecting the integrity of the others.

Of course, this scheme has a number of problems, or at least the current libasset design fits rather poorly with this style of usage.

First, there's nothing to _prevent_ assets in one civilization from over-riding assets in another, and in fact the artist would have to be very careful and disciplined in order to prevent this (I suspect that an appropriate naming convention might be of some use in this case.)

Secondly, the current libasset model is to present the game with a single uniform hierarchical namespace. There's currently no mechanism, however, for getting a directory listing of this space (I'm planning on adding this eventually), so for now the game has to know in advance which assets it wants to load. Of course, it can find this information inside of _other_ assets, but ultimately there must be some hard-coded set of "root assets" which the game knows about.

One way in which the current scheme could be adapted to a "plugin-style" model is as follows: Have the game start by iterating through the plugins directory itself, adding each zipfile to the path (there's no reason the asset path has to be constructed by hand, after all), and remembering the names of each zipfile encountered. Then, each zipfile would have inside of it a directory who's name was the same as the name of the zipfile. Inside that directory would be (among other things) a ".info" file containing information about the plugin. For example, the zipfile "Romans.zip" would have an asset named "Romans/.info" which would contain root-level information about that plugin. Then, using the names of the zipfiles remembered earlier, load each of the ".info" assets, one for each civilization.

As you can see, that solution is a bit kludgy, but it's a direct consequence of the way the namespace is organized, which was intended to support the kind of "override" semantics as illustrated in example 1. I could probably without too much trouble add some support in the library which would make this kludge easier, although probably no less ugly.

There is also the issue of search ordering. For example, suppose you had three seperate .zip files, all in the same directory, and all of which contained an asset with the same name. Which zip file gets searched first? Is the search order deterministic, or arbitrary?

So I guess what I'm getting at is that I'm not sure what the Right Thing is as regards to your feature request. I'd be interested in your suggestions on this matter.

> Also, and I realise this would be a lot of work, is there any
> possibility of read/write support? This would be useful for
> configuration and especially for saved game states. Some of
> these can be huge and complex, sometimes with only part
> being in memory at any time, so a random-access archive is
> needed.

Saving game states in a zipfile is definately _not_ the right thing in my opinion. Zipfiles tend to fragment when modified incrementally. You really want to seperate writable data from read-only data (unless your entire world is writable in which case I say 1) I can't wait to see it, 2) You're in good company [Ultima 2 was total-world writable], and 3) libasset is not the solution for you.)

What you really need is a small database. I have a couple of small db's that I have written for games which I might be able to make available (they work fine, but they need a lot of cleaning up and that's going to take time.)

In the mean time, consider using gdbm or the Sleepycat dbm.
 
> Thanks, Mike.
>
> --
> searle@longacre.demon.co.uk

From: Michael Searle <searle@longacre.demon.co.uk>
Subject: Re: libasset
To: Talin <Talin@ACM.org>

On Tue, Nov 02, 1999 at 05:46:03AM +0000, Talin wrote:
> Hey, Mike...
>
> First off, thanks for your interest and comments!
>
> > I just downloaded libasset 0.1 - looks like there's a problem
> > with the distribution, as configure needs a scripts
> > directory which isn't there. It builds OK without
> > configure, make or warnings though, and works fine on
> > FreeBSD 4.0. (BTW, what other systems does it run on?)
>
> Hmmm. That's interesting. If you do a mkdir on the scripts directory,
> and then do an "autoconf --add-missing" it seems to work OK. I'll see
> what I can do about making the install more friendly. I'm not an
> autoconf guru by any means.

I didn't expect autoconf. I assumed I could just run configure (this
is the usual way, and not everyone has autoconf installed anyway)

> I don't know yet what other systems libasset runs on, however since it
> only uses vanilla ANSI C I suspect it will run on any platform that is
> a) POSIX-compatible, and b) has a zlib port.

That's good to know. (I don't think alloca is very portable though - may
be only in the test programs)

>
> > My feature-request - it would be nice if the path could include
> > a directory of zipfiles (and internal directories?)
> > to make it easier to add extensions. As it is now, an add-on
> > package would either have to be referred to by name in the path,
> > or unpacked manually before using it. With a directory, all
> > packages could be just moved into the resource directory and
> > used automatically. Distinguishing such a directory from a
> > directory used directly could be a problem, but you could just
> > check both cases.
>
> One issue is that there seems to be two different models of how people
> think of "extensions". The model you are thinking of is similar to that
> of Photoshop plug-ins, where you generally want to have every possible
> plug-in loaded into the program. The model I was thinking of is more
> like DOOM .wad files - you generally don't want to load all possible wad
> files into a game, but rather you have a "canonical" wad file with
> perhaps a few variations represented as "deltas" (i.e. diffs/patches) of
> the main file. So in this case, you would only want to have specific
> sets of extensions active at any given time out of the set of all
> possible extensions that you have available.

Actually I am thinking of something more like wad files - but not just
for arbitrary patches, I intended to use these for levels as well
(which will not be loaded at once, but are all part of the same game
,all have the same interface and have dependencies (eg connections between
levels)).

> Note that the model used is going to depend greatly upon the game. Let's
> take two hypothetical examples:
>
> Example 1: A first-person shooter called "BOOM", in which the goal is to
> shoot down alien monsters, and where the the game artwork is stored in a
> zipped asset file. Harry Hacker has decided that he wants to convert the
> regular vanilla "BOOM" into "South Park BOOM" and wants to replace all
> of the artwork. So he re-does all of the artwork, and creates a zipfile
> which is a delta of the canonical BOOM zipfile.
>
> Ben Bitcruncher decides to take it a step further and create a delta of
> Harry' delta, called "South Park BOOM 2: Kenny's Revenge". Note that
> applying Ken's delta to a canonical BOOM assetfile, without Harry's
> delta, is probably nonsensical, or at least will give you only a part of
> the experience.
>
> Meanwhile Cathy Codewright decides to make "BOOM: The Phantom Menace" by
> replacing all of the artwork with appropriate (tm) Star Wars characters.
> Note that running both Cathy and Harry's deltas at the same time is also
> probably nonsensical - what you'll probably see is only one set of
> artwork.
>
> This is the primary scenario for which libasset was designed.
>
> Example 2: A civilization-style game called "Barbarism: 2000", where
> each civilization is stored in a seperate zipfile. All of the artwork,
> sounds, and behavior data for a particular civilization are stored in a
> single zipfile, and you can add new civilizations by adding new
> zipfiles. In a sense, each of the civilizations are true Photoshop-style
> plug-ins; Each can be added without affecting the integrity of the
> others.
>
> Of course, this scheme has a number of problems, or at least the current
> libasset design fits rather poorly with this style of usage.
>
> First, there's nothing to _prevent_ assets in one civilization from
> over-riding assets in another, and in fact the artist would have to be
> very careful and disciplined in order to prevent this (I suspect that an
> appropriate naming convention might be of some use in this case.)

A civilisation would have to have a particular format with a defined interface
to be used by the game, and all dependencies can probably go through this
interface. In this case there should be no problems with putting each in
a separate directory.

> Secondly, the current libasset model is to present the game with a
> single uniform hierarchical namespace. There's currently no mechanism,
> however, for getting a directory listing of this space (I'm planning on
> adding this eventually), so for now the game has to know in advance
> which assets it wants to load. Of course, it can find this information
> inside of _other_ assets, but ultimately there must be some hard-coded
> set of "root assets" which the game knows about.

OK, it would still be easier to add a new zip and modify one root file
than change the main zip, but a directory listing is a much nicer
way to do this.

> One way in which the current scheme could be adapted to a "plugin-style"
> model is as follows: Have the game start by iterating through the
> plugins directory itself, adding each zipfile to the path (there's no
> reason the asset path has to be constructed by hand, after all), and
> remembering the names of each zipfile encountered. Then, each zipfile
> would have inside of it a directory who's name was the same as the name
> of the zipfile. Inside that directory would be (among other things) a
> ".info" file containing information about the plugin. For example, the
> zipfile "Romans.zip" would have an asset named "Romans/.info" which
> would contain root-level information about that plugin. Then, using the
> names of the zipfiles remembered earlier, load each of the ".info"
> assets, one for each civilization.

That would be OK for a Photoshop style directory, with lots of different
types of plugins all loaded, but you could often get away with something much
simpler.

In the case of a per level zip, you have only one level loaded at once
and it is always in the same format. So, neither the subdirectory
structure or the .info file is needed.

If levels require other level's assets, or if multiple plugins of the same
type can be loaded (eg civilisations) the subdirectory would be needed,
but IMO the best way of doing multiple plugin types is not to have a .info
but put them in separate directories - this also makes it clearer what type
each plugin is.

> As you can see, that solution is a bit kludgy, but it's a direct
> consequence of the way the namespace is organized, which was intended to
> support the kind of "override" semantics as illustrated in example 1. I
> could probably without too much trouble add some support in the library
> which would make this kludge easier, although probably no less ugly.

However it is done, it sounds like something better done in the library
than in the caller. What? Me, lazy? :)

> There is also the issue of search ordering. For example, suppose you had
> three seperate .zip files, all in the same directory, and all of which
> contained an asset with the same name. Which zip file gets searched
> first? Is the search order deterministic, or arbitrary?
>
> So I guess what I'm getting at is that I'm not sure what the Right Thing
> is as regards to your feature request. I'd be interested in your
> suggestions on this matter.

A deterministic order within a single directory sounds fragile and
mostly unnecessary, but Kenny's Revenge requires an order. A simple/stupid
way would be just to sort alphabetically, with dependent patches
being given tagged filenames (0001-southpark.zip, 0002-kennysrevenge.zip).

This could be difficult to use if you had a lot of highly interlinked
patches - you could use a dependency list per zip and do it automatically
from there, but this seems like massive overkill for something which is
probably not much of a problem. It also causes problems for the bulk of
the plugins which don't need ordering.

> > Also, and I realise this would be a lot of work, is there any
> > possibility of read/write support? This would be useful for
> > configuration and especially for saved game states. Some of
> > these can be huge and complex, sometimes with only part
> > being in memory at any time, so a random-access archive is
> > needed.
>
> Saving game states in a zipfile is definately _not_ the right thing in
> my opinion. Zipfiles tend to fragment when modified incrementally. You
> really want to seperate writable data from read-only data (unless your
> entire world is writable in which case I say 1) I can't wait to see it,
> 2) You're in good company [Ultima 2 was total-world writable], and 3)
> libasset is not the solution for you.)

I am separating writable data - I'm talking about a separate zip used as
a save file. (Although I don't currently plan on a total writable world,
an example would be Nethack - a large save game can be bigger (compressed)
than the exe and all static data (uncompressed) together, and contains
maybe 70 stored levels.)

I thought when I went to zip files, I would be getting away from
fragmentation problems. When I found that there wasn't any suitable
library for this last year, I ended up writing my own read/write archiver,
complete with a defragger that ran whenever the file was closed, ick.

>
> What you really need is a small database. I have a couple of small db's
> that I have written for games which I might be able to make available
> (they work fine, but they need a lot of cleaning up and that's going to
> take time.)
>
> In the mean time, consider using gdbm or the Sleepycat dbm.
>

I considered using BSD db (similar to dbm IIRC) but it seems far too
complex for a simple save file format. I'd have to do compression by
hand as well. I'll have another look, but maybe the game specific dbs
would be a better match? (What's the fragmentation efficiency on these?
AFAIK most of these use 20-50% in indexing, which doesn't sound much better
than a zipfile.)

> > Thanks, Mike.
> >
> > --
> > searle@longacre.demon.co.uk

--
searle@longacre.demon.co.uk
 

Return to Talin's project page.
Return to Talin's home page.