PDA

View Full Version : New SPC Compression?


blargg
10-12-2004, 12:45 PM
I've written an SPC compressor and would like comments on it. The main motivation for creating it was to have the compression benefits of RAR archives while having individual files for each track for easier browsing and playlist creation.

The compressor takes the directory of SPCs from a given game, analyzes them for common data, and moves this to a shared file in the same directory. Most games use the same instrument samples and driver code for each song, so this results in significant space savings. For example:
<pre>1MB Earthbound RAR archive
8.6MB Total of decompressed files
5.6MB Total of files individually gzipped
0.8MB Total of files after running my compressor</pre>
The resulting files look like this:
<pre>2K Cave3.spc
13K CaveOfThePast.spc
3K ChaosTheater.spc
3K CoffeeBreak.spc
etc.
436K Earthbound.dat (this is the shared data)</pre>
The shared file is named after the game, so one could have SPC files from multiple games in the same directory using this compression scheme. Also, the format for each track has the standard SPC header uncompressed to allow SPC scanner/tag tools to work.

The basic question is, would this be useful to anyone or is RAR and perhaps SPZ (each SPC gz-compressed) sufficient?


<P ID="signature"></P>

CEpeep
10-12-2004, 12:49 PM
> The basic question is, would this be useful to anyone or is
> RAR and perhaps SPZ (each SPC gz-compressed) sufficient?
>

If it doesn't need a special player, I'd try it out for sure.

<P ID="signature"><hr>
Every gun that is made, every warship launched, every rocket fired, signifies in the final sense theft from those who hunger and are not fed, those who are cold and are not clothed.
-Eisenhower</P>

blargg
10-12-2004, 01:18 PM
> If it doesn't need a special player, I'd try it out for sure.

Unfortunately it will require specific support in the SPC player. That's one reason I posted; if it seems like a useful format I'll talk with player authors about integrating the code. BTW, it has been tested and works well with a private build of a player I'm working on, so it's not just an idea.


<P ID="signature"></P>

Wren
10-12-2004, 03:14 PM
If it's a new format (ie. needs a special player), it might be better to go with a more "MIDI" like ideal. For example, store all of the instruments in an instrument file, possibly named "earthbound.ins" but keep the track data in seperate files that reference the instrument file. This way you could still quickly select a specific piece without scanning through an all-in-one file for a song. You'll get about the same savings in space, but with a more convenient layout.

-Wren

<P ID="signature"></P>

MooglyGuy
10-12-2004, 04:34 PM
> If it's a new format (ie. needs a special player), it might
> be better to go with a more "MIDI" like ideal. For example,
> store all of the instruments in an instrument file, possibly
> named "earthbound.ins" but keep the track data in seperate
> files that reference the instrument file.

Read his post again, that's exactly how it works. The music player program and the instruments are parsed out to a separate file, and only the track data remains in each track.


<P ID="signature"></P>

blargg
10-12-2004, 04:37 PM
> If it's a new format (ie. needs a special player), it might
> be better to go with a more "MIDI" like ideal. For example,
> store all of the instruments in an instrument file, possibly
> named "earthbound.ins" but keep the track data in seperate
> files that reference the instrument file. [...]

It's nothing exotic like that; it's just a compression scheme like rar except it is specialized for SPCs (and perhaps PSFs too) and stores the data in multiple files. The player simply invokes the load_compressed_file function for the compressed SPC file and gets back the full 65792 bytes for the original uncompressed SPC data, much like it currently does for a gzipped file. The player then plays it like any other SPC.


<P ID="signature"></P>

The 9th Sage
10-12-2004, 06:14 PM
> 1MB Earthbound RAR archive
> 8.6MB Total of decompressed files
> 5.6MB Total of files individually gzipped
> 0.8MB Total of files after running my compressor

Hm....sounds kind of interesting, but I have a question. Is that Earthbound RAR a solid RAR archive, or just a regular archive?

<P ID="signature">http://www.xanga.com/home.aspx?user=zeldaddhttp://i.xanga.com/ZeldaDD/vibrosword.gif

My Weblog...vibrating swords now in stock!</a></P>

Wren
10-12-2004, 07:26 PM
Heh, read that wrong. I was under the impression that the files he listed were the pre-compressed versions and that earthbound.dat was a compilation of all of them.

In this case, I'd definately say that this is a nice format for SPC storage. I'll use it if I can get a fb2k or xmms plugin for it.

-Wren

<P ID="signature"></P>

blargg
10-12-2004, 08:17 PM
The Earthbound RAR archive I used as an example is solid archived (it's the one from the SPC section of this site).

The compression algorithm's main task is to find blocks of data common to two or more files, add them in the shared file and put references in the files. It also eliminates runs of repeated 0 or 255 bytes (these were common in SPC files), and compresses each file with zlib as a final step (without the zlib compression the files are 1.5 to 2 times the size). I was thinking that this compression isn't that specific to SPCs, so it could be used for any ripped music format with lots of shared data, i.e. for PlayStation (PSF?) rips and N64 if there is/will be a format for it.

Since the main shared data is the music driver and common instrument samples, these will end up in the shared file, but they won't be specially marked or anything, so unfortunately the algorithm doesn't serve as a high-level instrument extractor as mentioned in an earlier reply.

hcs
10-12-2004, 08:43 PM
> I was thinking that this compression isn't that
> specific to SPCs, so it could be used for any ripped music
> format with lots of shared data, i.e. for PlayStation (PSF?)
> rips and N64 if there is/will be a format for it.

USF uses a method very similar to this to store all common data in ROM and RAM in a single file, the USFlib.
Which was inspired by, but is more flexible than, the miniPSF/PSFlib method.

<P ID="signature">-http://hcs.freeshell.org/index.cgihcs</a></P><P ID="edit"><FONT class="small">Edited by hcs on 10/12/04 04:44 PM.</FONT></P>

Ethereal
10-13-2004, 12:25 AM
Well, if you use .ace compression, you can do this WITHOUT any custom (non-ace) algorithms. I currently have some archives that have large duplication rates in them.

4900 files, 7.2GB -> 2.5GB
4231 files, 5.7GB -> 1.6GB
8820 files, 1.9GB -> 268 MB

<P ID="signature"></P>

D--
10-13-2004, 06:36 AM
Talk a WinAmp SPC plugin author into supporting it and I would sure as hell use it.

(no offsense if you ARE an SPC plugin author, I have no idea. Zophar's Domain doesn't list the authors of any of the software in its archive, so I have no idea who is responsible for what)

<hr>
<img border=0 src=http://www.oakland.edu/~dcsoboda/img/avatar/you_lose.gif>
The Second Dimension (http://www.the2d.comhttp://www.the2d.com/</a>)

The 9th Sage
10-14-2004, 02:18 AM
> The Earthbound RAR archive I used as an example is solid
> archived (it's the one from the SPC section of this site).

I was just wondering because the archive I have compresses to around 760 some odd KB. A solid RAR archive at the highest compression level.

<P ID="signature">http://www.xanga.com/home.aspx?user=zeldaddhttp://i.xanga.com/ZeldaDD/vibrosword.gif

My Weblog...vibrating swords now in stock!</a></P>

Canar
10-14-2004, 03:52 AM
> The basic question is, would this be useful to anyone or is
> RAR and perhaps SPZ (each SPC gz-compressed) sufficient?

Try 7-zip. It's supported transparently in foobar2000 with the unpacker component. Its' compression usually beats RAR. Sometimes very significantly.

<P ID="signature"><center><a href='http://canar.no-ip.com'>http://canar.no-ip.com/teng_title.png
Visit my site and leave comments plzkthxbye.</a></center></P>

Wren
10-14-2004, 06:05 PM
Are you using 7-Zip or IZArc for 7z compression? I haven't yet tried either one, so I'm wondering which is better (IZArc seems to support more formats).

I remember an old DOS based archiver called UltraCompressor 2, which was very good. Unfortunately it hasn't been ported to any modern OSes nor is the source available to do so easily.

-Wren

<P ID="signature"></P>

Koji
10-16-2004, 06:39 AM
I support you! It's why I don't use RSNs much, even though they save quite a lot of space. If the feature was implemented in SNESamp, I'd most definitely get/convert my SPC sets to your format. :)

<P ID="signature"></P>