Re: [MLUG - DISCUSSION] Storage Challenge Competition winner

[Michael <EMAIL:PROTECTED>]

I wonder how repetitive their data was. You can often compress a 1GB log file into a couple megabytes just by using a normal gzip on it. Seems amazing until you remember how repetitive the log file data is. (And is a good reason to rotate and compress log files.)

I remember playing around with representing data as an algorithm that just treated data as numbers and used simple formulas that could generate the original data with just a couple starting numbers. Great compression but the problem was that it was horribly slow and was often slower than producing the original data or transmitting the original data. An important factor when evaluating compression algorithms I think.

I will look later when my screen is bigger, but this makes it sound
like the petabyte was an intermediate result; maybe they had to store
huge matrices in the middle of the algorithm to optimize the
compression. Most compression algorithms out there are heuristic
rather than optimal for big datasets.

_______________________________________________
discussion mailing list
EMAIL:PROTECTED
http://mlug.missouri.edu/mailman/listinfo/discussion