[PATCH 1/2] staging: Add Snappy compression library
zeev.tarantov at gmail.com
Tue Apr 19 19:01:24 PDT 2011
On Wed, Apr 20, 2011 at 04:09, Nitin Gupta <ngupta at vflare.org> wrote:
> On 04/19/2011 08:01 PM, Zeev Tarantov wrote:
>> On Tue, Apr 19, 2011 at 14:31, Nitin Gupta<ngupta at vflare.org> wrote:
>>> I'm in the process of writing a simple test for all these algorithms:
>>> This compresses input file page-by-page and dumps compressed sizes and
>>> taken to compress. With this data, I hope to come up with an adaptive
>>> which uses different compressors for different pages such that overall
>>> compression ratio stays good while not hogging CPU like when zlib is used
>> I have extended the block compressor tester I've written for Dan
>> to show this data.
>> It compresses a file one page at a time, computes a simple histogram
>> of compressed size, keeps elapsed cpu time and writes the compressed
>> blocks (and index) so the original file can be restored (to prevent
>> cheating, basically).
>> Code: https://github.com/zeevt/csnappy/blob/master/block_compressor.c
>> Because people don't click links, results inlined:
> fstats now does all this and gnuplot does histogram :)
> Anyways, I don't have any issues with links and don't have any problems copy
> pasting your stuff here if needed for context. Still, when posting your
> patches, it would be better to keep some of these performance numbers in the
> patch description.
fstats in mercurial is at the original version, with no snappy support
and with zlib reallocating memory inside the timed inner loop.
Anyway, are the benchmarks I posted enough? Should I use more
different kinds of data? What kind of data do people want to compress
Would making a mode of the tester that accepts pid instead of path and
compresses the read-only mapped pages in /proc/<pid>/maps be more
>> What do you think of the submitted patch as -is?
> csnappy as a compile time option is definitely welcome.
> Manual runtime switching is not really needed -- instead of more knobs for
> this, it would be better to have some simple scheme to automatically switch
> between available compressors.
Would you please (test and) ack this, then?
Or does it need changes?
RE: adaptive compression method.
What kind of simple scheme did you have in mind? I will gladly prototype it.
If you're serious about resorting to zlib is some cases, maybe it
should be tuned for in-memory compression instead of archiving: The
streaming interface slows it down. It has adler32 checksum zram
doesn't need. The defaults are not tuned for 4KB input
(max_lazy_match, good_match, nice_match, max_chain_length).
More information about the devel