RE: Data Compression

[Mike Krebs <mkrebs@xxxxxxxxxxxxxxxxxx>]

1. I believe HTTP says compression is negotiated. Your web services group should not be turning that on by default. Larger xml documents will benefit from compression/decompression. Small ones, it isn't worth the cpu cycles (YMMV).
2. In some experiments, I was able to use zlib to do the decompression. I don't remember what I had to do to make that happen, but it seemed trivial at the time. If I remember right, my experiments included modifying the HTTP header (some AddHeader thing already in HTTPAPI) and downloading the data to the IFS. From there, using zlib to do the decompression offline. I don't remember how I prepared the IFS file for doing the decompression.
3. Integrating into httpapi requires some knowledge of the internals. I think the compression occurs on the whole data string and therefore the decompression would follow after receiving all the data (you can't decompress "on the fly" unless you are sure of the "compression breaks" in each packet). I was stuck on this point when I moved to other projects, but it would seem that RecvDoc sounds like a good place for the decompression.

Mike Krebs

> -----Original Message-----
> From: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx [mailto:ftpapi-
> bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Karl Woods
> Sent: Wednesday, August 12, 2009 11:09 AM
> To: HTTPAPI and FTPAPI Projects
> Subject: RE: Data Compression
> 
> I started this thread back in March. I was just reminded by our Web
> Services group that they intend to compress data before sending it out.
> 
> Since HTTPAPI does not decompress data, could someone tell me what
> steps I need to take to decompress the data before HTTPAPI parses it?
> 
> Karl B. Woods
> Sr. Programmer/Analyst
> Professional Services
> 1960 S. Milestone Dr., Ste B
> Salt Lake City, UT 84104
> Tel: (801) 433-9426 (office)
> Fax: (801) 487-2775
> E-mail:  karl.woods@xxxxxxxxxxxxxx
> 
> 
> -----Original Message-----
> From: Scott Klement [mailto:sk@xxxxxxxxxxxxxxxx]
> Sent: Friday, March 06, 2009 6:06 PM
> To: HTTPAPI and FTPAPI Projects
> Subject: Re: Data Compression
> 
> hi Mike,
> 
> It would be RecvDoc() and/or RecvChunk() that would have to be
> modified,
> and not RecvResp().  Compression does not affect the HTTP keywords and
> other header information, it only affects the document itself.
> 
> and, I'm not familiar with ZLIB, so I'll take your word if you say it's
> compatible with gzip...  I know ZLIB is used in ZIP/UNZIP so I would've
> guessed it to be more like "deflate" encoding.  Maybe it does both?
> 
> 
> 
> Mike Krebs wrote:
> > Scott,
> >
> > Would adding decompression support to HTTPAPI be something like this?
> >
> > 1. Pick a decompression library/program (ie zlib which seems to be
> ported to
> > the standard library system and can be used to compress/uncompress
> gzip
> > encoded objects)
> > 2. Build a DECODE module similar to the encoder routines.
> > 3. Build in the content-header Accept-Encoding: gzip
> > 4. Extend the doget (or maybe RecvResp?) procedure to uncompress
> compressed
> > data. This would also touch parse_resp_chain to find out if the
> content was
> > gzipped similar to the way you handle 'chunked'?
> >
> > Mike Krebs
> >
> -----------------------------------------------------------------------
> This is the FTPAPI mailing list.  To unsubscribe, please go to:
> http://www.scottklement.com/mailman/listinfo/ftpapi
> -----------------------------------------------------------------------
> -----------------------------------------------------------------------
> This is the FTPAPI mailing list.  To unsubscribe, please go to:
> http://www.scottklement.com/mailman/listinfo/ftpapi
> -----------------------------------------------------------------------
-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------