Re: http_url_post speed concerns

[Scott Klement <klemscot@xxxxxxxxxxxx>]

Sender: Scott Klement <klemscot@xxxxxxxxxxxx>

On Fri, 13 Jun 2003, Bartell, Aaron L. (TC) wrote:
>
> I have a program making use of the http_url_post sub proc from the HTTP
> API's and have some concerns about speed. I have gone through the code and
> have a question as to why only one byte of data is being read from the
> socket connection at a time?  Take a look at this line of code "eval
> wwRec = recv(peSock: wwPos: 1: 0)".

Why?  Because it makes the code easier to write that way :)   You're
right, it won't perform very well, but the recvresp() routine is only
used once during the HTTP conversation, and on average, will be reading
less than 200 bytes.   So, making it as fast as possible wasn't a priority
for me.   By contrast, it was important to be that recvdoc() be efficient,
since that could be used to handle many megabytes during transfer.  That's
where most of the time will be.

The reason the code was easier to write this way is that it needs to read
until it encounters a x'0d0a0d0a' (CRLF twice) from the HTTP server.  The
length of the data between the start and the CRLF sequence is completely
variable.  It could be 0 bytes, it could be many bytes...  no way to know
how much to read.

You could read the data into a larger buffer (say 1024A) and then scan
that buffer for the x'0d0a0d0a' sequence, however, what if the length
happened to end up so that part of that string was in one read, and part
of it was in another?   So you'd have to add code to handle that
situation.   Then, what if the server sends more data after the
x'0d0a0d0a' sequence, and it ends up appearing in your 1k buffer
after the CRLF sequence?   You'd have to save that data, and use it
instead of a direct call to recv() in any/all subprocedures that could
read from the socket after calling recvresp().

>
> Any clarification is welcome. . .
>

To sum up:  You're right, reading one at a time is slow and inefficient.
However, it's much easier to code that way, and that routine is only a
small part of the conversation.

Solution:   If you'd like to re-write the recvresp and/or ssl_recvresp
procedures to be more efficient, you're welcome to do so.  Assuming
that your code looks like it'll work properly in all circumstances,
I'd be happy to include it in the distribution of HTTPAPI.

Remember, this is an open-source project.  It only survives if people are
willing to contribute back to the project.    I do my best to help you
guys out, but the only way that I'll get any benefit from giving you guys
this code is if you contribute something back!!

-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubsribe from the list send mail
to majordomo@xxxxxxxxxxxxx with the body: unsubscribe ftpapi mymailaddr
-----------------------------------------------------------------------