RE: http_url_post speed concerns

["Bartell, Aaron L. (TC)" <ALBartell@xxxxxxxxxxxxxx>]

Sender: "Bartell, Aaron L. (TC)" <ALBartell@xxxxxxxxxxxxxx>

>Why?  Because it makes the code easier to write that way :) 

Ha!  I was afraid you were going to say that.  I have written my own
processes to do the receive, but I switched over to yours because I didn't
know how to do the algorithm to figure out where the end of the request was
(Buck posted it once to the RPG list, but I never did take the time to
implement it http://archive.midrange.com/rpg400-l/200211/msg00525.html).  

>The reason the code was easier to write this way is that it needs to read
until it encounters a x'0d0a0d0a' (CRLF twice) from the HTTP server.

I am not sure if it is safe to write it this way, because you could
potentially have that sequence of characters in the content of the POST
response.

>...and on average, will be reading less than 200 bytes. 

How did you come up with that average?  I may be an exception, but I am
passing XML back and forth, and I am easily past 1000 bytes going out and
coming into each web service.  Oh, don't you just love XML;-)  It looks like
I would have to have my program that is hosting the web service specify that
info was chunked for it to come across as a "document" per your comment
about recvdoc() being much faster?

I hear ya on wanting to get help, I haven't had one ounce of outside help
yet for the RPG XML parser I am working on, of course I have only had that
open for a couple months and I don't have it in CVS yet (as you pointed out
Scott :-).  I wont have time to punch out any helpful code for a couple
weeks (conferences) but will give it a shot after I am done with those.

Aaron Bartell

-----Original Message-----
From: Scott Klement [mailto:klemscot@xxxxxxxxxxxx]
Sent: Friday, June 13, 2003 12:41 PM
To: 'ftpapi@xxxxxxxxxxxxx'
Subject: Re: http_url_post speed concerns


Sender: Scott Klement <klemscot@xxxxxxxxxxxx>

On Fri, 13 Jun 2003, Bartell, Aaron L. (TC) wrote:
>
> I have a program making use of the http_url_post sub proc from the HTTP
> API's and have some concerns about speed. I have gone through the code and
> have a question as to why only one byte of data is being read from the
> socket connection at a time?  Take a look at this line of code "eval
> wwRec = recv(peSock: wwPos: 1: 0)".

Why?  Because it makes the code easier to write that way :)   You're
right, it won't perform very well, but the recvresp() routine is only
used once during the HTTP conversation, and on average, will be reading
less than 200 bytes.   So, making it as fast as possible wasn't a priority
for me.   By contrast, it was important to be that recvdoc() be efficient,
since that could be used to handle many megabytes during transfer.  That's
where most of the time will be.

The reason the code was easier to write this way is that it needs to read
until it encounters a x'0d0a0d0a' (CRLF twice) from the HTTP server.  The
length of the data between the start and the CRLF sequence is completely
variable.  It could be 0 bytes, it could be many bytes...  no way to know
how much to read.

You could read the data into a larger buffer (say 1024A) and then scan
that buffer for the x'0d0a0d0a' sequence, however, what if the length
happened to end up so that part of that string was in one read, and part
of it was in another?   So you'd have to add code to handle that
situation.   Then, what if the server sends more data after the
x'0d0a0d0a' sequence, and it ends up appearing in your 1k buffer
after the CRLF sequence?   You'd have to save that data, and use it
instead of a direct call to recv() in any/all subprocedures that could
read from the socket after calling recvresp().

>
> Any clarification is welcome. . .
>

To sum up:  You're right, reading one at a time is slow and inefficient.
However, it's much easier to code that way, and that routine is only a
small part of the conversation.

Solution:   If you'd like to re-write the recvresp and/or ssl_recvresp
procedures to be more efficient, you're welcome to do so.  Assuming
that your code looks like it'll work properly in all circumstances,
I'd be happy to include it in the distribution of HTTPAPI.

Remember, this is an open-source project.  It only survives if people are
willing to contribute back to the project.    I do my best to help you
guys out, but the only way that I'll get any benefit from giving you guys
this code is if you contribute something back!!

-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubsribe from the list send mail
to majordomo@xxxxxxxxxxxxx with the body: unsubscribe ftpapi mymailaddr
-----------------------------------------------------------------------
-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubsribe from the list send mail
to majordomo@xxxxxxxxxxxxx with the body: unsubscribe ftpapi mymailaddr
-----------------------------------------------------------------------