Re: HttpApi Question

[Scott Klement <sk@xxxxxxxxxxxxxxxx>]

Mike,

HTTP contains protocol data, the meta-information that makes HTTP work, 
and then it also has the data that's being transferred, whcih I shall 
refer to as the 'payload' of the transaction. Since the payload is 8-bit 
binary data, it can contain any conceivable sequence of bytes... 
therefore there needs to be some way for HTTP to communicate when the 
'payload' has finished.... It can't tell that based on the contents of 
the data, because the data can literally contain anything.

There are 3 different ways that HTTP can communicate when the transfer 
of the 'payload' is complete:

1) The most common method is to send a content-length in the 
meta-information.  Then the receiver knows exactly how many bytes to 
expect.  It knows the file is complete when it has received that many bytes.

This is most common when downloading a file that's on disk... in this 
case the server knows at the outset exactly how big the file is (in 
bytes) and can just send the length.  But, imagine a situation where 
data is being generated by an application...  in this case, the server 
may not know the total size of the document at the outset.  The 
application may be sending megabytes of data, to the server, and the 
server won't want to buffer the whole thing just to calculate the length 
--  so it receives one buffer of data at a time from the application, 
then writes it to the client, then receives the next buffer, etc, until 
it reaches the end.  In this case it won't know until the end what the 
total length is... so content-length cannot be used.

2) It can use "chunked" transfer-coding.  In this case, the HTTP server 
will receive one buffer full from the application, and immediately send 
it on to the client.  Each buffer full is called a "chunk".  HTTP will 
send the length of the chunk, followed by the data, then will send the 
length of the next chunk, followed by more data, etc.  This repeats 
until the server has no more data to send, in which case it sends a 
length of 0 to tell the client that the transfer is complete.  This way, 
the server does not need to know the total payload size at the outset, 
it only needs to know how much it's sending in each chunk...

3) The original way of sending data (which is not recommended as of 
HTTP/1.1, but is still supported for backward compatibility with older 
HTTP specs) is to indicate the end of the data by disconnecting.  In 
this method, the server sends "connection: Close" in the meta-data to 
tell the client to expect that it will be disconnected when the transfer 
is compelte.  then, it just streams data until it has no more, and 
finally disconnects.   The server doesn't need to know the total length 
at the outset, because it does not send a length.  When the application 
stops sending it data, it sends it on to the client and then disconnects 
to indicate that it's done.

This 3rd method was the original way of doing it in old HTTP specs, but 
is not recommended because the client cannot detect whether the 
connection was dropped due to a communications error, or whether it was 
dropped due to the end of the payload being reached.  Although this 
method is deprecated, it is still allowed for backward compatibilty in 
the HTTP specs.

Anyway, it would seem that the server that Nancy is talking to is using 
this 3rd method.  In that case, the "Connection broken" is completely 
normal and expected -- it's how the eerver indicates the end of the data.

But, if THAT is the case, then Nancy would not be getting a return value 
of -1 from http_url_post -- she'd be getting a 1 = success. She should 
not be getting an error code from httpapi in that case.

She said that she's getting success immediately followed by an error -- 
and I don't understand what she means by that.  If she's actually 
getting an error in the return code from httpapi, then something is 
still wrong, and we need to dig deeper.

But, if she's getting a successful response from http_url_post, but is 
merely seeing the "connection broken" message in the debug/trace log, 
then there's no problem at all... everything is working correctly.  In 
this case, she is incorrectly interpreting a normal message in the debug 
log as an error....  she shouldn't be using the debug log to determine 
if an error occurred. She should be using the return code from 
http_url_post.

But, I don't know which of these situations (or potentially, a 
completely different one that I haven't thought of) is occurring. This 
is why I need clarification on what she means by "successful response 
followed immediately by an error".


On 10/9/2013 4:15 PM, Mike Krebs wrote:
> One thing I don't see is that your program is getting a length of the data the server is sending. Someone with protocol knowledge would have to answer if that is acceptable but it would mean that recvdoc would have to continue until it received "EOF" (which it will do - right Scott?). But it would not know if it received the entire document because it wouldn't know how long the document is. I wonder if that has something to do with the connection broken problem?

-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------