[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Ftpapi] Ideas for New HTTPAPI functions



Thomas,

Lots of good ideas, here. I also like the original idea -- at first, it sounded complicated, but the more that I think about it, the more I like it.

I will definitely consider these ideas, thanks for the feedback.

Everyone else: Please look over my ideas and the ones that Thomas sent, and give me your thoughts. Which sounds the best to you?

-SK


On 2/26/2017 11:52 AM, Thomas Raddatz wrote:
Hi Scott,

Now I understand your intention better. Here is another idea to consider.

What about stripping the procedure interface down to the following
parameters and using a type prefix similar to the URL protocol like this:

http_req(Method: Url: ReceivedData: [SendData);

Possible type identifiers:

string:// - The data that follows the type identifier is sent
             to the server.

file://   - The value following the type identifier specifies
             a file name. The data of the file identified by
             that file name is send to the server.

The default type identifier is 'string://'.

ReceivedData

Data that is received from the server. By default the received data is
stored as a string value in the "ReceivedData" field. The 'file://' type
identifier can be used to save the data to the specified file.

Examples:

http_req('GET': 'http://example.com/test.txt':
                 'file:///home/joe/receivedData.txt');

http_req('GET': 'http://example.com/test.txt':
                  receiveBuffer);

"receiveBuffer" could be empty, to use the 'string://' default value or
could specify 'string://'.

SendData

Data that is send as POST data to the server. By default the content of
field "SendData" is sent to the server. The 'file'//' type identifier
can be used to read the data from the specified file.

Examples:

http_req('POST': 'http://example.com/test.txt':
                  receiveBuffer:
                  sendBuffer);

http_req('POST': 'http://example.com/test.txt':
                  receiveBuffer:
                  'My POST data');

http_req('POST': 'http://example.com/test.txt':
                  'file:///home/joe/receivedData.txt':
                  'string://My POST data');

http_req('POST': 'http://example.com/test.txt':
                  'file:///home/joe/receivedData.txt':
                  'file:///home/joe/postData.txt');

Just another idea to make it easy and to stay flexible.

Regards,

Thomas.

Am 26.02.2017 um 09:08 schrieb Scott Klement:
Thomas,

I have not designed new "raw" routines because I think the existing
http_persist_req() routine provides that functionality already.
Passing a fd doesn't make sense, since I do not have a callback option!
HTTPAPI internally manages the file when writing so there's no need for
an fd to be passed along.

Your idea of providing http_StmfWriter/http_StringWriter, etc would make
sense for the "raw" routines, but I am looking for really simple
solutions, and this seems to add a level of complexity.  Someone who
wants this level of complexity can achieve it by using the "raw" type of
routines...

-SK


On 2/25/17 8:57 AM, Thomas Raddatz wrote:
Hi Scott,

I agree with you and I like your idea of changing/enhancing HTTPAPI.

When I understood you correctly, then the procedure interface shall be
changed to that:

http_req(Method: Url: [RecvStmf: [RecvString: [SendStmf: [SendString);

But what about the "file/socket descriptor" option that we now have with
the "raw/raw2" methods?

Following your idea that would lead to the following method signature:

http_req(Method: Url:
     [RecvStmf: [RecvString: [RecvFD: [SendStmf: [SendString: [SendFD);

Even without the additional FD parameters the user has to know that he
must not use all parameters but just a subset. He has to know that he
must specify "RecvStmf" exclusive or "RecvString" and that there are two
more optional parameters which are used exclusive or, too.

Also not very intuitive from my point of view.

Specific methods names are also not an option, because that would led to
a whole bunch of methods like this:

http_reqString()       - Receive string
http_reqStringString() - Receive string, send string
http_reqStringStmF()   - Receive string, send stream file
http_reqStmf()         - Receive stream file
http_reqStmfString()   - Receive stream file, send string
http_reqStmfStmF()     - Receive stream file, send stream file

Now imagine what happened when we would add the "file/socket descriptor"
parameters. Most ugly approach.

But what about the following:

http_req(Method: Url: Writer: Reader);

"Writer" could be:

StringWriter        - Stores the data in a string
StmfWriter          - Stores the data in a stream file
DescriptorWriter    - Writes the data to a given file/socket descriptor

"Reader" could be:

StringReader        - Reads the data from a string
StmfReader          - Reads the data from a stream file
DescriptorReader    - Reads the data from a given file/socket descriptor

HTTPAPI could expose the readers and writers as a large strings (data
structures) with a type field at the first position. So the procedure
interface of http_req() could be similar to this:

   d http_req...
   d                 pr
   d  i_method                     10a   varying const
   d  i_url                      1024a   varying const
   d  i_genericReader...
   d                             5000a   options(*varsize)
   d  i_genericWriter...
   d                             5000a   options(*varsize: *nopass)

A user could call http_req() like this:

http_req('GET': 'http://...': http_StringWriter(responseData));

For POST data it could look like this:

http_req('POST': 'http://...':
           http_StmfWriter(filename: ccsid: replace):
           http_StringReader('Hello World, my post data'));

And for raw data it could go like this:

http_req('POST': 'http://...':
    http_RawWriter(%paddr('myOpen'): %paddr('write'): %paddr('close')):
    http_RawReader(%paddr('myOpen'): %paddr('read'): %paddr('close')));

Perhaps it might be useful to add a optional "userData" parameter to
http_RawWriter() and http_RawReader().

http_req() could first check the "type" field of the data structure
created by e.g. http_StmfWriter() and then set up the open(), write()
and close() procedures accordingly:

   d genRdWr         ds                  qualified based(pGenRdWr)
   d  type                         10a

    pGenRdWr = %addr(i_genericReader);

    select;
    when (genRdWr.type = '*STRING');
      reader.pOpen = %paddr('StringReader_open');
      reader.pRead = %paddr('StringReader_read');
      reader.pClose = %paddr('StringReader_close');
    when (genRdWr.type = '*STMF');
      reader.pOpen = %paddr('StmfReader_open');
      reader.pRead = %paddr('StmfReader_read');
      reader.pClose = %paddr('StmfReader_close');
    when (genRdWr.type = '*FD');
      reader.pOpen = %paddr('GenericRawReader_open');
      reader.pRead = %paddr('GenericRawReader_read');
      reader.pClose = %paddr('GenericRawReader_close');
    when (genericSndRcv.type = '???');
      ...
    other;
      // send escape message
    endsl;

StringReader:

Most likely open(open() and close() do nothing and read() returns the
whole string at once. But of course it could also initialize an internal
"offset" field in open() and then return the string chunk by chunk in
read().

StmfReader:

I assume that this reader could just pass the open, read and close
requests to the Unix APIs open(), read() and close().

GenericRawReader:

Most likely this reader has to use a user defined open procedures and
could, in a lot of cases, use the Unix APIs read() and close() if the
user did not specify these procedures at http_RawReader().

The writer procedures would work similar to the reader procedures that I
just explained.

Regards,

Thomas.

Am 24.02.2017 um 23:54 schrieb Scott Klement:
Everyone,

For a long time, I've been wanting to find a better way to interface
with the HTTP protocol that HTTPAPI provides... currently, most people
are using these routines:

http_url_get(), http_url_get_raw(), http_url_get_xml()

http_url_post(), http_url_post_raw(), http_url_post_xml()

While these provide access to the GET and POST methods, I see several
big problems with them:

1) There are too many parameters for things like timeout, user agent,
soap action, etc.  To me, this makes the code awkward.

2) There is no really easy way to simply get the result as a string.  My
original design in 2001 was that a callback approach would be very
versatile -- and it is -- because you can code any logic you want for
where to store the data when it comes in.  But in perhaps 95% of the
cases, you want it in a string or in a stream file.  So having to code
the logic to write to a string every time seems cumbersome.

3) Only supporting GET/POST is limiting, especially with REST.

4) Automatically calling an XML parser is kinda neat, but not really
required anymore, since RPG now has one built-in.  Plus, XML is fading
in popularity in favor of JSON.  People get confused when they see XML
routines but no routines for other formats like JSON, MIME or
URL-encoded forms.  They think HTTPAPI is limited, when in fact, it was
designed to be as versatile as possible.

So I was thinking of a new interface that, really, is one core routine:

http_req() = make an HTTP request.


       Parameters would be:

method = HTTP method to use (GET, POST, PUT, DELETE, etc)
URL = the URL, of course


         Where to put the response... two parmeters for that:

RecvStmf = If you want the response put into a stream file (IFS) then
you pass this, otherwise use *OMIT
RecvString = If you want the response put into a string, you pass this,
Otherwise *OMIT


         Where to get request body from.. only used for POST/PUT methods,
         otherwise you can leave them off

SendStmf = stream file to send request body from, or *omit to send from
a string
SendString = string to send request body from, or *omit to send from
a stmf

For body the body and the response, you can either use one or the other,
not both.  This might lead to confusion?  But, it surely would be
simpler than what we have now.  It'd look like this:

req = '<something>some request to send</something>';
http_setOption('content-type': 'text/xml');
rc = http_req('POST': 'http://whatever': *omit: resp: *omit: req);

// resp now contains the response document (XML, JSON, HTML, whatever it
is.)

The http_setOption() routine allows you to set any of the options that
might've been extra parameters before.

We could also have some "simpler" wrappers around this routine for
working with strings and/or stream files.  I'm thinking that the
wrappers would (a) send an exception message for errors, so rc isn't
needed, (b) only accept one parameter for data to send, and one to
receive.  There'd be two wrappers, one for using strings, one for using
files.

Resp = http_string('GET': 'http://whatever');

Resp = http_string('POST': 'https://foo.bar': dataToSend);

http_stmf('GET': 'http://whatever': '/home/scott/myresponse.xml');

http_stmf('PUT': 'http://foo': '/tmp/response.dat': '/tmp/request.dat');


All the old routines would be retained, of course, so existing programs
would work without problems.  Instead of using the automatic XML
parsing, you'd just receive your data to a string, and then parse the
string afterwards.  (using RPG's built-ins, or HTTPAPIs
http_parse_xml_String routine, or SQL's XML stuff) Likewise for JSON you
could use YAJL, SQL, or your favorite 3rd party tool.  It's a string,
right?  You can feed it into anything.

Let me know your thoughts!


_______________________________________________
Ftpapi mailing list
Ftpapi@xxxxxxxxxxxxxxxxxxxxxx
http://scottklement.com/mailman/listinfo/ftpapi

_______________________________________________
Ftpapi mailing list
Ftpapi@xxxxxxxxxxxxxxxxxxxxxx
http://scottklement.com/mailman/listinfo/ftpapi

_______________________________________________
Ftpapi mailing list
Ftpapi@xxxxxxxxxxxxxxxxxxxxxx
http://scottklement.com/mailman/listinfo/ftpapi

_______________________________________________
Ftpapi mailing list
Ftpapi@xxxxxxxxxxxxxxxxxxxxxx
http://scottklement.com/mailman/listinfo/ftpapi


_______________________________________________
Ftpapi mailing list
Ftpapi@xxxxxxxxxxxxxxxxxxxxxx
http://scottklement.com/mailman/listinfo/ftpapi