[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: Ftpapi Digest, Vol 118, Issue 4
Hi Derick, I came across similar performance and storage issues on one of my recent projects (not with XML but large amount of data to be stored and accessed by different apps).
You can try few things:
Option-1. Use ZLIB123 utility to compress the XML documents and store in the table in VARCHAR or BLOB data type. ZLIB's data compression rate is off the chart, and compression and un-compression takes less than 20-30 milliseconds (it may vary depending upon the data size).
Option-2. Do the same but store the data in journal receiver using user-type code.
Here are the reason why I recommend option-2.
Storing data in BLOB or CLOB or VARCHAR, you may come across I/O gating and seizing especially if you are using re-use deleted record/space. Any time OS tries to reuse the deleted records for a varchar or blob column, it adjusts the space based on the new data which in return starts queueing other I/O requests until it is done with re-allocation. It will do the same when it needs to increase the table size. There is a remedy available for that too, but you will not be able to optimize the storage usage with that. There is an IBM article on gating and seizing.
Journal receiver with user-type entries can give you the flexibility of the storage/IO optimization along with easy access, and it will also save you the headaches of creating hourly folders. You can setup the journal size or number of entries limits for the receiver. It will create the chain automatically when it reaches the threshold. There are IBM API's available to access the receiver entries in chain mode or using entry sequence number.
Hopefully, this helps.
Athar
-----Original Message-----
From: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx [mailto:ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of ftpapi-request@xxxxxxxxxxxxxxxxxxxxxx
Sent: Wednesday, August 10, 2016 4:01 AM
To: ftpapi@xxxxxxxxxxxxxxxxxxxxxx
Subject: Ftpapi Digest, Vol 118, Issue 4
Send Ftpapi mailing list submissions to
ftpapi@xxxxxxxxxxxxxxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
http://scottklement.com/mailman/listinfo/ftpapi
or, via email, send a message with subject or body 'help' to
ftpapi-request@xxxxxxxxxxxxxxxxxxxxxx
You can reach the person managing the list at
ftpapi-owner@xxxxxxxxxxxxxxxxxxxxxx
When replying, please edit your Subject line so it is more specific than "Re: Contents of Ftpapi digest..."
Today's Topics:
1. HTTPAPI performance suggestions (Venter.Derick@xxxxxxxxxxxxxx)
----------------------------------------------------------------------
Message: 1
Date: Wed, 10 Aug 2016 10:00:38 +0200
From: Venter.Derick@xxxxxxxxxxxxxx
To: HTTPAPI and FTPAPI Projects <ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
Subject: HTTPAPI performance suggestions
Message-ID:
<OFFD285E73.BAE905AB-ON4225800B.00247A28-4225800B.002C00FD@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="us-ascii"
Hello everyone.
Hope someone can give us some suggestions to improve performance.
We are using HTTPAPI libraries to consume Corticon Rule Engine (Business decision making) web services on the IBM.
We are generating +- 185 000 XML's daily in our production environment.
This is causing lots of IO's on the IBM.
We generate and store the Request XML's in RPG on a folder in the IFS. The path and file name to the Request XML is then passed to a RPG function to call the web service and store the Response XML in the same IFS folder. We are generating folders per hour per day to keep the amount of XML files per folder to a minimum and to assist with archiving. Another RPG function then reads the Response XML file.
Is there maybe a faster way of doing this. The thing is we need history of the Request and Response XML files for at least 2 weeks. I am thinking of maybe storing the XML data in a table and not a .xml file.
Any ideas
Derick Venter
Applications Developer IV
Systems Integration
Tel: +27 (13) 247 2816 Fax: +27 (0) 86 573 2274
Cell: +27 (0) 83 458 6599
Email: derick.venter@xxxxxxxxxx
www.gijima.com
From: Scott Klement <sk@xxxxxxxxxxxxxxxx>
To: HTTPAPI and FTPAPI Projects <ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
Date: 2016/08/06 02:10 AM
Subject: Re: HTTPAPI -Header Parser
Sent by: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx
Hi Athar,
It's unusual for an application to need information from the headers, that's why you haven't seen much discussion in the archives. Normally header data is not needed for anything except httpapi, but in the rare occasion that it is, the application usually just needs a single header, so they just use http_header() to grab that one.
Can you explain why you'd like to iterate through all of them? That seems like a strange thing to do.
Your comment about searching for CRLF,CRLF is a good point. HTTPAPI should definitely NOT be looking at data after the HTTP headers.
Though, I don't think it does read past the CRLF,CRLF, because the calling routine only passes the headers to parse_resp_chain, it does not pass the entire response document. The variable names might be a bit misleading, however.
-SK
On 8/5/2016 1:01 PM, Athar Iqbal wrote:
> Hi,
>
>
> I am new to this forum, so please forgive me if I ask something that
> has already been asked. I did go through the archives and didn't
find
> any information.
>
>
> My question is on header parser function in HTTPAPI. Currently,
> header_parse() function reads all the headers and puts them in the
data
> structure which is available to the HEADERR4 module only. In order
to
> get the header information, we need to use http_header() function.
> Also, header_parse() goes through the complete response object to
pull
> headers from it. Wouldn't it be faster, if it looks for <CRLF><CRLF>
> and stop the search after that?
>
>
> I am developing a generic framework to consume web services offered
by
> different businesses such MelissaData or D&B. Developers on my team
may
> or may not know all the headers coming back in the response object.
I
> want to iterate though the all the header and put them in the list
and
> return it back to the calling procedure.
>
>
> Challenge I am coming across is that there is no iterator that can
get
> me all headers without specifying name. Also, could someone please
> explain, what is best use of USERDATA pointer in http_xproc()
>
>
> If someone has come across something like this, could you please
share.
> If not, I would like to contribute to HTTPAPI by enhancing existing
> code and send it back to the team to see if it is acceptable to
include
> in package.
>
>
> Again, I apologize in advance, if this has been already discussed
and
> if you can point me to some resource.
>
>
> Athar
>
>
> DISCLAIMER:
> This e-mail is intended for the use of the addressee(s) only and may
> contain privileged, confidential, or proprietary information that is
> exempt from disclosure under law. If you are not the intended
> recipient, please do not read, copy, use or disclose the contents of
> this
> communication to others. Please notify the sender that you have
> received this e-mail in error by replying to the e-mail. Please then
> delete the e-mail and destroy any copies of it. Thank you.
>
>
>
> -----------------------------------------------------------------------
> This is the FTPAPI mailing list. To unsubscribe, please go to:
> http://www.scottklement.com/mailman/listinfo/ftpapi
> -----------------------------------------------------------------------
-----------------------------------------------------------------------
This is the FTPAPI mailing list. To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------
-------------------------------------------------------------------------------------
This e-mail is subject to the Columbus Stainless [Pty] Ltd Email Legal Notices available at: http://www.columbus.co.za/EmailLegalNotice.htm.
-------------------------------------------------------------------------------------
This e-mail message has been scanned for Viruses and Content and cleared by MailMarshal
-------------------------------------------------------------------------------------
-------------- next part --------------
Hello everyone.
Hope someone can give us some suggestions to improve performance.
We are using HTTPAPI libraries to consume Corticon Rule Engine
(Business decision making) web services on the IBM.
We are generating +- 185 000 XML's daily in our production environment.
This is causing lots of IO's on the IBM.
We generate and store the Request XML's in RPG on a folder in the IFS.
The path and file name to the Request XML is then passed to a RPG
function to call the web service and store the Response XML in the same
IFS folder. We are generating folders per hour per day to keep the
amount of XML files per folder to a minimum and to assist with
archiving. Another RPG function then reads the Response XML file.
Is there maybe a faster way of doing this. The thing is we need history
of the Request and Response XML files for at least 2 weeks. I am
thinking of maybe storing the XML data in a table and not a .xml file.
Any ideas
Derick Venter
Applications Developer IV
[cid:_4_0B8667A40B8855E4002C00FC4225800B]
Systems Integration
Tel: +27 (13) 247 2816 Fax: +27 (0) 86 573 2274
Cell: +27 (0) 83 458 6599
Email: [1]derick.venter@xxxxxxxxxx
[2]www.gijima.com
From: Scott Klement <sk@xxxxxxxxxxxxxxxx>
To: HTTPAPI and FTPAPI Projects <ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
Date: 2016/08/06 02:10 AM
Subject: Re: HTTPAPI -Header Parser
Sent by: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx
__________________________________________________________________
Hi Athar,
It's unusual for an application to need information from the headers,
that's why you haven't seen much discussion in the archives. Normally
header data is not needed for anything except httpapi, but in the rare
occasion that it is, the application usually just needs a single
header,
so they just use http_header() to grab that one.
Can you explain why you'd like to iterate through all of them? That
seems like a strange thing to do.
Your comment about searching for CRLF,CRLF is a good point. HTTPAPI
should definitely NOT be looking at data after the HTTP headers.
Though, I don't think it does read past the CRLF,CRLF, because the
calling routine only passes the headers to parse_resp_chain, it does
not
pass the entire response document. The variable names might be a bit
misleading, however.
-SK
On 8/5/2016 1:01 PM, Athar Iqbal wrote:
> Hi,
>
>
> I am new to this forum, so please forgive me if I ask something
that
> has already been asked. I did go through the archives and didn't
find
> any information.
>
>
> My question is on header parser function in HTTPAPI. Currently,
> header_parse() function reads all the headers and puts them in
the data
> structure which is available to the HEADERR4 module only. In
order to
> get the header information, we need to use http_header()
function.
> Also, header_parse() goes through the complete response object to
pull
> headers from it. Wouldn't it be faster, if it looks for
<CRLF><CRLF>
> and stop the search after that?
>
>
> I am developing a generic framework to consume web services
offered by
> different businesses such MelissaData or D&B. Developers on my
team may
> or may not know all the headers coming back in the response
object. I
> want to iterate though the all the header and put them in the
list and
> return it back to the calling procedure.
>
>
> Challenge I am coming across is that there is no iterator that
can get
> me all headers without specifying name. Also, could someone
please
> explain, what is best use of USERDATA pointer in http_xproc()
>
>
> If someone has come across something like this, could you please
share.
> If not, I would like to contribute to HTTPAPI by enhancing
existing
> code and send it back to the team to see if it is acceptable to
include
> in package.
>
>
> Again, I apologize in advance, if this has been already discussed
and
> if you can point me to some resource.
>
>
> Athar
>
>
> DISCLAIMER:
> This e-mail is intended for the use of the addressee(s) only and
may
> contain privileged, confidential, or proprietary information that
is
> exempt from disclosure under law. If you are not the intended
> recipient, please do not read, copy, use or disclose the contents
of
> this
> communication to others. Please notify the sender that you have
> received this e-mail in error by replying to the e-mail. Please
then
> delete the e-mail and destroy any copies of it. Thank you.
>
>
>
>
-----------------------------------------------------------------------
> This is the FTPAPI mailing list. To unsubscribe, please go to:
> [3]http://www.scottklement.com/mailman/listinfo/ftpapi
>
-----------------------------------------------------------------------
-----------------------------------------------------------------------
This is the FTPAPI mailing list. To unsubscribe, please go to:
[4]http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------
__________________________________________________________________
This e-mail is subject to the Columbus Stainless [Pty] Ltd Email Legal
Notices available at:
[5]http://www.columbus.co.za/EmailLegalNotice.htm.
__________________________________________________________________
This e-mail message has been scanned for Viruses and Content and
cleared by MailMarshal
__________________________________________________________________
References
1. mailto:derick.venter@xxxxxxxxxx
2. http://www.gijima.com/
3. http://www.scottklement.com/mailman/listinfo/ftpapi
4. http://www.scottklement.com/mailman/listinfo/ftpapi
5. http://www.columbus.co.za/EmailLegalNotice.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 16588 bytes
Desc: not available
URL: <http://scottklement.com/pipermail/ftpapi/attachments/20160810/ccef0ac1/attachment.png>
------------------------------
-----------------------------------------------------------------------
This is the FTPAPI mailing list digest. To unsubscribe, go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------
End of Ftpapi Digest, Vol 118, Issue 4
**************************************
DISCLAIMER:
This e-mail is intended for the use of the addressee(s) only and may contain privileged, confidential, or proprietary information that is
exempt from disclosure under law. If you are not the intended recipient, please do not read, copy, use or disclose the contents of this
communication to others. Please notify the sender that you have received this e-mail in error by replying to the e-mail. Please then
delete the e-mail and destroy any copies of it. Thank you.
-----------------------------------------------------------------------
This is the FTPAPI mailing list. To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------