[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: Memory leak in EXPAT?
Couple of things to try ?
//example
p = XML_ParserCreate(*omit);
if (p = *NULL);
callp close(fd);
die('Couldn''t allocate memory for parser');
endif;
know there is an IBM api for allocating and deallocating, just can't
remember the name
RPGLE had opcodes which uses %alloc and %dealloc
Program OPTIMIZE(*FULL)
Free the result set
[1]www.pilotpen.us
Chris Hayden - Sr. Systems Analyst
Office: (904) 645-9999 ext.1252
Mobile: 904-654-4089
Pilot Corporation of America
3855 Regent Blvd, Jacksonville, Florida 32224 United States
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager. This message contains confidential information and
is intended only for the individual named. If you are not the named
addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete this e-mail from your system. If you
are not the intended recipient you are notified that disclosing,
copying, distributing or taking any action in reliance on the contents
of this information is strictly prohibited.
-----Original Message-----
From: ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx
[[2]mailto:ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx] On Behalf Of Nasser
Shukayr
Sent: Tuesday, April 12, 2016 3:01 PM
To: ftpapi@xxxxxxxxxxxxxxxxxxxxxx
Subject: RE: Memory leak in EXPAT?
I would love to set up an environment so that the issue can be easily
reproduced on a different system. Alas, the application is highly
proprietary and I just don't know of any way to reproduce it
externally.
Remaining realistic, here's a chronology:
1. An application program continuously reads a web service. Each
XML response is about 56K bytes. We believe that the web service
occasionally sends mal-formed XML. (Sometimes the document stops
abruptly, and any open tags are never closed.) We believe that each
time EXPAT gets invalid XML, a small amount of memory is consumed (not
freed up). After 11 to 13 hours, all heap space is exhausted and the
program crashes.
2. Using Advanced Job Scheduler, we stop and then re-start the job
every 8 hours. Now the program runs without crashing (however there
are three times per day, approx 15 minutes each time, when we don't
monitor the service even though we'd like to monitor it 24/7).
3. We changed the application to use IBM's XML Parser (XML-INTO).
We still get occasional parsing errors when the XML document is
incomplete.
4. We observed that the XML errors seem more frequent when we
receive multiple responses during the same clock-second. We changed
the program so that it waits at least one full second between
requests. This seemed to reduce the number of errors, however the
parsing errors still occur. We did NOT run this program until it
croaked, although that kind of experiment could be insightful.
5. We changed the program to make sure that the very last XML tag
is present. If that last tag is missing, then we ignore the document
because we know that the XML is malformed and it will not parse
properly. Yet we still get occasional parsing errors. The XML appears
to contain occasional control characters such as LF (Line Feed) and
occasional random < and > characters.
6. We can filter out the control characters (CR, LF, etc.) and see
what happens next.
7. According to the vendor, we're their only customer having
issues with the web service. Except for the fact that the program
crashes after 11 to 13 hours, we would be totally unaware of any
issues.
8. Our infrastructure team is checking to determine if there's any
kind of "noise" in the line which might corrupt XML responses from a
website. The program reliably croaks after running continuously for 11
to 13 hours . (11 hours on Production, 13 hours on Development) . We
believe that the program croaks after a finite number of bad XML
responses, and that this number is achieved sooner on Production,
because the Production CPU is faster than the Development CPU.
9. A completely separate application monitors a different web
service 24/7, and that application works perfectly. However, the
flawless web service is in JSON while the croaky one is in XML.
10. The troublesome service offers JSON as an option. Our next
step will be to change the program to request and receive the info in
JSON, to see if this solves the issues.
Bottom line: The evidence suggests that invalid XML can causes EXPAT
to consume (not free up) a small amount of memory. After a large
number of requests with invalid XML, EXPAT runs out of heap space.
The best solution is probably to correct the incoming data stream, so
that the XML is always proper. Still, it's certainly interesting that
a sufficient number of mal-formed XML messages seems to eventually
crash EXPAT.
Nasser Shukayr
I.T. Application Development Team Lead Heartland Co-op
-----Original Message-----
Date: Sat, 9 Apr 2016 12:14:53 -0500
From: Scott Klement <[3]sk@xxxxxxxxxxxxxxxx>
To: HTTPAPI and FTPAPI Projects <[4]ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: Memory leak in EXPAT?
Nasser,
There isn't just one place where it allocates memory, there are many
places. The little bit of information you provided points to the IBM
QC2ALLOC routine, which is part of the ILE environment used by all ILE
programs on the system, so that isn't specific enough.
I really need to know how to reproduce the problem.
-SK
On 4/8/2016 9:04 AM, Nasser Shukayr wrote:
> Thank you for the rapid reply!
>
> The heap space is exceeded when EXPAT tries to allocate a new buffer.
>
> MCH6903:
> To module . . . . . . . . . : QC2ALLOC
> To procedure . . . . . . . : do_malloc_default__FUL
> Statement . . . . . . . . . : 3
> Message . . . . : The heap space has reached its maximum
allowable size.
>
> I believe that EXPAT allocates resources when it starts to parse,
then frees up those resources after it finishes.
>
> Here are two possible theories for running out of heap space:
>
> 1. Theory: every call to EXPAT depletes (does not free up) a few
bytes. After many thousands of calls, all available heap space is
exhausted.
>
> or
>
> 2. Background: The XML document (from a web service) could
occasionally contain mal-formed XML. The document is approx. 65K bytes
and has info on about 90 different commodity price quotes. On rare
occasions in debug, I observed documents which end abruptly. Some
closing tags, and sometimes half of a data element, are missing.
Theory: When an invalid document gets passed to EXPAT (and it errors
out), it does not always free up all the allocated memory. It happens
every now and then. After 13 hours, it happens often enough to exhaust
all available heap space.
>
> On our production system, the program runs about 12 and a half hours
(give or take 45 minutes) until it croaks. On Development, it runs
about an hour longer, i.e. 13 and a half hours, plus or minus 45
minutes. Measurements were made during an 18-day period. I believe
that development is less active than production.
>
>
> Nasser Shukayr
> I.T. Application Development Team Lead
> [5]http://www.heartlandcoop.com
> 2829 Westown Parkway, Suite 350
> West Des Moines, Iowa 50266
> [6]NShukayr@xxxxxxxxxxxxxxxxx
> Office: 515.309.3857
>
>
> -----Original Message-----
>
> Message: 1
> Date: Thu, 7 Apr 2016 20:01:22 +0000
> From: Nasser Shukayr <[7]nshukayr@xxxxxxxxxxxxxxxxx>
> To: "[8]ftpapi@xxxxxxxxxxxxxxxxxxxxxx"
<[9]ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
> Subject: Memory leak in EXPAT?
>
> Setup: Program queries a web service constantly, receiving XML
response. Response is parsed with EXPAT. Program runs most hours of
the day.
> Issue: After 13 hours, the program croaks with MCH6903 (The heap
space has reached its maximum allowable size). (It's not always
exactly 13 hours ; it varies between 11:45 and 15:45 ) Temporary
solution: We changed the program so that after about 8 hours, it exits
and then re-starts as a brand-new copy.
>
> Why this is not ideal: We really need to monitor the web service3
continuously, even during the few seconds needed to exit and restart
the program.
>
> Has anyone else had a problem with running out of heap space after 13
hours of continuously using EXPAT?
>
> Many thanks,
>
>
> Nasser Shukayr
> I.T. Application Development Team Lead
> [10]http://www.heartlandcoop.com
> West Des Moines, Iowa 50266
> [11]NShukayr@xxxxxxxxxxxxxxxxx
>
> Message: 3
> Date: Thu, 7 Apr 2016 16:33:39 -0500
> From: Scott Klement <[12]sk@xxxxxxxxxxxxxxxx>
> To: HTTPAPI and FTPAPI Projects <[13]ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
> Subject: Re: Memory leak in EXPAT?
>
> Hello Nasser,
>
> This is the first time I can remember seeing this problem.
>
> So to reproduce it, I should create an XML document, and parse it
> repeatedly for 13 hours? Does it matter what is in the document? Do
I
> have to make an HTTP request each time, or does just parsing the XML
sufficient?
>
> -SK
-----------------------------------------------------------------------
This is the FTPAPI mailing list. To unsubscribe, please go to:
[14]http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------
References
1. http://www.pilotpen.us/
2. mailto:ftpapi-bounces@xxxxxxxxxxxxxxxxxxxxxx
3. mailto:sk@xxxxxxxxxxxxxxxx
4. mailto:ftpapi@xxxxxxxxxxxxxxxxxxxxxx
5. http://www.heartlandcoop.com/
6. mailto:NShukayr@xxxxxxxxxxxxxxxxx
7. mailto:nshukayr@xxxxxxxxxxxxxxxxx
8. mailto:ftpapi@xxxxxxxxxxxxxxxxxxxxxx
9. mailto:ftpapi@xxxxxxxxxxxxxxxxxxxxxx
10. http://www.heartlandcoop.com/
11. mailto:NShukayr@xxxxxxxxxxxxxxxxx
12. mailto:sk@xxxxxxxxxxxxxxxx
13. mailto:ftpapi@xxxxxxxxxxxxxxxxxxxxxx
14. http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------
This is the FTPAPI mailing list. To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------