Re: Memory leak in EXPAT?

[Nasser Shukayr <nshukayr@xxxxxxxxxxxxxxxxx>]

Thank you for the rapid reply!

The heap space is exceeded when EXPAT tries to allocate a new buffer.

MCH6903:
    To module . . . . . . . . . :   QC2ALLOC                                                   
    To procedure  . . . . . . . :   do_malloc_default__FUL                                     
    Statement . . . . . . . . . :   3                                                          
    Message . . . . :   The heap space has reached its maximum allowable size.                 

I believe that EXPAT allocates resources when it starts to parse, then frees up those resources after it finishes.

Here are two possible theories for running out of heap space:

1. Theory:  every call to EXPAT depletes (does not free up) a few bytes.  After many thousands of calls, all available heap space is exhausted.

or

2. Background:  The XML document (from a web service) could occasionally contain mal-formed XML.  The document is approx. 65K bytes and has info on about 90 different commodity price quotes.  On rare occasions in debug, I observed documents which end abruptly.  Some closing tags, and sometimes half of a data element, are missing.  Theory:  When an invalid document gets passed to EXPAT (and it errors out), it does not always free up all the allocated memory.  It happens every now and then.  After 13 hours, it happens often enough to exhaust all available heap space.

On our production system, the program runs about 12 and a half hours (give or take 45 minutes) until it croaks.  On Development, it runs about an hour longer, i.e. 13 and a half hours, plus or minus 45 minutes.  Measurements were made during an 18-day period.  I believe that development is less active than production.


Nasser Shukayr
    I.T. Application Development Team Lead
   http://www.heartlandcoop.com
   2829 Westown Parkway, Suite 350
   West Des Moines, Iowa 50266
NShukayr@xxxxxxxxxxxxxxxxx
Office: 515.309.3857


-----Original Message-----

Message: 1
Date: Thu, 7 Apr 2016 20:01:22 +0000
From: Nasser Shukayr <nshukayr@xxxxxxxxxxxxxxxxx>
To: "ftpapi@xxxxxxxxxxxxxxxxxxxxxx" <ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Memory leak in EXPAT?
	
Setup:  Program queries a web service constantly, receiving XML response.  Response is parsed with EXPAT.  Program runs most hours of the day.
Issue:  After 13 hours, the program croaks with MCH6903 (The heap space has reached its maximum allowable size).  (It's not always exactly 13 hours ; it varies between 11:45 and 15:45 ) Temporary solution:  We changed the program so that after about 8 hours, it exits and then re-starts as a brand-new copy.

Why this is not ideal:  We really need to monitor the web service3 continuously, even during the few seconds needed to exit and restart the program.

Has anyone else had a problem with running out of heap space after 13 hours of continuously using EXPAT?

Many thanks,


Nasser Shukayr
    I.T. Application Development Team Lead 
   http://www.heartlandcoop.com
   West Des Moines, Iowa 50266
NShukayr@xxxxxxxxxxxxxxxxx

Message: 3
Date: Thu, 7 Apr 2016 16:33:39 -0500
From: Scott Klement <sk@xxxxxxxxxxxxxxxx>
To: HTTPAPI and FTPAPI Projects <ftpapi@xxxxxxxxxxxxxxxxxxxxxx>
Subject: Re: Memory leak in EXPAT?

Hello Nasser,

This is the first time I can remember seeing this problem.

So to reproduce it, I should create an XML document, and parse it 
repeatedly for 13 hours?   Does it matter what is in the document? Do I 
have to make an HTTP request each time, or does just parsing the XML sufficient?

-SK

-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------