[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Corrupted contedt when using utf-8 with Hebrew values in XML

Boaz,
1.23 also re-calculated the pointers in exactly the same way. This is done because realloc() can potentially give you a different memory space (instead of just growing the existing one.) The only difference that was made in iconvdyn() is that it no longer uses such a large default buffer (double the data size). This was done because it was causing memory to extend far beyond the 16 mb limit of single level storage allocations, even for relatively small amounts of data, due to this crazy doubling -- this was causing a lot of problems for people, so I changed it to allocate smaller amounts at once.
Re-starting the translation from the beginning could be a real performance killer, especially if the data is many megabytes long. And it shouldn't be necessary to re-start it? Can you give me a test case where restarting the translation is necessary? 

-SK



On 9/6/2013 11:21 AM, Boaz mermelstein wrote:

    Thanks Scott

    First, I'm still running 1.23 on my production machine, while 1.24 on
    dev machine.

    Last week I switched to 1.24 on production, and had to go back to 1.23.


    I made some investigation, debug the code and found the place where the
    corruption occurred.

    It is in iconvdyn() procedure.

    Output buffer is not large enough to get the converted string.

    In 1.23, output buffer size was twice then input buffer length.

    In 1.24, new algorithm took place, but it is wrong.

    Yes you can add a chunk of 64K to output buffer each iteration, but do
    not play with pointers of input and output buffer.

    Since in conversion from single byte code page to multi bytes code
    page, you never know where conversion is stopped.

    So just increase output buffer and reconvert the whole thing from the
    beginning every time, until finished OK.


    Mike, yes , it is the same program runs twice. One with 1.23 version of
    HTTPAPIR4 *SRVPGM, the other with 1.24 version.


    Regards

    Boaz Mermelstein

    3/21 Hershko Avraham st. |  Rehovot   |  Israel

    Phone: +972 8 9458690  |  Mobile:+972 54 4441 891  |  Fax: +972 151 54
    4441 891

    http://www.smbt.co.il



-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------


-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubscribe, please go to:
http://www.scottklement.com/mailman/listinfo/ftpapi
-----------------------------------------------------------------------