Re: FTPAPI & text file transfers

[Christian <chrisv5@xxxxxx>]

Sender: Christian <chrisv5@xxxxxx>

Scott Klement wrote:

> > As others have found, if one uses ftp_binaryMode(sessionID: *ON) then 
> > the nice ASCII (or Latin-1, etc.) files in the IFS get garbled on the 
> > FTP server (PUT), or one ends up with rubbish in the IFS (GET).

> The word "BINARY" means "my file does not contain text characters, and 
> therefore it should not be translated from one character set to another."

I am very sorry, I have mixed up the parameters of ftp_BinaryMode in the 
first two examples. Very sorry. I know the FTP protocol very well, I 
guess I gave a bad impression, though. Above, I meant to say *OFF, i.e. 
put the server into ASCII mode. This will convert any file from/to 
EBCDIC, which is nice if we are working with source members, but 
generally bad if it's IFS files, possibly meant for/coming from an EDI 
converter, a HTML server, you name it...

> If you store it in your IFS in the format you want it to be in on the 
> Windows or Unix server, then binary makes perfect sense.  You're telling 
> it not to do any translation.

But one gets the problem with CRLF/LF conversion. A very nasty thing!

> > It has been suggested in the past to use ftp_binaryMode(SessionID: 
> > *OFF) in that case.
>
> Absolutely correct.  If you want translation to occur, don't use binary 
> mode.

I meant to say *ON, sending an ASCII file as BINARY...

> The recommendation to use binary mode comes from the fact that when 
> people have ASCII files in their IFS, it's usually because they created 
> them in the exact format that they want on the server.
>
> Either they've written a program that writes it in the exact format they 
> want it to be in, or they've used CPYTOIMPF or CPYTOSTMF to put it in 
> the precise format they want it in.  In that case, you don't want FTP to 
> mess it up by doing conversions on it.

Yes, that would be a workaround, but a rather annoying one, if generally 
dealing with file in ASCII/Latin-1 code pages. This is why I suggested 
using ASCII file transfer with the same code pages. Just never forget to 
switch back for the NLIST/LIST command wrappers!!

> Should I change the ToEBCDICF an ToASCIIF routines to check if the code 
> pages are the same, and if so, don't do ny translation?

This seems to be the right thing to do.

> Another problem with our current translation scheme in FTPAPI is that it 
> uses the same buffer for both input and output.   That'll cause problems 
> if you translate from a single-byte character set to a multiple byte 
> character set.  For example, from 819 to UTF-16 would corrupt the data.
>
> To me, this is a much bigger problem than the line break convention!

Oops! This hurts. I was not aware if this, I have only browsed through 
part of the source so far.

> With UTF-8, again, you run the risk of corruption of data since FTPAPI's 
> translation routines use the same buffer for input and output. For most 
> basic characters, like letters and numbers, you wouldn't have a problem. 
> However, any time UTF-8 needed to use two bytes for a character, you'd 
> have problems.

Would most definitely bite me and anyone else in Europe. While it is 
rather unlikely to deal with UTF-16 (isn't it actually UCS-2 on the 
iSeries?), more and more XML/HTML files are in UTF-8 nowadays.

> This is why I recommend that you do the ASCII/EBCDIC/UNICODE translation 
> outside of FTPAPI, and then use BINARY mode for the transfer.  That way, 
> you don't have to use FTPAPI's translation routines.

I love to have common tasks in a central spot. FTPAPI is the perfect 
place, IMHO.

> Granted, what I'm doing is working around a bug.  But, until the problem 
> is fixed, it's a better solution.  I don't have time to re-write FTPAPI 
> right now.

I could tackle that problem, if you do not mind. I know iconv well 
enough and I am not too bad in writing code ;-) At the same time I could 
optimize the special case of identical codepages. Maybe not within the 
next few hours, but definitely within the next few weeks. I am pretty 
motivated as well, esp. as we are now at V5R3, which is supposed to 
support UTF-8.

Also, I'd be glad to contribute something to the community.

Regards,
Christian
-----------------------------------------------------------------------
This is the FTPAPI mailing list.  To unsubsribe from the list send mail
to majordomo@xxxxxxxxxxxxx with the body: unsubscribe ftpapi mymailaddr
-----------------------------------------------------------------------