Mojibake

I finally found out why some Japanese pages have corrupted encoding when displayed in NubiNews. This happened very often when I test on slow 3G network, but happened rather rarely on WiFi.

The reason is this code:

encoding = "EUC-JP";
ins = httpUrlconnection.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(ins,
encoding), 1024);
char buff[] = new buff[1234];
in.read(buff);

The problem is: when the data is coming slowly, ins may have only part of a EUC-JP character (which could be 1, 2 or 3 bytes). When the EUC-JP converter sees these partial characters, instead of waiting for the remaining bytes, it just outputs an incorrect unicode character, which would lead to corruption of the subsequent characters as well.

My work around is to save the page fully in a file first. But anyway, I do believe this is a bug in the platform.