Saturday, October 17, 2009

Mojibake

I finally found out why some Japanese pages have corrupted encoding when displayed in NubiNews. This happened very often when I test on slow 3G network, but happened rather rarely on WiFi.

The reason is this code:

encoding = "EUC-JP";
ins = httpUrlconnection.getInputStream();
BufferedReader in = new BufferedReader(new InputStreamReader(ins,
encoding), 1024);
char buff[] = new buff[1234];
in.read(buff);

The problem is: when the data is coming slowly, ins may have only part of a EUC-JP character (which could be 1, 2 or 3 bytes). When the EUC-JP converter sees these partial characters, instead of waiting for the remaining bytes, it just outputs an incorrect unicode character, which would lead to corruption of the subsequent characters as well.

My work around is to save the page fully in a file first. But anyway, I do believe this is a bug in the platform.

2 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Hi,

    How can I get in touch with you?

    My name is Josh Schiffman, I run business development for Xtify – a mobile location technology company based in New York City.

    Xtify provides developers with technology to leverage the location of their users to create new feature capabilities and new monetization opportunities for their applications.

    What you can do with Xtify –
    • Provide location-relevant information, advertising and promotional messages
    • Incorporate new social, contextual and location-relevant aspects into your applications

    We provide -
    • Complete, free SDK
    • Full access to the Xtify Location Management Toolset including the Promotions Management Engine, Console, API’s and Analytics

    Why we provide this -
    • We want to prove the value of persistent location data to developers, marketers, (and our investors)

    Can we chat for a few minutes about the possibility of you incorporating Xtify?

    Thank you!

    Josh Schiffman

    Joshua Schiffman
    VP, Business Development
    Xtify, Inc.
    520 Broadway, 10th floor
    New York, NY 10012
    +1.646.395.1270 office
    +1.917.562.4271 mobile
    jschiffman@xtify.com

    ReplyDelete