From: "Elliotte Rusty Harold" <>
> I can't believe this was intentional. Frankly, I think that requiring
> normalization at all is too onerous.
I have been using IBM ICU4J for normalizing. It is pretty good, but
the distribution is now so bloated that you really have to select the
packages you need. I believe they are working on this aspect of it.
Here is a speed-up I am using for normalization. It minimizes the
impact of testing for normalization for characters which don't need
it.
If your Unicode characters are (combined) European or Chinese,
you don't need to check normalization. Here is an example Java routine
that I use (I believe the numbers are correct; corrections from Unicode
eggheads welcomed)
// inside loop reading characters
char c = characterArray[i];
if ((c >= 0x300) && ((c < 0x3100) || (c >= 0xF900))) {
if (detectNotNormalized(charArray, i)) barf();
} // else we don't need to worry
If you roll-your-own newline-discipline code, and add this there, it only
adds a single test and branch per character. Since the character will
presumably still be in a register or data cache, and the testing code will
be fitting into the instruction cache because of the loop, this may cause
almost no decrease in throughput, for suitable data. (Worthwhile to
verify the effect, of course.)
Similarly, when inserting an entity value, you can perform the same
test on the first character of that value and the first character
after the entity value. If these are in the safe range, there is no
normalization impact for that entity.
So I don't think that simple normalization and simple normalization-
checking is difficult or will cause performance penalties for
most Western and Chinese data. The performance penalty will be
borne by the documents that actually require the checks.
However, checking for include normalization is definitely more
tricky code wise, and I am not sure it provides much value.
But, again, it does not need to cause a performance hit on innocent
bystanders to have it. If there is a suitable loop to read characters,
normalization-checking should not be particularly onerous, AFAICS.
Cheers
Rick Jelliffe
