Bug 14759

Created attachment 42120 [details] [review]
This patch uses the proposed module - Text::Unaccent::PurePerl

Patch complexity is 'Medium' because this change has many architectural connections

I've also had issues getting Text::Unaccent to install on RedHat-like distros.  That said, at the moment Text::Unaccent::PurePerl is not packaged for Debian or Ubuntu.

It is also (presumably) slower than Text::Unaccent, although given that unac_string() is used only when registering a new patron, I don't speed matters much here.

I'd like to suggest an alternative way to get the job done - use Unicode::Normalize, which is already a Koha dependency:

http://stackoverflow.com/a/17561928/880696

If we do that, we can drop the dependency on Text::Unaccent entirely.

I didn't remember, but it looks like I introduced this dependency. I am in favor of reducing troublesome dependencies - so I am totally fine with an alternative solution.

Unfortunately the initial bug 7411 has no clear problem description, so it's hard to tell now why we made the change in the first place.

Maybe Biblibre could check the linked Mantis entry?
http://mantis.biblibre.com/view.php?id=7744
Adding Sophie to this bug.

I wrote a little test program to compare the options:

___BEGIN___
#!/usr/bin/perl

use Modern::Perl;
use Text::Unaccent qw//;
use Text::Unaccent::PurePerl qw//;
use utf8;
use Unicode::Normalize;

binmode STDOUT, ':utf8';
my @str = (
    'été',
    'umlaüt',
    'עברית',
    'חוֹלָם',
    '北京市',
    'Άά Έέ Ήή Ίί Όό Ύύ Ώώ',
    'مُدَرِّسَة'
);

sub unaccent {
    my $str = NFKD(shift);
    $str =~ s/\p{NonspacingMark}//g;
    return $str;
}

foreach (@str) {
    if ($_ eq 'مُدَرِّسَة') {
        # special case to avoid locking my terminal session (!)
        print "Text::Unaccent           - $_ => *refusing to let Text::Unaccent do this*\n";
    } else {
        print "Text::Unaccent           - $_ => " . Text::Unaccent::unac_string('utf-8', $_) . "\n";
    }
    print "Text::Unaccent::PurePerl - $_ => " . Text::Unaccent::PurePerl::unac_string($_) . "\n";
    print "Strip NonspacingMark     - $_ => " . unaccent($_) . "\n";
}
___END___

Here's its output:

Text::Unaccent           - été => ete
Text::Unaccent::PurePerl - été => ete
Strip NonspacingMark     - été => ete
Text::Unaccent           - umlaüt => umlaut
Text::Unaccent::PurePerl - umlaüt => umlaut
Strip NonspacingMark     - umlaüt => umlaut
Text::Unaccent           - עברית => ×¢×ר×ת
Text::Unaccent::PurePerl - עברית => עברית
Strip NonspacingMark     - עברית => עברית
Text::Unaccent           - חוֹלָם => ××Ö¹×Ö¸×
Text::Unaccent::PurePerl - חוֹלָם => חוֹלָם
Strip NonspacingMark     - חוֹלָם => חולם
Text::Unaccent           - 北京市 => å京å¸
Text::Unaccent::PurePerl - 北京市 => 北京市
Strip NonspacingMark     - 北京市 => 北京市
Text::Unaccent           - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => Îα Îε Îη Îι Îο Î¥Ï
 ΩÏ
Text::Unaccent::PurePerl - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => Αα Εε Ηη Ιι Οο Υυ Ωω
Strip NonspacingMark     - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => Αα Εε Ηη Ιι Οο Υυ Ωω
Text::Unaccent           - مُدَرِّسَة => *refusing to let Text::Unaccent do this*
Text::Unaccent::PurePerl - مُدَرِّسَة => مُدَرِّسَة
Strip NonspacingMark     - مُدَرِّسَة => مدرسة

Some conclusions:

[1] Text::Unaccent mangles non-Latin characters outright; that's enough reason to get rid of it.
[2] Both Text::Unaccent::PurePerl and stripping NonspacingMark characters are better -- they strip accents from Latin scripts, and don't mangle non-Latin.  Removing NonspacingMark characters is more aggressive; I think we need input from Arabic, Hebrew, and Greek suers as to whether that is acceptable -- or, alternatively, if we need a system preference, or need to bite the bullet and package Text::Unaccent::PurePerl.

Seeing the bad results for non-latin scripts I am promoting this from enhancement to bug. Not sure about the Arabic - I think we need a native speaker/reader.

My knowledge of Arabic is pretty much non-existent, but I recall a librarian I know once wanting Zebra to remove hamza for search purposes... (https://en.wikipedia.org/wiki/Arabic_diacritics)

What's the purpose of Text::Unaccent currently? It's only used when adding members? Is it for generating a userid? Do we really need to remove accents for that?

(In reply to David Cook from comment #7)
> Is it for generating a userid?

Yes

> Do we really need to remove accents
> for that?

Per bug 7411, there was apparently an issue searching on usernames with diacritics, although in retrospect that may simply have been an issue with mismatched Unicode normalization forms -- impossible to tell now.

The current patcheset for bug 7679 also proposes to use Text::Unaccent, but I'm dubious about that one.

(In reply to Galen Charlton from comment #8)
> (In reply to David Cook from comment #7)
> > Do we really need to remove accents
> > for that?
> 
> Per bug 7411, there was apparently an issue searching on usernames with
> diacritics, although in retrospect that may simply have been an issue with
> mismatched Unicode normalization forms -- impossible to tell now.
> 
> The current patcheset for bug 7679 also proposes to use Text::Unaccent, but
> I'm dubious about that one.

It's surprising that Text::Unaccent doesn't appear to be working correctly, since it is using iconv for the heavy lifting, and iconv seems to be pretty good when it comes to character conversions.

I can't speak to Hebrew or Greek (while I thought I wasn't bad with the modern Greek alphabet, I didn't know they used accents...), Arabic is sure interesting.

So we have the following string:
مُدَرِّسَة

If we run the following:
echo "مُدَرِّسَة" | xxd -p

We get this hex:
d985d98fd8afd98ed8b1d990d991d8b3d98ed8a90a

If we look at the first couple bytes there using a UTF-8 table (http://www.utf8-chartable.de/unicode-utf8-table.pl)

d985 = م = ARABIC LETTER MEEM
d98f = ُُ = ARABIC DAMMA
Together, these are written like مُ 

However, if you add the letter "dal":
d8af = د = ARABIC LETTER DAL

You'll get something like the following:
مُد

We'd recognize that from the "English end/Arabic start" of the string: "مُدَرِّسَة"

I had forgotten that Hebrew only has consonants in its alphabet, and it appears Arabic is the same. So that "damma" indicates a vowel sound but isn't a letter per se. I'd say it's a diacritic and this would agree: https://en.wikipedia.org/wiki/Arabic_diacritics#.E1.B8.8Cammah

So the output for "Strip Nonspacing Mark" looks good in the very first case at least:

Strip NonspacingMark     - مُدَرِّسَة => مدرسة

Although I don't know if it makes sense semantically as I don't read Arabic. If I understand correctly, you can omit vowel sounds from written Arabic and rely purely on context for meaning? (https://en.wikipedia.org/wiki/Arabic_alphabet#Vowels)

At a glance, the Strip NonspacingMark looks OK for Greek too as those diacritics appear to be there purely for pronunciation like in languages written in the Roman alphabet. (https://en.wikipedia.org/wiki/Modern_Greek#Phonology_and_orthography)

So I think I've tracked down the C code behind Text::Unaccent:

https://github.com/gitpan/Text-Unaccent/blob/master/unac.c

The only reference I see to "damma" is in the U+FE70...U+FEFF code point range which appears to list isolated forms which is not what we're dealing with in these examples.

While I haven't reviewed the code extensively, it looks like the tables used for Text::Unaccent are lacking...

If you replace the following line in Galen's script:

use Text::Unaccent qw//;

with

use Text::Unaccent qw/unac_debug/;
unac_debug($Text::Unaccent::DEBUG_HIGH);

You'll get more details of how Text::Unaccent is working (or not working as it were).

Here's the output I get for the Arabic:

unac.c:13708: unac_data0[5] & unac_positions[0][6]: 0x0645 => untouched
unac.c:13708: unac_data0[15] & unac_positions[0][16]: 0x064f => untouched
unac.c:13708: unac_data34[15] & unac_positions[34][16]: 0x062f => untouched
unac.c:13708: unac_data0[14] & unac_positions[0][15]: 0x064e => untouched
unac.c:13708: unac_data34[17] & unac_positions[34][18]: 0x0631 => untouched
unac.c:13708: unac_data0[16] & unac_positions[0][17]: 0x0650 => untouched
unac.c:13708: unac_data0[17] & unac_positions[0][18]: 0x0651 => untouched
unac.c:13708: unac_data34[19] & unac_positions[34][20]: 0x0633 => untouched
unac.c:13708: unac_data0[14] & unac_positions[0][15]: 0x064e => untouched
unac.c:13708: unac_data34[9] & unac_positions[34][10]: 0x0629 => untouched
Text::Unaccent           - مُدَرِّسَة => Ù�Ù�دÙ�رÙ�Ù�سÙ�Ø©
Strip NonspacingMark     - مُدَرِّسَة => مدرسة

Here's the output I get for the Greek:
unac.c:13708: unac_data21[6] & unac_positions[21][7]: 0x0386 => 0x0391
unac.c:13708: unac_data22[12] & unac_positions[22][13]: 0x03ac => 0x03b1
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[8] & unac_positions[21][9]: 0x0388 => 0x0395
unac.c:13708: unac_data22[13] & unac_positions[22][14]: 0x03ad => 0x03b5
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[9] & unac_positions[21][10]: 0x0389 => 0x0397
unac.c:13708: unac_data22[14] & unac_positions[22][15]: 0x03ae => 0x03b7
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[10] & unac_positions[21][11]: 0x038a => 0x0399
unac.c:13708: unac_data22[15] & unac_positions[22][16]: 0x03af => 0x03b9
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[12] & unac_positions[21][13]: 0x038c => 0x039f
unac.c:13708: unac_data23[12] & unac_positions[23][13]: 0x03cc => 0x03bf
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[14] & unac_positions[21][15]: 0x038e => 0x03a5
unac.c:13708: unac_data23[13] & unac_positions[23][14]: 0x03cd => 0x03c5
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[15] & unac_positions[21][16]: 0x038f => 0x03a9
unac.c:13708: unac_data23[14] & unac_positions[23][15]: 0x03ce => 0x03c9
Text::Unaccent           - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => Î�α Î�ε Î�η Î�ι Î�ο Î¥Ï� ΩÏ�
Strip NonspacingMark     - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => Αα Εε Ηη Ιι Οο Υυ Ωω

Interestingly we can see a reference to the Greek cpaital letter alpha with the tonos diacritic: 

* 0386 GREEK CAPITAL LETTER ALPHA WITH TONOS
* 	0391 GREEK CAPITAL LETTER ALPHA

Indeed, in the output, we can see that 0x0386 was changed to 0x0391... although admittedly I don't know exactly how. It looks like a binary operation that uses a bitmask to produce a certain value... we don't need to know 100% how that mechanism is working right now... just that it works as described above.

--

So in the Arabic example... everything was "untouched" and yet the output is garbled. That's certainly an encoding issue... 

Indeed, look at the following:

dcook@koha:~/experiments> echo "مُدَرِّسَة" | iconv -f latin1 -t utf-8
��د�ر��س�ة

That is the same output as Text::Unaccent:

Text::Unaccent           - مُدَرِّسَة => Ù�Ù�دÙ�رÙ�Ù�سÙ�Ø©

So somewhere along the line that UTF-8 string is getting double-encoded.

Check this out:

dcook@koha:~/experiments> echo "مُدَرِّسَة" | iconv -f latin1 -t utf-8 | iconv -f utf-8 -t latin1
مُدَرِّسَة

I think the double-encoding is down to us using "binmode STDOUT, ':utf8';" (which tells Perl to output UTF-8 encoded bytes instead of Latin-1 (or some other single byte encoding it normally uses) and "use utf8" which tells Perl that the source code uses UTF-8...

Removing those gets us the following:

Text::Unaccent           - été => ete

Strip NonspacingMark     - été => A▒tA▒
Text::Unaccent           - umlaüt => umlaut
Wide character in print at unaccent.pl line 47.
Strip NonspacingMark     - umlaüt => umlaA1⁄4t
Text::Unaccent           - עברית => עברית

Strip NonspacingMark     - עברית => עב▒ י▒a
Text::Unaccent           - חוֹלָם => חוֹלָם
Strip NonspacingMark     - חוֹלָם => חוO1לO ם
Text::Unaccent           - 北京市 => 北京市

Strip NonspacingMark     - 北京市 => a▒▒ao▒a ▒
Text::Unaccent           - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => Αα Εε Ηη Ιι Οο Υυ Ωω

Strip NonspacingMark     - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => I▒I▒ I▒I▒ I▒I▒ I▒I  I▒I▒ I▒I▒ I▒I▒
Text::Unaccent           - مُدَرِّسَة => مُدَرِّسَة

Strip NonspacingMark     - مُدَرِّسَة => U▒U▒▒ U▒رU▒U▒▒3U▒ة

At a glance, Text::Unaccent looks like it works for French, German, and Greek... but doesn't touch Hebrew, Japanese(?), or Arabic.

Here's that output again with the debugging:

unac.c:13708: unac_data3[10] & unac_positions[3][10]: 0x00e9 => 0x0065
unac.c:13708: unac_data0[20] & unac_positions[0][21]: 0x0074 => untouched
unac.c:13708: unac_data3[10] & unac_positions[3][10]: 0x00e9 => 0x0065
Text::Unaccent           - été => ete

Strip NonspacingMark     - été => A▒tA▒
unac.c:13708: unac_data0[21] & unac_positions[0][22]: 0x0075 => untouched
unac.c:13708: unac_data0[13] & unac_positions[0][14]: 0x006d => untouched
unac.c:13708: unac_data0[12] & unac_positions[0][13]: 0x006c => untouched
unac.c:13708: unac_data0[1] & unac_positions[0][2]: 0x0061 => untouched
unac.c:13708: unac_data3[29] & unac_positions[3][29]: 0x00fc => 0x0075
unac.c:13708: unac_data0[20] & unac_positions[0][21]: 0x0074 => untouched
Text::Unaccent           - umlaüt => umlaut
Wide character in print at unaccent.pl line 47.
Strip NonspacingMark     - umlaüt => umlaA1⁄4t
unac.c:13708: unac_data0[2] & unac_positions[0][3]: 0x05e2 => untouched
unac.c:13708: unac_data0[17] & unac_positions[0][18]: 0x05d1 => untouched
unac.c:13708: unac_data0[8] & unac_positions[0][9]: 0x05e8 => untouched
unac.c:13708: unac_data0[25] & unac_positions[0][26]: 0x05d9 => untouched
unac.c:13708: unac_data0[10] & unac_positions[0][11]: 0x05ea => untouched
Text::Unaccent           - עברית => עברית

Strip NonspacingMark     - עברית => עב▒ י▒a
unac.c:13708: unac_data0[23] & unac_positions[0][24]: 0x05d7 => untouched
unac.c:13708: unac_data0[21] & unac_positions[0][22]: 0x05d5 => untouched
unac.c:13708: unac_data0[25] & unac_positions[0][26]: 0x05b9 => untouched
unac.c:13708: unac_data0[28] & unac_positions[0][29]: 0x05dc => untouched
unac.c:13708: unac_data0[24] & unac_positions[0][25]: 0x05b8 => untouched
unac.c:13708: unac_data0[29] & unac_positions[0][30]: 0x05dd => untouched
Text::Unaccent           - חוֹלָם => חוֹלָם
Strip NonspacingMark     - חוֹלָם => חוO1לO ם
unac.c:13708: unac_data0[23] & unac_positions[0][24]: 0x5317 => untouched
unac.c:13708: unac_data0[12] & unac_positions[0][13]: 0x4eac => untouched
unac.c:13708: unac_data0[2] & unac_positions[0][3]: 0x5e02 => untouched
Text::Unaccent           - 北京市 => 北京市

Strip NonspacingMark     - 北京市 => a▒▒ao▒a ▒
unac.c:13708: unac_data21[6] & unac_positions[21][7]: 0x0386 => 0x0391
unac.c:13708: unac_data22[12] & unac_positions[22][13]: 0x03ac => 0x03b1
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[8] & unac_positions[21][9]: 0x0388 => 0x0395
unac.c:13708: unac_data22[13] & unac_positions[22][14]: 0x03ad => 0x03b5
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[9] & unac_positions[21][10]: 0x0389 => 0x0397
unac.c:13708: unac_data22[14] & unac_positions[22][15]: 0x03ae => 0x03b7
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[10] & unac_positions[21][11]: 0x038a => 0x0399
unac.c:13708: unac_data22[15] & unac_positions[22][16]: 0x03af => 0x03b9
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[12] & unac_positions[21][13]: 0x038c => 0x039f
unac.c:13708: unac_data23[12] & unac_positions[23][13]: 0x03cc => 0x03bf
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[14] & unac_positions[21][15]: 0x038e => 0x03a5
unac.c:13708: unac_data23[13] & unac_positions[23][14]: 0x03cd => 0x03c5
unac.c:13708: unac_data0[0] & unac_positions[0][1]: 0x0020 => untouched
unac.c:13708: unac_data21[15] & unac_positions[21][16]: 0x038f => 0x03a9
unac.c:13708: unac_data23[14] & unac_positions[23][15]: 0x03ce => 0x03c9
Text::Unaccent           - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => Αα Εε Ηη Ιι Οο Υυ Ωω

Strip NonspacingMark     - Άά Έέ Ήή Ίί Όό Ύύ Ώώ => I▒I▒ I▒I▒ I▒I▒ I▒I  I▒I▒ I▒I▒ I▒I▒
unac.c:13708: unac_data0[5] & unac_positions[0][6]: 0x0645 => untouched
unac.c:13708: unac_data0[15] & unac_positions[0][16]: 0x064f => untouched
unac.c:13708: unac_data34[15] & unac_positions[34][16]: 0x062f => untouched
unac.c:13708: unac_data0[14] & unac_positions[0][15]: 0x064e => untouched
unac.c:13708: unac_data34[17] & unac_positions[34][18]: 0x0631 => untouched
unac.c:13708: unac_data0[16] & unac_positions[0][17]: 0x0650 => untouched
unac.c:13708: unac_data0[17] & unac_positions[0][18]: 0x0651 => untouched
unac.c:13708: unac_data34[19] & unac_positions[34][20]: 0x0633 => untouched
unac.c:13708: unac_data0[14] & unac_positions[0][15]: 0x064e => untouched
unac.c:13708: unac_data34[9] & unac_positions[34][10]: 0x0629 => untouched
Text::Unaccent           - مُدَرِّسَة => مُدَرِّسَة

Strip NonspacingMark     - مُدَرِّسَة => U▒U▒▒ U▒رU▒U▒▒3U▒ة

Analyzing what "use utf8" does and it's... interesting.


#use utf8;
#binmode STDOUT, ':utf8';
say "Hex = ".unpack("H*",$_);

Hex = d985d98fd8afd98ed8b1d990d991d8b3d98ed8a9
Text::Unaccent           - مُدَرِّسَة => مُدَرِّسَة

echo "مُدَرِّسَة" | xxd -p
d985d98fd8afd98ed8b1d990d991d8b3d98ed8a90a

[That last 0a byte is just a LF character (ie \n)]

use utf8;
#binmode STDOUT, ':utf8';
say "Hex = ".unpack("H*",$_);

Hex = 454f2f4e315051334e29
Text::Unaccent           - مُدَرِّسَة => Ù�Ù�دÙ�رÙ�Ù�سÙ�Ø©

#use utf8;
binmode STDOUT, ':utf8';
say "Hex = ".unpack("H*",$_);

Hex = d985d98fd8afd98ed8b1d990d991d8b3d98ed8a9
Text::Unaccent           - ��د�ر��س�ة => ��د�ر��س�ة

use utf8;
binmode STDOUT, ':utf8';
say "Hex = ".unpack("H*",$_);

Hex = 454f2f4e315051334e29
Text::Unaccent           - مُدَرِّسَة => Ù�Ù�دÙ�رÙ�Ù�سÙ�Ø©
--

I have no idea what 454f2f4e315051334e29 is... it's not UTF-8 or Latin1. In fact, if you try to read it as either... you'll just read that EO/N1PQ3N).

Ahh, I was missing this error message: Character in 'H' format wrapped in unpack at unaccent.pl line 46.

Here's some more info using Devel::Peek::Dump():
PV = 0x1ba6b20 "\331\205\331\217\330\257\331\216\330\261\331\220\331\221\330\263\331\216\330\251"\0 [UTF8 "\x{645}\x{64f}\x{62f}\x{64e}\x{631}\x{650}\x{651}\x{633}\x{64e}\x{629}"]

Indeed, if we look back at our UTF-8 table: http://www.utf8-chartable.de/unicode-utf8-table.pl?start=1536

0645 is the code point for ARABIC LETTER MEEM which would be encoded as d9 85.

454f2f4e315051334e29 is clearly a butchering of the internal string of Unicode codepoints "\x{645}\x{64f}\x{62f}\x{64e}\x{631}\x{650}\x{651}\x{633}\x{64e}\x{629}" where only the low-byte values of the code point is being shown.

--

Ahh... I think I might have figured it out.

When you use "use utf8":

$_ = PV = 0xf25f60 "\331\205\331\217\330\257\331\216\330\261\331\220\331\221\330\263\331\216\330\251"\0 [UTF8 "\x{645}\x{64f}\x{62f}\x{64e}\x{631}\x{650}\x{651}\x{633}\x{64e}\x{629}"]
Text::Unaccent::unac_string('UTF-8', $_) = PV = 0x2a0a0c0 "\331\205\331\217\330\257\331\216\330\261\331\220\331\221\330\263\331\216\330\251"\0

If you print out the content of "Text::Unaccent::unac_string('UTF-8', $_)" on its own, you'll get مُدَرِّسَة.

However, if you mix $_ and $unaccented in a single concatenated string, you're going to wind up with a correct $_ but a double-encoded $unaccented.

If you look at the concatenated string, you'll get a PV of:

PV = 0x29028c0 "Text::Unaccent - \331\205\331\217\330\257\331\216\330\261\331\220\331\221\330\263\331\216\330\251 - \303\231\302\205\303\231\302\217\303\230\302\257\303\231\302\216\303\230\302\261\303\231\302\220\303\231\302\221\303\230\302\263\303\231\302\216\303\230\302\251 \n"\0 [UTF8 "Text::Unaccent - \x{645}\x{64f}\x{62f}\x{64e}\x{631}\x{650}\x{651}\x{633}\x{64e}\x{629} - \x{d9}\x{85}\x{d9}\x{8f}\x{d8}\x{af}\x{d9}\x{8e}\x{d8}\x{b1}\x{d9}\x{90}\x{d9}\x{91}\x{d8}\x{b3}\x{d9}\x{8e}\x{d8}\x{a9} \n"]

So in that UTF8 section you have $_ represented by Unicode codepoints while the UTF-8 encoded bytes of $unaccepted have been transformed into a string of codepoints using a hexadecimal byte for each code point.

If you wanted to concatenate them both in the string, you'd first have to run "$unaccented = decode('UTF-8', $unaccented)". Then your concatenated string would internally look like: 

PV = 0x27812a0 "Text::Unaccent - \331\205\331\217\330\257\331\216\330\261\331\220\331\221\330\263\331\216\330\251 - \331\205\331\217\330\257\331\216\330\261\331\220\331\221\330\263\331\216\330\251 \n"\0 [UTF8 "Text::Unaccent - \x{645}\x{64f}\x{62f}\x{64e}\x{631}\x{650}\x{651}\x{633}\x{64e}\x{629} - \x{645}\x{64f}\x{62f}\x{64e}\x{631}\x{650}\x{651}\x{633}\x{64e}\x{629} \n"]

And that would be correct:

Text::Unaccent - مُدَرِّسَة - مُدَرِّسَة
Strip NonspacingMark     - مُدَرِّسَة => مدرسة

I mean... the output still doesn't do us much good, but that explains the mangling.

While we gave Text::Unaccent a Perl string with a UTF8 flag set, it took that string through to some C code using a XS interface, did a few things (depending on the scenario), and then passed back a Perl string without a UTF8 flag set, which seems to confuse Perl.

If we do a utf8::upgrade($unaccented) earlier, it still creates a string with incorrect code points...

More interesting things... 

You can still have a Perl string with a UTF8 flag set, even when you're not using "use utf8"...

My example:

my $arabic =  "\x{0645}";
PV = 0x190e950 "\331\205"\0 [UTF8 "\x{645}"]


Interestingly, if I don't use "use utf8", and use a UTF8 encoded character in my source code, I get a string without a UTF8 flag:

my $arabic_text = "ﻡ";
PV = 0x1ea92b0 "\331\205"\0

I imagine use of the \x{} construct must do a utf8::upgrade...

--

In any case, if I put $arabic and $arabic_text into the same string, I get the following:

my $arabic_result = "Arabic = $arabic_text = $arabic";
say $arabic_result;

Arabic = Ù� = م 
PV = 0x29feda0 "Arabic = \303\231\302\205 = \331\205"\0 [UTF8 "Arabic = \x{d9}\x{85} = \x{645}"]


However, if I try "$arabic_text = decode("UTF-8",$arabic_text")", which according to http://perldoc.perl.org/Encode.html means: $characters = decode('UTF-8', $octets), then I get the following:

Arabic = م = م
PV = 0x15efe50 "Arabic = \331\205 = \331\205"\0 [UTF8 "Arabic = \x{645} = \x{645}"]

Alternatively, I could have done "$arabic = encode("UTF-8",$arabic);", which would yield this result:

Arabic = م = م
PV = 0x832210 "Arabic = \331\205 = \331\205"\0

This explains the UTF8 flag a bit: http://perldoc.perl.org/Encode.html#The-UTF8-flag

So yeah... that's cool... who knew that was a thing, eh?

(In reply to Galen Charlton from comment #5)
> Some conclusions:
> 
> [1] Text::Unaccent mangles non-Latin characters outright; that's enough
> reason to get rid of it.

As I pointed out in my overly long comments, it doesn't appear that Text::Unaccent is actually mangling non-Latin characters. 

Rather, in your example, it looks like Perl doesn't correctly handle the concatenated string composed of one string with a UTF8 flag set and one string without a UTF8 flag set. 

It looks like Perl tries to do a utf8::upgrade() on the string without the UTF8 flag set (ie the one returned from Text::Unaccent's C code), and instead of reading it as an octet string and correctly translating into a UTF8 string of corresponding Unicode code points, it reads each octet in as a code point, which creates a completely different string for display purposes even though the underlying octets are the same. 

When given the octets d9 and 85 (ie the Arabic letter Meem), it creates a "UTF8 string" with the code points of "\x{d9}\x{85}" when it should create a "UTF8 string" with the code point "\x{645}".

Instead of creating "\x{645}", Perl reads the octets d9 and 85 in as "\x{d9}\x{85}"

This only appears to be a problem when you put the Text::Unaccent string in the same string as a Perl string with a UTF8 flag. If you were to break them into two separate lines, they'd display correctly in the terminal. Or you could use Encode::decode("UTF-8",$unaccented) to create a Perl string with a UTF8 flag with the proper code point "\x{645}";

> [2] Both Text::Unaccent::PurePerl and stripping NonspacingMark characters
> are better -- they strip accents from Latin scripts, and don't mangle
> non-Latin.  Removing NonspacingMark characters is more aggressive; I think
> we need input from Arabic, Hebrew, and Greek suers as to whether that is
> acceptable -- or, alternatively, if we need a system preference, or need to
> bite the bullet and package Text::Unaccent::PurePerl.

I suspect that Text::Unaccent and Text::Unaccent::PurePerl are mostly the same, but that Text::Unaccent::PurePerl doesn't lose the UTF8 flag on the input string. We could avoid Text::Unaccent::PurePerl if we simply use "Encode::decode("UTF-8",$unaccented)" when using Text::Unaccent to translate the internal byte string into an internal UTF8 string. While it might not be required that we do that, doing so would probably prevent future buggy behaviour from occurring.

That said, Text::Unaccent and Text::Unaccent::PurePerl don't necessarily look good enough. They miss diacritics in Arabic at least, although I think we definitely need input from Arabic, Hebrew, and CJK users regarding how stripping NonspacingMark affects those strings. My guess is that it's fine to strip the diacritics out of Arabic, but there are people much more qualified than me to answer that question on the listserv. 

Greek actually looks OK with Text::Unaccent if the encoding is handled. We can see that a bit more clearly with the following lines:

use Text::Unaccent qw/unac_debug/;
unac_debug($Text::Unaccent::DEBUG_HIGH);

(In reply to David Cook from comment #13)
> As I pointed out in my overly long comments, it doesn't appear that
> Text::Unaccent is actually mangling non-Latin characters. 
> 
> Rather, in your example, it looks like Perl doesn't correctly handle the
> concatenated string composed of one string with a UTF8 flag set and one
> string without a UTF8 flag set. 

Other way around: Text::Unaccent is not, as it would be much preferable, emitting Perl Unicode strings; rather, it is emitting octet-sequences.  A good pattern is aim for is using *only* Unicode strings within core code, and relegating use of Encode and friends to input and output; Text::Unaccent would get in the way of that.

(In reply to Galen Charlton from comment #14)
> Other way around: Text::Unaccent is not, as it would be much preferable,
> emitting Perl Unicode strings; rather, it is emitting octet-sequences.

Sorry, I must have been unclear; I meant to say that Text::Unaccent is emitting octet-sequences (hence why using encode() on the string returned by Text::Unaccent would create a Perl Unicode string).

And that Perl itself was causing problems when it tried to create a new string from an octet sequence string and a Perl Unicode string.
  
> A good pattern is aim for is using *only* Unicode strings within core code,
> and relegating use of Encode and friends to input and output; Text::Unaccent
> would get in the way of that.

Fair enough. I'm not in favour of Text::Unaccent per se. I was curious why it seemed to mangle some strings, and I shared what answers I found. 

I suspect Unicode::Normalize will really be the way to go, as you suggest. It seems much more comprehensive than Text::Unaccent and Text::Unaccent::PurePerl. I imagine we just need feedback from people experienced in Arabic, Hebrew, and CJK languages.

I ran the script from comment #5 on some more Hebrew text, I hope I did not forget any diacritic marks. 
I don't know what Text::Unaccent is doing. Text::Unaccent::PurePerl on the other hand seems to be doing too little, if at all. 

All the outputs from 'Strip NonspacingMark' seems correct - it's perfectly readable and have all the diacritics removed.

I modified the script a bit, so it's easy to compare the three options:

Text::Unaccent           - קָמָץ => קָ×ָץ
Text::Unaccent           - פַתח => פַת×
Text::Unaccent           - עִבְרִית => ×¢Ö´×ְרִ×ת
Text::Unaccent           - חוֹלָם => ××Ö¹×Ö¸×
Text::Unaccent           - זָנָב, תָּכְנִית => ×ָנָ×, תָּ×Ö°× Ö´×ת
Text::Unaccent           - צָהֳרַיִם => צָ×ֳרַ×Ö´×
Text::Unaccent           - קַל => קַ×
Text::Unaccent           - חֲלוֹם => ×Ö²××Ö¹×
Text::Unaccent           - מֶלֶךְ => ×Ö¶×Ö¶×Ö°
Text::Unaccent           - נֶאֱמָן => × Ö¶×Ö±×Ö¸×
Text::Unaccent           - לֵב => ×Öµ×
Text::Unaccent           - יִכְתְּבוּ => ×Ö´×ְתְּ××Ö¼
Text::Unaccent           - שִׁיר => שִ××ר
Text::Unaccent           - דֻּבִּים => ×Ö»Ö¼×Ö´Ö¼××
Text::Unaccent           - חֹלִי => ×Ö¹×Ö´×
Text::Unaccent           - סוּס => ס×ּס
Text::Unaccent           - נוֹף => × ×Ö¹×£
Text::Unaccent           - גַמּד => ×Ö·×Ö¼×
Text::Unaccent           - מְסַבֵּךְ => ×ְסַ×ÖµÖ¼×Ö°
Text::Unaccent           - שולחנהּ => ש×××× ×Ö¼
Text::Unaccent           - שֵׁם => שֵ××
Text::Unaccent           - עֶשֶׂר => עֶשֶ×ר
Text::Unaccent           - אֵלֶּה, אָנָּא, הֵמָּה, לָמָּה, שָׁמָּה, בָּתִּים, שָׁבַרְתִּי, תַּלְתַּל, לְבַד, חַג, לַיְלָה => ×Öµ×Ö¶Ö¼×, ×ָנָּ×, ×Öµ×Ö¸Ö¼×, ×Ö¸×Ö¸Ö¼×, שָ××Ö¸Ö¼×, ×ָּתִּ××, שָ××ַרְתִּ×, תַּ×ְתַּ×, ×Ö°×Ö·×, ×Ö·×, ×Ö·×Ö°×Ö¸×
Text::Unaccent::PurePerl - קָמָץ => קָמָץ
Text::Unaccent::PurePerl - פַתח => פַתח
Text::Unaccent::PurePerl - עִבְרִית => עִבְרִית
Text::Unaccent::PurePerl - חוֹלָם => חוֹלָם
Text::Unaccent::PurePerl - זָנָב, תָּכְנִית => זָנָב, תָּכְנִית
Text::Unaccent::PurePerl - צָהֳרַיִם => צָהֳרַיִם
Text::Unaccent::PurePerl - קַל => קַל
Text::Unaccent::PurePerl - חֲלוֹם => חֲלוֹם
Text::Unaccent::PurePerl - מֶלֶךְ => מֶלֶךְ
Text::Unaccent::PurePerl - נֶאֱמָן => נֶאֱמָן
Text::Unaccent::PurePerl - לֵב => לֵב
Text::Unaccent::PurePerl - יִכְתְּבוּ => יִכְתְּבוּ
Text::Unaccent::PurePerl - שִׁיר => שִׁיר
Text::Unaccent::PurePerl - דֻּבִּים => דֻּבִּים
Text::Unaccent::PurePerl - חֹלִי => חֹלִי
Text::Unaccent::PurePerl - סוּס => סוּס
Text::Unaccent::PurePerl - נוֹף => נוֹף
Text::Unaccent::PurePerl - גַמּד => גַמּד
Text::Unaccent::PurePerl - מְסַבֵּךְ => מְסַבֵּךְ
Text::Unaccent::PurePerl - שולחנהּ => שולחנהּ
Text::Unaccent::PurePerl - שֵׁם => שֵׁם
Text::Unaccent::PurePerl - עֶשֶׂר => עֶשֶׂר
Text::Unaccent::PurePerl - אֵלֶּה, אָנָּא, הֵמָּה, לָמָּה, שָׁמָּה, בָּתִּים, שָׁבַרְתִּי, תַּלְתַּל, לְבַד, חַג, לַיְלָה => אֵלֶּה, אָנָּא, הֵמָּה, לָמָּה, שָׁמָּה, בָּתִּים, שָׁבַרְתִּי, תַּלְתַּל, לְבַד, חַג, לַיְלָה
Strip NonspacingMark     - קָמָץ => קמץ
Strip NonspacingMark     - פַתח => פתח
Strip NonspacingMark     - עִבְרִית => עברית
Strip NonspacingMark     - חוֹלָם => חולם
Strip NonspacingMark     - זָנָב, תָּכְנִית => זנב, תכנית
Strip NonspacingMark     - צָהֳרַיִם => צהרים
Strip NonspacingMark     - קַל => קל
Strip NonspacingMark     - חֲלוֹם => חלום
Strip NonspacingMark     - מֶלֶךְ => מלך
Strip NonspacingMark     - נֶאֱמָן => נאמן
Strip NonspacingMark     - לֵב => לב
Strip NonspacingMark     - יִכְתְּבוּ => יכתבו
Strip NonspacingMark     - שִׁיר => שיר
Strip NonspacingMark     - דֻּבִּים => דבים
Strip NonspacingMark     - חֹלִי => חלי
Strip NonspacingMark     - סוּס => סוס
Strip NonspacingMark     - נוֹף => נוף
Strip NonspacingMark     - גַמּד => גמד
Strip NonspacingMark     - מְסַבֵּךְ => מסבך
Strip NonspacingMark     - שולחנהּ => שולחנה
Strip NonspacingMark     - שֵׁם => שם
Strip NonspacingMark     - עֶשֶׂר => עשר
Strip NonspacingMark     - אֵלֶּה, אָנָּא, הֵמָּה, לָמָּה, שָׁמָּה, בָּתִּים, שָׁבַרְתִּי, תַּלְתַּל, לְבַד, חַג, לַיְלָה => אלה, אנא, המה, למה, שמה, בתים, שברתי, תלתל, לבד, חג, לילה

Thx Yuval! It looks to me like using the new method would be a big step in the right direction.

Way to go Yuval!

Try replacing the following line:

print "Text::Unaccent           - $_ => " . Text::Unaccent::unac_string('utf-8', $_) . "\n";

with these lines:

print "Text::Unaccent           - $_ => ";
print Text::Unaccent::unac_string('utf-8', $_)."\n";

--

I suspect that will make the output of Text::Unaccent and Text::Unaccent::PurePerl the same. 

The epic-length posts I wrote earlier were about how Perl wasn't handling the output of Text::Unaccent as expected. 

--

Replacing this line:

use Text::Unaccent qw//;

with these lines:

use Text::Unaccent qw/unac_debug/;
unac_debug($Text::Unaccent::DEBUG_HIGH);

That will also tell you what Text::Unaccent is doing (or probably not doing).

(In reply to Katrin Fischer from comment #17)
> Thx Yuval! It looks to me like using the new method would be a big step in
> the right direction.

I agree.

Text::Unaccent and Text::Unaccent::PurePerl don't appear to be comprehensive enough to deal with many languages. While it seems to handle Latin and Greek characters, it doesn't do so well with Arabic and Hebrew.

Note that nothing seems to happen with the (Japanese?) ideograms that Galen tested. I wonder if accents are even a thing with CJK languages... I've asked a friend who knows Chinese for her input on that one. Oh, I know some people with Japanese experience as well... I should ask them.

I think we should also ask Vietnamese users, as Vietnamese has a lot of diacritics... and I think they might actually be quite significant. (https://en.wikipedia.org/wiki/Vietnamese_alphabet#Tone_marks)

I'll update the listserv to ask for people with Vietnamese knowledge too... as that could potentially answer Galen's question about whether or not we should even be unaccenting userid values...

> I suspect that will make the output of Text::Unaccent and
> Text::Unaccent::PurePerl the same. 
>

Not really, it stays the same garbled mess.

> unac_debug($Text::Unaccent::DEBUG_HIGH);
> 
> That will also tell you what Text::Unaccent is doing (or probably not doing).

I tested on one string:

unac.c:13708: unac_data0[7] & unac_positions[0][8]: 0x05e7 => untouched
unac.c:13708: unac_data0[24] & unac_positions[0][25]: 0x05b8 => untouched
unac.c:13708: unac_data0[30] & unac_positions[0][31]: 0x05de => untouched
unac.c:13708: unac_data0[24] & unac_positions[0][25]: 0x05b8 => untouched
unac.c:13708: unac_data0[5] & unac_positions[0][6]: 0x05e5 => untouched
Text::Unaccent           - קָמָץ => קָ×ָץ


> Note that nothing seems to happen with the (Japanese?) ideograms that Galen
> tested. I wonder if accents are even a thing with CJK languages...

I am definitely not an authoritative source, but I know a tiny bit of Japanese. The letters above are Kanji alphabet, and to the best of my knowledge do not have diacritics. BUT Japanese has two more alphabets, Hiragana and Katakana, both use diacritics, which CANNOT be removed, or they change the sound (and potentially the meaning).
For example, in the word Hiragana, the first syllable is ひ (Hi, pronounce Hee). This same syllable, with two ticks is び, and it sounds like Bee. A circle makes it ぴ - sounds like Pee. Testing those three:

Text::Unaccent           - ひびぴ => ã²ã²ã²
Text::Unaccent::PurePerl - ひびぴ => ひひひ
Strip NonspacingMark     - ひびぴ => ひひひ

So we've changed 'Hee Bee Pee' to 'Hee Hee Hee'. The same result (and same syllables) for Katakana:

Text::Unaccent           - ヒビピ => ããã
Text::Unaccent::PurePerl - ヒビピ => ヒヒヒ
Strip NonspacingMark     - ヒビピ => ヒヒヒ

So diacritics, at least in those two alphabets, should not be removed, to the best of my knowledge.

(In reply to Yuval Hager from comment #20)
> > I suspect that will make the output of Text::Unaccent and
> > Text::Unaccent::PurePerl the same. 
> >
> 
> Not really, it stays the same garbled mess.
> 

That's odd.

In that case, you could try replacing the following line:

print "Text::Unaccent           - $_ => " . Text::Unaccent::unac_string('utf-8', $_) . "\n";

with these lines:

use Encode;
my $unaccented = Text::Unaccent::unac_string('utf-8', $_);
$unaccented = encode("UTF-8",$unaccented);

print "Text::Unaccent           - $_ => $unaccented \n";

The garbled mess is, basically, because we're using "use utf8" and Text::Unaccent returns strings without a UTF8 flag.

> > unac_debug($Text::Unaccent::DEBUG_HIGH);
> > 
> > That will also tell you what Text::Unaccent is doing (or probably not doing).
> 
> I tested on one string:
> 
> unac.c:13708: unac_data0[7] & unac_positions[0][8]: 0x05e7 => untouched
> unac.c:13708: unac_data0[24] & unac_positions[0][25]: 0x05b8 => untouched
> unac.c:13708: unac_data0[30] & unac_positions[0][31]: 0x05de => untouched
> unac.c:13708: unac_data0[24] & unac_positions[0][25]: 0x05b8 => untouched
> unac.c:13708: unac_data0[5] & unac_positions[0][6]: 0x05e5 => untouched
> Text::Unaccent           - קָמָץ => קָ×ָץ
> 
> 
> > Note that nothing seems to happen with the (Japanese?) ideograms that Galen
> > tested. I wonder if accents are even a thing with CJK languages...
> 
> I am definitely not an authoritative source, but I know a tiny bit of
> Japanese. The letters above are Kanji alphabet, and to the best of my
> knowledge do not have diacritics. BUT Japanese has two more alphabets,
> Hiragana and Katakana, both use diacritics, which CANNOT be removed, or they
> change the sound (and potentially the meaning).
> For example, in the word Hiragana, the first syllable is ひ (Hi, pronounce
> Hee). This same syllable, with two ticks is び, and it sounds like Bee. A
> circle makes it ぴ - sounds like Pee. Testing those three:
> 

I was just reading some comments from a friend who was suggesting the same thing. 

> Text::Unaccent           - ひびぴ => ã²ã²ã²
> Text::Unaccent::PurePerl - ひびぴ => ひひひ
> Strip NonspacingMark     - ひびぴ => ひひひ
> 
> So we've changed 'Hee Bee Pee' to 'Hee Hee Hee'. The same result (and same
> syllables) for Katakana:
> 
> Text::Unaccent           - ヒビピ => ããã
> Text::Unaccent::PurePerl - ヒビピ => ヒヒヒ
> Strip NonspacingMark     - ヒビピ => ヒヒヒ
> 
> So diacritics, at least in those two alphabets, should not be removed, to
> the best of my knowledge.

In that case, I really wonder whether we should actually be removing accents for any languages, and instead look at why we started stripping accents in the first place.

Text::Unaccent is clearly not removing accents for many languages, so clearly it can't be that big of a problem, no?

I can confirm what has been said about Japanese - removing the diacritics from katakana or hiragana makes an unwanted difference.

Looking at the answers so far, it seems like they can't be a perfect solution working as expected for all languages - maybe we need to make this optional?

Hi , 
For Arabic, Yes it is true if you deleted all the diacritics ( َ  ُ  ِ  ْ  ٍ  ٌ  ً ) that will be fine in most cases , 
but in more details  the diacritics are affecting the meaning of any word (ie : مُدَرِّسة means a female teacher ) if we alter other diacritics and make it : 
مَدرَسة , it will mean a school . 
but in general they are not widely used in our daily life as we can understand the meaning of any word from its context .

the ancients Arabs were not using any type of diacritics and it been used only to help the new learner of Arabic language . 

so if the output will be without any diacritics I think in most cases that would be OK . 

I hope that helps ,

(In reply to Katrin Fischer from comment #22)
> I can confirm what has been said about Japanese - removing the diacritics
> from katakana or hiragana makes an unwanted difference.
> 
> Looking at the answers so far, it seems like they can't be a perfect
> solution working as expected for all languages - maybe we need to make this
> optional?

I still think it might be worthwhile to re-open http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=7411, although that's probably easier said than done now that it's problem behaviour has been corrected...

Clearly, Text::Unaccent hasn't been doing anything for CJK or Arabic anyway, and everything has been fine. So I don't know if we really need to be unaccenting the userid anyway. Can the problem really be localized to French? Or were French libraries the only ones to notice the original problem?

The original comment on this bug also talks about replacing Text::Unaccent with Text::Unaccent::PurePerl because there were issues on 64 bit CentOS, but the community doesn't really support anything other than Debian/Ubuntu anyway....

Text::Unaccent does not build on 64bit systems, the tests fail because of errors in the ccode. There has been a patch for that for four years but it looks like the upstream code is moribund. If you look at the test results it now fails on all linux test builds. The module has not been kept up to date to handle modern perl strings. I think the debian version may be patched to fix the bug in 64bit tests but it is buggy and should not be relied on. Suggest moving to Text::Unaccent::PurPerl be prioritized

(In reply to Colin Campbell from comment #25)
> Text::Unaccent does not build on 64bit systems, the tests fail because of
> errors in the ccode. There has been a patch for that for four years but it
> looks like the upstream code is moribund. If you look at the test results it
> now fails on all linux test builds. The module has not been kept up to date
> to handle modern perl strings. I think the debian version may be patched to
> fix the bug in 64bit tests but it is buggy and should not be relied on.
> Suggest moving to Text::Unaccent::PurPerl be prioritized

Where are we using Text::Unaccent? Is it still just in userid strings?

(In reply to David Cook from comment #26)
> (In reply to Colin Campbell from comment #25)
> > Text::Unaccent does not build on 64bit systems, the tests fail because of
> > errors in the ccode. There has been a patch for that for four years but it
> > looks like the upstream code is moribund. If you look at the test results it
> > now fails on all linux test builds. The module has not been kept up to date
> > to handle modern perl strings. I think the debian version may be patched to
> > fix the bug in 64bit tests but it is buggy and should not be relied on.
> > Suggest moving to Text::Unaccent::PurPerl be prioritized
> 
> Where are we using Text::Unaccent? Is it still just in userid strings?

I think that is the only location

Some test fails ATM when building a package in Buster (unstable) and looking into that I ended up here. Looking into packaging Text::Unaccent::PurePerl I found it has not had an update since 2013 so we would exchange one dead package for another.

(In reply to Mirko Tietgen from comment #28)
> Some test fails ATM when building a package in Buster (unstable) and looking
> into that I ended up here. Looking into packaging Text::Unaccent::PurePerl I
> found it has not had an update since 2013 so we would exchange one dead
> package for another.

I'm thinking "Strip NonspacingMark" instead of Text::Accent* might be the solution?

(In reply to Colin Campbell from comment #25)
> Text::Unaccent does not build on 64bit systems, the tests fail because of
> errors in the ccode. There has been a patch for that for four years but it
> looks like the upstream code is moribund. If you look at the test results it
> now fails on all linux test builds. The module has not been kept up to date
> to handle modern perl strings. I think the debian version may be patched to
> fix the bug in 64bit tests but it is buggy and should not be relied on.
> Suggest moving to Text::Unaccent::PurPerl be prioritized

I just ran into this again. 

Can't build on a 64 bit system running Perl 5.26. If you do force the build and install, you'll just get difficult to diagnose 500 errors in Koha.

As Colin mentioned, there's been known issues for this for ages: https://rt.cpan.org/Public/Dist/Display.html?Name=Text-Unaccent

I've gone to https://packages.debian.org/stretch/libtext-unaccent-perl to see if I can find the patch that Debian uses.

I reckon http://deb.debian.org/debian/pool/main/libt/libtext-unaccent-perl/libtext-unaccent-perl_1.08-1.3.diff.gz is the patch but the Debian CDN keeps timing out on me. 

When I try with curl, it seems like the CDN is refusing connections on port 80 and 443, so something looks like it's up with Debian...

In any case, this seems like a major sticking point for anyone not using Debian-based distros.

For now, I'm just removing Text::Unaccent where necessary.

Went to the Australia mirror...

http://ftp.au.debian.org/debian/pool/main/libt/libtext-unaccent-perl/

http://ftp.au.debian.org/debian/pool/main/libt/libtext-unaccent-perl/libtext-unaccent-perl_1.08-1.3.diff.gz

Patch looks consistent with the following:

https://bugs.launchpad.net/ubuntu/+source/libtext-unaccent-perl/+bug/460640

https://rt.cpan.org/Public/Bug/Display.html?id=21177

I don't think the maintainer has done anything on CPAN or Debian for many years. Looks like other Debian folk fixed the Debian package, and CPAN is just abandoned.

Looks like in addition to the problems of building on 64 bit systems it fails to build on the most recent versions of Perl. I think this is a result of the removal of . from the default path

I'd really like to understand why we need to unaccent the login's in the first place... was it because of how mysql stores the data perhaps and then does a lookup?

(In reply to Martin Renvoize from comment #34)
> I'd really like to understand why we need to unaccent the login's in the
> first place... was it because of how mysql stores the data perhaps and then
> does a lookup?

I think that was not the problem - or can't imagine it would be. What I could think of is having limited 'keyboards' on self checks and similar that don't allow to log in with diacritics. The username is also used by external services for authentication possibly... maybe adapting to that?

Perhaps we could make this a pluggable configuration option? We could even leave it with Text::Unaccent, and then us non-Debian using folk could manually switch to using Text::Unaccent::PurePerl?

Created attachment 96974 [details] [review]
Bug 14759: Replace Text::Unaccent with Unicode::Normalize

As shown in the comments on the bug, it appears that Unicode::Normalize
is the most reliable way to strip accents from strings for this use
case.

*** Bug 24292 has been marked as a duplicate of this bug. ***

I have applied this patch after having the exact same problem as described in bug 24292. The patch solved the problem and enabled upgrading of the installation from 19.05.04 to 19.05.06. However the patch was applied "manually" editing the code as describes so I can't do a sign-off with a clear conscience. This is un-orthodox but I needed the patch to update a test server, which worked, but I do not have a sandbox or similar to recreate the problem.

Created attachment 97099 [details] [review]
Bug 14759: Replace Text::Unaccent with Unicode::Normalize

As shown in the comments on the bug, it appears that Unicode::Normalize
is the most reliable way to strip accents from strings for this use
case.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Created attachment 97100 [details] [review]
Bug 14759: Add test

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Martin, I think you forgot to add the new dependency.

(In reply to Jonathan Druart from comment #42)
> Martin, I think you forgot to add the new dependency.

C4/Charset.pm:use Unicode::Normalize;
C4/Installer/PerlDependencies.pm:    'Unicode::Normalize' => {
C4/Record.pm:use Unicode::Normalize; # _entity_encode
Koha/Patron.pm:use Unicode::Normalize;
misc/migration_tools/bulkmarcimport.pl:use Unicode::Normalize;

What did you mean?

Created attachment 97174 [details] [review]
Bug 14759: Replace Text::Unaccent with Unicode::Normalize

As shown in the comments on the bug, it appears that Unicode::Normalize
is the most reliable way to strip accents from strings for this use
case.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 97175 [details] [review]
Bug 14759: Add test

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

(In reply to Marcel de Rooy from comment #43)
> (In reply to Jonathan Druart from comment #42)
> > Martin, I think you forgot to add the new dependency.
> 
> C4/Charset.pm:use Unicode::Normalize;
> C4/Installer/PerlDependencies.pm:    'Unicode::Normalize' => {
> C4/Record.pm:use Unicode::Normalize; # _entity_encode
> Koha/Patron.pm:use Unicode::Normalize;
> misc/migration_tools/bulkmarcimport.pl:use Unicode::Normalize;
> 
> What did you mean?

Indeed, I missed that it was already in our dependency list.

Nice work everyone!

Pushed to master for 20.05

Pushed to 19.11.x branch for 19.11.03

backported to 19.05.x for 19.05.08

Backported to 18.11.x for 18.11.14