Bug 2629 - Diacritics not being ignored when searching
Summary: Diacritics not being ignored when searching
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: 3.6
Hardware: All All
: PATCH-Sent (DO NOT USE) minor (vote)
Assignee: Albert Oller
QA Contact: Marcel de Rooy
Depends on: 3216
  Show dependency treegraph
Reported: 2008-09-19 10:52 UTC by Chris Cormack
Modified: 2017-06-14 22:11 UTC (History)
20 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:

Add charmap mappings for characters with stroke (2.64 KB, patch)
2011-07-21 14:55 UTC, Ian Walls
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 2629: add char-with-stroke support to word-phrase-utf.chr (3.17 KB, patch)
2011-08-11 15:39 UTC, Katrin Fischer
Details | Diff | Splinter Review
Proposed Patch (970 bytes, patch)
2011-09-26 20:42 UTC, Ian Walls
Details | Diff | Splinter Review
Bug 2629: Add diacritic support for Ů (U with ring) (1.01 KB, patch)
2011-11-04 00:18 UTC, Chris Cormack
Details | Diff | Splinter Review
word-phrase-utf.chr (4.20 KB, patch)
2011-11-30 00:28 UTC, Albert Oller
Details | Diff | Splinter Review
Bug-2629-Diacritics-not-being-ignored-when-searching.patch (1.07 KB, patch)
2011-11-30 14:00 UTC, Albert Oller
Details | Diff | Splinter Review
example of diacritic causing a failed search (1.62 KB, application/octet-stream)
2011-12-28 18:52 UTC, Albert Oller
marc for record 62385 (1.16 KB, application/octet-stream)
2011-12-28 19:11 UTC, Albert Oller
[Signed Off] Bug 2629 - Diacritics not being ignored when searching (1.29 KB, patch)
2011-12-31 22:46 UTC, Liz Rea
Details | Diff | Splinter Review
Bug 2629 - Diacritics not being ignored when searching - Map ů to u (949 bytes, patch)
2012-09-20 12:24 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2010-05-21 00:53:39 UTC

---- Reported by bchurch@ptfs.com 2008-09-19 10:52:27 ----

Koha is not ignoring characters with diacritics when searching in both OPAC and the staff client. For example, if you search for the French playwright "Molière", you will only find records if the term contains the grave (è) accent. You will not retrieve records with the terms "Molière" AND "Moliere".

The problem is even more serious when you search authority records. If you have an authority record for Molière, you cannot retrieve the authority record at all. If you enter "Molière" or "Moliere", you will not find the authority. The only way to find the authority is to look for personal author names starting with "moli".

---- Additional Comments From henridamien@koha-fr.org 2008-10-06 13:07:47 ----

This problem is probably owed to the fact that you have two ways to write /é/ in UTF-8.
é as combined char '+e
é as character \x00E9
with \x00E9, everything should go fine.
With combined char, it is going bad.
Can you detail ?

---- Additional Comments From bibliwho@gmail.com 2009-09-24 13:45:46 ----

Wow, it's been a year since this "major" P1 bug was submitted ... any updates? Fixed in 3.2?

---- Additional Comments From brian@jhu.edu 2009-09-24 15:14:16 ----

Should we switch default.idx over to icuchain indexing to take advantage of ICU's Unicode normaliztion?  I think that the yaz/zebra issues with it have been fixed.  The following word.xml is working for me, but could use more non-diacritic special characters.

<icu_chain locale="en">
        <transform rule="[:Control:] Any-Remove" />
        <casemap rule="l" />
        <tokenize rule="w" />
        <transliterate rule="\- > ' '" />
        <transform rule="NFKD; [[:WhiteSpace:][:Punctuation:][:Diacritic:]] Remove" />
        <transliterate rule="ø > o"/>
        <transliterate rule="ð > th"/>
        <transliterate rule="ʒ > z"/>
        <transliterate rule="ȝ > g"/>
        <transliterate rule="æ > ae"/>
        <transliterate rule="œ > oe"/>
        <transliterate rule="þ > th"/>
        <display />

---- Additional Comments From nengard@gmail.com 2010-02-27 14:13:22 ----

Where are we with solving this?  I have customers asking about it - but this is over my area of expertise - and I don't want to have a developer step on anyone's toes if someone is working on this.

---- Additional Comments From henridamien@koha-fr.org 2010-02-27 18:05:12 ----

using icu.xml configuration by default would definitely solve this problem for zebra searches.
See http://lists.katipo.co.nz/public/koha/2010-January/022269.html
It should be noted though at at the moment zebra search with icu cannot provide ppl with right truncation.
But search is globally much more robust in any case.

---- Additional Comments From oleonard@myacpl.org 2010-02-27 18:31:43 ----

"zebra search with icu cannot provide ppl with right truncation," meaning searches under that configuration would return no results for "dogs" when searching for "dog" ? That doesn't sound more robust.

--- Bug imported by chris@bigballofwax.co.nz 2010-05-21 00:53 UTC  ---

This bug was previously known as _bug_ 2629 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=2629

Actual time not defined. Setting to 0.0
The original reporter of this bug does not have
   an account here. Reassigning to the person who moved
   it here: chris@bigballofwax.co.nz.
   Previous reporter was bchurch@ptfs.com.
CC member adrea@moablibrary.org does not have an account here
CC member arm@hanover.ca does not have an account here
CC member bibliwho@gmail.com does not have an account here
CC member brian@jhu.edu does not have an account here
CC member jwagner@ptfs.com does not have an account here
CC member mary.moules@liblime.com does not have an account here
CC member scook@hartford.edu does not have an account here

Comment 1 Nicole C. Engard 2010-09-03 12:41:35 UTC
Any word on the status of this bug? Is anyone working on resolving it?
Comment 2 dev_patches 2010-09-09 18:17:02 UTC
In fact, I meant left truncation. and not right truncation.
But well icu configuration could be a way to cope with that.
Comment 3 Ian Walls 2011-07-21 13:51:05 UTC
ICU chains are not required to solve this issue; the word-phrase-utf.chr configuration file contains a mapping of diacritic characters to 'plain' characters, including è.

To get this config file to kick in, however, the MARC records must be Unicode Normalized to NFC.  Since 3.4, any time AddBiblio or ModBiblio is called, the subroutine SetUTF8Flag is called, which normalizes each subfield in each field of the MARC record to NFC (or NFD, if the param is given).  This should not be an issue on a new installation of 3.4, but running misc/maintenance/touch_all_biblios.pl would ensure the records are all properly normalized.  For anyone upgrading to 3.4, since the upgrae requires the use of remove_items_from_biblioitems.pl, which makes use of ModBiblio, upgrading should also resolve this issue, for most diacritics.  

I say most, because I've recently discovered that several diacritic marks, including letters with stroke sign, are not included in the mapping.  This means that searches for the Polish name "Lutoslawski" will not return any hits if the name is spelled with diacritics: "Lutosławski".  Patch forthcoming.
Comment 4 dev_patches 2011-07-21 13:58:05 UTC
ICU is an elegant solution to index all types of characters.
charmap is good... But if you need to add every kind of characters, you will have a very big charmap file and maintenance will be even harder.
Comment 5 Ian Walls 2011-07-21 14:04:57 UTC
Until Koha's ICU configuration is fixed to work with fuzzy searching and right truncation, the word-phrase-utf.chr configuration is best solution to getting this kind of searching implemented.  I agree that listing out each character to map is kind of tedious, but I don't think that it's a blocker, and new characters aren't being created faster than we can keep up with.
Comment 6 dev_patches 2011-07-21 14:18:41 UTC
well the very many languages you may want to search and index... hebrew, arabic and more.... tends to be also a blocker for some libraries... using charmaps for them is not only tedious but also leads to some search crux.
Moreover, fuzzy search and right truncation, though a tough drawback, still is better than returning anything.
Comment 7 Ian Walls 2011-07-21 14:28:18 UTC
How about this:

1.  Commit a solid ICU config file to Koha
2.  Add a question to the installer whether to use Charmaps or ICU for indexing, and make default.idx respect that choice.

This'll let people choose whatever configuration works best for their local language, and much like what happened with Zebra v. NoZebra, the "better" config file will win out over time.
Comment 8 dev_patches 2011-07-21 14:41:38 UTC
why not ?
In my opinion and from what I could test and all the feedback I had,
the icu configuration file i posted on list was quite robust.
then having a debconf option to choose the indexing schema could be fine.
Mind you, the icu_chain should not be used though for p indexation.
default.idx was also posted on list.
Comment 9 Ian Walls 2011-07-21 14:54:07 UTC
I've used the icu.xml file, and aside from the fuzzy search/right truncation issues, it seems to work quite well when implemented.  I'd say let's start with that file.
Comment 10 Ian Walls 2011-07-21 14:55:47 UTC Comment hidden (obsolete)
Comment 11 Katrin Fischer 2011-08-11 15:39:49 UTC Comment hidden (obsolete)
Comment 12 Ian Walls 2011-08-11 15:42:14 UTC
Patch just adds new character support, and an updated note; Katrin's tests verify no regression.  Marking as Passed QA.
Comment 13 Chris Cormack 2011-08-14 08:10:32 UTC
Pushed, I have only tested cursorily, please test more thoroughly
Comment 14 Ian Walls 2011-09-26 20:42:22 UTC Comment hidden (obsolete)
Comment 15 Melia Meggs 2011-10-28 20:33:59 UTC
I think Ian's patch from 9/26 still needs to be tested and signed off, so I am changing the patch status to "needs signoff" again.
Comment 16 Chris Cormack 2011-11-04 00:18:42 UTC Comment hidden (obsolete)
Comment 17 Ian Walls 2011-11-04 08:21:03 UTC
Testing out newly implemented QA Contact feature, and assigning to Marcel to QA (since this latest patch is mine).
Comment 18 Marcel de Rooy 2011-11-10 14:23:36 UTC
Passed QA. Works. Simple patch.
Comment 19 Paul Poulain 2011-11-16 16:43:42 UTC
Patch pushed
Comment 20 Ian Walls 2011-11-17 16:20:42 UTC
There is currently no support for ċ in Koha, and other 'dot-above' characters may also have issues.
Comment 21 Paul Poulain 2011-11-17 16:37:43 UTC
I think this is a forever lasting ticket... until you try ICU. We (in France) have switched to icu, and word-phrase become useless
Comment 22 Ian Walls 2011-11-17 17:10:12 UTC

The patch for ICU that is floating around works pretty well in this regard, but breaks Fuzzy searching for Keywords.  I've implemented it for several of our libraries, but many were not pleased and we needed to revert.

So, if the ICU patch has been updated to fix this issue, we could include it as a alternate config.  Otherwise, I think it's best to stick with what works (word-phrase-utf.chr) even if we have to keep adding new character maps.
Comment 23 Katrin Fischer 2011-11-17 17:35:05 UTC
I know libraries that are not too happy with fuzzy search and ICU is needed if you want to search for other scripts like Hebrew. I think implementing this as an option in Koha would be great, even if we can't fix the FuzzySearch.
Comment 24 Paul Poulain 2011-11-17 17:37:57 UTC
In our setups, fuzzy is desactivated too. I suspect the fuzzy in english is good, and in other language poor. Some fuzzy suggestions were more understandable if you tried to say them in english (and imagine what zebra could have made)
Comment 25 Albert Oller 2011-11-30 00:28:14 UTC Comment hidden (obsolete)
Comment 26 Marcel de Rooy 2011-11-30 13:34:03 UTC
(In reply to comment #25)
> Created attachment 6460 [details] [review]
> word-phrase-utf.chr
> /etc/zebradb/etc/word-phrase-utf.chr

Albert, please send in your change as a git formatted patch. Thanks.
After that, please set Importance to Patch-Sent and Patch Status to Needs Signoff.
Comment 27 Albert Oller 2011-11-30 14:00:28 UTC Comment hidden (obsolete)
Comment 28 Jared Camins-Esakov 2011-12-02 01:14:24 UTC
I cannot replicate the problem that the second patch is trying to fix. As far as I can tell, all the accent characters it addresses already work properly. Please provide a more complete description and sample record.
Comment 29 Albert Oller 2011-12-09 21:03:35 UTC
Hi Jared,

The original problem reported was for this:

Anam ċara  from Harrison Memorial Library (Carmel)

Anam ċara : (Record no. 62385)
[ view plain ]
fixed length control field 	00991nam a2200325Ia 4500
control field 	ocm40357390
Personal name 	O'Donohue, John,
Dates associated with a name 	1956-
Title 	Anam ċara :
Remainder of title 	a book of Celtic wisdom /
Statement of responsibility, etc 	John O'Donohue

The ċ was the only character reported as causing a problem, but I found a few extra characters that were missing, that I added in to prevent future tickets.

Comment 30 Jared Camins-Esakov 2011-12-20 18:40:01 UTC

Could you please attach the record to the bug? Carmel has the download link removed from their OPAC.

Comment 31 Albert Oller 2011-12-28 18:52:40 UTC
Created attachment 6945 [details]
example of diacritic causing a failed search
Comment 32 Albert Oller 2011-12-28 19:11:32 UTC
Created attachment 6946 [details]
marc for record 62385
Comment 33 Liz Rea 2011-12-31 22:46:28 UTC Comment hidden (obsolete)
Comment 34 Liz Rea 2011-12-31 22:48:26 UTC
ps. thank you very much for the record to test this patch with. Made testing a lot easier, I appreciate it.
Comment 35 Katrin Fischer 2012-01-08 08:15:20 UTC
Bug 3216 implements ICU as an option during install. 
With ICU searching for diacritics can be solved in a general way.
Comment 36 Ian Walls 2012-01-09 12:07:17 UTC
Just adds support for additional characters in the charmap. Marking Passed QA
Comment 37 Paul Poulain 2012-01-10 22:34:59 UTC
Comment on attachment 6171 [details] [review]
Bug 2629: Add diacritic support for Ů (U with ring)

patch obsoleted, already applied
Comment 38 Paul Poulain 2012-01-10 22:37:36 UTC
Patch pushed, as it works, 
BUT katrin is right: you should investigate ICU, that solves all diacritics (well, there are some remaining problems, and that's why we should get rid of zebra, but that's another matter. And using ICU is better than trying to have everything in word-phrase !
Comment 39 Jared Camins-Esakov 2012-05-23 12:25:07 UTC
Included in the 3.6 branch prior to 3.6.4.
Comment 40 Kyle M Hall 2012-09-20 12:24:47 UTC
Created attachment 12385 [details] [review]
Bug 2629 - Diacritics not being ignored when searching - Map ů to u
Comment 41 wajasu 2012-10-04 18:38:19 UTC
As ICU is being persued, I am only noting an alternate possible idea in case that route does not come about.

We might be able to write a perl script that uses Unicde::UCD or such routines to scan the DB searchable fields, and convert them to NFD form to detect diacritics(decomposed form) or another alogrithm, and compile a table accordingly.  Then we could generate the /etc/zebradb/etc/word-phrase-utf.chr  map.   It may suffice as a stop gap measure, that once run in a large library, it might cover 99.9% of the cases, catering to the context of each library.  So instead of a patch per character, the site would periodically run the script to handle such cases.

But ICU is where we are persuing for now.
Comment 42 Marc Véron 2012-10-05 13:23:47 UTC
Applying the patch gives following error:
fatal: cannot convert from UTF-8utf-8 to UTF-8

It seems that the patch maps this letter twice? See line 1, 3 and 5 snip:
-map ů                   u
-map Ů                   u
+map ů           u
+map Ů           u
+map ů           u
Comment 43 wajasu 2012-10-19 18:53:40 UTC
fatal: cannot convert from UTF-8utf-8 to UTF-8
usually happens because when the patch was made the .gitconfig probably had

       headers = "Content-Type: text/plain; charset=UTF-8"

and a newer git was used somewhere inthe process.
So it got added to the patch header.

One can delete the duplicate line in the header of the patch, to cleanly apply it.

If you are using a newer/recent version of git, they fixed things, so it doesn't need this, anymore, so you can comment that line out in your .gitconfig if its there.

#       headers = "Content-Type: text/plain; charset=UTF-8"

So for older patches you still come across this duplicate line the patch header.
Comment 44 Jared Camins-Esakov 2012-11-01 20:01:32 UTC
What exactly is this latest patch supposed to do? It is unclear from inspection.
Comment 45 Mathieu Saby 2013-09-28 12:47:56 UTC
A note about right truncation and ICU : 
The existing issue was fixed in version 2.0.53 of Zebra (2012/12/03)
source : http://www.indexdata.com/zebra/doc/NEWS

I don't know how to take profit of that fix in Koha?

M. Saby
Comment 46 Owen Leonard 2016-08-10 14:35:16 UTC
To my understanding this may be valid if you are not using ICU, but it would probably be better to address specific issues in a separate bug.