Bug 14824 - Fix sorting of Norwegian vowels
Summary: Fix sorting of Norwegian vowels
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Magnus Enger
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-09-15 08:14 UTC by Magnus Enger
Modified: 2016-06-21 21:40 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 14824 - Fix sorting of Norwegian vowels (2.11 KB, patch)
2015-09-15 08:42 UTC, Magnus Enger
Details | Diff | Splinter Review
Bug 14824 - Fix sorting of Norwegian vowels (2.17 KB, patch)
2015-10-22 19:15 UTC, Mirko Tietgen
Details | Diff | Splinter Review
[PASSED QA] Bug 14824 - Fix sorting of Norwegian vowels (2.23 KB, patch)
2015-10-25 09:03 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Enger 2015-09-15 08:14:34 UTC
Currently, if you sort bibliographic search results by title, the Norwegian wovels ÆØÅ will not be sorted correctly. Specifically, with a result list containing 4 results, I was getting this for an A-Z (or A-Å) sort: 

Å...
Æ...
H...
Ø...

Which is just about as incorrect as it gets. 

As documented here: http://www.indexdata.com/zebra/doc/character-map-files.html sorting in Zebra is governed by the "lowercase" and "uppercase" setting in sort-string-utf.chr. That page suggests this for scandinavian languages: 

lowercase {0-9}{a-y}üzæäøöå
uppercase {0-9}{A-Y}ÜZÆÄØÖÅ

But with that config I got warnings during rebuild zebra: 

10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'ä'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'Ä'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'ö'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'Ö'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'ü'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'Ü'

I assume this is because these characters are already mapped to other chars. 

So I will propose a patch that uses this simplified setting:

lowercase {0-9}{a-z}æøå
uppercase {0-9}{A-Z}ÆØÅ
Comment 1 Magnus Enger 2015-09-15 08:42:46 UTC Comment hidden (obsolete)
Comment 2 Mirko Tietgen 2015-10-22 19:15:12 UTC Comment hidden (obsolete)
Comment 3 Katrin Fischer 2015-10-25 09:03:31 UTC
Created attachment 43943 [details] [review]
[PASSED QA] Bug 14824 - Fix sorting of Norwegian vowels

Currently, Norwegian vowels are not sorted correctly, even when
you have chosen "nb" as the ZEBRA_LANGUAGE during installation.

To test:
- Make sure you have three records with titles that begin with ÆØÅ
  respectively
- Do a search that turns up those three records and some others, and
  sort the results by title, bot ascending and descending.
- Verify that ÆØÅ is shown in some weird order.
- Edit your sort-string-utf.chr* so it is in line with the current
  patch. It should include these two lines:
  lowercase {0-9}{a-z}æøå
  uppercase {0-9}{A-Z}ÆØÅ
- Restart Zebra and reindex all the records, e.g.:
  $ sudo koha-restart-zebra <instancename>
  $ sudo koha-rebuild-zebra -f -v <instancename>
- Do the search again, make sure you order by title and check that
  ÆØÅ are sorted in the order of 1. Æ 2. Ø 3. Å, and after all other
  characters

* = If you are on a gitified install, you need to edit this file:
/etc/koha/zebradb/lang_defs/<your ZEBRA_LANGUAGE>/sort-string-utf.chr
NOT the file in your git clone (yeah, i wasted some time there...)

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 4 Tomás Cohen Arazi 2015-10-26 14:22:14 UTC
Patch pushed to master.

Thanks Magnus!