Bug 14824

Summary:	Fix sorting of Norwegian vowels
Product:	Koha	Reporter:	Magnus Enger <magnus>
Component:	Searching	Assignee:	Magnus Enger <magnus>
Status:	CLOSED FIXED	QA Contact:	Testopia <testopia>
Severity:	normal
Priority:	P5 - low	CC:	mirko, tomascohen
Version:	Main
Hardware:	All
OS:	All
Change sponsored?:	---	Patch complexity:	Trivial patch
Documentation contact:		Documentation submission:
Text to go in the release notes:		Version(s) released in:
Circulation function:
Attachments:	Bug 14824 - Fix sorting of Norwegian vowels Bug 14824 - Fix sorting of Norwegian vowels [PASSED QA] Bug 14824 - Fix sorting of Norwegian vowels

Description Magnus Enger 2015-09-15 08:14:34 UTC

Currently, if you sort bibliographic search results by title, the Norwegian wovels ÆØÅ will not be sorted correctly. Specifically, with a result list containing 4 results, I was getting this for an A-Z (or A-Å) sort: 

Å...
Æ...
H...
Ø...

Which is just about as incorrect as it gets. 

As documented here: http://www.indexdata.com/zebra/doc/character-map-files.html sorting in Zebra is governed by the "lowercase" and "uppercase" setting in sort-string-utf.chr. That page suggests this for scandinavian languages: 

lowercase {0-9}{a-y}üzæäøöå
uppercase {0-9}{A-Y}ÜZÆÄØÖÅ

But with that config I got warnings during rebuild zebra: 

10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'ä'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'Ä'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'ö'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'Ö'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'ü'
10:05:00-15/09 zebraidx(26924) [warn] duplicate entry for charmap from 'Ü'

I assume this is because these characters are already mapped to other chars. 

So I will propose a patch that uses this simplified setting:

lowercase {0-9}{a-z}æøå
uppercase {0-9}{A-Z}ÆØÅ

Comment 1 Magnus Enger 2015-09-15 08:42:46 UTC Comment hidden (obsolete)

Created attachment 42549 [details] [review]
Bug 14824 - Fix sorting of Norwegian vowels

Currently, Norwegian vowels are not sorted correctly, even when
you have chosen "nb" as the ZEBRA_LANGUAGE during installation.

To test:
- Make sure you have three records with titles that begin with ÆØÅ
  respectively
- Do a search that turns up those three records and some others, and
  sort the results by title, bot ascending and descending.
- Verify that ÆØÅ is shown in some weird order.
- Edit your sort-string-utf.chr* so it is in line with the current
  patch. It should include these two lines:
  lowercase {0-9}{a-z}æøå
  uppercase {0-9}{A-Z}ÆØÅ
- Restart Zebra and reindex all the records, e.g.:
  $ sudo koha-restart-zebra <instancename>
  $ sudo koha-rebuild-zebra -f -v <instancename>
- Do the search again, make sure you order by title and check that
  ÆØÅ are sorted in the order of 1. Æ 2. Ø 3. Å, and after all other
  characters

* = If you are on a gitified install, you need to edit this file:
/etc/koha/zebradb/lang_defs/<your ZEBRA_LANGUAGE>/sort-string-utf.chr
NOT the file in your git clone (yeah, i wasted some time there...)

Comment 2 Mirko Tietgen 2015-10-22 19:15:12 UTC Comment hidden (obsolete)

Created attachment 43780 [details] [review]
Bug 14824 - Fix sorting of Norwegian vowels

Currently, Norwegian vowels are not sorted correctly, even when
you have chosen "nb" as the ZEBRA_LANGUAGE during installation.

To test:
- Make sure you have three records with titles that begin with ÆØÅ
  respectively
- Do a search that turns up those three records and some others, and
  sort the results by title, bot ascending and descending.
- Verify that ÆØÅ is shown in some weird order.
- Edit your sort-string-utf.chr* so it is in line with the current
  patch. It should include these two lines:
  lowercase {0-9}{a-z}æøå
  uppercase {0-9}{A-Z}ÆØÅ
- Restart Zebra and reindex all the records, e.g.:
  $ sudo koha-restart-zebra <instancename>
  $ sudo koha-rebuild-zebra -f -v <instancename>
- Do the search again, make sure you order by title and check that
  ÆØÅ are sorted in the order of 1. Æ 2. Ø 3. Å, and after all other
  characters

* = If you are on a gitified install, you need to edit this file:
/etc/koha/zebradb/lang_defs/<your ZEBRA_LANGUAGE>/sort-string-utf.chr
NOT the file in your git clone (yeah, i wasted some time there...)

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Comment 3 Katrin Fischer 2015-10-25 09:03:31 UTC

Created attachment 43943 [details] [review]
[PASSED QA] Bug 14824 - Fix sorting of Norwegian vowels

Currently, Norwegian vowels are not sorted correctly, even when
you have chosen "nb" as the ZEBRA_LANGUAGE during installation.

To test:
- Make sure you have three records with titles that begin with ÆØÅ
  respectively
- Do a search that turns up those three records and some others, and
  sort the results by title, bot ascending and descending.
- Verify that ÆØÅ is shown in some weird order.
- Edit your sort-string-utf.chr* so it is in line with the current
  patch. It should include these two lines:
  lowercase {0-9}{a-z}æøå
  uppercase {0-9}{A-Z}ÆØÅ
- Restart Zebra and reindex all the records, e.g.:
  $ sudo koha-restart-zebra <instancename>
  $ sudo koha-rebuild-zebra -f -v <instancename>
- Do the search again, make sure you order by title and check that
  ÆØÅ are sorted in the order of 1. Æ 2. Ø 3. Å, and after all other
  characters

* = If you are on a gitified install, you need to edit this file:
/etc/koha/zebradb/lang_defs/<your ZEBRA_LANGUAGE>/sort-string-utf.chr
NOT the file in your git clone (yeah, i wasted some time there...)

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Comment 4 Tomás Cohen Arazi (tcohen) 2015-10-26 14:22:14 UTC

Patch pushed to master.

Thanks Magnus!