Bug 16952 - Add sorting rules for Czech language to Zebra
Summary: Add sorting rules for Czech language to Zebra
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Josef Moravec
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-07-21 09:31 UTC by Josef Moravec
Modified: 2017-06-14 22:13 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 16952: Czech language definitions for sorting in Zebra (4.38 KB, patch)
2016-07-21 09:44 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 16952: Czech language definitions for sorting in Zebra (4.48 KB, patch)
2016-10-20 07:13 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 16952: Czech language definitions for sorting in Zebra (4.53 KB, patch)
2016-10-22 05:06 UTC, Radek Šiman (R-Bit Technology, s.r.o.)
Details | Diff | Splinter Review
Bug 16952: Czech language definitions for sorting in Zebra (4.66 KB, patch)
2016-10-25 08:20 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Josef Moravec 2016-07-21 09:31:21 UTC

    
Comment 1 Josef Moravec 2016-07-21 09:44:21 UTC
Created attachment 53556 [details] [review]
Bug 16952: Czech language definitions for sorting in Zebra

Test plan:
1) Apply patch
2) Create new instance with parameter --zebralang cs
3) Insert some record with basic latin characters and some with "czech" characters (for example: "č" - should be sorted after "c", "š" - should be sorted after "s")
4) Try to search in katalog (staff and opac) and sort by other field then relevance - title or author for instance
5) Records should be sorted correctly by Czech rules
6) Look at code and confirm it is ok
Comment 2 Josef Moravec 2016-10-18 10:16:58 UTC
If you aren't building and installing packages after applying patches (which is most cases I think) you may need copy the new configuration file to /etc directory manually:

mkdir /etc/koha/zebradb/lang_defs/cs

cp etc/zebradb/lang_defs/cs/sort-string-utf.chr /etc/koha/zebradb/lang_defs/cs/
Comment 3 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-18 10:36:57 UTC
The right copy command for kohadevbox is following:
cp /home/vagrant/kohaclone/etc/zebradb/lang_defs/cs/sort-string-utf.chr /etc/koha/zebradb/lang_defs/cs/
Comment 4 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-18 10:57:04 UTC
In my opinion, czech alphabet needs better sorting of characters like "a" vs. "á". For instance "Cáchovnice" should be sorted after "Carevna", but it's not:
http://prntscr.com/cvrem1
I think we cannot consider eéě as the same one to be perfect.

This is the correct ordering of czech characters:
https://cs.wikipedia.org/wiki/Česká_abeceda
Comment 6 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-18 11:23:12 UTC
Regardning your last post, the sorting is much more sofisticated. The simplified (your) alphabet is used only when we actually CAN decide about hte correct order. If we cannot, it's required to use special signs like in this example:
- Sálá
- Sála
In this case the secondary ordering rule should be taken into account, I think.
Comment 7 Josef Moravec 2016-10-18 12:08:02 UTC
(In reply to Radek Šiman (R-Bit Technology, s.r.o.) from comment #6)
> Regardning your last post, the sorting is much more sofisticated. The
> simplified (your) alphabet is used only when we actually CAN decide about
> hte correct order. If we cannot, it's required to use special signs like in
> this example:
> - Sálá
> - Sála
> In this case the secondary ordering rule should be taken into account, I
> think.

According to Zebra documentation here: http://www.indexdata.com/zebra/doc/character-map-files.html

It looks like zebra is not capable doing 2-level sorting...

So propably this could be better than nothing, as it looks like secondary sorting is edge case...

what do you think?
Comment 8 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-18 12:25:01 UTC
Yes, I agree, that we should sort the texts somehow reasonably. What do you think about using my first proposal, ie. sorting like aábcčdďeéě...? This seems to me as more accurate approach, even if it's not as defined simplified standard rules.
Comment 9 Josef Moravec 2016-10-18 13:16:10 UTC
(In reply to Radek Šiman (R-Bit Technology, s.r.o.) from comment #8)
> Yes, I agree, that we should sort the texts somehow reasonably. What do you
> think about using my first proposal, ie. sorting like aábcčdďeéě...? This
> seems to me as more accurate approach, even if it's not as defined
> simplified standard rules.

I can't agree with you, it would be more unaccurate approach, because if use the current patches, we are able to perfectly match the primary level of sortng as defined by ČSN 97 6030. But with your approach with doesn't implement anything...

Moreover, the primary level works in most cases and merging primary and secondary levels rules together just ends like a mess in my opinion.
Comment 10 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-18 14:21:35 UTC
Ok, let's use ČSN's primary level only. Continuing the signoff process...
Comment 11 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-20 05:36:28 UTC
Unfortunately it seems, there are some issues to be solved. For instance "Kýbl" and "Kapr" are sorted in incorrect order.
http://prntscr.com/cwj9mj
Comment 12 Josef Moravec 2016-10-20 07:13:07 UTC
Created attachment 56683 [details] [review]
Bug 16952: Czech language definitions for sorting in Zebra

Test plan:
1) Apply patch
2) Create new instance with parameter --zebralang cs
3) Insert some record with basic latin characters and some with "czech" characters (for example: "č" - should be sorted after "c", "š" - should be sorted after "s")
4) Try to search in katalog (staff and opac) and sort by other field then relevance - title or author for instance
5) Records should be sorted correctly by Czech rules
6) Look at code and confirm it is ok
Comment 13 Josef Moravec 2016-10-20 07:19:02 UTC
Thanks for testing!

I've rebased on master and added some mappings.
Comment 14 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-21 13:28:49 UTC
What about this case...?
- Kůň
- Kuna

http://prntscr.com/cx4101

I'm not sure if this is the right order if taking ČSN Level 1 into account. What do you thnk?
Comment 15 Josef Moravec 2016-10-21 14:48:27 UTC
(In reply to Radek Šiman (R-Bit Technology, s.r.o.) from comment #14)
> What about this case...?
> - Kůň
> - Kuna
> 
> http://prntscr.com/cx4101
> 
> I'm not sure if this is the right order if taking ČSN Level 1 into account.
> What do you thnk?

I think it is right order according to this, example 2 and 3:
https://cs.wikipedia.org/wiki/Abecedn%C3%AD_%C5%99azen%C3%AD#P.C5.99.C3.ADklad_.C5.99azen.C3.AD

and this example
- motýl noční
- motýlek
from http://prirucka.ujc.cas.cz/?id=900
Comment 16 Radek Šiman (R-Bit Technology, s.r.o.) 2016-10-22 05:06:38 UTC
Created attachment 56733 [details] [review]
Bug 16952: Czech language definitions for sorting in Zebra

Test plan:
1) Apply patch
2) Create new instance with parameter --zebralang cs
3) Insert some record with basic latin characters and some with "czech" characters (for example: "č" - should be sorted after "c", "š" - should be sorted after "s")
4) Try to search in katalog (staff and opac) and sort by other field then relevance - title or author for instance
5) Records should be sorted correctly by Czech rules
6) Look at code and confirm it is ok

Signed-off-by: radiuscz <radek.siman@centrum.cz>
Comment 17 Jonathan Druart 2016-10-25 08:20:03 UTC
Created attachment 56833 [details] [review]
Bug 16952: Czech language definitions for sorting in Zebra

Test plan:
1) Apply patch
2) Create new instance with parameter --zebralang cs
3) Insert some record with basic latin characters and some with "czech" characters (for example: "č" - should be sorted after "c", "š" - should be sorted after "s")
4) Try to search in katalog (staff and opac) and sort by other field then relevance - title or author for instance
5) Records should be sorted correctly by Czech rules
6) Look at code and confirm it is ok

Signed-off-by: radiuscz <radek.siman@centrum.cz>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
I did not test this patch, but trust in the author and signoffer
Comment 18 Kyle M Hall 2016-10-28 15:33:27 UTC
Pushed to master for 16.11, thanks Josef!
Comment 19 Frédéric Demians 2016-11-02 18:27:54 UTC
Pushed in 16.05. Will be in 16.05.06.