Bug 11927 - Add greek support to CHR
Summary: Add greek support to CHR
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Fridolin SOMERS
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-03-12 11:44 UTC by Fridolin SOMERS
Modified: 2016-06-21 21:37 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 11927 - Add greek chr lang_def file (5.05 KB, patch)
2014-03-12 11:57 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 11927 - Add gr install option (2.71 KB, patch)
2014-03-12 11:57 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Greek ISO2709 records sample (857.99 KB, application/download)
2014-12-04 13:19 UTC, Fridolin SOMERS
Details
Bug 11927 - Add gr install option (2.94 KB, patch)
2014-12-04 14:39 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 11927 - Add greek chr lang_def file (5.37 KB, patch)
2014-12-04 14:39 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 11927 - Add greek to word-phrase-utf.chr (2.39 KB, patch)
2014-12-04 14:40 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 11927 - Small corrections on word-phrase-utf.chr (3.37 KB, patch)
2014-12-04 14:40 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 11927 - Add gr install option (3.00 KB, patch)
2014-12-28 07:14 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11927 - Add greek chr lang_def file (5.43 KB, patch)
2014-12-28 07:15 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11927 - Add greek to word-phrase-utf.chr (2.45 KB, patch)
2014-12-28 07:15 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 11927 - Small corrections on word-phrase-utf.chr (3.42 KB, patch)
2014-12-28 07:15 UTC, Chris Cormack
Details | Diff | Splinter Review
[PASSED QA] Bug 11927 - Add gr install option (3.08 KB, patch)
2015-01-09 12:11 UTC, Kyle M Hall
Details | Diff | Splinter Review
[PASSED QA] Bug 11927 - Add gr install option (3.08 KB, patch)
2015-01-09 12:23 UTC, Kyle M Hall
Details | Diff | Splinter Review
[PASSED QA] Bug 11927 - Add greek chr lang_def file (5.51 KB, patch)
2015-01-09 12:23 UTC, Kyle M Hall
Details | Diff | Splinter Review
[PASSED QA] Bug 11927 - Add greek to word-phrase-utf.chr (2.52 KB, patch)
2015-01-09 12:23 UTC, Kyle M Hall
Details | Diff | Splinter Review
[PASSED QA] Bug 11927 - Small corrections on word-phrase-utf.chr (3.49 KB, patch)
2015-01-09 12:23 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11927 - Add greek support to CHR (followup) (1018 bytes, patch)
2015-01-21 16:25 UTC, Fridolin SOMERS
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin SOMERS 2014-03-12 11:44:49 UTC
Add the sort-string-utf.chr for sorting Greek catalog (it can also contain latin records).

Proposed by Giannis Kourmoulis <ikourmou@lib.auth.gr>.
Comment 1 Fridolin SOMERS 2014-03-12 11:57:26 UTC Comment hidden (obsolete)
Comment 2 Fridolin SOMERS 2014-03-12 11:57:44 UTC Comment hidden (obsolete)
Comment 3 Dobrica Pavlinusic 2014-03-13 10:17:18 UTC
Where can I get a few records with greek alphabet to test this patch?
Comment 4 Fridolin SOMERS 2014-03-19 07:32:20 UTC
I set to discusion because it is not finalized.
I will provide a greek record.
Comment 5 Fridolin SOMERS 2014-12-04 13:19:26 UTC
Created attachment 34118 [details]
Greek ISO2709 records sample
Comment 6 Fridolin SOMERS 2014-12-04 14:39:43 UTC Comment hidden (obsolete)
Comment 7 Fridolin SOMERS 2014-12-04 14:39:53 UTC Comment hidden (obsolete)
Comment 8 Fridolin SOMERS 2014-12-04 14:40:02 UTC Comment hidden (obsolete)
Comment 9 Fridolin SOMERS 2014-12-04 14:40:12 UTC Comment hidden (obsolete)
Comment 10 Fridolin SOMERS 2014-12-04 14:56:49 UTC
I've revised patches and add greek support to word-phrase-utf.chr (used in zebra CHR indexing mode).
It allows to search using greek characters.
In some case, the use of ICU (translitteration) is unwanted because it returns too much results in a mixed greek/latin catalog.

I've set in mapping sigma forms 'Σ' and 'ς' equivalent to 'σ'.
So one may use (even its not correct) 'σ' instead of 'ς'.

Sort sorting purposes, sort-string-utf.chr defines some prefixes that should be ignored during sorting.
For example in English "the, a, an ...".
I've defined a few definite articles (from http://en.wikipedia.org/wiki/Ancient_Greek_grammar#Definite_article).
Comment 11 mathieu saby 2014-12-07 01:11:34 UTC
Kalimera
I studied ancient greek but not modern.

It looks good, but I think your article list are Ancient Greek. I think Modern greek are a bit different (no rough breathing for ex.) and there's a new indefinite article (ένας, μία, ένα). For ex, "A big bottle of water" = ena megalo nero (a very useful sentence in summer...)
http://en.wikipedia.org/wiki/Modern_Greek_grammar#Articles

The rough and soft breathing (esprit rude/doux in french) are no more in use since 1982, but maybe with legacy records it could be interesting to manage them in some way? http://en.wikipedia.org/wiki/Rough_breathing

Mathieu Saby
Comment 12 Zeno Tajoli 2014-12-11 08:00:37 UTC
Only a possibility.
Your problem with ICU are connect with the standard setup of 
etc/zebradb/etc/words-icu.xml
and 
etc/zebradb/etc/phrases-icu.xml
?
Because as know ICU or CHR selected, the file 
etc/zebradb/lang_defs/gr/sort-string-utf.chr  it will be used in any case (for sorting).

So I think it is a good develop to have words-icu.xml and phrases-icu.xml
specifics for same language.
Comment 13 Fridolin SOMERS 2014-12-11 09:22:32 UTC
(In reply to Zeno Tajoli from comment #12)
> Only a possibility.
> Your problem with ICU are connect with the standard setup of 
> etc/zebradb/etc/words-icu.xml
> and 
> etc/zebradb/etc/phrases-icu.xml
> ?
> Because as know ICU or CHR selected, the file 
> etc/zebradb/lang_defs/gr/sort-string-utf.chr  it will be used in any case
> (for sorting).
> 
> So I think it is a good develop to have words-icu.xml and phrases-icu.xml
> specifics for same language.

Nop, the greek must not be transliteration for sort AND search.

The only other solution would be to disable transliteration for greek into ICU configuration, but I've not found howto.
Comment 14 Zeno Tajoli 2014-12-13 08:19:29 UTC
Do you have try to costumize etc/zebradb/etc/words-icu.xml doing tests with
yaz-icu (http://www.indexdata.com/yaz/doc/yaz-icu.html) ?

I think that the first test is to setup <icu_chain locale = "xxx"> 

Bye
Comment 15 Colin Campbell 2014-12-15 10:56:44 UTC
In a straightforward ICU install no transliteration is done. Non latin alphabets can be searched using their own characters. It uses a unicode defined sequence, so a catalogue containing more than one alphabet should be able to be searched in that alphabet.
Comment 16 Chris Cormack 2014-12-28 07:14:52 UTC Comment hidden (obsolete)
Comment 17 Chris Cormack 2014-12-28 07:15:08 UTC Comment hidden (obsolete)
Comment 18 Chris Cormack 2014-12-28 07:15:20 UTC Comment hidden (obsolete)
Comment 19 Chris Cormack 2014-12-28 07:15:48 UTC Comment hidden (obsolete)
Comment 20 Kyle M Hall 2015-01-09 12:11:58 UTC Comment hidden (obsolete)
Comment 21 Kyle M Hall 2015-01-09 12:23:31 UTC
Created attachment 35043 [details] [review]
[PASSED QA] Bug 11927 - Add gr install option

Add greek as lang definition in installer.

Developed in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using "gr" in "Primary language for Zebra indexing"
- check gr is used in etc/zebradb/zebra-*.cfg

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 22 Kyle M Hall 2015-01-09 12:23:42 UTC
Created attachment 35044 [details] [review]
[PASSED QA] Bug 11927 - Add greek chr lang_def file

Add the sort-string-utf.chr for sorting Greek catalog (it can also contain latin records).

Developped in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using "gr" in "Primary language for Zebra indexing"
- Index a greek catalog
- Sort by title and check sorting is correct

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 23 Kyle M Hall 2015-01-09 12:23:45 UTC
Created attachment 35045 [details] [review]
[PASSED QA] Bug 11927 - Add greek to word-phrase-utf.chr

Add greek support in word-phrase-utf.chr for searching in a Greek catalog (it can also contain latin records).

Developped in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using CHR zebra indexing
- Index a greek catalog
- Look for results with mixed uppercase, lowercase and diacritics in title

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 24 Kyle M Hall 2015-01-09 12:23:47 UTC
Created attachment 35046 [details] [review]
[PASSED QA] Bug 11927 - Small corrections on word-phrase-utf.chr

Small fixes :
more space characters : ¡¿
uppercase AE missing in equivalent
some trailling spaces

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 25 Manos PETRIDIS 2015-01-15 14:11:09 UTC
Will the option to use "gr" as "Primary language for Zebra indexing" be available to current (package) installations when they are upgraded to versions incorporating Bug 11927? 
I understand there is no such system preference and any changes will have to be applied manually.
Comment 26 Tomás Cohen Arazi 2015-01-21 14:14:41 UTC
Patches pushed to master.

Thanks Frido!
Comment 27 Fridolin SOMERS 2015-01-21 16:25:28 UTC
Created attachment 35446 [details] [review]
Bug 11927 - Add greek support to CHR (followup)

Small error in word-phrase-utf.chr.
It generates this logs :
17:03:25-21/01 zebraidx(10636) [warn] Map: 'ς' has no mapping
17:03:25-21/01 zebraidx(10636) [warn] duplicate entry for charmap from 'Σ'
Comment 28 Fridolin SOMERS 2015-01-21 16:27:31 UTC
last patch needs SO.
It should fix UT t/db_dependant/Search.t
Comment 29 Robin Sheat 2015-01-21 23:12:42 UTC
(In reply to Manos PETRIDIS from comment #25)
> Will the option to use "gr" as "Primary language for Zebra indexing" be
> available to current (package) installations when they are upgraded to
> versions incorporating Bug 11927? 
> I understand there is no such system preference and any changes will have to
> be applied manually.

It's already available:

  --zebralang lang          Choose the primary language for Zebra indexing. Valid
                            values are en (default), es, fr, gr, nb, ru and uk.

assuming that's the same as what you're talking about.
Comment 30 Tomás Cohen Arazi 2015-01-24 19:26:17 UTC
Followup pushed to master.

Thanks Fridolin!