Bug 11927

Where can I get a few records with greek alphabet to test this patch?

I set to discusion because it is not finalized.
I will provide a greek record.

Created attachment 34118 [details]
Greek ISO2709 records sample

I've revised patches and add greek support to word-phrase-utf.chr (used in zebra CHR indexing mode).
It allows to search using greek characters.
In some case, the use of ICU (translitteration) is unwanted because it returns too much results in a mixed greek/latin catalog.

I've set in mapping sigma forms 'Σ' and 'ς' equivalent to 'σ'.
So one may use (even its not correct) 'σ' instead of 'ς'.

Sort sorting purposes, sort-string-utf.chr defines some prefixes that should be ignored during sorting.
For example in English "the, a, an ...".
I've defined a few definite articles (from http://en.wikipedia.org/wiki/Ancient_Greek_grammar#Definite_article).

Kalimera
I studied ancient greek but not modern.

It looks good, but I think your article list are Ancient Greek. I think Modern greek are a bit different (no rough breathing for ex.) and there's a new indefinite article (ένας, μία, ένα). For ex, "A big bottle of water" = ena megalo nero (a very useful sentence in summer...)
http://en.wikipedia.org/wiki/Modern_Greek_grammar#Articles

The rough and soft breathing (esprit rude/doux in french) are no more in use since 1982, but maybe with legacy records it could be interesting to manage them in some way? http://en.wikipedia.org/wiki/Rough_breathing

Mathieu Saby

Only a possibility.
Your problem with ICU are connect with the standard setup of 
etc/zebradb/etc/words-icu.xml
and 
etc/zebradb/etc/phrases-icu.xml
?
Because as know ICU or CHR selected, the file 
etc/zebradb/lang_defs/gr/sort-string-utf.chr  it will be used in any case (for sorting).

So I think it is a good develop to have words-icu.xml and phrases-icu.xml
specifics for same language.

(In reply to Zeno Tajoli from comment #12)
> Only a possibility.
> Your problem with ICU are connect with the standard setup of 
> etc/zebradb/etc/words-icu.xml
> and 
> etc/zebradb/etc/phrases-icu.xml
> ?
> Because as know ICU or CHR selected, the file 
> etc/zebradb/lang_defs/gr/sort-string-utf.chr  it will be used in any case
> (for sorting).
> 
> So I think it is a good develop to have words-icu.xml and phrases-icu.xml
> specifics for same language.

Nop, the greek must not be transliteration for sort AND search.

The only other solution would be to disable transliteration for greek into ICU configuration, but I've not found howto.

Do you have try to costumize etc/zebradb/etc/words-icu.xml doing tests with
yaz-icu (http://www.indexdata.com/yaz/doc/yaz-icu.html) ?

I think that the first test is to setup <icu_chain locale = "xxx"> 

Bye

In a straightforward ICU install no transliteration is done. Non latin alphabets can be searched using their own characters. It uses a unicode defined sequence, so a catalogue containing more than one alphabet should be able to be searched in that alphabet.

Created attachment 35043 [details] [review]
[PASSED QA] Bug 11927 - Add gr install option

Add greek as lang definition in installer.

Developed in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using "gr" in "Primary language for Zebra indexing"
- check gr is used in etc/zebradb/zebra-*.cfg

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Created attachment 35044 [details] [review]
[PASSED QA] Bug 11927 - Add greek chr lang_def file

Add the sort-string-utf.chr for sorting Greek catalog (it can also contain latin records).

Developped in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using "gr" in "Primary language for Zebra indexing"
- Index a greek catalog
- Sort by title and check sorting is correct

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Created attachment 35045 [details] [review]
[PASSED QA] Bug 11927 - Add greek to word-phrase-utf.chr

Add greek support in word-phrase-utf.chr for searching in a Greek catalog (it can also contain latin records).

Developped in collaboration with Giannis Kourmoulis <ikourmou@lib.auth.gr>

Test plan :
- Install using CHR zebra indexing
- Index a greek catalog
- Look for results with mixed uppercase, lowercase and diacritics in title

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Created attachment 35046 [details] [review]
[PASSED QA] Bug 11927 - Small corrections on word-phrase-utf.chr

Small fixes :
more space characters : ¡¿
uppercase AE missing in equivalent
some trailling spaces

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Will the option to use "gr" as "Primary language for Zebra indexing" be available to current (package) installations when they are upgraded to versions incorporating Bug 11927? 
I understand there is no such system preference and any changes will have to be applied manually.

Patches pushed to master.

Thanks Frido!

Created attachment 35446 [details] [review]
Bug 11927 - Add greek support to CHR (followup)

Small error in word-phrase-utf.chr.
It generates this logs :
17:03:25-21/01 zebraidx(10636) [warn] Map: 'ς' has no mapping
17:03:25-21/01 zebraidx(10636) [warn] duplicate entry for charmap from 'Σ'

last patch needs SO.
It should fix UT t/db_dependant/Search.t

(In reply to Manos PETRIDIS from comment #25)
> Will the option to use "gr" as "Primary language for Zebra indexing" be
> available to current (package) installations when they are upgraded to
> versions incorporating Bug 11927? 
> I understand there is no such system preference and any changes will have to
> be applied manually.

It's already available:

  --zebralang lang          Choose the primary language for Zebra indexing. Valid
                            values are en (default), es, fr, gr, nb, ru and uk.

assuming that's the same as what you're talking about.

Followup pushed to master.

Thanks Fridolin!