Bug 21357 - Filter elisions from index and search terms in Elasticsearch
Summary: Filter elisions from index and search terms in Elasticsearch
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Ere Maijala
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-09-15 20:27 UTC by Ere Maijala
Modified: 2020-03-20 18:21 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 21357: Add elision filtering to default ES index config (2.47 KB, patch)
2018-09-15 20:40 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 21357: Add elision filtering to default ES index config (2.61 KB, patch)
2018-09-25 14:01 UTC, Séverine Queune
Details | Diff | Splinter Review
french ES index config (1.84 KB, application/x-yaml)
2018-09-28 10:32 UTC, Fridolin SOMERS
Details
Bug 21357: Add elision filtering to default ES index config (2.66 KB, patch)
2019-03-15 12:46 UTC, Björn Nylén
Details | Diff | Splinter Review
Bug 21357: Add elision filtering to default ES index config (2.66 KB, patch)
2019-04-12 12:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 21357: Add uppercase articles to the elision filter (1.03 KB, patch)
2019-04-12 12:45 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 21357: Case-insensive articles to the elision filter (899 bytes, patch)
2019-05-29 15:43 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 21357: Allow to use multiple ES language analyzers (7.03 KB, patch)
2020-01-30 15:11 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 21357: Add elision filtering to default ES index config (2.76 KB, patch)
2020-01-31 20:25 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 21357: Case-insensive articles to the elision filter (948 bytes, patch)
2020-01-31 20:25 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 21357: Allow to use multiple ES language analyzers (7.09 KB, patch)
2020-01-31 20:25 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 21357: Add elision filtering to default ES index config (2.81 KB, patch)
2020-03-20 18:06 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 21357: Case-insensive articles to the elision filter (1000 bytes, patch)
2020-03-20 18:06 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 21357: Allow to use multiple ES language analyzers (7.13 KB, patch)
2020-03-20 18:06 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 19482 - DB changes (2.04 KB, patch)
2020-03-20 18:19 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 19482 - Add support for defining 'mandatory' mappings (22.86 KB, patch)
2020-03-20 18:19 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 19482: SCHEMA CHANGES _ DO NOT PUSH (1.59 KB, patch)
2020-03-20 18:19 UTC, Bouzid Fergani
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Ere Maijala 2018-09-15 20:27:52 UTC
To make searching easier at least in French, it would be nice to have the default configuration handle elisions properly (e.g. l'avion). Coming up with a proposed patch.
Comment 1 Ere Maijala 2018-09-15 20:40:24 UTC
Created attachment 78889 [details] [review]
Bug 21357: Add elision filtering to default ES index config

To test:

1. Rebuild Elasticsearch index with the new config
2. Add a record with "l'avion" in the title
3. Verify that the record can be found with both "l'avion" and "avion"
Comment 2 Séverine Queune 2018-09-25 14:01:12 UTC
Created attachment 79375 [details] [review]
Bug 21357: Add elision filtering to default ES index config

To test:

1. Rebuild Elasticsearch index with the new config
2. Add a record with "l'avion" in the title
3. Verify that the record can be found with both "l'avion" and "avion"

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Comment 3 Fridolin SOMERS 2018-09-27 14:57:34 UTC
Hi, bravo.

But its really specific to french.

I set in discussion because I think we should build a french-specific configuration, based on :
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-lang-analyzer.html#french-analyzer
This adds french stemming and stop words.

Since one can define a path for configuration in koha-conf.xml, what about creating a lang-def folder in etc ?
Comment 4 Katrin Fischer 2018-09-27 15:05:55 UTC
I think a lot of libraries have French materials, is it specific to French catalouging or just to data?
Comment 5 Ere Maijala 2018-09-28 07:51:17 UTC
If it's decided this shouldn't be part of the default rules, we could perhaps still include these as a commented-out example, right?
Comment 6 Fridolin SOMERS 2018-09-28 10:29:42 UTC
(In reply to Ere Maijala from comment #5)
> If it's decided this shouldn't be part of the default rules, we could
> perhaps still include these as a commented-out example, right?

I'd prefer to propose an entire french-specific file.
I attach the one I'm working on.
Comment 7 Fridolin SOMERS 2018-09-28 10:32:07 UTC
Created attachment 79564 [details]
french ES index config

We could add this in etc/elasticsearch/fr or etc/searchengine/fr ?
Comment 8 Katrin Fischer 2018-09-28 11:46:15 UTC
(In reply to Fridolin SOMERS from comment #6)
> (In reply to Ere Maijala from comment #5)
> > If it's decided this shouldn't be part of the default rules, we could
> > perhaps still include these as a commented-out example, right?
> 
> I'd prefer to propose an entire french-specific file.
> I attach the one I'm working on.

Are we sure those rules would not be helpful for other installations as well? I can understand if it's specific to UNIMARC, but otherwise it might be more helpful to develop good defaults together.

At the moment we have the problem with Zebra and the sort files - some language specific sort files are really good and others have only minimal information. If you have a library with multiple languages (we do) one file that fits multiple would be better.
Comment 9 Fridolin SOMERS 2018-09-28 11:54:59 UTC
(In reply to Katrin Fischer from comment #8)
> (In reply to Fridolin SOMERS from comment #6)
> > (In reply to Ere Maijala from comment #5)
> > > If it's decided this shouldn't be part of the default rules, we could
> > > perhaps still include these as a commented-out example, right?
> > 
> > I'd prefer to propose an entire french-specific file.
> > I attach the one I'm working on.
> 
> Are we sure those rules would not be helpful for other installations as
> well? I can understand if it's specific to UNIMARC, but otherwise it might
> be more helpful to develop good defaults together.
> 
> At the moment we have the problem with Zebra and the sort files - some
> language specific sort files are really good and others have only minimal
> information. If you have a library with multiple languages (we do) one file
> that fits multiple would be better.

Stemming for example is very language specific.
Stop words even more, for example "or" means gold in french.

But I was thinking that an install may merge several language (elisions, stemmers and stop words) files to index for example a French+German catalog.
Comment 10 Katrin Fischer 2018-09-28 12:18:58 UTC
I think ideally things shoudl be as easy as possible. I think all of our libraries have materials in various languages - the Goethe institute libraires being a prominent example. I don't want to run each of them with a different Elasticsearch configuration if possible.
Comment 11 Fridolin SOMERS 2018-09-28 12:32:50 UTC
(In reply to Katrin Fischer from comment #10)
> I think ideally things shoudl be as easy as possible. I think all of our
> libraries have materials in various languages - the Goethe institute
> libraires being a prominent example. I don't want to run each of them with a
> different Elasticsearch configuration if possible.

Me too, but I fear side-effects.
Like stopwords, its very useful to have them language-specific, but they are real words in other languages.

Waiting for other opinions.
Or if we want this bug to focus on elision I'm OK, I will open another bug.
Comment 12 Katrin Fischer 2018-09-28 12:56:51 UTC
Serious question: Why would we want stop words?
Comment 13 Katrin Fischer 2018-09-28 12:58:54 UTC
(In reply to Katrin Fischer from comment #12)
> Serious question: Why would we want stop words?

To explain: to me stop words seem to stem from a time where search would die searching for something like "the" - I think modern search engines are better than that and we have always been happy that you can find the band "the the" with Koha and haven't had stop words since moving to Zebra.

For stemming: I think stemming is a bit difficult, but could we not make it depend on the language of the interface? Ideally it should depend on the language you enter something in (if I use a German GUI and search a French book...)... or could we stem for mulitple languages at once?
Comment 14 Nicolas Legrand 2018-09-28 13:44:38 UTC
Ay up friends,

We are a language library and we have more than 350 different languages and lots of writing systems. Our interface is only in French (the language of our country) or in English (the science's new lingua franca). I we had installed other languages, we should have installed all languages available, which even today seems overkill, while everyone seems OK with French and English. Therefore, having stemming based on the interface language doesn't look a good option for us.

OK maybe we're weird, nonetheless I think most University libraries have books in a lot of different languages and not just one or two.

With our context, stemming itself seems quite trying. We aren't yet eager about it.

So before having the full stemmed engine picky about which language is what, what have done Ere (thank you !) will certainly suit us. If it is not generic enough to be in Koha... Well we will hack it on our instance.
Comment 15 Katrin Fischer 2018-09-28 13:58:07 UTC
> OK maybe we're weird, nonetheless I think most University libraries have
> books in a lot of different languages and not just one or two.

Exactly my point too - you are not weird!

Not everyone can hack their installations. I just wanted to make sure that we don't start adding separate configurations when we just would be better of with one standard file for now.
Comment 16 Fridolin SOMERS 2018-10-01 06:20:17 UTC
Ok I will propose the stemming configuration on wiki page.

This patch is OK, but for me the "apostrophe" filter is not needed.

I've checked with analyse API on ES :
curl -X GET 'http://localhost:9200/koha_master_biblios/_analyze?pretty=yes&analyzer=analyser_phrase' -d "dans l'avion"

With patch it returns :    "token" : "dans lavion"
Without patch it returns : "token" : "dans l'avion"

Checking with only "l'avion" both with and without patch returns "avion".
Comment 17 Ere Maijala 2018-10-01 07:39:25 UTC
The apostrophe filter is for the other cases where an apostrophe is not a part of an elision. It's not stripped in the char filter like other punctuation so that the elision filter can do its job but needs to be done afterwards. Try analysis for e.g. "Merriam-Webster's Collegiate Dictionary".

I agree that this bug should only be about elisions. Stop words are another thing completely, and there are strong reasons for not using a stop word list like Katrin described.
Comment 18 Fridolin SOMERS 2018-11-05 11:59:17 UTC
(In reply to Ere Maijala from comment #17)
> The apostrophe filter is for the other cases where an apostrophe is not a
> part of an elision. It's not stripped in the char filter like other
> punctuation so that the elision filter can do its job but needs to be done
> afterwards. Try analysis for e.g. "Merriam-Webster's Collegiate Dictionary".
> 
> I agree that this bug should only be about elisions. Stop words are another
> thing completely, and there are strong reasons for not using a stop word
> list like Katrin described.

I think we should use ICU transformation for that.
I will try to find the syntax.
Comment 19 Ere Maijala 2018-12-07 08:31:36 UTC
I'd like to either move this forward. I think elision handling is important even if the language used does not use them since they could be used in names etc.
Comment 20 Björn Nylén 2019-03-15 12:46:05 UTC
Created attachment 86670 [details] [review]
Bug 21357: Add elision filtering to default ES index config

To test:

1. Rebuild Elasticsearch index with the new config
2. Add a record with "l'avion" in the title
3. Verify that the record can be found with both "l'avion" and "avion"

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Comment 21 Björn Nylén 2019-03-15 12:51:47 UTC
We signed off this bug as we think it's a good feature as is. We just recently had a coworker noting this behaviour.
Comment 22 Nicolas Legrand 2019-04-08 15:28:24 UTC
This is a most awaited feature for us. Note it doesn't work for « L'avion ». So we need maybe something like this :

diff --git a/admin/searchengine/elasticsearch/index_config.yaml b/admin/searchengine/elasticsearch/index_config.yaml
index f47fb22..05d89a9 100644
--- a/admin/searchengine/elasticsearch/index_config.yaml
+++ b/admin/searchengine/elasticsearch/index_config.yaml
@@ -54,6 +54,12 @@ index:
           - 'qu'
           - 's'
           - 't'
+          - 'C'
+          - 'D'
+          - 'J'
+          - 'L'
+          - 'QU'
+          - 'T'
       apostrophe:
         type: pattern_replace
         pattern: "'"
Comment 23 Ere Maijala 2019-04-12 12:45:34 UTC
Created attachment 87901 [details] [review]
Bug 21357: Add elision filtering to default ES index config

To test:

1. Rebuild Elasticsearch index with the new config
2. Add a record with "l'avion" in the title
3. Verify that the record can be found with both "l'avion" and "avion"

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Comment 24 Ere Maijala 2019-04-12 12:45:38 UTC
Created attachment 87902 [details] [review]
Bug 21357: Add uppercase articles to the elision filter
Comment 25 Ere Maijala 2019-04-12 12:46:28 UTC
Right, thanks for pointing that out. I added uppercase versions for all articles.
Comment 26 Séverine Queune 2019-04-15 14:06:33 UTC
Hi Ere,

Everything works fine for the single letters + apostrophe for both upper and lower case, but it doesn't work for the double letters QU : here the title of my notice : Qu'ils me laissent tranquille !
If I search :
  - QU'ILS --> failed
  - Qu'ils --> ok
  - qu'ils --> failed (same list of result I had for QU'ILS)
  - qU'ILS --> ok

I created another exemple and had no problem due to uper or lower case :
  - Aujourd'hui --> ok
  - AUJOURD'HUI --> ok
  - AUJOURd'HUI --> ok

As the 'D' and 'd' is on your list, could my first exemple behave correctly if you add only 'U' and 'u' on the settings ?
Comment 27 Fridolin SOMERS 2019-05-29 15:07:09 UTC
(In reply to Ere Maijala from comment #24)
> Created attachment 87902 [details] [review] [review]
> Bug 21357: Add uppercase articles to the elision filter

I think the correct way is using : 
"articles_case": true

From https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-lang-analyzer.html#french-analyzer
Comment 28 Fridolin SOMERS 2019-05-29 15:39:01 UTC
At last I'm going to work on this again.
I understand way more Elasticsearch now :D

I know now that we can use french elision on a non-french catalog, even with an unwanted impact the ranking will give priority to correct match.
So we can provide this in default configuration.

Thanks to all for your existing participation on this bug.
Comment 29 Fridolin SOMERS 2019-05-29 15:43:49 UTC
Created attachment 90191 [details] [review]
Bug 21357: Case-insensive articles to the elision filter

Test with :
GET index/_analyze
{
  "filter": ["elision"],
  "text": "qu'ils"
}

With different cases :
QU'ILS
Qu'ils
qu'ils
qU'ILS
Comment 30 Fridolin SOMERS 2019-05-29 15:50:07 UTC
Ohhh playing with ES API with explain analyze :
https://www.elastic.co/guide/en/elasticsearch/reference/current/_explain_analyze.html

I see that elision is only with the begining of the token.
It explains a lot on analyzer_phrase (comment 16)

I continue my tests.
Comment 31 Séverine Queune 2019-09-19 13:45:39 UTC
(In reply to Fridolin SOMERS from comment #30)
> Ohhh playing with ES API with explain analyze :
> https://www.elastic.co/guide/en/elasticsearch/reference/current/
> _explain_analyze.html
> 
> I see that elision is only with the begining of the token.
> It explains a lot on analyzer_phrase (comment 16)
> 
> I continue my tests.

Hey Frido !
Ok for you to test again during Hackfest ?
Comment 32 Julian Maurice 2020-01-30 15:10:40 UTC
I think that enabling french analyzer (or parts of french analyzer) by default is not the best solution. Sure it will work with french catalogs, but what will happen when one want to add support for another language (or, for the weird people above, 350 other languages! :)) ?

I think a better solution is to use multi-fields with multiple analyzers, as suggested here : https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html#_multi_fields_with_multiple_analyzers

The idea is to have multiple "subfields" for each field, one subfield per language. For instance:
- `title` will be language agnostic
- `title.lang_en` will use an english analyzer
- `title.lang_fr` will use a french analyzer

When searching, Koha can search in all these fields, and the query will be analyzed by all analyzers.

I've been testing this idea this week, and I made a small patch that I will submit here. It is only a proof of concept for now, but I hope it will help
Comment 33 Julian Maurice 2020-01-30 15:11:13 UTC
Created attachment 98169 [details] [review]
Bug 21357: Allow to use multiple ES language analyzers

Test plan:
1. Apply this patch
2. If you have a custom `field_config.yaml` report the diff in your
custom config
3. Create biblios in english and french
4. misc/search_tools/rebuild_elasticsearch.pl
5. Test biblio search at intranet with stemming and ellision in mind

Examples of test queries:
- "journal actualité" => should return results with "Journaux
d'actualités" in it
- "lord lady" => should return results with "Lords and Ladies" in it
Comment 34 Julian Maurice 2020-01-30 15:12:02 UTC
Changing status to needs signoff to get opinions
Comment 35 Katrin Fischer 2020-01-30 21:49:03 UTC
Hi Julian, does it mean it searches the different representations simultanously? 
I am still not quite sure about 'French specific'. 

It feels like every catalog containing some French materials would be happy to have "l'avion" work the way you describe - we would. In my experience most catalogs contain records in different languages.

I wonder how it would work for English, thinking of words like "can't" or "doesn't".
Comment 36 Ere Maijala 2020-01-31 07:13:44 UTC
I can't really see the benefit since, as far as I can see, elision handling is not prone to cause conflicts with other language analysis. Separating analysis for different languages also won't work for mixed-language fields. Think about names and a (very fictional) example phrase "Images from movie l'Avion". You'd get either elision filtering or English stemming but not both. For sure it will still be found with a simple keyword search, but it breaks at least adjacent word searches and relevance ranking.
Comment 37 Julian Maurice 2020-01-31 09:02:09 UTC
(In reply to Katrin Fischer from comment #35)
> Hi Julian, does tit mean it searches the different representations
> simultanously? 

Only one query to ES is needed, if that's what you mean by simultaneously
 
> I wonder how it would work for English, thinking of words like "can't" or
> "doesn't".

The built-in english analyzer does not do anything with words that end with "n't" but it should possible to configure a custom english analyzer that treats "can't" and "cannot" the same way.

(In reply to Ere Maijala from comment #36)
> I can't really see the benefit since, as far as I can see, elision handling
> is not prone to cause conflicts with other language analysis.

Ellision might not cause troubles (but what about names like "D'Amato" ?).
I'm thinking about the next step : stemming is very different from one language to another and we need to find a way to have stemming for multi-language catalogs.

> Separating analysis for different languages also won't work for
> mixed-language fields.  Think about names and a (very fictional)
> example phrase "Images from movie l'Avion". You'd get either elision
> filtering or English stemming but not both. For sure it will still be
> found with a simple keyword search, but it breaks at least adjacent
> word searches and relevance ranking.

Nothing would work perfectly with mixed-language fields. But in this particular example, you could have another subfield `lang_en_fr` that does english stemming and french elision
Comment 38 Ere Maijala 2020-01-31 09:14:26 UTC
(In reply to Julian Maurice from comment #37)
> Ellision might not cause troubles (but what about names like "D'Amato" ?).
> I'm thinking about the next step : stemming is very different from one
> language to another and we need to find a way to have stemming for
> multi-language catalogs.

I agree that stemming is difficult, and I'e tried to purposefully limit this bug to elisions. 

> > Separating analysis for different languages also won't work for
> > mixed-language fields.  Think about names and a (very fictional)
> > example phrase "Images from movie l'Avion". You'd get either elision
> > filtering or English stemming but not both. For sure it will still be
> > found with a simple keyword search, but it breaks at least adjacent
> > word searches and relevance ranking.
> 
> Nothing would work perfectly with mixed-language fields. But in this
> particular example, you could have another subfield `lang_en_fr` that does
> english stemming and french elision

And maybe lang_fi_fr, lang_sv_fr etc. This gets complicated pretty quickly. And I'm afraid separating the different language analysis chains doesn't solve the issue with stemming etc. because you'd need to avoid indexing into "wrong" fields which would require you to know what language the string to be indexed is in.
Comment 39 Julian Maurice 2020-01-31 09:39:33 UTC
> because you'd need to avoid indexing into "wrong" fields which would
> require you to know what language the string to be indexed is in.

I don't think you have to avoid indexing into the wrong fields (but I agree it would be better). You just need to have at least one that match.
You might end up with false positives (more results that you should have) but in my opinion it's still better than not having the results you expected.

I realize that stemming is out of the scope of this bug, but even if we talk only about ellision, I think it would be better to have one field analyzed with ellision and one field without ellision, and to search in both fields every time
Comment 40 Nick Clemens 2020-01-31 11:09:29 UTC
(In reply to Julian Maurice from comment #39)

What about using a record field to determine the language of the record and applying different analyser settings based on that? 008/35-37 or 041 in marc21, I don't know if there is a UNIMARC equivalent
Comment 41 Julian Maurice 2020-01-31 11:21:57 UTC
(In reply to Nick Clemens from comment #40)
> (In reply to Julian Maurice from comment #39)
> 
> What about using a record field to determine the language of the record and
> applying different analyser settings based on that? 008/35-37 or 041 in
> marc21, I don't know if there is a UNIMARC equivalent

In UNIMARC there is field 101 that could be used for that.
Comment 42 Ere Maijala 2020-01-31 12:34:56 UTC
At least with MARC 21 that's the language of the catalogued item, not the language of the metadata record, right? We've been through this with our discovery interface and had to give up trying to guess the language. Additionally, a record can contain metadata fields in multiple languages.
Comment 43 Nick Clemens 2020-01-31 13:07:42 UTC
(In reply to Ere Maijala from comment #42)
> At least with MARC 21 that's the language of the catalogued item, not the
> language of the metadata record, right? We've been through this with our
> discovery interface and had to give up trying to guess the language.
> Additionally, a record can contain metadata fields in multiple languages.

True, but for at least certain fields like 'title' shouldn't the language match? i.e. the french version should have the french title and the english version the english title?

Maybe the problem here is 'MARC is bad' :-)
Comment 44 Nicolas Legrand 2020-01-31 14:42:42 UTC
Julian, for non latin scripts we are doing a latin transliteration and so are suppose to do all the ABES partners in France. Look at 

https://catalogue.bulac.fr/cgi-bin/koha/opac-detail.pl?biblionumber=976426 http://www.sudoc.fr/18117295X

If I check the language here, it's Hebrew. I've got a 200$a with a Hebrew title in hebrew and another one with a Hebrew title in Latin alphabet.

To know what to do, I have to look at 200$6 and 200$7 which are ABES non standard ways to describe non latin scripts (I like to think we are not using UNIMARC but ABESMARC).

http://documentation.abes.fr/sudoc/regles/Catalogage/Regles_Multiecritures.htm#Ss-zones$6$7

Voilà. Don't know what to think yet, but now you know.
Comment 45 Michal Denar 2020-01-31 20:25:01 UTC
Created attachment 98244 [details] [review]
Bug 21357: Add elision filtering to default ES index config

To test:

1. Rebuild Elasticsearch index with the new config
2. Add a record with "l'avion" in the title
3. Verify that the record can be found with both "l'avion" and "avion"

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 46 Michal Denar 2020-01-31 20:25:06 UTC
Created attachment 98245 [details] [review]
Bug 21357: Case-insensive articles to the elision filter

Test with :
GET index/_analyze
{
  "filter": ["elision"],
  "text": "qu'ils"
}

With different cases :
QU'ILS
Qu'ils
qu'ils
qU'ILS

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 47 Michal Denar 2020-01-31 20:25:11 UTC
Created attachment 98246 [details] [review]
Bug 21357: Allow to use multiple ES language analyzers

Test plan:
1. Apply this patch
2. If you have a custom `field_config.yaml` report the diff in your
custom config
3. Create biblios in english and french
4. misc/search_tools/rebuild_elasticsearch.pl
5. Test biblio search at intranet with stemming and ellision in mind

Examples of test queries:
- "journal actualité" => should return results with "Journaux
d'actualités" in it
- "lord lady" => should return results with "Lords and Ladies" in it

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 48 Bouzid Fergani 2020-03-20 18:06:33 UTC
Created attachment 101305 [details] [review]
Bug 21357: Add elision filtering to default ES index config

To test:

1. Rebuild Elasticsearch index with the new config
2. Add a record with "l'avion" in the title
3. Verify that the record can be found with both "l'avion" and "avion"

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 49 Bouzid Fergani 2020-03-20 18:06:39 UTC
Created attachment 101306 [details] [review]
Bug 21357: Case-insensive articles to the elision filter

Test with :
GET index/_analyze
{
  "filter": ["elision"],
  "text": "qu'ils"
}

With different cases :
QU'ILS
Qu'ils
qu'ils
qU'ILS

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 50 Bouzid Fergani 2020-03-20 18:06:46 UTC
Created attachment 101307 [details] [review]
Bug 21357: Allow to use multiple ES language analyzers

Test plan:
1. Apply this patch
2. If you have a custom `field_config.yaml` report the diff in your
custom config
3. Create biblios in english and french
4. misc/search_tools/rebuild_elasticsearch.pl
5. Test biblio search at intranet with stemming and ellision in mind

Examples of test queries:
- "journal actualité" => should return results with "Journaux
d'actualités" in it
- "lord lady" => should return results with "Lords and Ladies" in it

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 51 Bouzid Fergani 2020-03-20 18:19:05 UTC Comment hidden (obsolete)
Comment 52 Bouzid Fergani 2020-03-20 18:19:14 UTC Comment hidden (obsolete)
Comment 53 Bouzid Fergani 2020-03-20 18:19:20 UTC Comment hidden (obsolete)