Bug 27153 - ElasticSearch should search keywords apostrophe blind
Summary: ElasticSearch should search keywords apostrophe blind
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal with 30 votes (vote)
Assignee: Nick Clemens
QA Contact: Martin Renvoize
URL:
Keywords: release-notes-needed, rel_23_05_candidate, rel_23_11_candidate
Depends on:
Blocks:
 
Reported: 2020-12-04 14:55 UTC by Jonathan Field
Modified: 2024-04-22 12:06 UTC (History)
12 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.11.00


Attachments
Bug 27153: Default to apostrophe blind search (1.18 KB, patch)
2020-12-07 16:24 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 27153: POC (1.87 KB, patch)
2022-08-29 22:34 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 27153: DB Updates (2.54 KB, patch)
2023-09-01 17:57 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 27153: DO NOT PUSH: Schema updates (1.66 KB, patch)
2023-09-01 17:57 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 27153: Interface updates (8.86 KB, patch)
2023-09-01 17:57 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 27153: Add filter option to Elasticsearch indexing (12.73 KB, patch)
2023-09-01 17:57 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 27153: Adjust mappings to add title without punctuation (2.01 KB, patch)
2023-09-01 17:57 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 27153: Adjust kohastructure for new primary key (1.39 KB, patch)
2023-09-05 17:26 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 27153: Adjust kohastructure for new primary key (1.46 KB, patch)
2023-09-11 16:17 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: DO NOT PUSH: Schema updates (1.72 KB, patch)
2023-09-11 17:55 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Interface updates (8.93 KB, patch)
2023-09-11 17:55 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Add filter option to Elasticsearch indexing (12.79 KB, patch)
2023-09-11 17:55 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Adjust mappings to add title without punctuation (2.07 KB, patch)
2023-09-11 17:56 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: DO NOT PUSH: Schema updates (1.72 KB, patch)
2023-09-11 17:56 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Adjust kohastructure for new primary key (1.46 KB, patch)
2023-09-11 17:56 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Interface updates (8.93 KB, patch)
2023-09-11 17:56 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Add filter option to Elasticsearch indexing (12.79 KB, patch)
2023-09-11 17:56 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Adjust mappings to add title without punctuation (2.07 KB, patch)
2023-09-11 17:56 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: Adjust kohastructure for new primary key (1.46 KB, patch)
2023-09-11 17:56 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 27153: DB Updates (2.61 KB, patch)
2023-10-24 11:37 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 27153: DO NOT PUSH: Schema updates (1.79 KB, patch)
2023-10-24 11:37 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 27153: Interface updates (8.96 KB, patch)
2023-10-24 11:37 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 27153: Add filter option to Elasticsearch indexing (12.86 KB, patch)
2023-10-24 11:37 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 27153: Adjust mappings to add title without punctuation (2.14 KB, patch)
2023-10-24 11:37 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 27153: Adjust kohastructure for new primary key (1.52 KB, patch)
2023-10-24 11:37 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Field 2020-12-04 14:55:54 UTC
We are often asked why keyword searches don't ignore apostrophes. 

If you searched for "carries war" on Google, the first hits that come up are for Carrie's War by Nina Bawden. 

If you do a keyword search in Koha for "carries war" you get 0 hits. If you search for "carrie's war" you will find copies of this book.

Koha with ElasticSearch should search apostrophe-blind (ignore the apostrophe).  

This was a perennial problem with zebra. In zebra you had to change words-icu.xml to ignore apostrophes.

We should deliver an ES index_config.yaml that handles this scenario.
Comment 1 Katrin Fischer 2020-12-06 13:00:10 UTC
Hi Jonathan, I believe this to be a duplicate of bug 21357 - maybe you could add your voice to that bug for a simplified solution (ignore ')

*** This bug has been marked as a duplicate of bug 21357 ***
Comment 2 Martin Renvoize 2020-12-07 16:24:05 UTC
Created attachment 114234 [details] [review]
Bug 27153: Default to apostrophe blind search

Keyword searching in elasticsearch should be apostrophe blind; this
patch updates the default configuration to filter apostrophe's.
Comment 3 Martin Renvoize 2020-12-07 16:29:03 UTC
In my understanding elisions and apostrophes are not equal and as such distinct bugs can/should be opened to handle the two cases.

This case is for apostrophes only and leaves elisions, which appear to be more difficult to reach a consensus, to the other bug.

As such, I don't think this is a duplicate at all.. I've uploaded Jonathans patch.
Comment 4 Katrin Fischer 2020-12-09 23:20:01 UTC
I don't mind a simple solution, but can you explain the difference?
Comment 5 Nick Clemens 2020-12-10 11:06:04 UTC
With this simple apostrophe filter:
Dont let the pigeon drive the batmobile
will find:
Don't let the pigeon drive the batmobile
but, you get no results for:
ti:"Don't let the pigeon drive the batmobile"

I like the idea of a simple solution, but I am not sure it works for everyone. While elisions are different than contractions I believe they both use an apostrophe, so this will affect both.

We are planning on testing the 'english' analyzer with one of our partners to help with this issue, I suspect that this may not have a one-size-fits-all solution - but will require the user to decide on which configuration works best in their library
Comment 6 Fridolin Somers 2020-12-10 16:30:17 UTC
(In reply to Katrin Fischer from comment #4)
> I don't mind a simple solution, but can you explain the difference?

Elision removes the text : l'europe = europe
This apostrophe filter creates : l'europe = leurope

Wrong in french but surely ranking will bet better for the exact match.

I have to test combined with ellision.

But :
Maybe it is bad for authorities : "Pierre d'Anjou" and Pierre Danjou".
Comment 7 Katrin Fischer 2020-12-12 10:52:49 UTC
Could we treat it ' as a space and 'no space' at the same time maybe? 
And how do we currently handle this with Zebra?

I guess we'd want

Don't
Dont
don t
Pierre d'Anjou
Pierre dAnjou
Pierre d Anjou

to be the same?
Comment 8 Barbara Johnson 2021-08-13 16:14:24 UTC
This is a constant problem for our patrons who are used to searching Amazon or Google without having to worry about apostrophes.  Fixing this issue would really improve the ability for our patrons to find items in our catalog.  The status of this bug is listed as Needs Signoff.  Is there a test plan?
Comment 9 Fridolin Somers 2021-08-13 20:36:47 UTC
Actually Zebra as a rule in words-icu.xml :
  <transliterate rule="\'>\ "/>
But it may not be optimal to do the exact same thing in Elasticsearch.
Comment 10 David Cook 2021-08-15 23:47:52 UTC
It might be a good idea to harmonize the normalization used between Zebra and Elasticsearch just for consistency...
Comment 11 Fridolin Somers 2021-08-16 19:35:57 UTC
(In reply to David Cook from comment #10)
> It might be a good idea to harmonize the normalization used between Zebra
> and Elasticsearch just for consistency...

+1

So we just need to change replacement with a space character ?

This will require a unit test I bet.
Comment 12 Jennifer Millikan 2022-08-29 19:35:52 UTC
I am brand new to Koha, but I was told this bug is why I have to include apostrophes while searching. I tried searching for Don't Hate the Player title and came up with zero results. I did not use the apostrophe because I have never had to before in multiple catalog systems. I repeated the search with the apostrophe and found the title. I agree with this and think the searching should work both ways with or without the apostrophe. Thank you.
Comment 13 Nick Clemens 2022-08-29 22:34:59 UTC
Created attachment 139969 [details] [review]
Bug 27153: POC

This is a small example to show that we can add an extra field to our default indexing that
removes punctuation.

If we then add that field to the fields we push, we can sarch both the regular and the punctuation stripped version

To complete this we would need to add a 'search variations' field to the ES config page, for the user to decide which
fields should be searched in all variations, and which shouldn't

Or we could add this to all 'simple searches' from the search bar, and only search exact on advanced searches.

I am happy to explore either option
Comment 14 David Cook 2022-08-30 00:14:08 UTC
I had a librarian (from an English-only library) asking for this the other day, and I was wondering why we have apostrophes replaced with a space (both for CHR and ICU indexing with Zebra).

But Frido's example makes a lot of sense:

(In reply to Fridolin Somers from comment #6)
> Elision removes the text : l'europe = europe
> This apostrophe filter creates : l'europe = leurope
> 
> Wrong in french but surely ranking will bet better for the exact match.
 
And in English, we'd have "father's" become "fathers", which means "father" wouldn't match unless you're using right truncation (which Koha typically does out of the box I suppose). 

But then there's a Ukrainian word під'їзд which if you broke it into під їзд, you'd get hits for під which is a totally unrelated word.  

--

I know people have praised Google here, but it's not perfect either. 

Try searching "l'arbre під'їзд" and try searching "під'їзд l'arbre", and you'll get wildly different results. 

It seems that Google tries to determine the language of the search query (possibly based off the first word), and then analyze the search string based off that. 

For "l'arbre під'їзд", you get many results with "L'Arbre", "des arbres", "arbre", etc. Most results seem to include Russian rather than Ukrainian. 

But for "під'їзд l'arbre", you only get a few results that have the string "L'Arbre"and most of the results are Ukrainian. 

So Google is likely determining the language of the search string and then applying a language-specific analyzer. 

"My mom" in Chinese is 我的妈妈. 的 is the character that denotes possession of mom (妈妈) by me/I (我). You might think then you could replace 的 with a space to separate the 2 nouns... except 的 isn't always used that way. The word 目的 means "goal". Doing anything to 的 would compromise the word, unless you're able to understand the context that it's used in. 

If you google "我的妈妈", eventually it strips off "我的" and just searches for "妈妈" since that's the main noun in the phrase. 

Search is hard.
Comment 15 David Cook 2022-08-30 00:24:20 UTC
(In reply to Katrin Fischer from comment #7)
> Could we treat it ' as a space and 'no space' at the same time maybe? 

Without language-specific analysis, this might be the best way to go...

(In reply to Nick Clemens from comment #13)
> Created attachment 139969 [details] [review] [review]
> Bug 27153: POC
> 
> This is a small example to show that we can add an extra field to our
> default indexing that
> removes punctuation.
> 
> If we then add that field to the fields we push, we can sarch both the
> regular and the punctuation stripped version
 
I'm not very familiar with the Elasticsearch integration, but that POC doesn't touch the indexing, does it? Isn't it just modifying the search? (I don't know that it's touching apostrophes either?)

If you searched "carries war" but it had been indexed as "carrie s war", then you still wouldn't get a hit?
Comment 16 Nick Clemens 2022-08-31 06:25:22 UTC
(In reply to David Cook from comment #15)
> (In reply to Katrin Fischer from comment #7)
> > Could we treat it ' as a space and 'no space' at the same time maybe? 
> Without language-specific analysis, this might be the best way to go...

That's sort of what the POC does, but just for "as apostrophe" "as no space" - we could add a third field for converting apostrophes to spaces
> 
> (In reply to Nick Clemens from comment #13)
> > Created attachment 139969 [details] [review] [review] [review]
> > Bug 27153: POC
> I'm not very familiar with the Elasticsearch integration, but that POC
> doesn't touch the indexing, does it? 
It does, it adds an extra field, with a punctuation filter to the indexes for 'default' fields - that filter removes all ASCII punctuation characters

> Isn't it just modifying the search?

It then adds that field to the fields being searched in a 'keyword' search 


> If you searched "carries war" but it had been indexed as "carrie s war",
> then you still wouldn't get a hit?
Not with this one, but "carrie's war" or "carries war" would work

The nice thing is, adding an extra index means we can generically search all versions of the field, but can also target specific ones:
title.punc_removed: carries
title.raw: carrie's

So staff can be specific, while making standard search more friendly
Comment 17 Jonathan Field 2022-08-31 16:19:18 UTC
I think the POC is a good compromise. Difficult are this but we need to do something to address it as it comes up constantly. We're happy to go with this.
Comment 18 David Cook 2022-08-31 23:53:22 UTC
I tried to understand this without applying the patches, but it's too hard without the context, so let's see...

I assume that the second patch "POC" is an alternative patch and not an additional patch...

First patch:
- adds apostrophe filter to "analyzer_standard" which is used for all default searches...
- apostrophe strips out apostrophes

Second patch:
- Adds a "punc_removed" field (which uses the analyzer_stdno analyzer which already has a punctuation filter) to "default" under "search". If I understood Elasticsearch and Koha's integration better, I would probably understand this, but I don't currently.
- Reading through Koha/SearchEngine/Elasticsearch.pm and https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html it looks like "phrase", "raw", "ci_raw" and "punc_removed" are "fields" only used when queried specifically
- In Koha/SearchEngine/Elasticsearch/QueryBuilder.pm adds title.punc_removed field to all Elastic queries. We do something similar in Zebra in C4::Search::_build_weighted_query() where we add title fields to search. 

I wonder a bit if adding title.punc_removed there will have unintended consequences but it seems similar to Zebra so might not be a big drama.

Regarding Zebra, I don't think we can get full feature parity here. However, we could potentially add a Title-punc_removed index and update ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl to strip punctuation for it, and then add that into C4::Search::_build_weighted_query() (or elsewhere). 

--

I think the "POC" patch would need some testing to make sure there aren't any unintended consequences, but overall it sounds like a reasonable proposition.
Comment 19 David Cook 2023-06-15 04:34:38 UTC
(In reply to David Cook from comment #18)
> Regarding Zebra, I don't think we can get full feature parity here. However,
> we could potentially add a Title-punc_removed index and update
> ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl to strip punctuation for it,
> and then add that into C4::Search::_build_weighted_query() (or elsewhere). 

Alternatively, we could store a punctuation removed version into the same index. That way you end up indexing both "Carrie's war" and "Carries War". 

But then do you do that for all indexes? That would double the size of your index...
Comment 20 Nick Clemens 2023-09-01 17:57:13 UTC
Created attachment 155145 [details] [review]
Bug 27153: DB Updates

We start by adding the field to the table

It is possible a library would want two versions of the same field in
the indexing. We need to make the filter part of the primary key in
order to allow this
Comment 21 Nick Clemens 2023-09-01 17:57:16 UTC
Created attachment 155146 [details] [review]
Bug 27153: DO NOT PUSH: Schema updates
Comment 22 Nick Clemens 2023-09-01 17:57:18 UTC
Created attachment 155147 [details] [review]
Bug 27153: Interface updates

This patch adds a new column to the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)

To test:
1 - Confirm the new 'filter' column shows
2 - Update an existing field to set filter to punctuation
3 - Confirm it can be saved
4 - Add a new field
5 - Confirm it saves correctly
6 - Unset filter for a field
7 - Confirm it saves
Comment 23 Nick Clemens 2023-09-01 17:57:21 UTC
Created attachment 155148 [details] [review]
Bug 27153: Add filter option to Elasticsearch indexing

This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.

Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.

Tests are updated.
Comment 24 Nick Clemens 2023-09-01 17:57:23 UTC
Created attachment 155149 [details] [review]
Bug 27153: Adjust mappings to add title without punctuation

This patch finally adjust the default mappings to add a new field to
the elastic index with the title stripped of punctuation

This method optionally allows the library to place the filtered version in the same
search field, or a new search field. The default mappings will include the filtered version
in a keyword search, but not a targeted search

To test:
1 - Import some affected records via z3950, e.g.:
    Carrie's war
    1,000 Japanese words
2 - Search (using Elasticsearch) for the titles without including punctuation
    Carries war
    1000 Japanese words
3 - No results
4 - Reload mappings and reindex
    perl misc/search_tools/rebuild_elasticsearch.pl -v -r
5 - Search again
6 - Success!
7 - Search title specifically:
    ti:Carries war
8 - No results
9 - Adjust mappings.yaml to add second mapping for 245 to title index:
    - facet: ''
      marc_field: 245abp
      marc_type: marc21
      sort: 1
      suggestible: 1
      filter: punctuation
10 - Reload mappings and reindex
11 - Repeat 7
12 - Success
Comment 25 Nick Clemens 2023-09-05 17:26:03 UTC
Created attachment 155230 [details] [review]
Bug 27153: Adjust kohastructure for new primary key
Comment 26 ByWater Sandboxes 2023-09-11 16:17:40 UTC
Created attachment 155502 [details] [review]
Bug 27153: Adjust kohastructure for new primary key

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 27 ByWater Sandboxes 2023-09-11 17:55:53 UTC
Created attachment 155508 [details] [review]
Bug 27153: DO NOT PUSH: Schema updates

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 28 ByWater Sandboxes 2023-09-11 17:55:55 UTC
Created attachment 155509 [details] [review]
Bug 27153: Interface updates

This patch adds a new column to the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)

To test:
1 - Confirm the new 'filter' column shows
2 - Update an existing field to set filter to punctuation
3 - Confirm it can be saved
4 - Add a new field
5 - Confirm it saves correctly
6 - Unset filter for a field
7 - Confirm it saves

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 29 ByWater Sandboxes 2023-09-11 17:55:58 UTC
Created attachment 155510 [details] [review]
Bug 27153: Add filter option to Elasticsearch indexing

This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.

Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.

Tests are updated.

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 30 ByWater Sandboxes 2023-09-11 17:56:01 UTC
Created attachment 155511 [details] [review]
Bug 27153: Adjust mappings to add title without punctuation

This patch finally adjust the default mappings to add a new field to
the elastic index with the title stripped of punctuation

This method optionally allows the library to place the filtered version in the same
search field, or a new search field. The default mappings will include the filtered version
in a keyword search, but not a targeted search

To test:
1 - Import some affected records via z3950, e.g.:
    Carrie's war
    1,000 Japanese words
2 - Search (using Elasticsearch) for the titles without including punctuation
    Carries war
    1000 Japanese words
3 - No results
4 - Reload mappings and reindex
    perl misc/search_tools/rebuild_elasticsearch.pl -v -r
5 - Search again
6 - Success!
7 - Search title specifically:
    ti:Carries war
8 - No results
9 - Adjust mappings.yaml to add second mapping for 245 to title index:
    - facet: ''
      marc_field: 245abp
      marc_type: marc21
      sort: 1
      suggestible: 1
      filter: punctuation
10 - Reload mappings and reindex
11 - Repeat 7
12 - Success

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 31 ByWater Sandboxes 2023-09-11 17:56:02 UTC
Created attachment 155512 [details] [review]
Bug 27153: DO NOT PUSH: Schema updates

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 32 ByWater Sandboxes 2023-09-11 17:56:03 UTC
Created attachment 155513 [details] [review]
Bug 27153: Adjust kohastructure for new primary key

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 33 ByWater Sandboxes 2023-09-11 17:56:04 UTC
Created attachment 155514 [details] [review]
Bug 27153: Interface updates

This patch adds a new column to the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)

To test:
1 - Confirm the new 'filter' column shows
2 - Update an existing field to set filter to punctuation
3 - Confirm it can be saved
4 - Add a new field
5 - Confirm it saves correctly
6 - Unset filter for a field
7 - Confirm it saves

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 34 ByWater Sandboxes 2023-09-11 17:56:07 UTC
Created attachment 155515 [details] [review]
Bug 27153: Add filter option to Elasticsearch indexing

This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.

Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.

Tests are updated.

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 35 ByWater Sandboxes 2023-09-11 17:56:09 UTC
Created attachment 155516 [details] [review]
Bug 27153: Adjust mappings to add title without punctuation

This patch finally adjust the default mappings to add a new field to
the elastic index with the title stripped of punctuation

This method optionally allows the library to place the filtered version in the same
search field, or a new search field. The default mappings will include the filtered version
in a keyword search, but not a targeted search

To test:
1 - Import some affected records via z3950, e.g.:
    Carrie's war
    1,000 Japanese words
2 - Search (using Elasticsearch) for the titles without including punctuation
    Carries war
    1000 Japanese words
3 - No results
4 - Reload mappings and reindex
    perl misc/search_tools/rebuild_elasticsearch.pl -v -r
5 - Search again
6 - Success!
7 - Search title specifically:
    ti:Carries war
8 - No results
9 - Adjust mappings.yaml to add second mapping for 245 to title index:
    - facet: ''
      marc_field: 245abp
      marc_type: marc21
      sort: 1
      suggestible: 1
      filter: punctuation
10 - Reload mappings and reindex
11 - Repeat 7
12 - Success

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 36 ByWater Sandboxes 2023-09-11 17:56:11 UTC
Created attachment 155517 [details] [review]
Bug 27153: Adjust kohastructure for new primary key

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Comment 37 Martin Renvoize 2023-10-24 11:37:03 UTC
Created attachment 157732 [details] [review]
Bug 27153: DB Updates

We start by adding the field to the table

It is possible a library would want two versions of the same field in
the indexing. We need to make the filter part of the primary key in
order to allow this

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 38 Martin Renvoize 2023-10-24 11:37:06 UTC
Created attachment 157733 [details] [review]
Bug 27153: DO NOT PUSH: Schema updates

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 39 Martin Renvoize 2023-10-24 11:37:09 UTC
Created attachment 157734 [details] [review]
Bug 27153: Interface updates

This patch adds a new column to the 'Bibliographic records' tab in
Administration > Search engine configuration (Elasticsearch)

To test:
1 - Confirm the new 'filter' column shows
2 - Update an existing field to set filter to punctuation
3 - Confirm it can be saved
4 - Add a new field
5 - Confirm it saves correctly
6 - Unset filter for a field
7 - Confirm it saves

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 40 Martin Renvoize 2023-10-24 11:37:13 UTC
Created attachment 157735 [details] [review]
Bug 27153: Add filter option to Elasticsearch indexing

This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.

Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.

Tests are updated.

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 41 Martin Renvoize 2023-10-24 11:37:17 UTC
Created attachment 157736 [details] [review]
Bug 27153: Adjust mappings to add title without punctuation

This patch finally adjust the default mappings to add a new field to
the elastic index with the title stripped of punctuation

This method optionally allows the library to place the filtered version in the same
search field, or a new search field. The default mappings will include the filtered version
in a keyword search, but not a targeted search

To test:
1 - Import some affected records via z3950, e.g.:
    Carrie's war
    1,000 Japanese words
2 - Search (using Elasticsearch) for the titles without including punctuation
    Carries war
    1000 Japanese words
3 - No results
4 - Reload mappings and reindex
    perl misc/search_tools/rebuild_elasticsearch.pl -v -r
5 - Search again
6 - Success!
7 - Search title specifically:
    ti:Carries war
8 - No results
9 - Adjust mappings.yaml to add second mapping for 245 to title index:
    - facet: ''
      marc_field: 245abp
      marc_type: marc21
      sort: 1
      suggestible: 1
      filter: punctuation
10 - Reload mappings and reindex
11 - Repeat 7
12 - Success

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 42 Martin Renvoize 2023-10-24 11:37:20 UTC
Created attachment 157737 [details] [review]
Bug 27153: Adjust kohastructure for new primary key

Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 43 Martin Renvoize 2023-10-24 11:38:26 UTC
This adds a nice flexible and configurable way of achieving the desired goal. No regressions found, the QA scripts are happy and unit tests are all passing.

Passing QA
Comment 44 Tomás Cohen Arazi 2023-10-24 18:45:13 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 45 Fridolin Somers 2023-10-26 07:02:48 UTC
Enhancement not pushed to 23.05.x
Comment 46 Nick Clemens 2023-10-26 14:12:58 UTC
I marked as 'normal' now - while this adds new functionality I think it resolves a bug for users where records are not being found.

Reconsider?
Comment 47 Fridolin Somers 2023-10-28 20:22:28 UTC
(In reply to Fridolin Somers from comment #45)
> Enhancement not pushed to 23.05.x

Sure, I'll try backport.
Any ES feature is welcome ;)
Comment 48 Nick Clemens 2023-11-08 15:31:36 UTC
(In reply to Fridolin Somers from comment #47)
> (In reply to Fridolin Somers from comment #45)
> > Enhancement not pushed to 23.05.x
> 
> Sure, I'll try backport.
> Any ES feature is welcome ;)

Ping?
Comment 49 Fridolin Somers 2023-11-10 07:53:32 UTC
I'll work on it
Comment 50 Nick Clemens 2024-03-01 12:29:45 UTC
Current patches apply cleanly to 23.05.x and work in my testing - can this be considered for backport?
Comment 51 Laura O'Neil 2024-04-22 12:06:50 UTC
+1 for backport to 23.05, this would be very helpful to a lot of libraries.