We are often asked why keyword searches don't ignore apostrophes. If you searched for "carries war" on Google, the first hits that come up are for Carrie's War by Nina Bawden. If you do a keyword search in Koha for "carries war" you get 0 hits. If you search for "carrie's war" you will find copies of this book. Koha with ElasticSearch should search apostrophe-blind (ignore the apostrophe). This was a perennial problem with zebra. In zebra you had to change words-icu.xml to ignore apostrophes. We should deliver an ES index_config.yaml that handles this scenario.
Hi Jonathan, I believe this to be a duplicate of bug 21357 - maybe you could add your voice to that bug for a simplified solution (ignore ') *** This bug has been marked as a duplicate of bug 21357 ***
Created attachment 114234 [details] [review] Bug 27153: Default to apostrophe blind search Keyword searching in elasticsearch should be apostrophe blind; this patch updates the default configuration to filter apostrophe's.
In my understanding elisions and apostrophes are not equal and as such distinct bugs can/should be opened to handle the two cases. This case is for apostrophes only and leaves elisions, which appear to be more difficult to reach a consensus, to the other bug. As such, I don't think this is a duplicate at all.. I've uploaded Jonathans patch.
I don't mind a simple solution, but can you explain the difference?
With this simple apostrophe filter: Dont let the pigeon drive the batmobile will find: Don't let the pigeon drive the batmobile but, you get no results for: ti:"Don't let the pigeon drive the batmobile" I like the idea of a simple solution, but I am not sure it works for everyone. While elisions are different than contractions I believe they both use an apostrophe, so this will affect both. We are planning on testing the 'english' analyzer with one of our partners to help with this issue, I suspect that this may not have a one-size-fits-all solution - but will require the user to decide on which configuration works best in their library
(In reply to Katrin Fischer from comment #4) > I don't mind a simple solution, but can you explain the difference? Elision removes the text : l'europe = europe This apostrophe filter creates : l'europe = leurope Wrong in french but surely ranking will bet better for the exact match. I have to test combined with ellision. But : Maybe it is bad for authorities : "Pierre d'Anjou" and Pierre Danjou".
Could we treat it ' as a space and 'no space' at the same time maybe? And how do we currently handle this with Zebra? I guess we'd want Don't Dont don t Pierre d'Anjou Pierre dAnjou Pierre d Anjou to be the same?
This is a constant problem for our patrons who are used to searching Amazon or Google without having to worry about apostrophes. Fixing this issue would really improve the ability for our patrons to find items in our catalog. The status of this bug is listed as Needs Signoff. Is there a test plan?
Actually Zebra as a rule in words-icu.xml : <transliterate rule="\'>\ "/> But it may not be optimal to do the exact same thing in Elasticsearch.
It might be a good idea to harmonize the normalization used between Zebra and Elasticsearch just for consistency...
(In reply to David Cook from comment #10) > It might be a good idea to harmonize the normalization used between Zebra > and Elasticsearch just for consistency... +1 So we just need to change replacement with a space character ? This will require a unit test I bet.
I am brand new to Koha, but I was told this bug is why I have to include apostrophes while searching. I tried searching for Don't Hate the Player title and came up with zero results. I did not use the apostrophe because I have never had to before in multiple catalog systems. I repeated the search with the apostrophe and found the title. I agree with this and think the searching should work both ways with or without the apostrophe. Thank you.
Created attachment 139969 [details] [review] Bug 27153: POC This is a small example to show that we can add an extra field to our default indexing that removes punctuation. If we then add that field to the fields we push, we can sarch both the regular and the punctuation stripped version To complete this we would need to add a 'search variations' field to the ES config page, for the user to decide which fields should be searched in all variations, and which shouldn't Or we could add this to all 'simple searches' from the search bar, and only search exact on advanced searches. I am happy to explore either option
I had a librarian (from an English-only library) asking for this the other day, and I was wondering why we have apostrophes replaced with a space (both for CHR and ICU indexing with Zebra). But Frido's example makes a lot of sense: (In reply to Fridolin Somers from comment #6) > Elision removes the text : l'europe = europe > This apostrophe filter creates : l'europe = leurope > > Wrong in french but surely ranking will bet better for the exact match. And in English, we'd have "father's" become "fathers", which means "father" wouldn't match unless you're using right truncation (which Koha typically does out of the box I suppose). But then there's a Ukrainian word під'їзд which if you broke it into під їзд, you'd get hits for під which is a totally unrelated word. -- I know people have praised Google here, but it's not perfect either. Try searching "l'arbre під'їзд" and try searching "під'їзд l'arbre", and you'll get wildly different results. It seems that Google tries to determine the language of the search query (possibly based off the first word), and then analyze the search string based off that. For "l'arbre під'їзд", you get many results with "L'Arbre", "des arbres", "arbre", etc. Most results seem to include Russian rather than Ukrainian. But for "під'їзд l'arbre", you only get a few results that have the string "L'Arbre"and most of the results are Ukrainian. So Google is likely determining the language of the search string and then applying a language-specific analyzer. "My mom" in Chinese is 我的妈妈. 的 is the character that denotes possession of mom (妈妈) by me/I (我). You might think then you could replace 的 with a space to separate the 2 nouns... except 的 isn't always used that way. The word 目的 means "goal". Doing anything to 的 would compromise the word, unless you're able to understand the context that it's used in. If you google "我的妈妈", eventually it strips off "我的" and just searches for "妈妈" since that's the main noun in the phrase. Search is hard.
(In reply to Katrin Fischer from comment #7) > Could we treat it ' as a space and 'no space' at the same time maybe? Without language-specific analysis, this might be the best way to go... (In reply to Nick Clemens from comment #13) > Created attachment 139969 [details] [review] [review] > Bug 27153: POC > > This is a small example to show that we can add an extra field to our > default indexing that > removes punctuation. > > If we then add that field to the fields we push, we can sarch both the > regular and the punctuation stripped version I'm not very familiar with the Elasticsearch integration, but that POC doesn't touch the indexing, does it? Isn't it just modifying the search? (I don't know that it's touching apostrophes either?) If you searched "carries war" but it had been indexed as "carrie s war", then you still wouldn't get a hit?
(In reply to David Cook from comment #15) > (In reply to Katrin Fischer from comment #7) > > Could we treat it ' as a space and 'no space' at the same time maybe? > Without language-specific analysis, this might be the best way to go... That's sort of what the POC does, but just for "as apostrophe" "as no space" - we could add a third field for converting apostrophes to spaces > > (In reply to Nick Clemens from comment #13) > > Created attachment 139969 [details] [review] [review] [review] > > Bug 27153: POC > I'm not very familiar with the Elasticsearch integration, but that POC > doesn't touch the indexing, does it? It does, it adds an extra field, with a punctuation filter to the indexes for 'default' fields - that filter removes all ASCII punctuation characters > Isn't it just modifying the search? It then adds that field to the fields being searched in a 'keyword' search > If you searched "carries war" but it had been indexed as "carrie s war", > then you still wouldn't get a hit? Not with this one, but "carrie's war" or "carries war" would work The nice thing is, adding an extra index means we can generically search all versions of the field, but can also target specific ones: title.punc_removed: carries title.raw: carrie's So staff can be specific, while making standard search more friendly
I think the POC is a good compromise. Difficult are this but we need to do something to address it as it comes up constantly. We're happy to go with this.
I tried to understand this without applying the patches, but it's too hard without the context, so let's see... I assume that the second patch "POC" is an alternative patch and not an additional patch... First patch: - adds apostrophe filter to "analyzer_standard" which is used for all default searches... - apostrophe strips out apostrophes Second patch: - Adds a "punc_removed" field (which uses the analyzer_stdno analyzer which already has a punctuation filter) to "default" under "search". If I understood Elasticsearch and Koha's integration better, I would probably understand this, but I don't currently. - Reading through Koha/SearchEngine/Elasticsearch.pm and https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html it looks like "phrase", "raw", "ci_raw" and "punc_removed" are "fields" only used when queried specifically - In Koha/SearchEngine/Elasticsearch/QueryBuilder.pm adds title.punc_removed field to all Elastic queries. We do something similar in Zebra in C4::Search::_build_weighted_query() where we add title fields to search. I wonder a bit if adding title.punc_removed there will have unintended consequences but it seems similar to Zebra so might not be a big drama. Regarding Zebra, I don't think we can get full feature parity here. However, we could potentially add a Title-punc_removed index and update ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl to strip punctuation for it, and then add that into C4::Search::_build_weighted_query() (or elsewhere). -- I think the "POC" patch would need some testing to make sure there aren't any unintended consequences, but overall it sounds like a reasonable proposition.
(In reply to David Cook from comment #18) > Regarding Zebra, I don't think we can get full feature parity here. However, > we could potentially add a Title-punc_removed index and update > ./etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl to strip punctuation for it, > and then add that into C4::Search::_build_weighted_query() (or elsewhere). Alternatively, we could store a punctuation removed version into the same index. That way you end up indexing both "Carrie's war" and "Carries War". But then do you do that for all indexes? That would double the size of your index...
Created attachment 155145 [details] [review] Bug 27153: DB Updates We start by adding the field to the table It is possible a library would want two versions of the same field in the indexing. We need to make the filter part of the primary key in order to allow this
Created attachment 155146 [details] [review] Bug 27153: DO NOT PUSH: Schema updates
Created attachment 155147 [details] [review] Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves
Created attachment 155148 [details] [review] Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated.
Created attachment 155149 [details] [review] Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success
Created attachment 155230 [details] [review] Bug 27153: Adjust kohastructure for new primary key
Created attachment 155502 [details] [review] Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155508 [details] [review] Bug 27153: DO NOT PUSH: Schema updates Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155509 [details] [review] Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155510 [details] [review] Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated. Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155511 [details] [review] Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155512 [details] [review] Bug 27153: DO NOT PUSH: Schema updates Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155513 [details] [review] Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155514 [details] [review] Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155515 [details] [review] Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated. Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155516 [details] [review] Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 155517 [details] [review] Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Created attachment 157732 [details] [review] Bug 27153: DB Updates We start by adding the field to the table It is possible a library would want two versions of the same field in the indexing. We need to make the filter part of the primary key in order to allow this Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 157733 [details] [review] Bug 27153: DO NOT PUSH: Schema updates Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 157734 [details] [review] Bug 27153: Interface updates This patch adds a new column to the 'Bibliographic records' tab in Administration > Search engine configuration (Elasticsearch) To test: 1 - Confirm the new 'filter' column shows 2 - Update an existing field to set filter to punctuation 3 - Confirm it can be saved 4 - Add a new field 5 - Confirm it saves correctly 6 - Unset filter for a field 7 - Confirm it saves Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 157735 [details] [review] Bug 27153: Add filter option to Elasticsearch indexing This patch uses the filter option from the mappings to add a value_callback to reord processing for indexing. Fields defined with 'punctuation' filter will have all punctuation stripped when conveted to documents. Tests are updated. Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 157736 [details] [review] Bug 27153: Adjust mappings to add title without punctuation This patch finally adjust the default mappings to add a new field to the elastic index with the title stripped of punctuation This method optionally allows the library to place the filtered version in the same search field, or a new search field. The default mappings will include the filtered version in a keyword search, but not a targeted search To test: 1 - Import some affected records via z3950, e.g.: Carrie's war 1,000 Japanese words 2 - Search (using Elasticsearch) for the titles without including punctuation Carries war 1000 Japanese words 3 - No results 4 - Reload mappings and reindex perl misc/search_tools/rebuild_elasticsearch.pl -v -r 5 - Search again 6 - Success! 7 - Search title specifically: ti:Carries war 8 - No results 9 - Adjust mappings.yaml to add second mapping for 245 to title index: - facet: '' marc_field: 245abp marc_type: marc21 sort: 1 suggestible: 1 filter: punctuation 10 - Reload mappings and reindex 11 - Repeat 7 12 - Success Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 157737 [details] [review] Bug 27153: Adjust kohastructure for new primary key Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This adds a nice flexible and configurable way of achieving the desired goal. No regressions found, the QA scripts are happy and unit tests are all passing. Passing QA
Pushed to master for 23.11. Nice work everyone, thanks!
Enhancement not pushed to 23.05.x
I marked as 'normal' now - while this adds new functionality I think it resolves a bug for users where records are not being found. Reconsider?
(In reply to Fridolin Somers from comment #45) > Enhancement not pushed to 23.05.x Sure, I'll try backport. Any ES feature is welcome ;)
(In reply to Fridolin Somers from comment #47) > (In reply to Fridolin Somers from comment #45) > > Enhancement not pushed to 23.05.x > > Sure, I'll try backport. > Any ES feature is welcome ;) Ping?
I'll work on it
Current patches apply cleanly to 23.05.x and work in my testing - can this be considered for backport?
+1 for backport to 23.05, this would be very helpful to a lot of libraries.