Bug 26051

Summary: Elasticsearch uses the wrong field for callnumber sorting
Product: Koha Reporter: Fridolin Somers <fridolin.somers>
Component: Searching - ElasticsearchAssignee: Fridolin Somers <fridolin.somers>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P5 - low CC: alex.arnaud, andrew, anne-cecile.bories, black23, bwsdonna, david, flyingendpaper, martin.renvoize, nick, severine.queune, victor
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
This fixes the sorting of search results by call number when using Elasticsearch. Currently it does not sort correctly (uses local-classification instead of cn-sort) and may also cause error messages "No results found" and "Error: Unable to perform your search. Please try again.". This also matches the behaviour used by Zebra.
Version(s) released in:
21.05.00,20.11.04,20.05.10
Circulation function:
Bug Depends on:    
Bug Blocks: 29632    
Attachments: Bug 26051: Elasticsearch field local-classification sort=1 in mapping.yaml
Bug 26051: Elasticsearch field local-classification sort=1 in mapping.yaml
Bug 26051: [alternate] We should sort callnumber based on cn-sort
Bug 26051: Add sort=1 on cn-sort in mappings.yaml
Bug 26051: We should sort callnumber based on cn-sort
Bug 26051: Add sort=1 on cn-sort in mappings.yaml
Bug 26051: We should sort callnumber based on cn-sort
cam12263 - plugin Pando - Ajout d'un plugin pour intégrer les utilisateurs Koha avec Pando
Bug 26051: Add sort=1 on cn-sort in mappings.yaml
Bug 26051: We should sort callnumber based on cn-sort

Description Fridolin Somers 2020-07-23 12:50:41 UTC
With Elasticsearch the search field use for sorting by callnumber is "local-classification" (from Koha::SearchEngine::Elasticsearch::QueryBuilder::_convert_sort_fields).
So this search field must be sortable in default mappings.yaml.
Comment 1 Fridolin Somers 2020-07-23 12:55:41 UTC
Created attachment 107237 [details] [review]
Bug 26051: Elasticsearch field local-classification sort=1 in mapping.yaml

With Elasticsearch the search field use for sorting by callnumber is "local-classification" (from Koha::SearchEngine::Elasticsearch::QueryBuilder::_convert_sort_fields).
So this search field must be sortable in default mappings.yaml.

If not you see this error in logs :
No mapping found for [local-classification__sort]

Test plan :
1) Install Koha with Elasticsearch with mappings.yaml from sources (admin/searchengine/elasticsearch/mappings.yaml)
2) Reset mappings
3) Rebuild indexes
4) Perform a search
5) Check sorting by callnumber is OK
Comment 2 Victor Grousset/tuxayo 2020-12-27 16:41:06 UTC
Created attachment 114707 [details] [review]
Bug 26051: Elasticsearch field local-classification sort=1 in mapping.yaml

With Elasticsearch the search field use for sorting by callnumber is "local-classification" (from Koha::SearchEngine::Elasticsearch::QueryBuilder::_convert_sort_fields).
So this search field must be sortable in default mappings.yaml.

If not you see this error in logs :
No mapping found for [local-classification__sort]

Test plan :
1) Install Koha with Elasticsearch with mappings.yaml from sources (admin/searchengine/elasticsearch/mappings.yaml)
     It should be the case on all developement instalations
2) Reset mappings (Administration › Search engine configuration (Elasticsearch))
3) Rebuild indexes: misc/search_tools/rebuild_elasticsearch.pl -v -d
4) Perform a search
5) Check sorting by callnumber is OK (add some call numbers if
     necessary)

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 3 Victor Grousset/tuxayo 2020-12-27 16:41:48 UTC
Completed a bit the plan.
Comment 4 Nick Clemens (kidclamp) 2021-01-08 13:17:40 UTC
Created attachment 114957 [details] [review]
Bug 26051: [alternate] We should sort callnumber based on cn-sort

In zebra we sort by callnumber using 8007 cn-sort

We should do the same in elasticsearch

To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!
Comment 5 Nick Clemens (kidclamp) 2021-01-13 12:44:37 UTC
Victor and Frido, can you look at my alternate patch?
Comment 6 Fridolin Somers 2021-01-21 16:13:17 UTC
I'll do ASAP ;)
Comment 7 Fridolin Somers 2021-01-22 08:05:13 UTC
Created attachment 115662 [details] [review]
Bug 26051: Add sort=1 on cn-sort in mappings.yaml
Comment 8 Fridolin Somers 2021-01-22 08:05:27 UTC
Created attachment 115664 [details] [review]
Bug 26051: We should sort callnumber based on cn-sort

In zebra we sort by callnumber using 8007 cn-sort

We should do the same in elasticsearch

To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Comment 9 Fridolin Somers 2021-01-22 08:07:10 UTC
You patch makes sens :)

I've added a patch to define cn-sort sort=1 in mappings.yaml.
Comment 10 Nick Clemens (kidclamp) 2021-01-29 20:22:54 UTC
*** Bug 26192 has been marked as a duplicate of this bug. ***
Comment 11 David Nind 2021-01-30 00:34:39 UTC
Created attachment 116086 [details] [review]
Bug 26051: Add sort=1 on cn-sort in mappings.yaml

Signed-off-by: David Nind <david@davidnind.com>
Comment 12 David Nind 2021-01-30 00:34:43 UTC
Created attachment 116087 [details] [review]
Bug 26051: We should sort callnumber based on cn-sort

In zebra we sort by callnumber using 8007 cn-sort

We should do the same in elasticsearch

To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Signed-off-by: David Nind <david@davidnind.com>
Comment 13 David Nind 2021-01-30 00:52:36 UTC
Testing notes (using koha-testing-docker)
- the error message appears in /var/log/koha/kohadev/plack-intranet-error.log 
- enabling Elasticsearch by changing system preference SearchEngine
- package command to reindex: koha-elasticsearch --rebuild -d -b -a -v kohadev
- before applying the patch also got "No results found" and "Error: Unable to perform your search. Please try again." when trying to sort in the staff interface (including the advanced search form) and the OPAC
Comment 14 Ivan Dziuba 2021-02-03 21:01:13 UTC
Created attachment 116295 [details] [review]
cam12263 - plugin Pando - Ajout d'un plugin pour intégrer les utilisateurs Koha avec Pando
Comment 15 David Nind 2021-02-04 06:07:22 UTC
Hi Ivan.

I'm assuming your patch with InLibro's carousel plugin was added in error to this bug?

David
Comment 16 Martin Renvoize (ashimema) 2021-02-22 16:15:38 UTC
Created attachment 117165 [details] [review]
Bug 26051: Add sort=1 on cn-sort in mappings.yaml

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 17 Martin Renvoize (ashimema) 2021-02-22 16:15:41 UTC
Created attachment 117166 [details] [review]
Bug 26051: We should sort callnumber based on cn-sort

In zebra we sort by callnumber using 8007 cn-sort

We should do the same in elasticsearch

To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 18 Martin Renvoize (ashimema) 2021-02-22 16:16:25 UTC
Surprised this one slipped through the net for as long as it has..

Works as expected, no regressions found.

Passing QA

(I also got rid of that final errant attachment)
Comment 19 Jonathan Druart 2021-02-23 12:21:17 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 20 Fridolin Somers 2021-02-25 09:36:43 UTC
Pushed to 20.11.x for 20.11.04
Comment 21 Andrew Fuerste-Henry 2021-03-05 13:50:46 UTC
Pushed to 20.05.x for 20.05.10
Comment 22 Victor Grousset/tuxayo 2021-03-18 23:48:23 UTC
Not backported to oldoldstable (19.11.x). Feel free to ask if it's needed.