Bug 33594 - Sorting results by Title A-Z might use wrong title field
Summary: Sorting results by Title A-Z might use wrong title field
Status: Needs documenting
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact: Katrin Fischer
URL:
Keywords:
Depends on: 33206
Blocks:
  Show dependency treegraph
 
Reported: 2023-04-23 21:40 UTC by Katrin Fischer
Modified: 2023-06-08 19:28 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This changes the behavour of title search and sorting (Title (A-Z)) in the OPAC and staff interface when using Elasticsearch or Open Search. The title search now only uses 245 (for MARC 21) and 200 (for UNIMARC). Previously other title fields may have affected the search order, for example 240$a in MARC21. To make this change for existing installations, reset the search engine mappings (Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (scroll down to the bottom of the page)) and rebuild the search index (koha-elasticsearch --rebuild -d -b -a <instancename>). If you have customized the search engine configuration, remember to record or back these up BEFORE resetting the mappings.
Version(s) released in:
23.05.00


Attachments
Bug 33594: Update mappings and comment (156.99 KB, patch)
2023-05-04 11:36 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33594: Only sort on title main heading (10.22 KB, patch)
2023-05-05 13:11 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33594: Update mappings and comment (157.04 KB, patch)
2023-05-07 20:21 UTC, David Nind
Details | Diff | Splinter Review
Bug 33594: Only sort on title main heading (10.26 KB, patch)
2023-05-07 20:21 UTC, David Nind
Details | Diff | Splinter Review
Bug 33594: Only sort on title main heading (10.32 KB, patch)
2023-05-08 21:52 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Katrin Fischer 2023-04-23 21:40:03 UTC
When sorting by title we expect the sorting to use 245, but this is not always the case. title in Elasticsearch is a broadly defined index, also containing other title fields.

Example:
* Search for 'e' with Elasticsearch in the sample data
* Result will look something like this:

14. Agesilaos and the crisis of Sparta Paul Cartledge 

15. Xenophōntos Agēsilaos. Agesilaus of Xenophon / the text revised, with critical and explanatory notes, introduction, analyses and indices, by H. Hailstone. 

16. Alcestis / Euripides ; edited with introduction and commentary by A.M. Dale. 

So what happened here? Look at the MARC of 15. - 240 was used:

240 	1 	0 	_a Agesilaus.
_l Greek.
_f 1879
245 	1 	0 	_a Xenophōntos Agēsilaos. Agesilaus of Xenophon /
_c the text revised, with critical and explanatory notes, introduction, analyses and indices, by H. Hailstone.


Maybe we need a specific index definition for sorting?
Comment 1 Janusz Kaczmarek 2023-04-24 07:32:00 UTC
I have encountered this some time ago.  For me worked customizing the mappings.yaml as follows:

       - facet: ''
         marc_field: 240
         marc_type: marc21
-        sort: ~
+        sort: ''
         suggestible: ''

I.e. changing ~ into '' for all MARC fields except 245. 

For some to-me-mysterious reason, ~ (= undef) is explicitly interpreted as "yes, include it into the sort search field" by Koha/SearchEngine/Elasticsearch.pm:

                # Sort is a bit special as it can be true, false, undef.
                # We care about "true" or "undef",
                # "undef" means to do the default thing, which is make it sortable.


So, since this is by design and mappings.yaml is configurable, I didn't treat it as a bug.  Just use a customized copy of mappings.

Please, check if changing ~ into '' works for you.  Please, consider also if it is worth to be a bug.  If so, the solution is there.
Comment 2 Janusz Kaczmarek 2023-04-24 11:39:42 UTC
(In reply to Janusz Kaczmarek from comment #1)
> 
> I.e. changing ~ into '' for all MARC fields except 245. 
> 

I meant, for title search field, of course.
Comment 3 Katrin Fischer 2023-05-03 13:44:12 UTC
(In reply to Janusz Kaczmarek from comment #2)
> (In reply to Janusz Kaczmarek from comment #1)
> > 
> > I.e. changing ~ into '' for all MARC fields except 245. 
> > 
> 
> I meant, for title search field, of course.

Hi Janusz, thx for the insight, this is really helpful. I also found bug 32479. Would you agree that all ~ shoudl be replaced by "" or 0?
Comment 4 Katrin Fischer 2023-05-03 21:24:05 UTC
I feel it's a bug and we should fix it, but still a little confused about the different values we could use here for sort and what effect they will have. Can someone explain a little more?
Comment 5 Nick Clemens 2023-05-04 11:34:14 UTC
(In reply to Katrin Fischer from comment #4)
> I feel it's a bug and we should fix it, but still a little confused about
> the different values we could use here for sort and what effect they will
> have. Can someone explain a little more?

27316 moved the options for setting the field in staff interface from "Undef/Yes/No" to simply "Yes/No" = 0/1

We did not update the mappings file, that should be done here

The code continues to support ~/Undef so that we don't break existing setups - we should update the comment Janusz noted to explain that 'UNdef' should be set as '1'
Comment 6 Nick Clemens 2023-05-04 11:36:20 UTC
Created attachment 150649 [details] [review]
Bug 33594: Update mappings and comment
Comment 7 Nick Clemens 2023-05-04 11:39:35 UTC
This changes a few other fields, but it really is just a dump from the DB, those changes are essentially how the fields are going to be treated anyways, it was generated using:
perl misc/search_tools/export_elasticsearch_mappings.pl > admin/searchengine/elasticsearch/mappings.yaml
Comment 8 Janusz Kaczmarek 2023-05-04 20:19:53 UTC
Nick, thank you for your contribution.  Changing "sort: ~" into "sort: 1" makes the mappings.yaml to be more explicit and therefor less mysterious.  

But the patch does not change the way Koha with ES sorts titles, so it does not fix the problem.  I think, everybody would expect that when choosing sort order by title (both in staff and OPAC interface) the sorting will by done according to the main title, i.e. 245 in MARC 21 and 200 in UNIMARC. 

So, IMHO, in searchfield 'title' only 245 (for MARC 21) and 200 (for UNIMARC) should have 'sort: 1', and the rest of tags indexed with title -- 'sort: 0'.

I would appreciate your feedback.
Comment 9 Katrin Fischer 2023-05-04 20:37:44 UTC
(In reply to Janusz Kaczmarek from comment #8)
> Nick, thank you for your contribution.  Changing "sort: ~" into "sort: 1"
> makes the mappings.yaml to be more explicit and therefor less mysterious.  
> 
> But the patch does not change the way Koha with ES sorts titles, so it does
> not fix the problem.  I think, everybody would expect that when choosing
> sort order by title (both in staff and OPAC interface) the sorting will by
> done according to the main title, i.e. 245 in MARC 21 and 200 in UNIMARC. 
> 
> So, IMHO, in searchfield 'title' only 245 (for MARC 21) and 200 (for
> UNIMARC) should have 'sort: 1', and the rest of tags indexed with title --
> 'sort: 0'.
> 
> I would appreciate your feedback.

I'd agree with that expectation. The other titles don't even all show, so it can get quite confusing.
Comment 10 Nick Clemens 2023-05-05 12:45:41 UTC
(In reply to Janusz Kaczmarek from comment #8)
> Nick, thank you for your contribution.  Changing "sort: ~" into "sort: 1"
> makes the mappings.yaml to be more explicit and therefor less mysterious.  
> 
> But the patch does not change the way Koha with ES sorts titles, so it does
> not fix the problem.  I think, everybody would expect that when choosing
> sort order by title (both in staff and OPAC interface) the sorting will by
> done according to the main title, i.e. 245 in MARC 21 and 200 in UNIMARC. 
> 
> So, IMHO, in searchfield 'title' only 245 (for MARC 21) and 200 (for
> UNIMARC) should have 'sort: 1', and the rest of tags indexed with title --
> 'sort: 0'.
> 
> I would appreciate your feedback.

Agreed - it looks like the Elastic default on a multi-value sort field is min (ascending) or max (descending)
Comment 11 Nick Clemens 2023-05-05 13:11:20 UTC
Created attachment 150749 [details] [review]
Bug 33594: Only sort on title main heading

This patch simply remvoes sort from all elements that are not strictly
the main title

Note:
If multiple fields are set as sort, they are collapsed into a single entry in
the {field}__sort field in the ES index. The order will be determined by the order in
the marc record

To test:
1 - Apply patch
2 - perl misc/search_tools/rebuild_elasticsearch -r -v
3 - Search the catalog
4 - Sort by title
5 - Confirm records are correct
6 - Add a 240 (before the 245) with subfield a 'AAAAA'
7 - Confirm sorting is not affected
8 - View record details, click 'Elasticsearch record: Show'
9 - Find 'title__sort' and confirm it looks correct (does not include AAAAA)
Comment 12 David Nind 2023-05-07 20:21:24 UTC
Created attachment 150792 [details] [review]
Bug 33594: Update mappings and comment

Signed-off-by: David Nind <david@davidnind.com>
Comment 13 David Nind 2023-05-07 20:21:26 UTC
Created attachment 150793 [details] [review]
Bug 33594: Only sort on title main heading

This patch simply remvoes sort from all elements that are not strictly
the main title

Note:
If multiple fields are set as sort, they are collapsed into a single entry in
the {field}__sort field in the ES index. The order will be determined by the order in
the marc record

To test:
1 - Apply patch
2 - perl misc/search_tools/rebuild_elasticsearch -r -v
3 - Search the catalog
4 - Sort by title
5 - Confirm records are correct
6 - Add a 240 (before the 245) with subfield a 'AAAAA'
7 - Confirm sorting is not affected
8 - View record details, click 'Elasticsearch record: Show'
9 - Find 'title__sort' and confirm it looks correct (does not include AAAAA)

Signed-off-by: David Nind <david@davidnind.com>
Comment 14 David Nind 2023-05-07 20:40:59 UTC
Testing notes (using KTD):

1. I used the example in the bug description (#c0) to test.

2. After step 1, I needed to reset the mappings (Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings (scroll down to the bottom of the page)).

3. Draft release notes. I've made an attempt at this, but this area is not something I really understand. Please update to fix what I've got wrong!
Comment 15 Katrin Fischer 2023-05-08 21:52:26 UTC
Created attachment 150813 [details] [review]
Bug 33594: Only sort on title main heading

This patch simply remvoes sort from all elements that are not strictly
the main title

Note:
If multiple fields are set as sort, they are collapsed into a single entry in
the {field}__sort field in the ES index. The order will be determined by the order in
the marc record

To test:
1 - Apply patch
2 - perl misc/search_tools/rebuild_elasticsearch -r -v
3 - Search the catalog
4 - Sort by title
5 - Confirm records are correct
6 - Add a 240 (before the 245) with subfield a 'AAAAA'
7 - Confirm sorting is not affected
8 - View record details, click 'Elasticsearch record: Show'
9 - Find 'title__sort' and confirm it looks correct (does not include AAAAA)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 16 Tomás Cohen Arazi 2023-05-18 14:04:43 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 17 Matt Blenkinsop 2023-06-08 16:44:11 UTC
Enhancement - not backporting to 22.11.x
Comment 18 Katrin Fischer 2023-06-08 19:28:06 UTC
It would be nice to have this reconsidered, but it will only affect new installations.