Bug 40658 - When sorting by local-number we should use the sort field
Summary: When sorting by local-number we should use the sort field
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Nick Clemens (kidclamp)
QA Contact: Testopia
URL:
Keywords:
Depends on: 23875
Blocks:
  Show dependency treegraph
 
Reported: 2025-08-15 12:06 UTC by Nick Clemens (kidclamp)
Modified: 2025-12-25 20:38 UTC (History)
3 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 40658: Use local-number__sort and not local-number directly (3.13 KB, patch)
2025-08-15 12:18 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 40658: Atomic update to ensure local-number is sortable (2.47 KB, patch)
2025-08-15 13:17 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 40658: Use local-number__sort and not local-number directly (3.84 KB, patch)
2025-08-18 13:00 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 40658: Atomic update to ensure local-number is sortable (2.47 KB, patch)
2025-08-18 13:00 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 40658: Use local-number__sort and not local-number directly (3.90 KB, patch)
2025-11-10 11:51 UTC, Bernard
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2025-08-15 12:06:51 UTC
When adding 'local-number' as a fallback sort I explicitly added sorting on 'local-number' - it should be 'local-number__sort'

This avoids problems when mappings may be older and local-number itself is indexed as a string and not a number
Comment 1 Nick Clemens (kidclamp) 2025-08-15 12:18:24 UTC
Created attachment 185441 [details] [review]
Bug 40658: Use local-number__sort and not local-number directly

This patch makes sure we inject the __sort form of local-number, rather
than the field directly - this ensurs searching works even if mappings mayu
be different and local-number itself is not numeric

To test:
1 - Edit mappings.yaml and set local-number from 'number' to 'string'
    NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here
2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v
3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string
4 - Search and get an error
    Error: Unable to perform your search. Please try again.
    In the logs:
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
5 - Apply patch, restart all
6 - Search again
7 - Success!
Comment 2 Nick Clemens (kidclamp) 2025-08-15 13:17:57 UTC
Created attachment 185447 [details] [review]
Bug 40658: Atomic update to ensure local-number is sortable

I don't know if this is appropriate for master as it requires Koha objects,
but this is helpful for upgrading sites to avoid any problems from old mappings
for local-number
Comment 3 Nick Clemens (kidclamp) 2025-08-18 13:00:35 UTC
Created attachment 185511 [details] [review]
Bug 40658: Use local-number__sort and not local-number directly

This patch makes sure we inject the __sort form of local-number, rather
than the field directly - this ensurs searching works even if mappings mayu
be different and local-number itself is not numeric

To test:
1 - Edit mappings.yaml and set local-number from 'number' to 'string'
    NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here
2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v
3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string
4 - Search and get an error
    Error: Unable to perform your search. Please try again.
    In the logs:
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
5 - Apply patch, restart all
6 - Search again
7 - Success!
Comment 4 Nick Clemens (kidclamp) 2025-08-18 13:00:38 UTC
Created attachment 185512 [details] [review]
Bug 40658: Atomic update to ensure local-number is sortable

I don't know if this is appropriate for master as it requires Koha objects,
but this is helpful for upgrading sites to avoid any problems from old mappings
for local-number
Comment 5 Brendan Lawlor 2025-09-19 18:58:53 UTC
I tried to test this, up until step 4 went as planned, but I couldn't get the search to work after applying the patch.

I started ktd with ktd --es7 up

I got:
No results found
You did not specify any search criteria.

Error: Unable to perform your search. Please try again.

Maybe I'm missing some system preference or process?
Comment 7 Andrii Nugged 2025-11-04 08:26:08 UTC
(In reply to Nick Clemens (kidclamp) from comment #4)
> Created attachment 185512 [details] [review] [review]
> Bug 40658: Atomic update to ensure local-number is sortable

Is atomic update ElasticSearch hardcoded? What if in case when we have Zebra?
Comment 8 Bernard 2025-11-10 11:51:13 UTC
Created attachment 189385 [details] [review]
Bug 40658: Use local-number__sort and not local-number directly

This patch makes sure we inject the __sort form of local-number, rather
than the field directly - this ensurs searching works even if mappings mayu
be different and local-number itself is not numeric

To test:
1 - Edit mappings.yaml and set local-number from 'number' to 'string'
    NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here
2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v
3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string
4 - Search and get an error
    Error: Unable to perform your search. Please try again.
    In the logs:
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
5 - Apply patch, restart all
6 - Search again
7 - Success!

Signed-off-by: Bernard Scaife <bernard.scaife@openfifth.co.uk>
Comment 9 David Nind 2025-12-25 20:38:31 UTC
Hi Bernard.

It looks like you signed this off in November.

Could you change the status to Signed Off? That way, you will also get credit on the dashboard (https://dashboard.koha-community.org/leaderboard).

The patch still applies.

Thanks!

David