Bug 13976 - Sorting search results by popularity is alphabetical
Summary: Sorting search results by popularity is alphabetical
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens (kidclamp)
QA Contact: Marcel de Rooy
URL:
Keywords: release-notes-needed
Depends on:
Blocks:
 
Reported: 2015-04-09 15:04 UTC by Fridolin Somers
Modified: 2023-12-28 20:44 UTC (History)
12 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This patch fixes the sorting of searches by popularity, ensuring that results are sorted numerically. Note: The popularity search requires the use of either the syspref UpdateTotalIssuesOnCirc or the update_totalissue.pl cronjob
Version(s) released in:
23.05.00,22.11.05,22.05.12,21.11.20


Attachments
Bug 13976: Zeropad populariy for Zebra searching (4.33 KB, patch)
2022-09-28 19:31 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 13976: Sort popularity numerically in Zebra (3.79 KB, patch)
2022-11-21 19:14 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 13976: Sort popularity numerically in Zebra (3.84 KB, patch)
2022-12-02 18:32 UTC, PTFS Europe Sandboxes
Details | Diff | Splinter Review
Bug 13976: Sort popularity numerically in Zebra (1.39 KB, patch)
2023-02-01 16:00 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 13976: Sort popularity numerically in Zebra (1.49 KB, patch)
2023-03-27 14:01 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin Somers 2015-04-09 15:04:26 UTC
After performing a search, sorting by popularity is an alphabetical sort instead of a numerical.

For example, sorting the values of totalissues 5, 7, 9, 66, 85, 91, 808 will give : 91->9->85->808->7->66->5

The Zebra configuration is the default UNIMARC  with :
melm 942$0      totalissues:n,totalissues:s

I've looked into doc of http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#sort%28%29
It seems there is no option for numerical sort.
Comment 1 Mark Tompsett 2015-04-09 15:40:15 UTC
I wonder which one it maps to:
http://www.indexdata.com/yaz/doc/zoom.query.html#zoom-sort-strategy
Comment 3 Fridolin Somers 2015-10-19 16:37:12 UTC
(In reply to M. Tompsett from comment #2)
> http://www.indexdata.com/yaz/doc/zoom.resultsets.html#zoom.z3950.resultset.
> sort
> 
> http://www.indexdata.com/yaz/doc/yaz-client.html#sortspec
> 
> Hmmm... There is a problem, isn't there? :)

What do you mean ?
Comment 4 BASE Library Consortium 2018-03-29 10:01:46 UTC
Still relevant - Koha 16.05 and 17.11.03 search for a title and apply sort filter Popularity (most to least) - results are sorted by biblionumber ASC
Comment 5 Katrin Fischer 2018-03-30 09:29:28 UTC
Sorting by biblionumber asc is expected if the popularity values on the records are the same, which could explain. But I think the problem is still real. Not sure if it can't be solved with Zebra tho. Adding Tomas.
Comment 6 Fridolin Somers 2020-03-24 15:24:22 UTC
I bet its solved with Elasticsearch. Can someone confirm ?
Comment 7 Nick Clemens (kidclamp) 2022-09-21 13:48:45 UTC
We need to zeropad the popularity like we did for biblionumbers on 30879
Comment 8 Nick Clemens (kidclamp) 2022-09-28 19:31:59 UTC
Created attachment 141068 [details] [review]
Bug 13976: Zeropad populariy for Zebra searching

This patch is much eased in testing by bug 31644 which will allow
you to modify totalissues field via modification

To test:
1 - Create a new template at Tools->Marc modification template
2 - Add action: Copy 999$c to 942$0
3 - Create a report:
    SELECT biblionuber FROM biblios
4 - Run report, show all, do a batch modification to all records using template above
5 - Search for 'a' (make sure you are using Zebra)
6 - Sort by popularity
7 - Note records are sorted wrong
8 - Apply patch
9 - Copy files:
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
10 - Restart all, reindex zebra
    sudo koha-rebuild-zebra -v -f kohadev
11 - Reload search results
12 - Success! Sorted correctly

Bonus: Test unimarc - same changes
Comment 9 Nick Clemens (kidclamp) 2022-11-21 19:14:10 UTC
Created attachment 144115 [details] [review]
Bug 13976: Sort popularity numerically in Zebra

To test:
1 - Create a new template at Tools->Marc modification template
2 - Add action: Copy 999$c to 942$0
3 - Create a report:
    SELECT biblionuber FROM biblios
4 - Run report, show all, do a batch modification to all records using template above
5 - Search for 'a' (make sure you are using Zebra)
6 - Sort by popularity
7 - Note records are sorted wrong
8 - Apply patch
9 - Restart all
10 - Reload search results
11 - Success! Sorted correctly
Comment 10 PTFS Europe Sandboxes 2022-12-02 18:32:55 UTC
Created attachment 144406 [details] [review]
Bug 13976: Sort popularity numerically in Zebra

To test:
1 - Create a new template at Tools->Marc modification template
2 - Add action: Copy 999$c to 942$0
3 - Create a report:
    SELECT biblionuber FROM biblios
4 - Run report, show all, do a batch modification to all records using template above
5 - Search for 'a' (make sure you are using Zebra)
6 - Sort by popularity
7 - Note records are sorted wrong
8 - Apply patch
9 - Restart all
10 - Reload search results
11 - Success! Sorted correctly

Signed-off-by: Anke <anke.bruns@gwdg.de>
Comment 11 Anke Bruns 2022-12-02 18:38:23 UTC
Note for testers: The aim of the batch modification is to populate the number of totalissues in the biblioitems tab to have some entries there without having to perform manual checkouts; the biblionumber is used to populate this field.

This means also that checking if the sorting was correct can be done by just looking at the results' URLs where the biblionumber also appears.

Thanks to Victor and his patch testing session!
Comment 12 Nick Clemens (kidclamp) 2022-12-02 19:05:47 UTC
(In reply to Anke Bruns from comment #11)
> Note for testers: The aim of the batch modification is to populate the
> number of totalissues in the biblioitems tab to have some entries there
> without having to perform manual checkouts; the biblionumber is used to
> populate this field.
> 
> This means also that checking if the sorting was correct can be done by just
> looking at the results' URLs where the biblionumber also appears.
> 
> Thanks to Victor and his patch testing session!

Thanks Anke, and Victor, for explaining the bit I forgot to explain :-)
Comment 13 Fridolin Somers 2022-12-21 07:27:52 UTC
Hi,

Why does patch impact koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slim2intranetDetail.xsl ?

For the records here is the Zebra doc in attribute 4 :
https://software.indexdata.com/zebra/doc/querymodel-rpn.html#querymodel-bib1-structure
Comment 14 Martin Renvoize 2023-02-01 15:52:59 UTC
Agreed, there's something off going on here.. looks like a mistake adding the XSML change in this patch?
Comment 15 Nick Clemens (kidclamp) 2023-02-01 16:00:14 UTC
Created attachment 145963 [details] [review]
Bug 13976: Sort popularity numerically in Zebra

To test:
1 - Create a new template at Tools->Marc modification template
2 - Add action: Copy 999$c to 942$0
3 - Create a report:
    SELECT biblionuber FROM biblios
4 - Run report, show all, do a batch modification to all records using template above
5 - Search for 'a' (make sure you are using Zebra)
6 - Sort by popularity
7 - Note records are sorted wrong
8 - Apply patch
9 - Restart all
10 - Reload search results
11 - Success! Sorted correctly

Signed-off-by: Anke <anke.bruns@gwdg.de>
Comment 16 Nick Clemens (kidclamp) 2023-02-01 16:04:56 UTC
(In reply to Martin Renvoize from comment #14)
> Agreed, there's something off going on here.. looks like a mistake adding
> the XSML change in this patch?

Removed those
Comment 17 Jonathan Druart 2023-02-23 09:57:59 UTC
I think we need a test for this change.

(In reply to Nick Clemens from comment #7)
> We need to zeropad the popularity like we did for biblionumbers on 30879

Don't we finally need it?
Comment 18 Nick Clemens (kidclamp) 2023-02-23 15:37:08 UTC
(In reply to Jonathan Druart from comment #17)
> I think we need a test for this change.

Will try to add one when I get a chance

> 
> (In reply to Nick Clemens from comment #7)
> > We need to zeropad the popularity like we did for biblionumbers on 30879
> 
> Don't we finally need it?

David Cook found we could natively sort these in Zebra, no need for the padding
Comment 19 Marcel de Rooy 2023-03-24 09:29:35 UTC
QAing
Comment 20 Marcel de Rooy 2023-03-24 10:39:40 UTC Comment hidden (obsolete)
Comment 21 Marcel de Rooy 2023-03-24 10:45:01 UTC Comment hidden (obsolete)
Comment 22 Marcel de Rooy 2023-03-24 10:59:38 UTC
Mentioning the pref C4::Context->preference('UpdateTotalIssuesOnCirc') would have been helpful too :)
Comment 23 Marcel de Rooy 2023-03-24 11:02:25 UTC Comment hidden (obsolete)
Comment 24 Marcel de Rooy 2023-03-24 11:03:35 UTC Comment hidden (obsolete)
Comment 25 Marcel de Rooy 2023-03-27 13:52:45 UTC
(In reply to Marcel de Rooy from comment #23)
> Keeping this blocked for now, will resume. Hoping for feedback btw.
> Still thinking if the combination of :n and :s that I see on totalissues
> might be related; e.g. not-onloan-count does not have the :s and works.

This is not true. We definitely need the :s for the sorting index. Looks like I got it working after all.
Comment 26 Marcel de Rooy 2023-03-27 14:01:44 UTC
Created attachment 148777 [details] [review]
Bug 13976: Sort popularity numerically in Zebra

To test:
1 - Create a new template at Tools->Marc modification template
2 - Add action: Copy 999$c to 942$0
3 - Create a report:
    SELECT biblionuber FROM biblios
4 - Run report, show all, do a batch modification to all records using template above
5 - Search for 'a' (make sure you are using Zebra)
6 - Sort by popularity
7 - Note records are sorted wrong
8 - Apply patch
9 - Restart all
10 - Reload search results
11 - Success! Sorted correctly

Signed-off-by: Anke <anke.bruns@gwdg.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 27 Marcel de Rooy 2023-03-27 14:02:47 UTC
Please add release notes and make some reference to the UpdateTotalIssuesOnCirc preference ;)
Comment 28 Tomás Cohen Arazi 2023-03-31 10:42:57 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 29 Katrin Fischer 2023-04-01 10:35:17 UTC
Really happy to see this one fixed :)
Comment 30 Jacob O'Mara 2023-04-16 17:23:58 UTC
Nice work, thanks everyone!

Pushed to 22.11.x for the next release.
Comment 31 Lucas Gass 2023-04-24 17:29:20 UTC
Backported to 22.05.x for upcoming 22.05.12
Comment 32 Arthur Suzuki 2023-05-03 19:08:56 UTC
applied to 21.11.x for 21.11.20