Bug 31756 - Biblionumber not numerically sorting natively when using Zebra
Summary: Biblionumber not numerically sorting natively when using Zebra
Status: CLOSED WONTFIX
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Zebra (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: David Cook
QA Contact:
URL:
Keywords:
Depends on: 30879
Blocks:
  Show dependency treegraph
 
Reported: 2022-10-12 23:52 UTC by David Cook
Modified: 2023-06-08 22:26 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 31756: Use native Zebra numeric sort for biblionumber (1.45 KB, patch)
2022-10-13 00:07 UTC, David Cook
Details | Diff | Splinter Review
Bug 31756: Undo zeropad workaround for Local-Number:s for MARC21 (2.37 KB, patch)
2022-10-13 00:07 UTC, David Cook
Details | Diff | Splinter Review
Bug 31756: Undo zeropad workaround for Local-Number:s for UNIMARC (1.80 KB, patch)
2022-10-13 00:08 UTC, David Cook
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2022-10-12 23:52:28 UTC
When bug 30879 was pushed, we didn't know how to natively sort numbers, but we do now. 

This issue seeks to replace the workaround with the native implementation.
Comment 1 David Cook 2022-10-13 00:07:52 UTC
Created attachment 141759 [details] [review]
Bug 31756: Use native Zebra numeric sort for biblionumber

This patch uses native Zebra numeric sort for biblionumber
to provide ascending and descending biblionumber sort in Koha.

To test:
1. Apply patches
2. koha-plack --restart kohadev
3. Browse to: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?
idx=kw&q=a&sort_by=biblionumber_dsc&count=20
4. Confirm records sorted correctly
5. Browse to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?
idx=kw&q=a&sort_by=biblionumber_asc&count=20
6. Confirm records sorted correctly
Comment 2 David Cook 2022-10-13 00:07:56 UTC
Created attachment 141760 [details] [review]
Bug 31756: Undo zeropad workaround for Local-Number:s for MARC21
Comment 3 David Cook 2022-10-13 00:08:01 UTC
Created attachment 141761 [details] [review]
Bug 31756: Undo zeropad workaround for Local-Number:s for UNIMARC
Comment 4 David Cook 2022-10-13 00:08:44 UTC
Actually, the test plan should be the same as bug 30879:

To test:
1 - Apply patches
2 - copy etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl to /etc/koha/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl
3 - Restart all, reindex zebra
4 - Browse to: http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=kw&q=a&sort_by=biblionumber_dsc&count=20
5 - Confirm records sorted correctly
6 - Browse to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=kw&q=a&sort_by=biblionumber_asc&count=20
7 - Confirm records sorted correctly

And same again for UNIMARC...
Comment 5 David Nind 2022-10-13 23:18:59 UTC
Hi David.

Things seem to be working for MARC21.

However, for UNIMARC:
- I changed the count number =100
- After the first 10 pages of results (that is, from page 11 and the 1001 search result) the order changes to asc starting from 1

It works fine for asc.

Note: when copying the definitions in step 2, I replaced marc21 with unimarc in the file paths.

David
Comment 6 David Cook 2022-10-14 03:01:49 UTC
(In reply to David Nind from comment #5)
> Hi David.
> 
> Things seem to be working for MARC21.
> 
> However, for UNIMARC:
> - I changed the count number =100
> - After the first 10 pages of results (that is, from page 11 and the 1001
> search result) the order changes to asc starting from 1
> 
> It works fine for asc.
> 
> Note: when copying the definitions in step 2, I replaced marc21 with unimarc
> in the file paths.
> 
> David

Could you elaborate a bit more? You say the order changes to "asc". Does that mean it was "desc" before? 

I don't think that's an issue with these patches.

Also, in ./etc/zebradb/zebra-biblios-dom.cfg, you'll probably notice that "sortmax" is set to "1000". That means that only the first 1000 records in a result set will be sorted, so that might be what you're encountering.

That means only the first 1000 records are sorted.
Comment 7 David Nind 2022-10-14 05:51:14 UTC
(In reply to David Cook from comment #6)
 
> Could you elaborate a bit more? You say the order changes to "asc". Does
> that mean it was "desc" before? 

Yes.

For UNIMARC, when I used the URL with dsc in it:
- Results 1 to 1,000 appear to sort correctly, with biblo records in descending order
- For result 1,001 the biblio number of the record displayed is 1, and then it continues in ascending order for the rest of the paged set of results (until "last")

If there is a setting that limits the sorting to 1,000, then I would expect this to happen with the URL with asc in it as well, but it doesn't - everything appears to be as expected with all the results set 4,000+ sorting correctly.

Maybe for the asc URL this is the "default" for the Zebra config, would that explain why they act differently?
Comment 8 David Cook 2022-10-14 06:14:34 UTC
(In reply to David Nind from comment #7)
> (In reply to David Cook from comment #6)
>  
> > Could you elaborate a bit more? You say the order changes to "asc". Does
> > that mean it was "desc" before? 
> 
> Yes.
> 
> For UNIMARC, when I used the URL with dsc in it:
> - Results 1 to 1,000 appear to sort correctly, with biblo records in
> descending order
> - For result 1,001 the biblio number of the record displayed is 1, and then
> it continues in ascending order for the rest of the paged set of results
> (until "last")
> 
> If there is a setting that limits the sorting to 1,000, then I would expect
> this to happen with the URL with asc in it as well, but it doesn't -
> everything appears to be as expected with all the results set 4,000+ sorting
> correctly.
> 
> Maybe for the asc URL this is the "default" for the Zebra config, would that
> explain why they act differently?

Yeah, the default sort in Zebra is ascending by internal record number which is the same as biblionumber. 

What happens with MARC21? Surely it must be the same behaviour...
Comment 9 David Nind 2022-10-14 06:49:05 UTC
(In reply to David Cook from comment #8)
 
> Yeah, the default sort in Zebra is ascending by internal record number which
> is the same as biblionumber. 
> 
> What happens with MARC21? Surely it must be the same behaviour...

There are only 400 odd records in the sample data set, so doesn't show as an issue.

I've had no luck exporting all the records from Koha and then importing at the moment, so can't test this out...
Comment 10 Nick Clemens (kidclamp) 2022-10-14 19:37:13 UTC
I don't see any problems with 1000+ records in MARC21

I wonder though, do we need to remove the XSLT changes? The sorting works the same with or without them, and helps those who might not know enough about YAZ options but use Z39.50 to search Koha. I suggest dropping the last two patches.

Sorting will still work without a reindex. Happy to add my stamp once David is good for sign off
Comment 11 David Cook 2022-10-16 23:15:11 UTC
(In reply to Nick Clemens from comment #10)
> I don't see any problems with 1000+ records in MARC21
> 
> I wonder though, do we need to remove the XSLT changes? The sorting works
> the same with or without them, and helps those who might not know enough
> about YAZ options but use Z39.50 to search Koha. I suggest dropping the last
> two patches.
 
I'm happy for the last two patches to be dropped.
Comment 12 David Cook 2022-10-16 23:16:06 UTC
(In reply to David Nind from comment #5)
> However, for UNIMARC:
> - I changed the count number =100
> - After the first 10 pages of results (that is, from page 11 and the 1001
> search result) the order changes to asc starting from 1
 
Have you tried testing this before applying the patch? Does it only happy with these patches?
Comment 13 Katrin Fischer 2022-11-14 22:34:51 UTC
What's the status here? Should the last 2 patches be obsoleted and this be retested?
Comment 14 David Cook 2022-11-15 01:36:47 UTC
(In reply to Katrin Fischer from comment #13)
> What's the status here? Should the last 2 patches be obsoleted and this be
> retested?

I suppose I'm fairly indifferent overall. I'm probably OK with this just being Failed QA / Resolved Won't Fix.