Bug 34128 - Clean up biblioitems.serial data
Summary: Clean up biblioitems.serial data
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-06-27 09:37 UTC by Matt Blenkinsop
Modified: 2023-07-26 09:17 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 34128: Prevent holdings table from incorrectly ordering items (1.86 KB, patch)
2023-06-27 09:56 UTC, Matt Blenkinsop
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Blenkinsop 2023-06-27 09:37:11 UTC
The presence of a series entry in the 490 field causes the ordering of items to change in the holdings table. The different ordering is designed to display serial/journal items correctly but it shouldn't be triggered by a 490 entry for a "Books" item type

To replicate:
1) In cataloging, click on New record
2) Fill in all the required fields with generic data, ensuring that the 942c field (Koha item type) is "Books"
3) In the 490a field - fill in any string as the Series statement
4) Click save
5) You should now be on the Add item page. We need to add 3+ items, making sure that the Home and Current libraries for each item are in reverse alphabetical order.
e.g. 
Item 1:
 - Home library: Midway
 - Current library: Midway
Item 2:
 - Home library: Fairview
 - Current library: Fairview
Item 1:
 - Home library: Centerville
 - Current library: Centerville
6) In the top searchbar, find your new item that you have just added
7) On the item detail page, the holdings table will be displaying the items in reverse alphabetical order by library rather than in alphabetical order as it should be by default
Comment 1 Matt Blenkinsop 2023-06-27 09:56:46 UTC
Created attachment 152711 [details] [review]
Bug 34128: Prevent holdings table from incorrectly ordering items

This patch stops the hildings table from incorrectly ordering books as serials

Test plan:
1) In cataloging, click on New record
2) Fill in all the required fields with generic data, ensuring that the 942c field (Koha item type) is "Books"
3) In the 490a field - fill in any string as the Series statement
4) Click save
5) You should now be on the Add item page. We need to add 3+ items, making sure that the Home and Current libraries for each item are in reverse alphabetical order.
e.g.
Item 1:
 - Home library: Midway
 - Current library: Midway
Item 2:
 - Home library: Fairview
 - Current library: Fairview
Item 1:
 - Home library: Centerville
 - Current library: Centerville
6) In the top searchbar, find your new item that you have just added
7) On the item detail page, the holdings table will be displaying the items in reverse alphabetical order by library rather than in alphabetical order as it should be by default.
8) Apply patch
9) Refresh the item detail page -  the items should now be correctly ordered.
Comment 2 Katrin Fischer 2023-06-28 21:18:04 UTC
Hm, I believe there are some things here that are a little odd. 490 doesn't indicate if something is a serial - it only indicates it's part of a series (a volume of a series). I believe I haven't seen serial flag set in MARC21 installations, but I might be mistaken. Could a library have changed the Koha-to-MARC mappings falsely resulting in it to validate to true?

Also we should not hardcode a framework code. Koha comes with some examples, but libraries can and will built their own with any code. 

Also: Test plan needs updating (mentions 942 set to Books)

I'd suggest checking the mappings!
Comment 3 Katrin Fischer 2023-06-28 21:19:23 UTC
Note: default mapping for biblioitems.serial appears to be 942$s!
Comment 4 Phil Ringnalda 2023-06-29 00:52:51 UTC
Koha does try to set biblio.serial based on seriestitle, in https://git.koha-community.org/Koha-community/Koha/src/commit/532cf4ab9f5549c1e8f41d6c7191f9e1a9ed9e3c/C4/Biblio.pm#L233

And from a quick sandbox test, it succeeds (had to use a sandbox, since my production install has seriestitle only mapped from 440, not 490, so I lack both seriestitle and serial).

Blame says that Koha has been that way at least since "rel_3_0 moved to HEAD" 17 years ago. It's not sane, but it's a longstanding lack of sanity.
Comment 5 Phil Ringnalda 2023-06-29 23:08:29 UTC
Amusingly, there's also is_serial, which CheckPrevCheckout added because it wanted to know about actual serials where you don't want to warn someone checking out the current issue of People that they checked out the issue from last week, and the issue from the week before, and so on.

Alas, it says something is_serial if biblio->serial, or if leader/07 is "s", rather than only checking the leader (which is also bad for it, since for people who actually have 490 mapped to seriestitle, and thus set serial for it, people who have forgotten that they already read number 34 in a series won't be warned about that). Maybe we need to add really_is_serial?

Or for that matter, really_really_is_serial, since what this code actually needs to know is whether or not sorting by serialid.publisheddate makes sense, which it only does if all, or maybe at least two, of the itemnumbers exist in serialitems.itemnumber. I have plenty of biblios with leader/07="s", but since I don't use the serials module, serialid.publisheddate doesn't do anything for me. Or I guess in a less painful way to look up for every load of every items table, if the biblionumber exists in a serial.biblionumber.
Comment 6 Katrin Fischer 2023-06-30 20:41:21 UTC
I believe that mapping seriestitle probably made sense in UNIMARC as the serial flag seems to stem from there. It would be interesting to see what is mapped in UNIMARC and if that makes more sense.

Can we agree that a volume in a series is not a serial?
Comment 7 Phil Ringnalda 2023-07-01 03:52:31 UTC
By default, UNIMARC maps biblio.serial to both 099$s and 942$s, neither one editable by default, and maps biblio.seriestitle to nothing at all. So in UNIMARC you only get biblio.serial if you either make one of the $s subfields editable, or you map biblio.seriestitle to something thinking that doing so will do you some good.

I can agree that being a volume in a series has no relationship whatsoever to being a serial, if that helps. According to the first session in https://www.loc.gov/catworkshop/courses/naco-full%20series-RDA/course%20table.html in 2013 when they last updated their numbers, 89.4% of the series headings in the LC/NACO name authority database were monograph series ("Annual report of the tin production in Moldavia"), and monograph series are serials by definition, unlike multipart monograph series ("Continents of the world") that have a fixed ending. So I can't agree that "a volume in a series is not a serial," only that "a volume in a series either is or is not a serial."

I would sign off a patch that removes setting biblio.serial based on biblio.seriestitle in a heartbeat. More than a heartbeat: it's only set in AddBiblio, not ModBiblio, so you have to have a seriestitle when you first save a record to have serial set.

Even better, ModBiblio actually gets rid of it. The first time you save a biblio without a seriestitle, serial is set to 0, but if you save it again it is reset to null; the first time you save a biblio with a seriestitle, serial is set to 1, but if you save it again it is set to null, making the whole seriestitle -> serial thing useless.

Workaround for Matt's clients who are annoyed by unsorted items from a 490 existing: edit the biblio and save it without making any changes. Poof, no longer a serial.
Comment 8 Phil Ringnalda 2023-07-01 04:54:18 UTC
Oy. It hasn't been that way for over 17 years, it's been that way for 16 years, since https://git.koha-community.org/Koha-community/Koha/commit/af466ca41ab02150f4f6c60bf6262b079b2e6e85 looked at the pointless code that was

my $series       = 0;

if ( $biblio->{'seriestitle'} ) { $series = 1 }

and said "well, $series is unused, there's no biblio.series for it to set, but serial is just two letters differnt than series so they must have meant

# set the series flag
my $serial = 0;
if ( $biblio->{'seriestitle'} ) { $serial = 1 }
"

so this whole stupid thing is because someone looked at dead code and instead of deleting it, made it do something very wrong instead.
Comment 9 Phil Ringnalda 2023-07-01 05:33:01 UTC
Filed bug 34182 for removing the bogus code, though that won't help here since the damage has already been done to existing installations.
Comment 10 Phil Ringnalda 2023-07-03 14:37:45 UTC
I've never used the serials module. Is it reasonable to expect that there will be biblios with some items added through it which have a serialid, and other items not added through it, either from before a single library started using it, or from other libraries in a consortium not using it?
Comment 11 Martin Renvoize 2023-07-14 18:48:36 UTC
Should we retask this one for creating a database update that fixes any bad data found.. I'm not sure how easy it would be to identify such cases
Comment 12 Katrin Fischer 2023-07-16 10:43:14 UTC
I would love to see clean-up, but not sure how we can best do it.

* Whatever we do, we want to limit to MARC21 installations.
* We don't know if the library has been setting the flag manually in some cases.
* We will need to access MARC data, which might make any clean-up really slow on big databases.


Ideas:

Would running misc/batchRebuildBiblioTables.pl.fix the issue now? Then we could add a database statement recommending to run it.

Could we assume that we only if 942$s exists/is filled, we should set biblioitems.serial? I think the previously false code, didn't create that field, right or wrong?


Thinking more about it... what does the alternative sorting do exactly?
It seems to counteract OPACHoldingsDefaultSortField?