Bug 26869 - Enable batch record modification to create 952/item on existing bibs
Summary: Enable batch record modification to create 952/item on existing bibs
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement with 10 votes (vote)
Assignee: Pedro Amorim
QA Contact: Testopia
URL:
Keywords: Sandbox
Depends on:
Blocks:
 
Reported: 2020-10-29 20:56 UTC by Andrew Fuerste-Henry
Modified: 2024-06-25 17:26 UTC (History)
19 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Screenshot of my MARC modification template for OverDrive records (147.61 KB, image/png)
2020-10-30 10:37 UTC, Owen Leonard
Details
Bug 26869: Add items from MARC in BatchUpdateBiblio (2.87 KB, patch)
2024-02-23 15:02 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 26869: Add items from MARC in BatchUpdateBiblio (2.85 KB, patch)
2024-02-23 15:48 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 26869: Add items from MARC in BatchUpdateBiblio (2.85 KB, patch)
2024-02-23 15:51 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 26869: Add MARC modification template mapping validation (3.63 KB, patch)
2024-05-10 12:36 UTC, Pedro Amorim
Details | Diff | Splinter Review
Import file to test marc modification template (5.36 KB, text/plain)
2024-05-31 15:37 UTC, Esther Melander
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Fuerste-Henry 2020-10-29 20:56:24 UTC
Say I have a couple thousand records for eBooks. I imported them without adding items, but decided later that I'd like to have items to ease searching. I'd love to be able to add those to my bibs in a batch without exporting and re-importing them. I'm imagining an interface like additem.pl except items I define would be created across all the bibs I've indicated. Ideally, there'd be some mechanism for values like itype and call number to choose whether you want Koha to apply a static value for all the items or derive specific values from the bibs.
Comment 1 Andrew Fuerste-Henry 2020-10-29 20:57:41 UTC
Alternate path to the same end goal: If we had a mechanism to stage records for re-import and overlay without ever downloading. Define a list of bibs and tell Koha to copy those into the reservoir, applying a MARC mod template on the way.
Comment 2 Katrin Fischer 2020-10-29 22:02:38 UTC
You can use MARC modification templates and the batch record modification tool to add the items.
Comment 3 Andrew Fuerste-Henry 2020-10-29 22:17:45 UTC
Do you mean add a 952 to an existing bib using a marc mod template and batch bib modification? That's never worked for me. If I tell the marc mod template to add a 952, Koha just doesn't do it. No new item is created and the MARC isn't changed.
Comment 4 Katrin Fischer 2020-10-29 22:18:52 UTC
Hm, I haven't done it in a while, but I am pretty sure it worked at some point.
Comment 5 Owen Leonard 2020-10-30 10:37:21 UTC
Created attachment 112687 [details]
Screenshot of my MARC modification template for OverDrive records

(In reply to Andrew Fuerste-Henry from comment #3)
> Do you mean add a 952 to an existing bib using a marc mod template and batch
> bib modification?

I use a MARC modification template when importing OverDrive records, and I don't see why it wouldn't work in batch record modification.
Comment 6 Andrew Fuerste-Henry 2020-10-30 11:19:11 UTC
> I use a MARC modification template when importing OverDrive records, and I
> don't see why it wouldn't work in batch record modification.

Yeah, I know that works at import. It does not currently work for bibs already in Koha and, in my memory, never has. I'd assumed there was some basic structural reason why this was difficult/impossible, but if we can make it work that accomplishes what I'm after.
Comment 7 Katrin Fischer 2020-10-30 12:10:37 UTC
Thinking about it I might have done it on import too - I used it to add items to sample/test data for easier testing before we had the nice sample data available.

I think a reason could be that the import parses 952 tags and then moves them to them to items. The batch record modification is probably not able to process 952... but I think making it possible would be a good idea as a lot of the "infrastructure" we want would already be in place.
Comment 8 Andrew Fuerste-Henry 2020-10-30 12:26:43 UTC
Re-titled the bug. I think this is a good path toward what I want to do. Let's make the Batch Record Mod tool able to create a 952 and then translate it into an item, as the import process does.
Comment 9 Barbara Johnson 2020-10-30 15:50:40 UTC
I recently tried adding items to existing records using a MARC mod template and it didn’t work.  It took a lot of time to identify, extract, modify and import them back in.  So we would love the ability to do this from within Koha.
Comment 10 Nick Clemens (kidclamp) 2020-10-30 15:52:59 UTC
(In reply to Andrew Fuerste-Henry from comment #6)
> > I use a MARC modification template when importing OverDrive records, and I
> > don't see why it wouldn't work in batch record modification.
> 
> Yeah, I know that works at import. It does not currently work for bibs
> already in Koha and, in my memory, never has. I'd assumed there was some
> basic structural reason why this was difficult/impossible, but if we can
> make it work that accomplishes what I'm after.

We use C4::Biblio::ModBiblio here, so all item changes are stripped before saving
Comment 11 Joe Sikowitz 2021-02-04 20:29:27 UTC
I would also like to be able to do add an item this way. It would be very useful.
Comment 12 Aude Charillon 2022-11-23 11:38:50 UTC
I agree with previous comments that it would be useful to be able to add items to existing records using a MARC modification template. It just happens that sometimes you change how you deal with eBook records and all the existing ones need to be amended!
Comment 13 Susan 2022-11-23 16:44:10 UTC
I agree having the option to list items makes sense - It helps with limiting results to specific library having a journal/book online or in print format.
Comment 14 Chris Rowlands 2024-01-29 10:06:56 UTC
Just throwing my hat into the ring of people who would find this feature useful - particularly for changing ebook records as others have mentioned.
Comment 15 Pedro Amorim 2024-02-23 15:02:47 UTC Comment hidden (obsolete)
Comment 16 Pedro Amorim 2024-02-23 15:04:36 UTC
Hey, I've submitted a patch. It's a bit more than a proof of concept as I've tested it quite a bit and I believe it achieves what's required, but a more experienced set of eyes and testers would be great here. I'd love to hear some feedback on this.
Comment 17 Pedro Amorim 2024-02-23 15:48:05 UTC Comment hidden (obsolete)
Comment 18 Pedro Amorim 2024-02-23 15:51:46 UTC
Created attachment 162388 [details] [review]
Bug 26869: Add items from MARC in BatchUpdateBiblio

Test plan, clean k-t-d:

Before test, pick a record and confirm it has no items, for this test we're using biblio 79:
<staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79

1) Add a new MARC modification template, visit:
<staff_url>/cgi-bin/koha/tools/marc_modification_templates.pl
2) Click 'New template', enter a name and click 'Submit'
3) Click 'New action'
4) Pick 'Add new' and enter '952' as field and 'a' as subfield, enter 'CPL' as value
5) Click 'Add action'
6) Do a batch record modification, visit:
<staff_url>/cgi-bin/koha/tools/batch_record_modification.pl
7) Select the tab 'Enter a list of record numbers' and enter '79'
8) On the template, select the template previously created. Click 'continue'
9) Click 'Modify selected records'. Click 'View detail of the enqueued job'. Notice success message is shown with no errors
10) Check the biblio, visit:
<staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79
11) Notice it now has an item with homebranch 'CPL'

test with 952 but with other subfields, without subfield 'a' -> confirm 'FK constraint' error is shown, no modification made
test with more than one occurrence of 952$a -> confirm 'FK constraint' error is shown, no modification made
test with 952$a too long e.g. 'aaaaaaaa' -> confirm 'too long' error is shown
test with no 952 mapping -> confirm things work as intended
test with 952 mapping plus other tags -> confirm things work as intended

prove koha/t/db_dependent/Koha/BackgroundJobs/BatchUpdateBiblio.t
prove koha/t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblio.t
Comment 19 Esther Melander 2024-02-29 16:55:41 UTC
I would sign-off because the test plan works. This is an exciting enhancement but there were a few things I noticed that may be worth discussion first.

The test plan works through step 11 and an item is created. The item created defaulted to the 952$y item type Book, even though only the 952$a is in the action. That may not always be the correct item type, though it is probably better for something to be assigned than nothing.

Then I created a template with actions to add the 952$a and other subfields including adding the item type 952$y as configured. The items were created correctly.

I duplicated the previous template and removed the action to include the 952$a. Items were created with no error message. I tested removing other subfields and items were created.

The test plan implies the 952$a is required to create an item, but I was able to create an item with only the 952$8. The caveat is the item defaulted to the Book item type for the 952$y again.

If the item type is required, as it is added regardless, how can the user be alerted so they can select the correct item type before the items are added.
Comment 20 Andrew Fuerste-Henry 2024-02-29 17:27:57 UTC
If we were creating items from a 952 while importing a new bib, which item fields, if any, would be required? It makes sense to me to have this process match those requirements.
Comment 21 Pedro Amorim 2024-03-01 11:15:47 UTC
(In reply to Esther Melander from comment #19)
> The test plan works through step 11 and an item is created. The item created
> defaulted to the 952$y item type Book, even though only the 952$a is in the
> action. That may not always be the correct item type, though it is probably
> better for something to be assigned than nothing.

Good catch. Thanks. I took a deeper look and I'm convinced this is because of the following code in Item.pm sub store:

unless ( $self->itype ) {
    $self->itype($self->biblio->biblioitem->itemtype);
}
    
The item will default to the itemtype at the biblio level if not mapping for itype is exists.
In the case of this test plan, biblio 79 field 942$c is 'BK' so the code is grabbing that when storing the item. 
This follows core logic as the patch is using a core method AddItemFromMarc.
I think this is a good behavior for this use case. Let me know if you disagree!

> The test plan implies the 952$a is required to create an item, but I was
> able to create an item with only the 952$8.

Interesting! I'm sure I was unable to create an item without 952$a before but I've tried now and, as you say, the item is created regardless of 952$a mapping existing or not.
I had different approaches for this solution before the current patch, using different methods, so this is likely a remnant of that i.e. with a previous version of the patch this was the case, but not anymore it seems.

(In reply to Andrew Fuerste-Henry from comment #20)
> If we were creating items from a 952 while importing a new bib, which item
> fields, if any, would be required? It makes sense to me to have this process
> match those requirements.

This does make sense, I agree.
Are you referring to bulkmarcimport.pl or a different method? Can you please provide a test plan on creating items from a 952 while importing a new bib? If so, I'll have a go at testing that myself. Thanks!
Comment 22 Esther Melander 2024-03-01 18:24:45 UTC
I created a marc modification template per the test plan that creates an item with only 952$a. The new item   was added to a bibliographic record where the 942$c = Visual Materials. The 952$y did default to matching the 942$c. So this is perhaps not a blocker, but could be made a bit more clear in documentation.

As far as required fields, I think Koha will allow an item record to be created/exist with no required fields. However in practice, it seems the 952 $a, $b, $y are required in order for downstream circulation, holds, etc., to function properly. Perhaps, this is best addressed in documentation rather than forcing required fields. Or, perhaps there is a way to give best practices when creating items in general.
Comment 23 Alexandre Noel 2024-05-09 16:16:14 UTC
After doing the test plan I noticed that the OPAC view on the user side return an error 500.
<site_url>/cgi-bin/koha/opac-detail.pl?biblionumber=# (# : number of the tested biblio)
Comment 24 Katrin Fischer 2024-05-09 16:21:44 UTC
The problem might not be related to the patch. Any item that has no item type (no value in 952$y) will cause an error 500 in the OPAC.
Comment 25 Pedro Amorim 2024-05-10 12:36:04 UTC
Created attachment 166541 [details] [review]
Bug 26869: Add MARC modification template mapping validation

Follow the same test plan as above, but attempt to add an item without having at least homebranch, holdingbranch and type subfields.
Notice you now get an error with an appropriate message and the item is not added.
Comment 26 Esther Melander 2024-05-22 19:09:12 UTC
This is very close, but I wanted to share some notes from my testing.

Following the testing plan, created a template that added only a 952$a and I did receive a error message on why the item was not added.

My next template looked like this:

Add 952$a CPL
Add 952$b CPL
Add 952$y BK

This template does work when modifying the record from the Bibliographic detail page. It adds a single item record. However, if this same template is used in Stage MARC records, it adds three item records.

A better template would look like this:

Add 952$a CPL
Update existing or add new field 952$b CPL
Update existing or add new field 952$y BK

This template will add only one item record in both scenarios.

This is perhaps not a blocker and I will happily sign-off. Just wanted to make a note that the behavior is different and it should probably be consistent.
Comment 27 Mathieu Saby 2024-05-23 08:13:58 UTC
Hi
Does this patch only work for Marc21, or does it take into account Unimarc (where items are mapped to 995 field) ?
Comment 28 Katrin Fischer 2024-05-23 08:22:13 UTC
Pedro will be able to tell you more, but on first glance it seems good for UNIMARC:

+sub marc_record_contains_item_data {
+    my ($record) = @_;
+
+    my $itemfield   = C4::Context->preference('marcflavour') eq 'UNIMARC' ? '995' : '952';
+    my @item_fields = $record->field($itemfield);
+    return scalar @item_fields;
+}

We still need a sign-off, maybe you could test with a UNIMARC sandbox?
Comment 29 Mathieu Saby 2024-05-23 11:18:12 UTC
Impossible to test in Biblibre sandbox, but I suspect it is a general issue affecting sandboxes?

In provision logs I get this :

fatal: [localhost -> koha-box26869]: FAILED! => {"changed": true, "cmd": "koha-shell -c 'cd /kohadevbox/koha && yes | git bz apply 26869' box26869", "delta": "0:00:00.379079", "end": "2024-05-23 11:02:57.723063", "msg": "non-zero return code", "rc": 1, "start": "2024-05-23 11:02:57.343984", "stderr": "Error getting login cookie from browser:\n   Cannot find default Firefox profile\n\nConfigured browser: firefox3 (change with 'git config --global bz.browser <value>')\nPossible browsers: chromium, epiphany, firefox3, galeon, google-chrome", "stderr_lines": ["Error getting login cookie from browser:", "   Cannot find default Firefox profile", "", "Configured browser: firefox3 (change with 'git config --global bz.browser <value>')", "Possible browsers: chromium, epiphany, firefox3, galeon, google-chrome"], "stdout": "", "stdout_lines": []}
Comment 30 Pedro Amorim 2024-05-27 11:53:26 UTC
(In reply to Mathieu Saby from comment #27)
> Hi
> Does this patch only work for Marc21, or does it take into account Unimarc
> (where items are mapped to 995 field) ?

The code considers MARC flavour and should act accordingly.

(In reply to Mathieu Saby from comment #29)
> Impossible to test in Biblibre sandbox, but I suspect it is a general issue
> affecting sandboxes?

This indeed appears to be a general issue affecting sandboxes, community has been made aware and are looking into it.
Comment 31 Pedro Amorim 2024-05-27 11:58:49 UTC
(In reply to Esther Melander from comment #26)
> This is very close, but I wanted to share some notes from my testing.
> 
> Following the testing plan, created a template that added only a 952$a and I
> did receive a error message on why the item was not added.
> 
> My next template looked like this:
> 
> Add 952$a CPL
> Add 952$b CPL
> Add 952$y BK
> 
> This template does work when modifying the record from the Bibliographic
> detail page. It adds a single item record. However, if this same template is
> used in Stage MARC records, it adds three item records.
> 
> A better template would look like this:
> 
> Add 952$a CPL
> Update existing or add new field 952$b CPL
> Update existing or add new field 952$y BK
> 
> This template will add only one item record in both scenarios.
> 
> This is perhaps not a blocker and I will happily sign-off. Just wanted to
> make a note that the behavior is different and it should probably be
> consistent.

Interesting and fair points.
I'm not entirely sure what the best course of action would be here, if any is required.
Comparing the code from stage MARC and biblio records modification nothing obvious stands out to me, although these are certainly 2 different areas in the code.
Can you please provide a more detailed test plan (or include a MARC records file - if required) so that I can try and reproduce the exact use case and test further? Thank you.
Comment 32 David Cook 2024-05-29 03:51:43 UTC
(In reply to Esther Melander from comment #26)
> My next template looked like this:
> 
> Add 952$a CPL
> Add 952$b CPL
> Add 952$y BK
> 
> This template does work when modifying the record from the Bibliographic
> detail page. It adds a single item record. However, if this same template is
> used in Stage MARC records, it adds three item records.

In 23.11.x prod, I'm noticing a template like that does indeed create 3 items (one per action), when used with Stage MARC records. 
 
> A better template would look like this:
> 
> Add 952$a CPL
> Update existing or add new field 952$b CPL
> Update existing or add new field 952$y BK

Did you test this? It seems to me that you'd still wind up with 3 items, because there wouldn't be a pre-existing 952$b and 952$y?
Comment 33 David Cook 2024-05-29 03:57:14 UTC
(In reply to David Cook from comment #32)
> > A better template would look like this:
> > 
> > Add 952$a CPL
> > Update existing or add new field 952$b CPL
> > Update existing or add new field 952$y BK
> 
> Did you test this? It seems to me that you'd still wind up with 3 items,
> because there wouldn't be a pre-existing 952$b and 952$y?

I just tested this, and it looks like you're right.

I'll never understand the MARC Modification Templates...
Comment 34 Esther Melander 2024-05-31 15:37:19 UTC
Created attachment 167312 [details]
Import file to test marc modification template

Here is my test plan:

1. Create a marc modification template with these actions:

Add 952$a CPL
Add 952$b CPL
Add 952$y BK

2. Find a bibliographic record. Edit - Modify record using template.
3. Apply the marc modification template.

See that one item record is created.

4. Now Stage a MARC record (example file uploaded, flora.mrc). Select the marc modification template above. Import.

See that three item records are created. Each of the add actions creates an item record.

5. Now create a new marc modification template with these actions:

Add 952$a CPL
Update existing or add new field 952$b CPL
Update existing or add new field 952$y BK

6. Repeat the steps above, but note that only one item record is created each time.

As mentioned previously, this is perhaps not a blocker for this bug. Documentation can provide an example template like the second example. The inconsistency could be subject to a separate bug. I would be happy to sign off with some guidance on the best course of action.
Comment 35 Pedro Amorim 2024-06-20 15:36:15 UTC
Thank you, Esther, for the detailed instructions and sample file.
I believe that for the stage file import scenario, the setting 'How to process items:' being 'Always add items' (default) is what is causing each new MARC item entry (952 for MARC21) to forcibly create a new item.

Imo, the ideal solution here would be to also have a 'How to process items' option in the MARC modification template tool UI (the screen that shows after you select 'Modify record using template') from a bib record. This would work around/with the actions coming from the modification template in the same way the stage import flow currently does, ideally using the same code, for consistency.

In conclusion, I believe the behavior differs because the 'stage MARC record file for import' has additional logic on top of the MARC modification template actions, whereas the 'Modify record using template' only considers the actions from the MARC modification template and nothing else.

Would the suggestion above be better handled in a follow-up bug or here?

What do others think about this?
Comment 36 David Cook 2024-06-20 23:26:11 UTC
(In reply to Pedro Amorim from comment #35)
> What do others think about this?

I'm not sure at the moment. I find the whole situation to be a confusing and unintuitive. 

Of course, any changes to how it processes now will cause potential breaking changes for anyone who expects it to work as it works at the moment.
Comment 37 Andrew Fuerste-Henry 2024-06-21 13:39:55 UTC
(In reply to Pedro Amorim from comment #35)
> Thank you, Esther, for the detailed instructions and sample file.
> I believe that for the stage file import scenario, the setting 'How to
> process items:' being 'Always add items' (default) is what is causing each
> new MARC item entry (952 for MARC21) to forcibly create a new item.
> 
> Imo, the ideal solution here would be to also have a 'How to process items'
> option in the MARC modification template tool UI (the screen that shows
> after you select 'Modify record using template') from a bib record. This
> would work around/with the actions coming from the modification template in
> the same way the stage import flow currently does, ideally using the same
> code, for consistency.
> 
> In conclusion, I believe the behavior differs because the 'stage MARC record
> file for import' has additional logic on top of the MARC modification
> template actions, whereas the 'Modify record using template' only considers
> the actions from the MARC modification template and nothing else.
> 
> Would the suggestion above be better handled in a follow-up bug or here?
> 
> What do others think about this?

I like Pedro's idea of adding a "How to process items" dropdown to the Batch Record Modification tool so that the modification of existing bibs is handled using as much existing code from the record import process as possible, thereby ensuring consistency of behavior.

I would rather see that functionality added in this bug than in a follow-up. Introducing this feature with its current inconsistency of behavior relative to the import process seems like it will create confusion that will persist even after a follow-up bug makes the behavior consistent.
Comment 38 Aude Charillon 2024-06-25 16:41:01 UTC
(In reply to Pedro Amorim from comment #35)
> Would the suggestion above be better handled in a follow-up bug or here?
> 
> What do others think about this?

I apologise in advance for adding a differing opinion here... I think Pedro's suggestion to clarify the MARC modification templates functionality is nice - but I also think it belongs in a separate bug.

As Esther's thorough testing highlighted (in comment #34), there is already a way to make sure only one item is created: you simply need to use "Update existing or add new field" in subsequent actions rather than "Add". The Koha manual can easily be updated to give advice on this as well.