Summary: | Enable batch record modification to create 952/item on existing bibs | ||
---|---|---|---|
Product: | Koha | Reporter: | Andrew Fuerste-Henry <andrewfh> |
Component: | Cataloging | Assignee: | Pedro Amorim <pedro.amorim> |
Status: | Signed Off --- | QA Contact: | Testopia <testopia> |
Severity: | enhancement | ||
Priority: | P5 - low | CC: | aude.charillon, barbara.johnson, blawlor, catrina, chris.rowlands6, david, dcook, esther.melander, heather_hernandez, helm.consortium, joe, lauren_denny, lucy.vaux-harvey, m.de.rooy, martin.renvoize, mathsabypro, mspinney, pedro.amorim, severine.queune, susan.merner |
Version: | Main | Keywords: | Sandbox |
Hardware: | All | ||
OS: | All | ||
Change sponsored?: | --- | Patch complexity: | --- |
Documentation contact: | Documentation submission: | ||
Text to go in the release notes: | Version(s) released in: | ||
Circulation function: | |||
Attachments: |
Screenshot of my MARC modification template for OverDrive records
Bug 26869: Add items from MARC in BatchUpdateBiblio Bug 26869: Add items from MARC in BatchUpdateBiblio Bug 26869: Add items from MARC in BatchUpdateBiblio Bug 26869: Add MARC modification template mapping validation Import file to test marc modification template Bug 26869: Add items from MARC in BatchUpdateBiblio Bug 26869: Add MARC modification template mapping validation |
Description
Andrew Fuerste-Henry
2020-10-29 20:56:24 UTC
Alternate path to the same end goal: If we had a mechanism to stage records for re-import and overlay without ever downloading. Define a list of bibs and tell Koha to copy those into the reservoir, applying a MARC mod template on the way. You can use MARC modification templates and the batch record modification tool to add the items. Do you mean add a 952 to an existing bib using a marc mod template and batch bib modification? That's never worked for me. If I tell the marc mod template to add a 952, Koha just doesn't do it. No new item is created and the MARC isn't changed. Hm, I haven't done it in a while, but I am pretty sure it worked at some point. Created attachment 112687 [details] Screenshot of my MARC modification template for OverDrive records (In reply to Andrew Fuerste-Henry from comment #3) > Do you mean add a 952 to an existing bib using a marc mod template and batch > bib modification? I use a MARC modification template when importing OverDrive records, and I don't see why it wouldn't work in batch record modification. > I use a MARC modification template when importing OverDrive records, and I
> don't see why it wouldn't work in batch record modification.
Yeah, I know that works at import. It does not currently work for bibs already in Koha and, in my memory, never has. I'd assumed there was some basic structural reason why this was difficult/impossible, but if we can make it work that accomplishes what I'm after.
Thinking about it I might have done it on import too - I used it to add items to sample/test data for easier testing before we had the nice sample data available. I think a reason could be that the import parses 952 tags and then moves them to them to items. The batch record modification is probably not able to process 952... but I think making it possible would be a good idea as a lot of the "infrastructure" we want would already be in place. Re-titled the bug. I think this is a good path toward what I want to do. Let's make the Batch Record Mod tool able to create a 952 and then translate it into an item, as the import process does. I recently tried adding items to existing records using a MARC mod template and it didn’t work. It took a lot of time to identify, extract, modify and import them back in. So we would love the ability to do this from within Koha. (In reply to Andrew Fuerste-Henry from comment #6) > > I use a MARC modification template when importing OverDrive records, and I > > don't see why it wouldn't work in batch record modification. > > Yeah, I know that works at import. It does not currently work for bibs > already in Koha and, in my memory, never has. I'd assumed there was some > basic structural reason why this was difficult/impossible, but if we can > make it work that accomplishes what I'm after. We use C4::Biblio::ModBiblio here, so all item changes are stripped before saving I would also like to be able to do add an item this way. It would be very useful. I agree with previous comments that it would be useful to be able to add items to existing records using a MARC modification template. It just happens that sometimes you change how you deal with eBook records and all the existing ones need to be amended! I agree having the option to list items makes sense - It helps with limiting results to specific library having a journal/book online or in print format. Just throwing my hat into the ring of people who would find this feature useful - particularly for changing ebook records as others have mentioned. Created attachment 162383 [details] [review] Bug 26869: Add items from MARC in BatchUpdateBiblio Test plan, clean k-t-d: Before test, pick a record and confirm it has no items, for this test we're using biblio 79: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 1) Add a new MARC modification template, visit: <staff_url>/cgi-bin/koha/tools/marc_modification_templates.pl 2) Click 'New template', enter a name and click 'Submit' 3) Click 'New action' 4) Pick 'Add new' and enter '952' as field and 'a' as subfield, enter 'CPL' as value 5) Click 'Add action' 6) Do a batch record modification, visit: <staff_url>/cgi-bin/koha/tools/batch_record_modification.pl 7) Select the tab 'Enter a list of record numbers' and enter '76' 8) On the template, select the template previously created. Click 'continue' 9) Click 'Modify selected records'. Click 'View detail of the enqueued job'. Notice success message is shown with no errors 10) Check the biblio, visit: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 11) Notice it now has an item with homebranch 'CPL' test with 952 but with other subfields, without subfield 'a' -> confirm 'FK constraint' error is shown, no modification made test with more than one occurrence of 952$a -> confirm 'FK constraint' error is shown, no modification made test with 952$a too long e.g. 'aaaaaaaa' -> confirm 'too long' error is shown test with no 952 mapping -> confirm things work as intended test with 952 mapping plus other tags -> confirm things work as intended prove koha/t/db_dependent/Koha/BackgroundJobs/BatchUpdateBiblio.t prove koha/t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblio.t Hey, I've submitted a patch. It's a bit more than a proof of concept as I've tested it quite a bit and I believe it achieves what's required, but a more experienced set of eyes and testers would be great here. I'd love to hear some feedback on this. Created attachment 162387 [details] [review] Bug 26869: Add items from MARC in BatchUpdateBiblio Test plan, clean k-t-d: Before test, pick a record and confirm it has no items, for this test we're using biblio 79: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 1) Add a new MARC modification template, visit: <staff_url>/cgi-bin/koha/tools/marc_modification_templates.pl 2) Click 'New template', enter a name and click 'Submit' 3) Click 'New action' 4) Pick 'Add new' and enter '952' as field and 'a' as subfield, enter 'CPL' as value 5) Click 'Add action' 6) Do a batch record modification, visit: <staff_url>/cgi-bin/koha/tools/batch_record_modification.pl 7) Select the tab 'Enter a list of record numbers' and enter '76' 8) On the template, select the template previously created. Click 'continue' 9) Click 'Modify selected records'. Click 'View detail of the enqueued job'. Notice success message is shown with no errors 10) Check the biblio, visit: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 11) Notice it now has an item with homebranch 'CPL' test with 952 but with other subfields, without subfield 'a' -> confirm 'FK constraint' error is shown, no modification made test with more than one occurrence of 952$a -> confirm 'FK constraint' error is shown, no modification made test with 952$a too long e.g. 'aaaaaaaa' -> confirm 'too long' error is shown test with no 952 mapping -> confirm things work as intended test with 952 mapping plus other tags -> confirm things work as intended prove koha/t/db_dependent/Koha/BackgroundJobs/BatchUpdateBiblio.t prove koha/t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblio.t Created attachment 162388 [details] [review] Bug 26869: Add items from MARC in BatchUpdateBiblio Test plan, clean k-t-d: Before test, pick a record and confirm it has no items, for this test we're using biblio 79: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 1) Add a new MARC modification template, visit: <staff_url>/cgi-bin/koha/tools/marc_modification_templates.pl 2) Click 'New template', enter a name and click 'Submit' 3) Click 'New action' 4) Pick 'Add new' and enter '952' as field and 'a' as subfield, enter 'CPL' as value 5) Click 'Add action' 6) Do a batch record modification, visit: <staff_url>/cgi-bin/koha/tools/batch_record_modification.pl 7) Select the tab 'Enter a list of record numbers' and enter '79' 8) On the template, select the template previously created. Click 'continue' 9) Click 'Modify selected records'. Click 'View detail of the enqueued job'. Notice success message is shown with no errors 10) Check the biblio, visit: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 11) Notice it now has an item with homebranch 'CPL' test with 952 but with other subfields, without subfield 'a' -> confirm 'FK constraint' error is shown, no modification made test with more than one occurrence of 952$a -> confirm 'FK constraint' error is shown, no modification made test with 952$a too long e.g. 'aaaaaaaa' -> confirm 'too long' error is shown test with no 952 mapping -> confirm things work as intended test with 952 mapping plus other tags -> confirm things work as intended prove koha/t/db_dependent/Koha/BackgroundJobs/BatchUpdateBiblio.t prove koha/t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblio.t I would sign-off because the test plan works. This is an exciting enhancement but there were a few things I noticed that may be worth discussion first. The test plan works through step 11 and an item is created. The item created defaulted to the 952$y item type Book, even though only the 952$a is in the action. That may not always be the correct item type, though it is probably better for something to be assigned than nothing. Then I created a template with actions to add the 952$a and other subfields including adding the item type 952$y as configured. The items were created correctly. I duplicated the previous template and removed the action to include the 952$a. Items were created with no error message. I tested removing other subfields and items were created. The test plan implies the 952$a is required to create an item, but I was able to create an item with only the 952$8. The caveat is the item defaulted to the Book item type for the 952$y again. If the item type is required, as it is added regardless, how can the user be alerted so they can select the correct item type before the items are added. If we were creating items from a 952 while importing a new bib, which item fields, if any, would be required? It makes sense to me to have this process match those requirements. (In reply to Esther Melander from comment #19) > The test plan works through step 11 and an item is created. The item created > defaulted to the 952$y item type Book, even though only the 952$a is in the > action. That may not always be the correct item type, though it is probably > better for something to be assigned than nothing. Good catch. Thanks. I took a deeper look and I'm convinced this is because of the following code in Item.pm sub store: unless ( $self->itype ) { $self->itype($self->biblio->biblioitem->itemtype); } The item will default to the itemtype at the biblio level if not mapping for itype is exists. In the case of this test plan, biblio 79 field 942$c is 'BK' so the code is grabbing that when storing the item. This follows core logic as the patch is using a core method AddItemFromMarc. I think this is a good behavior for this use case. Let me know if you disagree! > The test plan implies the 952$a is required to create an item, but I was > able to create an item with only the 952$8. Interesting! I'm sure I was unable to create an item without 952$a before but I've tried now and, as you say, the item is created regardless of 952$a mapping existing or not. I had different approaches for this solution before the current patch, using different methods, so this is likely a remnant of that i.e. with a previous version of the patch this was the case, but not anymore it seems. (In reply to Andrew Fuerste-Henry from comment #20) > If we were creating items from a 952 while importing a new bib, which item > fields, if any, would be required? It makes sense to me to have this process > match those requirements. This does make sense, I agree. Are you referring to bulkmarcimport.pl or a different method? Can you please provide a test plan on creating items from a 952 while importing a new bib? If so, I'll have a go at testing that myself. Thanks! I created a marc modification template per the test plan that creates an item with only 952$a. The new item was added to a bibliographic record where the 942$c = Visual Materials. The 952$y did default to matching the 942$c. So this is perhaps not a blocker, but could be made a bit more clear in documentation. As far as required fields, I think Koha will allow an item record to be created/exist with no required fields. However in practice, it seems the 952 $a, $b, $y are required in order for downstream circulation, holds, etc., to function properly. Perhaps, this is best addressed in documentation rather than forcing required fields. Or, perhaps there is a way to give best practices when creating items in general. After doing the test plan I noticed that the OPAC view on the user side return an error 500. <site_url>/cgi-bin/koha/opac-detail.pl?biblionumber=# (# : number of the tested biblio) The problem might not be related to the patch. Any item that has no item type (no value in 952$y) will cause an error 500 in the OPAC. Created attachment 166541 [details] [review] Bug 26869: Add MARC modification template mapping validation Follow the same test plan as above, but attempt to add an item without having at least homebranch, holdingbranch and type subfields. Notice you now get an error with an appropriate message and the item is not added. This is very close, but I wanted to share some notes from my testing. Following the testing plan, created a template that added only a 952$a and I did receive a error message on why the item was not added. My next template looked like this: Add 952$a CPL Add 952$b CPL Add 952$y BK This template does work when modifying the record from the Bibliographic detail page. It adds a single item record. However, if this same template is used in Stage MARC records, it adds three item records. A better template would look like this: Add 952$a CPL Update existing or add new field 952$b CPL Update existing or add new field 952$y BK This template will add only one item record in both scenarios. This is perhaps not a blocker and I will happily sign-off. Just wanted to make a note that the behavior is different and it should probably be consistent. Hi Does this patch only work for Marc21, or does it take into account Unimarc (where items are mapped to 995 field) ? Pedro will be able to tell you more, but on first glance it seems good for UNIMARC: +sub marc_record_contains_item_data { + my ($record) = @_; + + my $itemfield = C4::Context->preference('marcflavour') eq 'UNIMARC' ? '995' : '952'; + my @item_fields = $record->field($itemfield); + return scalar @item_fields; +} We still need a sign-off, maybe you could test with a UNIMARC sandbox? Impossible to test in Biblibre sandbox, but I suspect it is a general issue affecting sandboxes? In provision logs I get this : fatal: [localhost -> koha-box26869]: FAILED! => {"changed": true, "cmd": "koha-shell -c 'cd /kohadevbox/koha && yes | git bz apply 26869' box26869", "delta": "0:00:00.379079", "end": "2024-05-23 11:02:57.723063", "msg": "non-zero return code", "rc": 1, "start": "2024-05-23 11:02:57.343984", "stderr": "Error getting login cookie from browser:\n Cannot find default Firefox profile\n\nConfigured browser: firefox3 (change with 'git config --global bz.browser <value>')\nPossible browsers: chromium, epiphany, firefox3, galeon, google-chrome", "stderr_lines": ["Error getting login cookie from browser:", " Cannot find default Firefox profile", "", "Configured browser: firefox3 (change with 'git config --global bz.browser <value>')", "Possible browsers: chromium, epiphany, firefox3, galeon, google-chrome"], "stdout": "", "stdout_lines": []} (In reply to Mathieu Saby from comment #27) > Hi > Does this patch only work for Marc21, or does it take into account Unimarc > (where items are mapped to 995 field) ? The code considers MARC flavour and should act accordingly. (In reply to Mathieu Saby from comment #29) > Impossible to test in Biblibre sandbox, but I suspect it is a general issue > affecting sandboxes? This indeed appears to be a general issue affecting sandboxes, community has been made aware and are looking into it. (In reply to Esther Melander from comment #26) > This is very close, but I wanted to share some notes from my testing. > > Following the testing plan, created a template that added only a 952$a and I > did receive a error message on why the item was not added. > > My next template looked like this: > > Add 952$a CPL > Add 952$b CPL > Add 952$y BK > > This template does work when modifying the record from the Bibliographic > detail page. It adds a single item record. However, if this same template is > used in Stage MARC records, it adds three item records. > > A better template would look like this: > > Add 952$a CPL > Update existing or add new field 952$b CPL > Update existing or add new field 952$y BK > > This template will add only one item record in both scenarios. > > This is perhaps not a blocker and I will happily sign-off. Just wanted to > make a note that the behavior is different and it should probably be > consistent. Interesting and fair points. I'm not entirely sure what the best course of action would be here, if any is required. Comparing the code from stage MARC and biblio records modification nothing obvious stands out to me, although these are certainly 2 different areas in the code. Can you please provide a more detailed test plan (or include a MARC records file - if required) so that I can try and reproduce the exact use case and test further? Thank you. (In reply to Esther Melander from comment #26) > My next template looked like this: > > Add 952$a CPL > Add 952$b CPL > Add 952$y BK > > This template does work when modifying the record from the Bibliographic > detail page. It adds a single item record. However, if this same template is > used in Stage MARC records, it adds three item records. In 23.11.x prod, I'm noticing a template like that does indeed create 3 items (one per action), when used with Stage MARC records. > A better template would look like this: > > Add 952$a CPL > Update existing or add new field 952$b CPL > Update existing or add new field 952$y BK Did you test this? It seems to me that you'd still wind up with 3 items, because there wouldn't be a pre-existing 952$b and 952$y? (In reply to David Cook from comment #32) > > A better template would look like this: > > > > Add 952$a CPL > > Update existing or add new field 952$b CPL > > Update existing or add new field 952$y BK > > Did you test this? It seems to me that you'd still wind up with 3 items, > because there wouldn't be a pre-existing 952$b and 952$y? I just tested this, and it looks like you're right. I'll never understand the MARC Modification Templates... Created attachment 167312 [details]
Import file to test marc modification template
Here is my test plan:
1. Create a marc modification template with these actions:
Add 952$a CPL
Add 952$b CPL
Add 952$y BK
2. Find a bibliographic record. Edit - Modify record using template.
3. Apply the marc modification template.
See that one item record is created.
4. Now Stage a MARC record (example file uploaded, flora.mrc). Select the marc modification template above. Import.
See that three item records are created. Each of the add actions creates an item record.
5. Now create a new marc modification template with these actions:
Add 952$a CPL
Update existing or add new field 952$b CPL
Update existing or add new field 952$y BK
6. Repeat the steps above, but note that only one item record is created each time.
As mentioned previously, this is perhaps not a blocker for this bug. Documentation can provide an example template like the second example. The inconsistency could be subject to a separate bug. I would be happy to sign off with some guidance on the best course of action.
Thank you, Esther, for the detailed instructions and sample file. I believe that for the stage file import scenario, the setting 'How to process items:' being 'Always add items' (default) is what is causing each new MARC item entry (952 for MARC21) to forcibly create a new item. Imo, the ideal solution here would be to also have a 'How to process items' option in the MARC modification template tool UI (the screen that shows after you select 'Modify record using template') from a bib record. This would work around/with the actions coming from the modification template in the same way the stage import flow currently does, ideally using the same code, for consistency. In conclusion, I believe the behavior differs because the 'stage MARC record file for import' has additional logic on top of the MARC modification template actions, whereas the 'Modify record using template' only considers the actions from the MARC modification template and nothing else. Would the suggestion above be better handled in a follow-up bug or here? What do others think about this? (In reply to Pedro Amorim from comment #35) > What do others think about this? I'm not sure at the moment. I find the whole situation to be a confusing and unintuitive. Of course, any changes to how it processes now will cause potential breaking changes for anyone who expects it to work as it works at the moment. (In reply to Pedro Amorim from comment #35) > Thank you, Esther, for the detailed instructions and sample file. > I believe that for the stage file import scenario, the setting 'How to > process items:' being 'Always add items' (default) is what is causing each > new MARC item entry (952 for MARC21) to forcibly create a new item. > > Imo, the ideal solution here would be to also have a 'How to process items' > option in the MARC modification template tool UI (the screen that shows > after you select 'Modify record using template') from a bib record. This > would work around/with the actions coming from the modification template in > the same way the stage import flow currently does, ideally using the same > code, for consistency. > > In conclusion, I believe the behavior differs because the 'stage MARC record > file for import' has additional logic on top of the MARC modification > template actions, whereas the 'Modify record using template' only considers > the actions from the MARC modification template and nothing else. > > Would the suggestion above be better handled in a follow-up bug or here? > > What do others think about this? I like Pedro's idea of adding a "How to process items" dropdown to the Batch Record Modification tool so that the modification of existing bibs is handled using as much existing code from the record import process as possible, thereby ensuring consistency of behavior. I would rather see that functionality added in this bug than in a follow-up. Introducing this feature with its current inconsistency of behavior relative to the import process seems like it will create confusion that will persist even after a follow-up bug makes the behavior consistent. (In reply to Pedro Amorim from comment #35) > Would the suggestion above be better handled in a follow-up bug or here? > > What do others think about this? I apologise in advance for adding a differing opinion here... I think Pedro's suggestion to clarify the MARC modification templates functionality is nice - but I also think it belongs in a separate bug. As Esther's thorough testing highlighted (in comment #34), there is already a way to make sure only one item is created: you simply need to use "Update existing or add new field" in subsequent actions rather than "Add". The Koha manual can easily be updated to give advice on this as well. I was testing this again today in a 24.06 sandbox, ready to sign off. It works perfectly when adding items from a bibliographic record. But the same template is now not working through Stage MARC records. The import file will stage but fail at import. I tested the same file in a 24.05 demo and it is working in both areas. Something has changed, and maybe is broken? Since the templates are shared in both places, it seems that both areas need to work. If someone else has time, could they try the test plan and see if they get the same results? (In reply to Esther Melander from comment #39) > I was testing this again today in a 24.06 sandbox, ready to sign off. It > works perfectly when adding items from a bibliographic record. But the same > template is now not working through Stage MARC records. The import file will > stage but fail at import. I tested the same file in a 24.05 demo and it is > working in both areas. Something has changed, and maybe is broken? Marking this as Failed QA due to Esther's comments. I've had more and more libraries adding items through Stage MARC records, so I think we can't break this. (In reply to Esther Melander from comment #39) > I was testing this again today in a 24.06 sandbox, ready to sign off. It > works perfectly when adding items from a bibliographic record. But the same > template is now not working through Stage MARC records. The import file will > stage but fail at import. I tested the same file in a 24.05 demo and it is > working in both areas. Something has changed, and maybe is broken? > > Since the templates are shared in both places, it seems that both areas need > to work. If someone else has time, could they try the test plan and see if > they get the same results? Hi, I tested this again on current master (24.06). This still works for me as it was working before, i.e. a MARC modification template where the 3 actions are 'add' will add one item for each action, whereas a MARC modification template where all 3 actions are "Update existing or add new" only adds 1 item with the 3 correct subfields. This is what I did (after creating the MMtemplate): 1) Visit http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl 2) Upload flora.mrc file, select template, click "Stage for import" 3) Once the job is done click "View batch" 4) Click "Import this batch into the catalog" 5) Wait for the job to finish. Refresh the page after a few seconds. Once complete, do a search for "flora": http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=flora 6) The newly created record should be there Created attachment 170305 [details] [review] Bug 26869: Add items from MARC in BatchUpdateBiblio Test plan, clean k-t-d: Before test, pick a record and confirm it has no items, for this test we're using biblio 79: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 1) Add a new MARC modification template, visit: <staff_url>/cgi-bin/koha/tools/marc_modification_templates.pl 2) Click 'New template', enter a name and click 'Submit' 3) Click 'New action' 4) Pick 'Add new' and enter '952' as field and 'a' as subfield, enter 'CPL' as value 5) Click 'Add action' 6) Do a batch record modification, visit: <staff_url>/cgi-bin/koha/tools/batch_record_modification.pl 7) Select the tab 'Enter a list of record numbers' and enter '79' 8) On the template, select the template previously created. Click 'continue' 9) Click 'Modify selected records'. Click 'View detail of the enqueued job'. Notice success message is shown with no errors 10) Check the biblio, visit: <staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79 11) Notice it now has an item with homebranch 'CPL' test with 952 but with other subfields, without subfield 'a' -> confirm 'FK constraint' error is shown, no modification made test with more than one occurrence of 952$a -> confirm 'FK constraint' error is shown, no modification made test with 952$a too long e.g. 'aaaaaaaa' -> confirm 'too long' error is shown test with no 952 mapping -> confirm things work as intended test with 952 mapping plus other tags -> confirm things work as intended prove koha/t/db_dependent/Koha/BackgroundJobs/BatchUpdateBiblio.t prove koha/t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblio.t Signed-off by: Chris Rowlands <chris.rowlands6@nhs.net> Created attachment 170306 [details] [review] Bug 26869: Add MARC modification template mapping validation Follow the same test plan as above, but attempt to add an item without having at least homebranch, holdingbranch and type subfields. Notice you now get an error with an appropriate message and the item is not added. Signed-off by: Chris Rowlands <chris.rowlands6@nhs.net> |