Summary: | Add option to bulkmarimport.pl for matching on original id | ||
---|---|---|---|
Product: | Koha | Reporter: | David Gustafsson <glasklas> |
Component: | Tools | Assignee: | David Gustafsson <glasklas> |
Status: | Signed Off --- | QA Contact: | Testopia <testopia> |
Severity: | enhancement | ||
Priority: | P5 - low | CC: | david, frank.hansen, martin.renvoize, schodkowy.omegi-0r |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | --- | Documentation contact: | |
Documentation submission: | Text to go in the release notes: |
This enhancement adds a `match_record_id` option to the ./misc/migration_tools/bulkmarcimport.pl script. If a record matches based on the ID, it is updated. Previously, without this option, a record (but not the items) would be duplicated.
Example use case: allows updates to previously exported records, without creating duplicate records.
|
|
Version(s) released in: | Circulation function: | ||
Bug Depends on: | 29440 | ||
Bug Blocks: | |||
Attachments: |
Bug 29654 - Add match_record_id option to bulkmarimport.pl
Bug 29654 - Add match_record_id option to bulkmarimport.pl Bug 29654: Add match_record_id option to bulkmarimport.pl Bug 29654: Add match_record_id option to bulkmarimport.pl Bug 29654: Add match_record_id option to bulkmarimport.pl |
Description
David Gustafsson
2021-12-07 23:01:14 UTC
If I'm not mistaken, currently in bulkmarkimport.pl, if no match is found for authorities the incoming record is used for matching against an existing Koha authority, this is not done for biblio records though. Instead of this inconsistent behaviour it would be better if there was an option to explicitly set if matching on incoming record ids should be performed or not. Created attachment 128335 [details] [review] Bug 29654 - Add match_record_id option to bulkmarimport.pl Not yet ready for review, will later add how to test. Created attachment 128365 [details] [review] Bug 29654 - Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated Created attachment 128575 [details] [review] Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated The patch doesn't apply anymore. Rebase please. Using index info to reconstruct a base tree... M misc/migration_tools/bulkmarcimport.pl Falling back to patching base and 3-way merge... Auto-merging misc/migration_tools/bulkmarcimport.pl CONFLICT (content): Merge conflict in misc/migration_tools/bulkmarcimport.pl error: Failed to merge in the changes. Patch failed at 0001 Bug 29654: Add match_record_id option to bulkmarimport.pl hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-29654-Add-matchrecordid-option-to-bulkmarimpor-rJcwEW.patch vagrant@kohadevbox:kohaclone(bug29654|AM 1/1)$ I must have accidentally added two different iterations of the same patch, now it should work. Sorry, patch doesn't apply (again): Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 29440: Refactor and clean up bulkmarcimport.pl Using index info to reconstruct a base tree... M C4/Biblio.pm Falling back to patching base and 3-way merge... Auto-merging C4/Biblio.pm CONFLICT (content): Merge conflict in C4/Biblio.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 29440: Refactor and clean up bulkmarcimport.pl hint: Use 'git am --show-current-patch=diff' to see the failed patch When you have resolved this problem run "git bz apply --continue". If you would prefer to skip this patch, instead run "git bz apply --skip". To restore the original branch and stop patching run "git bz apply --abort". Patch left in /tmp/Bug-29440-Refactor-and-clean-up-bulkmarcimportpl-zsN_GN.patch Cannot apply cleanly patches from bug 29440. Everything will be left dirty. git bz apply --continue will not continue the process if patches from other bug reports need to be applied. Sorry about being slow to reply. Tried applying the patch and applied cleanly for me, realized the conflict was in bug 29440 which since have been rebased. I'm not sure, as far as I understand the code, that this is the best approach to implementing it. It looks like it still tries to match the record by old logic, and only afterwards swaps out the ID. I don't think it should perform any matching when we specify the Koha record ID upfront, or at most it should check if it already exists... The patch no longer applies - sha1/fake ancestor error: git bz apply 29654 Bug 29654 - Add option to bulkmarimport.pl for matching on original id 128575 - Bug 29654: Add match_record_id option to bulkmarimport.pl Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 29654: Add match_record_id option to bulkmarimport.pl error: sha1 information is lacking or useless (misc/migration_tools/bulkmarcimport.pl). error: could not build fake ancestor Patch failed at 0001 Bug 29654: Add match_record_id option to bulkmarimport.pl Created attachment 178169 [details] [review] Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated Sponsored-by: Gothenburg University Library Rebased against main. Created attachment 182239 [details] [review] Bug 29654: Add match_record_id option to bulkmarimport.pl Add -match_record_id option to bulkmarkimport.pl. When enabled incoming record id will be assumed to match record in Koha. To test: 1) Before applying patch, export a biblio from staff interace 2) Import using the exported file: ./bulkmarcimport.pl -m="MARCXML" -v -b -file bib-xxxxx.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" 3) The import with be inserted and create a duplicate record in Koha. Check the import.log or search using the staff interface. (could also crash with duplicate key error if contains item) 4) Apply patch 5) Run the import again with the same command as above 6) A new record should been inserted 7) Remove all newly inserted duplicate records (it's important to keep the record the export was initially performed on) 6) Run the import again, but add the -match_record_id option 8) The original Koha record should now have been updated Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> Testing notes (using KTD): 1. Running the command in step 2 generates this output - a duplicate record is inserted, but no item (I used record 262 - Programming perl): ./misc/migration_tools/bulkmarcimport.pl -m="MARCXML" -v -b -file bib-262.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" Using --update without --match or --isbn seems to be useless. Characteristic MARC flavour: MARC21 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 399, <GEN3> chunk 2. .DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Duplicate entry '578' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 174 ERROR: Adding items to bib 439 failed: Duplicate ID at ./misc/migration_tools/bulkmarcimport.pl line 636. 1 MARC records done in 0.155983924865723 seconds /tmp/import.log: 1 id;operation;status 2 262;insert;ok 3 439;insert items;ERROR 4 file : bib-262.marcxml 5 1 MARC records done in 0.155983924865723 seconds 2. The patch still applies. 3. Running the command again for step 5 generates this output - another record is inserted without an item (so have two duplicate records): ./misc/migration_tools/bulkmarcimport.pl -m="MARCXML" -v -b -file bib-262.marcxml -insert -update -c=MARC21 -l "/tmp/import.log" Using --update without --match or --isbn seems to be useless. Characteristic MARC flavour: MARC21 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 399, <GEN3> chunk 2. .DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Duplicate entry '578' for key 'PRIMARY' at /kohadevbox/koha/Koha/Object.pm line 174 ERROR: Adding items to bib 440 failed: Duplicate ID at ./misc/migration_tools/bulkmarcimport.pl line 642. 1 MARC records done in 0.161586046218872 seconds /tmp/import.log: 1 id;operation;status 2 262;insert;ok 3 440;insert items;ERROR 4 file : bib-262.marcxml 5 1 MARC records done in 0.161586046218872 seconds 4. Running the command again with the -match_record_id option (after deleting the duplicate records) generates this output: ./misc/migration_tools/bulkmarcimport.pl -m="MARCXML" -v -b -file bib-262.marcxml -insert -update -c=MARC21 -match_record_id -l "/tmp/import.log" Using --update without --match or --isbn seems to be useless. Characteristic MARC flavour: MARC21 Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 399, <GEN3> chunk 2. . 1 MARC records done in 0.153377056121826 seconds /tmp/import.log: 1 id;operation;status 2 262;update;ok 3 file : bib-262.marcxml 4 1 MARC records done in 0.153377056121826 seconds 5. Note that with CataloguingLog system preference enabled (it is on by default in KTD), there is no record of any changes to the record - "No log found for Bibliographic record 262.". 6. I made a change to the exported record (edited the description in 504$a), and this change was reflected in the record. |