Enhance bulkmarcimport.pl to keep ids of previous Information Library System when loading both bibliographic and authority records. Amend and improve the documentation about the the various options. The ids could be in tag without subfields (like '001'). The ids must to be only numeric.
Based on your mailing list post, I have changed the status of the bug to 'seeking developers'. Now, a question: Based on my understanding of what you put in the bug, what you're looking for are changes to bulkmarcimport.pl to do the following: [1] Look for a legacy record ID in a field specified by the user. [2] Make sure that that original ID is retained, possibility moving it to a different field specified by the user. [3] Improve the documentation of bulkmarcimport.pl. And all of this to serve the following purpose: to ensure that 76X-78X links will continue to work. Am I understanding that correctly? And would you mind attaching to this bug an example bib record where this applies?
Created attachment 3845 [details] sample of 16 records. previous id in 001 and 999 subfield c
We are interested in developing this enhancement, so we changed bug status to "assigned". We will study a little what is needed, then we will add more details in next days.
(In reply to comment #3) > We are interested in developing this enhancement, so we changed bug status to > "assigned". We will study a little what is needed, then we will add more > details in next days. While there is nothing stopping you, please note that I believe that the original poster is looking for proposals to perform the development. If you wish to provide a price quote, please email Linda. If you wish to just do the coding outright, you can assign the bug to yourself.
(In reply to comment #0) > Enhance bulkmarcimport.pl to keep ids of previous Information Library > System when loading both bibliographic and authority records. Call me dense, but why not do this when converting the records produced by the previous ILS?
(In reply to comment #5) > (In reply to comment #0) > > Enhance bulkmarcimport.pl to keep ids of previous Information Library > > System when loading both bibliographic and authority records. > > Call me dense, but why not do this when converting the records produced by the > previous ILS? Well, preprocessing is certainly a valid option, but retaining the original LMS ID (when it is numeric, of course), is a minor improvement and simplifies the processing for Linda's particular use case.
I am not sure manipulating an internal database id that way is a good idea. And I am not sure this will always work. What if the old system's internal number is alphanumeric, has leading zeros in the number, is too long or something? What will happen when you delete or add a new record? I was told that relying on Control number in 001 and using that for linking is too slow compared to using an internal number. That said I have to say that it works for us and our libraries. What about adding a new column to the biblio table? This could be mapped to 001 or to any other field containing the old number in your data. And make it possible to construct a URL using that? Would this make things faster?
(In reply to comment #7) > What about adding a new column to the biblio table? This could be mapped to 001 > or to any other field containing the old number in your data. And make it > possible to construct a URL using that? Would this make things faster? In the specific use case, an option to bulkmarcimport.pl to set the biblionumber explicitly is sufficient, but the idea of storing other standard identifiers would be useful, though possibly as a new table instead of just a column.
(In reply to comment #7) >What if the old system's internal number >is alphanumeric, has leading zeros in the number, is too long ... ? well this enhancement is to use in migration, not during production. The idea is to create a perfect sync between old ILS ids and the values in: biblio.biblionumber, biblioitems.biblionumber, biblioitems.biblioitemnumber, items.biblionumber, items.biblioitemnumber and also the marc fields connenct with them (in marc21 standard KohaToMarc links 999$c and 999$d, for unimarc 001 and 090$a) so Q:What if the old system's internal number is alphanumeric ? A:This enhancement will not work. It works only with numbers. The sql defintion of biblio.biblionumber is: `biblionumber` int(11) NOT NULL auto_increment, Q:..has leading zeros in the number ? A: You need to delete leanding zeros before use bulkmarcimport.pl Q: .. is too long ? The max value in field biblio.biblionumber is 4294967295, if your numbers is bigger, bulkmarcimport.pl will no works. But we can plan to change biblio.biblionumber sql definition.
Details of proposed change - add four new parameters to bulkmarcimport.pl: • --reuse-legacy-rec-id - when specified, set the biblionumber (or authority iD) of a new imported record to the numeric ID present in the MARC record • --legacy-rec-id-tag - when specified, the tag containing the legacy bib or authority ID. If not specified, defaults to the biblionumber/authoritynumber tag set in the default MARC framework. • --legacy-rec-id-subf - like --legacy-rec-id-tag, but for the subfield • --overlay-on-rec-id - when specified, if an incoming record has the same bib ID as an exist Koha bib or authority, the Koha record will be replaced by the incoming record. This option is effectively only when --reuse-legacy-rec-id is on. If --reuse-legacy-rec-id is enabled but --overlay-on-rec-id is not, if an incoming record matches an existing Koha record, the record would instead be written to an exception file. If --reuse-legacy-rec-id is in effect but the ID found in the tag/subfield for the legacy record is not numeric, the record will be written to an exception file.
Work to date can be found at: http://git.esilibrary.com/?p=koha-equinox.git;a=shortlog;h=refs/heads/enh/bug6113
Description of change; determined that options could be simplified to: -reject-file FILE If specified, MARC records rejected during the course of processing are written to the file specified by this option. -recid-from-field TAG[:SUBFIELD] If specified, the specified tag/subfield is assumed to contain the record ID to assign to each loaded record. If a record with that ID already exists in the database, it will be replaced by the incoming record. If the incoming record lacks the ID field or the ID field doesn't contain an integer value, the record will be written the reject file. This option cannot be used at the same time as -match. The primary use case for -recid-from-field is for an initial load where it may be desirable to set the record ID during preprocessing. The secondary use case is for overlaying records on record ID (e.g., after authority record cleanup); the record ID lookups in that specific case can be faster than using the -match option.
I think that #12 is enough, but we need also to be more explicit on the help about the options --keepids=<FIELD> and -keepids
Created attachment 6527 [details] An utility to complement bulkmarcimport.pl Rather then improve bulkmarcimport.pl, sysno_keeper.pl works on biblio, biblioitems and items tables with a series of SQL commands, to reassign biblionumber and biblioitemnumber copying them from a control or data field (default 001).
Created attachment 21698 [details] [review] Bug 6113 [ENH] - enhancement to keep previous ids Created the test cases.
Created attachment 21699 [details] [review] Bug 6113 [ENH] - enhancement to keep previous ids Created the test cases.
Created attachment 21700 [details] [review] Bug 6113 [ENH] - enhancement to keep previous ids Forgot the extra warnings in C4::Biblio. Removed double warnings.
Created attachment 21701 [details] [review] Bug 6113 [ENH] - enhancement to keep previous ids Fixed warning messages.
Created attachment 21702 [details] [review] Bug 6113 [ENH] - enhancement to keep previous ids Modified the C4::Biblio::_koha_add_biblioitem() && _koha_add_biblio() to support INSERTing biblionumbers and biblioitemnumbers to the DB. Also validating that id's are positive integers and well below the critical Perl LONG_MAX limit so we don't get buffer overflow. Duplicate id INSERTion will fail, but in normal use scenarios this is not an issue, if "Koha to MARC mapping" has biblionumber and biblioitemnumber mapped under field 999. MySQL takes care of the primary key sequence refreshing. Updated the bulkmarcimport.pl -script to copy the desired legacy id MARC field to the field designated in "Koha to MARC mapping" -> biblionumber && biblioitemnumber. This is handy if you have the legacy id in just 001 and need to populate 999c and 999d to push them to DB. No need to make an extra modification run to source bibliographic records just to facilitate Koha migration better. Documented the new functionality in bulkmarcimport.pl
Created attachment 21703 [details] [review] Bug 6113 [ENH] - enhancement to keep previous ids Modified the C4::Biblio::_koha_add_biblioitem() && _koha_add_biblio() to support INSERTing biblionumbers and biblioitemnumbers to the Koha DB. Created validators to make sure the primary keys are positive integers and well below the Perl's LONG_MAX threshold to keep Koha from getting crazy. Duplicate id's are the users responsibility and in normal use conditions they are non-existing, unless the "Koha to MARC mapping" to biblionumber is a user-modifiable MARC field, which is TROUBLE! MySQL takes care of the primary key sequence refreshing. Updated the bulkmarcimport.pl -script to copy the desired legacy id MARC field to the biblionumber- and biblioitemnumber-fields designated in "Koha to MARC mapping", eg. copy field 001 to 999c and 999d. This is handy if you have the legacy id in just 001 and need to populate 999c and 999d to push them to DB. No need to make an extension to the existing migration scripts. Documented the new functionality in bulkmarcimport.pl Created test cases for C4::Biblio.pm modifications.
Sponsored-by: Joensuu Regional Library
Patch applied cleanly, go forth and signoff
Would it be possibly better to solve this using bug 8015 as a basis? It's a more extensible/flexible system and we could easily modify bulkmarcimport.pl to allow the use of modification templates. I know it means tossing this existing work, but I'd say it's usually better to have less code than more code.
>Would it be possibly better to solve this using bug 8015 as a basis? It's a more >extensible/flexible system and we could easily modify bulkmarcimport.pl to allow >the use of modification templates. I know it means tossing this existing work, >but I'd say it's usually better to have less code than more code. I think bug 8015 is a great basis for this, but I needed this functionality for our data migration and subsequently this bug was just a target of opportunity. Hopefully this helps someone else as well. It's fine by me that someone else will solve this problem with more style and potentially discard all the work I have made. Unfortunately this is way out of our scope, so that someone else wont be me.
Created attachment 24073 [details] [review] Bug 6113 [ENH] - enhancement to keep previous ids Modified the C4::Biblio::_koha_add_biblioitem() && _koha_add_biblio() to support INSERTing biblionumbers and biblioitemnumbers to the Koha DB. Created validators to make sure the primary keys are positive integers and well below the Perl's LONG_MAX threshold to keep Koha from getting crazy. Duplicate id's are the users responsibility and in normal use conditions they are non-existing, unless the "Koha to MARC mapping" to biblionumber is a user-modifiable MARC field, which is TROUBLE! MySQL takes care of the primary key sequence refreshing. Updated the bulkmarcimport.pl -script to copy the desired legacy id MARC field to the biblionumber- and biblioitemnumber-fields designated in "Koha to MARC mapping", eg. copy field 001 to 999c and 999d. This is handy if you have the legacy id in just 001 and need to populate 999c and 999d to push them to DB. No need to make an extension to the existing migration scripts. Documented the new functionality in bulkmarcimport.pl Created test cases for C4::Biblio.pm modifications.
This patch just created an issue for us in conjunction with Bug 12624. It was very useful in our initial migration but after that it will be worthless.
BTW the issue was existing biblios getting overwritten by new records from staging tables. So if C4::Biblio::_koha_add_biblio() returns an error, the rest of the instructions are carried out normally, eg. ModBiblio, which overwrites the existing marc record even if the "adding new record to a reserved biblionumber" just failed.
(In reply to Olli-Antti Kivilahti from comment #27) > This patch just created an issue for us in conjunction with Bug 12624. > > It was very useful in our initial migration but after that it will be > worthless. So, should this be left as "Patch doesn't apply" or perhaps change it to "Resolved - Won't Fix"?