Bug 6113 - Keep previous IDs when importing bibliographic and authority records
Summary: Keep previous IDs when importing bibliographic and authority records
Status: Patch doesn't apply
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic record staging/import (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Olli-Antti Kivilahti
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-04-07 13:17 UTC by lculber
Modified: 2023-08-04 20:24 UTC (History)
10 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
sample of 16 records. previous id in 001 and 999 subfield c (14.75 KB, application/mrc)
2011-04-08 17:41 UTC, lculber
Details
An utility to complement bulkmarcimport.pl (10.86 KB, text/plain)
2011-12-02 10:04 UTC, Stefano Bargioni
Details
Bug 6113 [ENH] - enhancement to keep previous ids (6.41 KB, patch)
2013-10-01 14:29 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 6113 [ENH] - enhancement to keep previous ids (6.41 KB, patch)
2013-10-01 14:30 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 6113 [ENH] - enhancement to keep previous ids (2.52 KB, patch)
2013-10-01 14:33 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 6113 [ENH] - enhancement to keep previous ids (1.91 KB, patch)
2013-10-01 14:33 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 6113 [ENH] - enhancement to keep previous ids (15.88 KB, patch)
2013-10-01 14:33 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 6113 [ENH] - enhancement to keep previous ids (21.71 KB, patch)
2013-10-01 15:34 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 6113 [ENH] - enhancement to keep previous ids (21.57 KB, patch)
2014-01-09 13:45 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description lculber 2011-04-07 13:17:58 UTC
Enhance bulkmarcimport.pl to keep ids of previous Information Library
System when loading both bibliographic and authority records.
Amend and improve the documentation about the the various options.
The ids could be in tag without subfields (like '001').
The ids must to be only numeric.
Comment 1 Galen Charlton 2011-04-08 13:01:21 UTC
Based on your mailing list post, I have changed the status of the bug to 'seeking developers'.  Now, a question:

Based on my understanding of what you put in the bug, what you're looking for are changes to bulkmarcimport.pl to do the following:

[1] Look for a legacy record ID in a field specified by the user.
[2] Make sure that that original ID is retained, possibility moving it to a different field specified by the user.
[3] Improve the documentation of bulkmarcimport.pl.

And all of this to serve the following purpose: to ensure that 76X-78X links will continue to work.  Am I understanding that correctly?  And would you mind attaching to this bug an example bib record where this applies?
Comment 2 lculber 2011-04-08 17:41:01 UTC
Created attachment 3845 [details]
sample of 16 records. previous id in 001 and 999 subfield c
Comment 3 Hyperborea 2011-04-12 10:08:04 UTC
We are interested in developing this enhancement, so we changed bug status to "assigned". We will study a little what is needed, then we will add more details in next days.
Comment 4 Galen Charlton 2011-04-12 11:34:57 UTC
(In reply to comment #3)
> We are interested in developing this enhancement, so we changed bug status to
> "assigned". We will study a little what is needed, then we will add more
> details in next days.

While there is nothing stopping you, please note that I believe that the original poster is looking for proposals to perform the development.  If you wish to provide a price quote, please email Linda.  If you wish to just do the coding outright, you can assign the bug to yourself.
Comment 5 MJ Ray (software.coop) 2011-04-15 18:52:06 UTC
(In reply to comment #0)
> Enhance bulkmarcimport.pl to keep ids of previous Information Library
> System when loading both bibliographic and authority records.

Call me dense, but why not do this when converting the records produced by the previous ILS?
Comment 6 Galen Charlton 2011-04-15 19:02:45 UTC
(In reply to comment #5)
> (In reply to comment #0)
> > Enhance bulkmarcimport.pl to keep ids of previous Information Library
> > System when loading both bibliographic and authority records.
> 
> Call me dense, but why not do this when converting the records produced by the
> previous ILS?

Well, preprocessing is certainly a valid option, but retaining the original LMS ID (when it is numeric, of course), is a minor improvement and simplifies the processing for Linda's particular use case.
Comment 7 Katrin Fischer 2011-04-15 19:10:53 UTC
I am not sure manipulating an internal database id that way is a good idea. And I am not sure this will always work. What if the old system's internal number is alphanumeric, has leading zeros in the number, is too long or something?
What will happen when you delete or add a new record? 

I was told that relying on Control number in 001 and using that for linking is too slow compared to using an internal number. That said I have to say that it works for us and our libraries. 

What about adding a new column to the biblio table? This could be mapped to 001 or to any other field containing the old number in your data. And make it possible to construct a URL using that? Would this make things faster?
Comment 8 Galen Charlton 2011-04-15 19:14:16 UTC
(In reply to comment #7)
> What about adding a new column to the biblio table? This could be mapped to 001
> or to any other field containing the old number in your data. And make it
> possible to construct a URL using that? Would this make things faster?

In the specific use case, an option to bulkmarcimport.pl to set the biblionumber explicitly is sufficient, but the idea of storing other standard identifiers would be useful, though possibly as a new table instead of just a column.
Comment 9 Zeno Tajoli 2011-04-16 11:13:21 UTC
(In reply to comment #7)

>What if the old system's internal number
>is alphanumeric, has leading zeros in the number, is too long ... ?

well this enhancement is to use in migration, not during production.
The idea is to create a perfect sync between old ILS ids and the values in:
biblio.biblionumber, 
biblioitems.biblionumber, 
biblioitems.biblioitemnumber,
items.biblionumber, 
items.biblioitemnumber
and also the marc fields connenct with them (in marc21 standard KohaToMarc links
999$c and 999$d, for unimarc 001 and 090$a)

so
Q:What if the old system's internal number is alphanumeric ?
A:This enhancement will not work. It works only with  numbers.
The sql defintion of biblio.biblionumber is:
`biblionumber` int(11) NOT NULL auto_increment,

Q:..has leading zeros in the number ?
A: You need to delete leanding zeros before use bulkmarcimport.pl

Q: .. is too long ?
The max value in field biblio.biblionumber is 4294967295, if your numbers is bigger, bulkmarcimport.pl will no works.
But we can plan to change biblio.biblionumber sql definition.
Comment 10 Galen Charlton 2011-04-19 17:13:45 UTC
Details of proposed change - add four new parameters to bulkmarcimport.pl:

•	--reuse-legacy-rec-id - when specified, set the biblionumber (or authority iD) of a new imported record to the numeric ID present in the MARC record
•	--legacy-rec-id-tag - when specified, the tag containing the legacy bib or authority ID.  If not specified, defaults to the biblionumber/authoritynumber tag set in the default MARC framework.
•	--legacy-rec-id-subf - like --legacy-rec-id-tag, but for the subfield
•	--overlay-on-rec-id - when specified, if an incoming record has the same bib ID as an exist Koha bib or authority, the Koha record will be replaced by the incoming record.  This option is effectively only when --reuse-legacy-rec-id is on.  If --reuse-legacy-rec-id is enabled but --overlay-on-rec-id is not, if an incoming record matches an existing Koha record, the record would instead be written to an exception file.

If --reuse-legacy-rec-id is in effect but the ID found in the tag/subfield for the legacy record is not numeric, the record will be written to an exception file.
Comment 11 Galen Charlton 2011-05-27 03:15:14 UTC
Work to date can be found at:

http://git.esilibrary.com/?p=koha-equinox.git;a=shortlog;h=refs/heads/enh/bug6113
Comment 12 Galen Charlton 2011-05-27 03:16:02 UTC
Description of change; determined that options could be simplified to:


    -reject-file FILE
    
    If specified, MARC records rejected during the course of
    processing are written to the file specified by this option.
    
    -recid-from-field TAG[:SUBFIELD]
    
    If specified, the specified tag/subfield is assumed to contain
    the record ID to assign to each loaded record.  If a record
    with that ID already exists in the database, it will be
    replaced by the incoming record.  If the incoming record
    lacks the ID field or the ID field doesn't contain an integer
    value, the record will be written the reject file.  This
    option cannot be used at the same time as -match.
    
    The primary use case for -recid-from-field is for an initial
    load where it may be desirable to set the record ID during
    preprocessing.  The secondary use case is for overlaying
    records on record ID (e.g., after authority record cleanup); the
    record ID lookups in that specific case can be faster than using
    the -match option.
Comment 13 Zeno Tajoli 2011-05-27 07:21:48 UTC
I think that #12 is enough, but we need also to be more explicit on the help about the options --keepids=<FIELD> and -keepids
Comment 14 Stefano Bargioni 2011-12-02 10:04:27 UTC
Created attachment 6527 [details]
An utility to complement bulkmarcimport.pl

Rather then improve bulkmarcimport.pl, sysno_keeper.pl works on biblio,
biblioitems and items tables with a series of SQL commands, to reassign
biblionumber and biblioitemnumber copying them from a control or data field
(default 001).
Comment 15 Olli-Antti Kivilahti 2013-10-01 14:29:42 UTC Comment hidden (obsolete)
Comment 16 Olli-Antti Kivilahti 2013-10-01 14:30:54 UTC Comment hidden (obsolete)
Comment 17 Olli-Antti Kivilahti 2013-10-01 14:33:05 UTC Comment hidden (obsolete)
Comment 18 Olli-Antti Kivilahti 2013-10-01 14:33:26 UTC Comment hidden (obsolete)
Comment 19 Olli-Antti Kivilahti 2013-10-01 14:33:48 UTC Comment hidden (obsolete)
Comment 20 Olli-Antti Kivilahti 2013-10-01 15:34:00 UTC Comment hidden (obsolete)
Comment 22 Olli-Antti Kivilahti 2013-10-01 15:43:31 UTC
Sponsored-by: Joensuu Regional Library
Comment 23 I'm just a bot 2013-10-01 18:44:49 UTC
Patch applied cleanly, go forth and signoff
Comment 24 Kyle M Hall 2013-10-25 15:50:57 UTC
Would it be possibly better to solve this using bug 8015 as a basis? It's a more extensible/flexible system and we could easily modify bulkmarcimport.pl to allow the use of modification templates. I know it means tossing this existing work, but I'd say it's usually better to have less code than more code.
Comment 25 Olli-Antti Kivilahti 2013-10-28 08:18:04 UTC
>Would it be possibly better to solve this using bug 8015 as a basis? It's a more >extensible/flexible system and we could easily modify bulkmarcimport.pl to allow >the use of modification templates. I know it means tossing this existing work, >but I'd say it's usually better to have less code than more code.

I think bug 8015 is a great basis for this, but I needed this functionality for our data migration and subsequently this bug was just a target of opportunity.
Hopefully this helps someone else as well.

It's fine by me that someone else will solve this problem with more style and potentially discard all the work I have made. Unfortunately this is way out of our scope, so that someone else wont be me.
Comment 26 Olli-Antti Kivilahti 2014-01-09 13:45:27 UTC
Created attachment 24073 [details] [review]
Bug 6113 [ENH] - enhancement to keep previous ids

Modified the C4::Biblio::_koha_add_biblioitem() && _koha_add_biblio() to support INSERTing biblionumbers and biblioitemnumbers to
the Koha DB. Created validators to make sure the primary keys are positive integers and well below the Perl's LONG_MAX threshold
to keep Koha from getting crazy. Duplicate id's are the users responsibility and in normal use conditions they are non-existing,
unless the "Koha to MARC mapping" to biblionumber is a user-modifiable MARC field, which is TROUBLE!
MySQL takes care of the primary key sequence refreshing.

Updated the bulkmarcimport.pl -script to copy the desired legacy id MARC field to the biblionumber- and biblioitemnumber-fields
designated in "Koha to MARC mapping", eg. copy field 001 to 999c and 999d.
This is handy if you have the legacy id in just 001 and need to populate 999c and 999d to push them to DB.
No need to make an extension to the existing migration scripts.

Documented the new functionality in bulkmarcimport.pl

Created test cases for C4::Biblio.pm modifications.
Comment 27 Olli-Antti Kivilahti 2014-07-31 13:14:50 UTC
This patch just created an issue for us in conjunction with Bug 12624.

It was very useful in our initial migration but after that it will be worthless.
Comment 28 Olli-Antti Kivilahti 2014-07-31 13:16:25 UTC
BTW the issue was existing biblios getting overwritten by new records from staging tables.

So if C4::Biblio::_koha_add_biblio() returns an error, the rest of the instructions are carried out normally, eg. ModBiblio, which overwrites the existing marc record even if the "adding new record to a reserved biblionumber" just failed.
Comment 29 Mark Tompsett 2015-04-04 21:35:27 UTC
(In reply to Olli-Antti Kivilahti from comment #27)
> This patch just created an issue for us in conjunction with Bug 12624.
> 
> It was very useful in our initial migration but after that it will be
> worthless.

So, should this be left as "Patch doesn't apply" or perhaps change it to "Resolved - Won't Fix"?