it's sometimes useful to do some changes on the fly on biblios or items. We will add a YAML file to do some mappings when running bulkmarcimport we will also add: * a test parameter * an insert parameter to allow INSERT only * and update parameter to allow biblios to be only updated * a filter on tags/subfields
Created attachment 3547 [details] [review] Proposed Patch
This bug is mentioned in: Bug 5635 changes for bulkmarcimport http://lists.koha-community.org/pipermail/koha-patches/2011-March/014372.html
patch does not apply anymore. Marking as such, will try to submit smaller improvements & a clear test plan...
This patch does not apply and should not be applied. It removes some things that are usefull in official bulkmarcimport and are not synch'ed with the patch. For example: - pod2usage( -verbose => 2 ); - exit; + print <<EOF ; +Small script to import bibliographic records into Koha. + +Parameters: + h this version/help screen + file /path/to/file/to/dump: the file to import + v verbose mode. 1 means "some infos", 2 means "MARC dumping" + fk Turn off foreign key checks during import. + n the number of records to import. If missing, all the file is imported + o file offset before importing, ie number of records to skip. + commit the number of records to wait before performing a 'commit' operation + l file logs actions done for each record and their status into file + t test mode: parses the file, saying what he would do, but doing nothing. + s skip automatic conversion of MARC-8 to UTF-8. This option is + provided for debugging. + c the characteristic MARC flavour. At the moment, only MARC21 and + UNIMARC are supported. MARC21 by default. + d delete EVERYTHING related to biblio in koha-DB before import. Tables: + biblio, biblioitems, titems + m format, MARCXML or ISO2709 (defaults to ISO2709) + yaml file format a yaml file with ids + keepids field store ids in field (usefull for authorities, where 001 contains the authid for Koha, that can contain a very valuable info for authorities coming from LOC or BNF. useless for biblios probably) + x source bib tag for reporting the source bib number + y source subfield for reporting the source bib number + idmap file for the koha bib and source id + keepids store ids in 009 (usefull for authorities, where 001 contains the authid for Koha, that can contain a very valuable info for authorities coming from LOC or BNF. useless for biblios probably) + b|biblios type of import : bibliographic records + a|authorities type of import : authority records + match matchindex,fieldtomatch matchpoint to use to deduplicate + fieldtomatch can be either 001 to 999 + or field and list of subfields as such 100abcde + test if set, test mode only, donot add anything in database + insert if set, only insert when possible + update if set, only updates (any biblio should have a matching record) + all if set, do whatever is required + i|isbn if set, a search will be done on isbn, and, if the same isbn is found, the biblio is not added. It's another + method to deduplicate. + match & i can be both set. +IMPORTANT: don't use this script before you've entered and checked your MARC + parameters tables twice (or more!). Otherwise, the import won't work + correctly and you will get invalid data. + +SAMPLE: + \$ export KOHA_CONF=/etc/koha.conf + \$ perl misc/migration_tools/bulkmarcimport.pl -d -commit 1000 \\ + -file /home/jmf/koha.mrc -n 3000 +EOF +exit; It also does a lot of indentation/perltidy changes that make it hard to read. Should be rewritten completly diff'ing community/master and biblibre/master bulkmarcimport & picking what's needed ! (meld will help !)
(In reply to comment #4) > This patch does not apply and should not be applied. It removes some things > that are usefull in official bulkmarcimport and are not synch'ed with the > patch. > For example: > - pod2usage( -verbose => 2 ); > - exit; > + print <<EOF ; ... > +EOF > +exit; This part contains just the documentation. However, in the two branches there is documentation at end of file. I will complete it with new elements. > > It also does a lot of indentation/perltidy changes that make it hard to read. > Should be rewritten completly diff'ing community/master and biblibre/master > bulkmarcimport & picking what's needed ! (meld will help !) ok, I'll try to do that at best
Created attachment 7485 [details] [review] proposed patch
Created attachment 7486 [details] to test the patch this attachement contains files to test the patch. extract here, and run it with : export KOHA_CONF=/home/koha/etc/koha-conf.xml && export PERL5LIB=/home/src && /usr/bin/perl /home/koha/src/misc/migration_tools/bulkmarcimport.pl -s -yaml /path_to_the_extract_directory_of_test5635/TR728R39C001.RAW.yaml -match ident,001 -keepids 009 -l /path_to_the_extract_directory_of_test5635/log/ILN2011-10-04-02-c.csv -a -file /path_to_the_extract_directory_of_test5635/fichiers/TR728R39C001.RAW -authtypes /path_to_the_extract_directory_of_test5635/authtypes.yml if you have an error like this : ".CCL parsing error (10014) Unknown qualifier ZOOM at /home/koha/src/C4/Search.pm" replace your ccl.properties with the file into ccl_used directory in the tar.gz file
(In reply to comment #7) > Created attachment 7486 [details] > to test the patch > > this attachement contains files to test the patch. > > extract here, and run it with : > export KOHA_CONF=/home/koha/etc/koha-conf.xml && export PERL5LIB=/home/src > && /usr/bin/perl /home/koha/src/misc/migration_tools/bulkmarcimport.pl -s > -yaml /path_to_the_extract_directory_of_test5635/TR728R39C001.RAW.yaml > -match ident,001 -keepids 009 -l > /path_to_the_extract_directory_of_test5635/log/ILN2011-10-04-02-c.csv -a > -file /path_to_the_extract_directory_of_test5635/fichiers/TR728R39C001.RAW > -authtypes /path_to_the_extract_directory_of_test5635/authtypes.yml > > if you have an error like this : ".CCL parsing error (10014) Unknown > qualifier ZOOM at /home/koha/src/C4/Search.pm" replace your ccl.properties > with the file into ccl_used directory in the tar.gz file Why do we need a special ccl.properties file in order to use this script? We never have in the past. It seems to me it would be better to patch ccl.properties in master, wouldn't it?
Patch does not apply.
Created attachment 11071 [details] [review] proposed patch updated patch
This patch is live in Mines ParisTech since 2012-07-24. Signing off
Please resubmit after translating the following comment in english: + # On récupère tous les souschamps des champs vedettes d'autorités
Created attachment 13671 [details] [review] the required followup (translated comment) it's the required followup (translate a comment in english)
I would like to re-iterate Jareds Question in comment 8. "Why do we need a special ccl.properties file in order to use this script? We never have in the past. It seems to me it would be better to patch ccl.properties in master, wouldn't it?" Elliott
(En réponse au commentaire 14) > "Why do we need a special ccl.properties file in order to use this script? no ; the ccl.properties file is here only if your ccl is not standard
Elliott, does Stephane's answer resolve your question? I agree it should work with the standard ccl file, but it looks like we all do :)
hey ho, this patch is signed since almost five months, it would be nice going to the next step (before the end of the world :) ).
The patch, created 6 months ago and signed since almost five months, is now "does not apply" (YEAH)
Created attachment 14499 [details] [review] proposed patch
Created attachment 14500 [details] [review] followup for 3.8.5
Created attachment 14502 [details] [review] proposed patch
Created attachment 14503 [details] [review] proposed patch for master (3.11.00.014) I hope the final proposed patch (with the precedently forgotten translated's comment from last followup).
Created attachment 14504 [details] to test last proposed patch to test the patch this attachement contains files to test the patch. extract here, and run it with : export KOHA_CONF=/home/koha/etc/koha-conf.xml && export PERL5LIB=/home/src && /usr/bin/perl /home/koha/src/misc/migration_tools/bulkmarcimport.pl -s -yaml /path_to_the_extract_directory_of_test5635/TR728R39C001.RAW.yaml -match ident,001 -keepids 009 -l /path_to_the_extract_directory_of_test5635/log/ILN2011-10-04-02-c.csv -a -file /path_to_the_extract_directory_of_test5635/fichiers/TR728R39C001.RAW -authtypes /path_to_the_extract_directory_of_test5635/authtypes.yml
Please note that when no authtypes are provided, bulkmarcimport will crash. A possible patch would be: @@ -544,6 +544,7 @@ sub get_heading_fields{ $debug && warn YAML::Dump($headingfields); } unless ($headingfields){ + my $dbh = C4::Context->dbh; $headingfields=$dbh->selectall_hashref("SELECT auth_tag_to_report, authtypecode from auth_types",'auth_tag_to_report',{Slice=>{}}); $headingfields={C4::Context->preference('marcflavour')=>$headingfields}; } Could you please fix this, Stéphane ?
Okay, my mistake, this was already fixed, but in a different way (by moving the dbh initialization).
Created attachment 16227 [details] [review] proposed patch for master (3.11.00.100) an amend on the commit message and a little update on documentation
test appplies in Mines ParisTech
Created attachment 16259 [details] [review] proposed patch for master (3.11.00.100) same than last but passed perlcritic
Created attachment 16364 [details] [review] [PASSED QA] Bug 5635: bulkmarcimport new parameters & features See the script's documentation for more details New parameters are: - authtypes - filter - insert - update - all Signed-off-by: Pascale Nalon <pascale.nalon@gmail.com> This patch is live in Mines ParisTech since 2012-07-24. Signing off Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> - Moved the sign-off from bugzilla to the commit message. - All tests and QA script pass. - Amended commit message to list new parameters. - Verified this patch works on a UNIMARC installation. - Verified normal import still works correct on a MARC21 installation.
This patch has been pushed to master.
Added a bad ModBiblio call. Look at see also.