Bug 5635 - bulkmarcimport new parameters & features
Summary: bulkmarcimport new parameters & features
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: master
Hardware: All All
: P1 - high enhancement (vote)
Assignee: Stéphane Delaune
QA Contact: Katrin Fischer
URL:
Keywords:
Depends on:
Blocks: 5992 20811
  Show dependency treegraph
 
Reported: 2011-01-19 20:53 UTC by Paul Poulain
Modified: 2018-07-20 09:54 UTC (History)
11 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Proposed Patch (18.11 KB, patch)
2011-03-30 20:51 UTC, Chris Cormack
Details | Diff | Splinter Review
proposed patch (14.05 KB, patch)
2012-02-07 16:26 UTC, Stéphane Delaune
Details | Diff | Splinter Review
to test the patch (26.54 KB, application/octet-stream)
2012-02-07 16:37 UTC, Stéphane Delaune
Details
proposed patch (14.11 KB, patch)
2012-07-23 08:54 UTC, Stéphane Delaune
Details | Diff | Splinter Review
the required followup (translated comment) (1.12 KB, patch)
2012-11-26 09:33 UTC, Stéphane Delaune
Details | Diff | Splinter Review
proposed patch (14.15 KB, patch)
2013-01-09 15:25 UTC, Stéphane Delaune
Details | Diff | Splinter Review
followup for 3.8.5 (1.14 KB, patch)
2013-01-09 15:42 UTC, Stéphane Delaune
Details | Diff | Splinter Review
proposed patch (14.67 KB, patch)
2013-01-09 16:47 UTC, Stéphane Delaune
Details | Diff | Splinter Review
proposed patch for master (3.11.00.014) (14.59 KB, patch)
2013-01-09 17:12 UTC, Stéphane Delaune
Details | Diff | Splinter Review
to test last proposed patch (14.91 KB, application/x-gzip)
2013-01-09 17:13 UTC, Stéphane Delaune
Details
proposed patch for master (3.11.00.100) (14.43 KB, patch)
2013-03-18 11:06 UTC, Stéphane Delaune
Details | Diff | Splinter Review
proposed patch for master (3.11.00.100) (15.37 KB, patch)
2013-03-18 15:00 UTC, Stéphane Delaune
Details | Diff | Splinter Review
[PASSED QA] Bug 5635: bulkmarcimport new parameters & features (15.90 KB, patch)
2013-03-19 10:30 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Poulain 2011-01-19 20:53:34 UTC
it's sometimes useful to do some changes on the fly on biblios or items. We will add a YAML file to do some mappings when running bulkmarcimport

we will also add: 
* a test parameter
* an insert parameter to allow INSERT only
* and update parameter to allow biblios to be only updated
* a filter on tags/subfields
Comment 1 Chris Cormack 2011-03-30 20:51:58 UTC Comment hidden (obsolete)
Comment 2 MJ Ray (software.coop) 2011-04-04 08:09:49 UTC
This bug is mentioned in:
Bug 5635 changes for bulkmarcimport http://lists.koha-community.org/pipermail/koha-patches/2011-March/014372.html
Comment 3 Paul Poulain 2011-09-27 15:57:14 UTC
patch does not apply anymore. Marking as such, will try to submit smaller improvements & a clear test plan...
Comment 4 Paul Poulain 2012-01-19 14:33:41 UTC
This patch does not apply and should not be applied. It removes some things that are usefull in official bulkmarcimport and are not synch'ed with the patch.
For example:
-    pod2usage( -verbose => 2 );
-    exit;
+    print <<EOF ;
+Small script to import bibliographic records into Koha.
+
+Parameters:
+  h      this version/help screen
+  file   /path/to/file/to/dump: the file to import
+  v      verbose mode. 1 means "some infos", 2 means "MARC dumping"
+  fk     Turn off foreign key checks during import.
+  n      the number of records to import. If missing, all the file is imported
+  o      file offset before importing, ie number of records to skip.
+  commit the number of records to wait before performing a 'commit' operation
+  l file logs actions done for each record and their status into file
+  t      test mode: parses the file, saying what he would do, but doing nothing.
+  s      skip automatic conversion of MARC-8 to UTF-8.  This option is 
+         provided for debugging.
+  c      the characteristic MARC flavour. At the moment, only MARC21 and 
+         UNIMARC are supported. MARC21 by default.
+  d      delete EVERYTHING related to biblio in koha-DB before import. Tables:
+         biblio, biblioitems, titems
+  m      format, MARCXML or ISO2709 (defaults to ISO2709)
+  yaml file  format a yaml file with ids
+  keepids field store ids in field (usefull for authorities, where 001 contains the authid for Koha, that can contain a very valuable info for authorities coming from LOC or BNF. useless for biblios probably)
+  x      source bib tag for reporting the source bib number
+  y      source subfield for reporting the source bib number
+  idmap  file for the koha bib and source id
+  keepids store ids in 009 (usefull for authorities, where 001 contains the authid for Koha, that can contain a very valuable info for authorities coming from LOC or BNF. useless for biblios probably)
+  b|biblios type of import : bibliographic records
+  a|authorities type of import : authority records
+  match  matchindex,fieldtomatch matchpoint to use to deduplicate
+          fieldtomatch can be either 001 to 999 
+                       or field and list of subfields as such 100abcde
+  test   if set, test mode only, donot add anything in database
+  insert if set, only insert when possible
+  update if set, only updates (any biblio should have a matching record)
+  all    if set, do whatever is required
+  i|isbn if set, a search will be done on isbn, and, if the same isbn is found, the biblio is not added. It's another
+         method to deduplicate. 
+         match & i can be both set.
+IMPORTANT: don't use this script before you've entered and checked your MARC 
+           parameters tables twice (or more!). Otherwise, the import won't work 
+           correctly and you will get invalid data.
+
+SAMPLE: 
+  \$ export KOHA_CONF=/etc/koha.conf
+  \$ perl misc/migration_tools/bulkmarcimport.pl -d -commit 1000 \\
+    -file /home/jmf/koha.mrc -n 3000
+EOF
+exit;

It also does a lot of indentation/perltidy changes that make it hard to read. Should be rewritten completly diff'ing community/master and biblibre/master bulkmarcimport & picking what's needed ! (meld will help !)
Comment 5 Stéphane Delaune 2012-02-07 13:23:23 UTC
(In reply to comment #4)
> This patch does not apply and should not be applied. It removes some things
> that are usefull in official bulkmarcimport and are not synch'ed with the
> patch.
> For example:
> -    pod2usage( -verbose => 2 );
> -    exit;
> +    print <<EOF ;
...
> +EOF
> +exit;
This part contains just the documentation. However, in the two branches there is documentation at end of file. I will complete it with new elements.
> 
> It also does a lot of indentation/perltidy changes that make it hard to read.
> Should be rewritten completly diff'ing community/master and biblibre/master
> bulkmarcimport & picking what's needed ! (meld will help !)
ok, I'll try to do that at best
Comment 6 Stéphane Delaune 2012-02-07 16:26:58 UTC Comment hidden (obsolete)
Comment 7 Stéphane Delaune 2012-02-07 16:37:41 UTC Comment hidden (obsolete)
Comment 8 Jared Camins-Esakov 2012-07-09 11:51:50 UTC
(In reply to comment #7)
> Created attachment 7486 [details]
> to test the patch
> 
> this attachement contains files to test the patch.
> 
> extract here, and run it with :
> export KOHA_CONF=/home/koha/etc/koha-conf.xml && export PERL5LIB=/home/src
> && /usr/bin/perl /home/koha/src/misc/migration_tools/bulkmarcimport.pl -s
> -yaml /path_to_the_extract_directory_of_test5635/TR728R39C001.RAW.yaml
> -match ident,001 -keepids 009 -l
> /path_to_the_extract_directory_of_test5635/log/ILN2011-10-04-02-c.csv -a
> -file /path_to_the_extract_directory_of_test5635/fichiers/TR728R39C001.RAW
> -authtypes /path_to_the_extract_directory_of_test5635/authtypes.yml
> 
> if you have an error like this : ".CCL parsing error (10014) Unknown
> qualifier ZOOM at /home/koha/src/C4/Search.pm" replace your ccl.properties
> with the file into ccl_used directory in the tar.gz file

Why do we need a special ccl.properties file in order to use this script? We never have in the past. It seems to me it would be better to patch ccl.properties in master, wouldn't it?
Comment 9 Gaetan Boisson 2012-07-23 08:00:15 UTC
Patch does not apply.
Comment 10 Stéphane Delaune 2012-07-23 08:54:47 UTC Comment hidden (obsolete)
Comment 11 Pascale Nalon 2012-07-31 07:57:26 UTC
This patch is live in Mines ParisTech since 2012-07-24.
Signing off
Comment 12 Jonathan Druart 2012-08-20 09:46:52 UTC
Please resubmit after translating the following comment in english:

+                        # On récupère tous les souschamps des champs vedettes d'autorités
Comment 13 Stéphane Delaune 2012-11-26 09:33:49 UTC Comment hidden (obsolete)
Comment 14 Elliott Davis 2012-12-07 20:15:14 UTC
I would like to re-iterate Jareds Question in comment 8.

"Why do we need a special ccl.properties file in order to use this script? We never have in the past. It seems to me it would be better to patch ccl.properties in master, wouldn't it?"

Elliott
Comment 15 Stéphane Delaune 2012-12-11 11:28:54 UTC
(En réponse au commentaire 14)
> "Why do we need a special ccl.properties file in order to use this script?

no ; the ccl.properties file is here only if your ccl is not standard
Comment 16 Katrin Fischer 2012-12-14 08:59:30 UTC
Elliott, does Stephane's answer resolve your question? I agree it should work with the standard ccl file, but it looks like we all do :)
Comment 17 Stéphane Delaune 2012-12-19 08:18:05 UTC
hey ho, this patch is signed since almost five months, it would be nice going to the next step (before the end of the world :) ).
Comment 18 Stéphane Delaune 2013-01-09 15:18:44 UTC
The patch, created 6 months ago and signed since almost five months, is now "does not apply" (YEAH)
Comment 19 Stéphane Delaune 2013-01-09 15:25:30 UTC Comment hidden (obsolete)
Comment 20 Stéphane Delaune 2013-01-09 15:42:43 UTC Comment hidden (obsolete)
Comment 21 Stéphane Delaune 2013-01-09 16:47:54 UTC Comment hidden (obsolete)
Comment 22 Stéphane Delaune 2013-01-09 17:12:22 UTC Comment hidden (obsolete)
Comment 23 Stéphane Delaune 2013-01-09 17:13:42 UTC
Created attachment 14504 [details]
to test last proposed patch

to test the patch

this attachement contains files to test the patch.

extract here, and run it with :
export KOHA_CONF=/home/koha/etc/koha-conf.xml && export PERL5LIB=/home/src && /usr/bin/perl /home/koha/src/misc/migration_tools/bulkmarcimport.pl -s -yaml /path_to_the_extract_directory_of_test5635/TR728R39C001.RAW.yaml -match ident,001 -keepids 009 -l /path_to_the_extract_directory_of_test5635/log/ILN2011-10-04-02-c.csv -a -file /path_to_the_extract_directory_of_test5635/fichiers/TR728R39C001.RAW -authtypes /path_to_the_extract_directory_of_test5635/authtypes.yml
Comment 24 Matthias Meusburger 2013-02-04 14:43:11 UTC
Please note that when no authtypes are provided, bulkmarcimport will crash.

A possible patch would be:

@@ -544,6 +544,7 @@ sub get_heading_fields{
         $debug && warn YAML::Dump($headingfields);
     }
     unless ($headingfields){
+        my $dbh = C4::Context->dbh;
         $headingfields=$dbh->selectall_hashref("SELECT auth_tag_to_report, authtypecode from auth_types",'auth_tag_to_report',{Slice=>{}});
         $headingfields={C4::Context->preference('marcflavour')=>$headingfields};
     }

Could you please fix this, Stéphane ?
Comment 25 Matthias Meusburger 2013-02-04 14:53:57 UTC
Okay, my mistake, this was already fixed, but in a different way (by moving the dbh initialization).
Comment 26 Stéphane Delaune 2013-03-18 11:06:10 UTC Comment hidden (obsolete)
Comment 27 Pascale Nalon 2013-03-18 11:32:35 UTC
test appplies in Mines ParisTech
Comment 28 Stéphane Delaune 2013-03-18 15:00:18 UTC Comment hidden (obsolete)
Comment 29 Katrin Fischer 2013-03-19 10:30:49 UTC
Created attachment 16364 [details] [review]
[PASSED QA] Bug 5635: bulkmarcimport new parameters & features

See the script's documentation for more details

New parameters are:
 - authtypes
 - filter
 - insert
 - update
 - all

Signed-off-by: Pascale Nalon <pascale.nalon@gmail.com>
This patch is live in Mines ParisTech since 2012-07-24.
Signing off

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
 - Moved the sign-off from bugzilla to the commit message.
 - All tests and QA script pass.
 - Amended commit message to list new parameters.
 - Verified this patch works on a UNIMARC installation.
 - Verified normal import still works correct on a MARC21
   installation.
Comment 30 Jared Camins-Esakov 2013-03-22 00:22:18 UTC
This patch has been pushed to master.
Comment 31 Marcel de Rooy 2018-07-20 09:54:05 UTC
Added a bad ModBiblio call. Look at see also.