The forthcoming patch changes bulkmarcimport.pl in two ways: 1) For the verbose option, verbose level 2 now means print the formatted marcxml for the records. Verbose 1 just prints dots and counts-per-hundred-records. 2) The new code for customization allows you to add a LocalChanges.pm in the misc/migration_tools folder. If this module exists, it is 'required' and a routine customize is called for each record. This makes it possible to add local adjustments to your marc record before they are processed in bulkmarcimport. I will attach a second example patch with such a module just to show the idea; this patch does not need signoff, etc.
Created attachment 23061 [details] One test record in marcxml
Created attachment 23062 [details] [review] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.)
Created attachment 23063 [details] [review] Bug 11278: [EXAMPLE: DO_NOT_PUSH] Example LocalChanges.pm This patch adds a LocalChanges.pm to migration_tools. To illustrate the concept of record customization during bulkmarcimport.
Created attachment 23133 [details] [review] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.)
Created attachment 23134 [details] [review] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation.
Hi Marcel, It does not seem to work with the default value for the custom parameter: % perl misc/migration_tools/bulkmarcimport.pl -file /tmp/zztest01.xml -v=2 -b -m XML -t | more TESTING MODE ONLY DOING NOTHING =============== Characteristic MARC flavour: UNIMARC LDR 00000uam a22000001i 4500 001 734 008 870805 041 _aeng 044 _cNL 070 _a86307796X 084 _a14 245 00 _aEmerging hierarchies : processes of stratification and early state formation in the Indonesian archipelago : prehistory and the ethnographic present _cdoor Ina Erna Slamet-Velsink 260 _a[S.l. _bs.n.] _c1986 300 _aviii, 310 p., 20 pl. _bill., maps _c22 cm 500 _aDoctoral diss. RU Leiden 500 _aBibliogr.: p. 278-296 500 _aIncludes index 650 4 _aarchaeology 650 4 _astates 650 4 _anation building 650 4 _acultural and social anthropology 651 4 _aindonesia 700 1 _aSlamet-Velsink, Ina Erna 852 _aCentrale Bibliotheek (CB) _cN 87-911 _p14000000001457 _zBeschikbaar 852 _aCentrale Bibliotheek (CB) _cN 96-317 _p14000002218786 _zBeschikbaar 1 MARC records done in 0.0246779918670654 seconds There is no 005 created. Marked as Failed QA.
(In reply to Jonathan Druart from comment #6) > Hi Marcel, > It does not seem to work with the default value for the custom parameter: > > % perl misc/migration_tools/bulkmarcimport.pl -file /tmp/zztest01.xml -v=2 > -b -m XML -t | more The default value will only be used if you supply the custom flag. Something like bulkmarcimport.pl -file ... -cust ... In your example the -custom parameter is not passed, so bulkmarcimport does not look at the customization routines (just normal behavior). Does this need a better explanation in the POD ?
Created attachment 24748 [details] [review] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Created attachment 24749 [details] [review] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Created attachment 24750 [details] [review] Bug 11278: (follow-up) Return an exit value (1) if the module is not found. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
(In reply to M. de Rooy from comment #7) > In your example the -custom parameter is not passed, so bulkmarcimport does > not look at the customization routines (just normal behavior). Yes sorry, I have just stupidly followed the test plan.
(In reply to Jonathan Druart from comment #11) Thanks for testing and follow-up.
Created attachment 26087 [details] [review] [PASSED QA] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 26088 [details] [review] [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 26090 [details] [review] [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 26099 [details] [review] [PASSED QA] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 26100 [details] [review] [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 26101 [details] [review] [PASSED QA] Bug 11278: (follow-up) Return an exit value (1) if the module is not found. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Does exactly what it says on the tin, and passes QA scripts too. I'm happy that the code all looks sane and reasonable, and have tested some larger imports with & without the given example filter module. I would love to see some of the bulkmarkimport features added to the stage_file.pl and commit_file.pl workflow for this action; but this is for another day.
Pushed to master. Thanks, Marcel!