|
Description
Marcel de Rooy
2013-11-21 12:01:58 UTC
Created attachment 23061 [details]
One test record in marcxml
Created attachment 23062 [details] [review] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Created attachment 23063 [details] [review] Bug 11278: [EXAMPLE: DO_NOT_PUSH] Example LocalChanges.pm This patch adds a LocalChanges.pm to migration_tools. To illustrate the concept of record customization during bulkmarcimport. Created attachment 23133 [details] [review] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Created attachment 23134 [details] [review] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Hi Marcel,
It does not seem to work with the default value for the custom parameter:
% perl misc/migration_tools/bulkmarcimport.pl -file /tmp/zztest01.xml -v=2 -b -m XML -t | more
TESTING MODE ONLY
DOING NOTHING
===============
Characteristic MARC flavour: UNIMARC
LDR 00000uam a22000001i 4500
001 734
008 870805
041 _aeng
044 _cNL
070 _a86307796X
084 _a14
245 00 _aEmerging hierarchies : processes of stratification and early state formation in the Indonesian archipelago : prehistory and the ethnographic present
_cdoor Ina Erna Slamet-Velsink
260 _a[S.l.
_bs.n.]
_c1986
300 _aviii, 310 p., 20 pl.
_bill., maps
_c22 cm
500 _aDoctoral diss. RU Leiden
500 _aBibliogr.: p. 278-296
500 _aIncludes index
650 4 _aarchaeology
650 4 _astates
650 4 _anation building
650 4 _acultural and social anthropology
651 4 _aindonesia
700 1 _aSlamet-Velsink, Ina Erna
852 _aCentrale Bibliotheek (CB)
_cN 87-911
_p14000000001457
_zBeschikbaar
852 _aCentrale Bibliotheek (CB)
_cN 96-317
_p14000002218786
_zBeschikbaar
1 MARC records done in 0.0246779918670654 seconds
There is no 005 created.
Marked as Failed QA.
(In reply to Jonathan Druart from comment #6) > Hi Marcel, > It does not seem to work with the default value for the custom parameter: > > % perl misc/migration_tools/bulkmarcimport.pl -file /tmp/zztest01.xml -v=2 > -b -m XML -t | more The default value will only be used if you supply the custom flag. Something like bulkmarcimport.pl -file ... -cust ... In your example the -custom parameter is not passed, so bulkmarcimport does not look at the customization routines (just normal behavior). Does this need a better explanation in the POD ? Created attachment 24748 [details] [review] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Created attachment 24749 [details] [review] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Created attachment 24750 [details] [review] Bug 11278: (follow-up) Return an exit value (1) if the module is not found. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> (In reply to M. de Rooy from comment #7) > In your example the -custom parameter is not passed, so bulkmarcimport does > not look at the customization routines (just normal behavior). Yes sorry, I have just stupidly followed the test plan. (In reply to Jonathan Druart from comment #11) Thanks for testing and follow-up. Created attachment 26087 [details] [review] [PASSED QA] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 26088 [details] [review] [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 26090 [details] [review] [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 26099 [details] [review] [PASSED QA] Bug 11278: Adjusting bulkmarcimport.pl for customization routine and verbose printing This patch makes two adjustments: [1] For the verbose option, verbose level 2 now means print the formatted marcxml per record. [2] If a module LocalChanges.pm is found in misc/migration_tools, the routine customize in this module is called for each marc record. This allows you to make local changes to these marc records before importing them. Test plan: [1] Test the verbose option: do no use -v on the command line, use -v 1 and -v 2 on the command line. (Do not yet copy LocalChanges.pm in the folder.) You may used the attached example file on Bugzilla: perl misc/migration_tools/bulkmarcimport.pl -file zztest01.xml -v 2 -b -m XML -t | more Note the option t for test; no records will be imported. [2] Copy LocalChanges.pm in the migration_tools folder. You may use the example provided on Bugzilla (in a patch). If you use the example module, check the contents of 001, 005 and 590 fields. (The verbose 2 option allows you to easily check that.) Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 26100 [details] [review] [PASSED QA] Bug 11278: Followup for customize command line parameter The initial patch for this bug did not include a specific command line option for customization. If a module LocalChanges.pm existed, it would be used without asking. This patch adds a command line option enabling the customization option and offering the extra possibility of using another module name. If no file name is passed, we default to LocalChanges. Without the -custom option, behavior is as it was. Also some POD lines are added to document the feature. Test plan: [1] Make a LocalChanges.pm in migration_tools. Verify that it is not used, if you do not enable the -cust parameter. [2] Run the script again with -cust. Verify that it is called now. [3] Copy LocalChanges.pm to Whatever.pm. Make some change. Run with -cust Whatever and verify that the new module is used. [4] Copy Whatever.pm to another dir, make some change. Run with -cust and the full name. Verify that the latest change was used. [5] Run without any option. Check the pod documentation. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 26101 [details] [review] [PASSED QA] Bug 11278: (follow-up) Return an exit value (1) if the module is not found. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Does exactly what it says on the tin, and passes QA scripts too. I'm happy that the code all looks sane and reasonable, and have tested some larger imports with & without the given example filter module. I would love to see some of the bulkmarkimport features added to the stage_file.pl and commit_file.pl workflow for this action; but this is for another day. Pushed to master. Thanks, Marcel! |