rebuild_zebra_sliced.sh don't work where Record length of 106041 is larger than the MARC spec allows
Created attachment 12386 [details] [review] Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML This avoid indexing failures due to "bad offset" or "bad length" error with ISO2709 format + minor improvements: - --length parameter is optional. If not given, it will execute the right sql query to find the number of records to index - new parameter --reset-index. If set, index is reset before indexing -- Exporting multiple records in the same ISO2709 file can cause problems. If one record is wrong (say a record is longer than 99999 bytes, for example), then all following records won't be parsed correctly. For example yaz-marcdump refuse to read records that follow a malformed record. This cause problems in rebuild_zebra_sliced.sh because yaz-marcdump is used to count the number of records contained in one file. So the workaround used here is to export in MARCXML. yaz-marcdump can't split MARCXML files so a piece of Perl code is used instead (I didn't succeed to achieve this job with POSIX tools) To test you can create a biblio record longer than 99999 bytes and try to index a range of biblionumbers that contain it. On master indexation of this record should fail and this should succeed with this patch. (Records longer than 99999 bytes seem to not alter indexation of following records. I've encountered this behaviour with badly encoded records)
(In reply to comment #1) > Created attachment 12386 [details] [review] > Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML > > This avoid indexing failures due to "bad offset" or "bad length" error > with ISO2709 format > > + minor improvements: > - --length parameter is optional. If not given, it will execute the > right sql query to find the number of records to index > - new parameter --reset-index. If set, index is reset before indexing > > sorry, i could not get this script working correctly, using... $ perl ./misc/migration_tools/rebuild_zebra_sliced.sh --reset-index using the normal 'rebuild_zebra.pl' worked OK, any ideas? (an attached test .mrc file might be helpful for testing this patch?)
Mason, did you get some error messages? I just tried misc/migration_tools/rebuild_zebra_sliced.sh --reset-index on a freshly updated install and it worked just fine.
Created attachment 14537 [details] [review] Bug 8746: Fix indexation in DOM index mode When in DOM index mode, files exported by `rebuild_zebra.pl -x` are wrapped by '<collection></collection>' tag. This is a problem because splitting files produces invalid files. This is fixed by adding the missing <collection> tags in each generated file. Another problem was that the wrong zebra configuration file was used. The script now uses C4::Context->zebraconfig($server)->{config} to know which configuration file has to be used.
Mason I think this patch should fix the problem you encountered.
I tried with small base (2K records). rebuil_zebra.pl works ok, and don't have anomalous records. Pre-patch with rebuild_zebra_sliced.sh -t biblios -s 1000 -f ... Splitting file in chunks of 1000 records Indexing /tmp/rebuild/export/biblio/exported_records_1000_00 Indexing /tmp/rebuild/export/biblio/exported_records_1000_01 Indexing /tmp/rebuild/export/biblio/exported_records_1000_02 Ok Post-patch, same arguments Splitting file in chunks of 1000 records Indexing /tmp/rebuild/export/biblio/exported_records_1000_00 Indexing /tmp/rebuild/export/biblio/exported_records_1000_01 Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01.log Split file and continue... Splitting file in chunks of 500 records Indexing /tmp/rebuild/export/biblio/exported_records_1000_01_500_00 Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01_500_00.log ... Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01_500_00_250_00_125_00_62_00_31_00_15_01_7_00_3_00.log Split file and continue... Splitting file in chunks of 1 records Indexing /tmp/rebuild/export/biblio/exported_records_1000_01_500_00_250_00_125_00_62_00_31_00_15_01_7_00_3_00_1_00 Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01_500_00_250_00_125_00_62_00_31_00_15_01_7_00_3_00_1_00.log ... Proceeds one by one. Use grs1
Created attachment 16392 [details] [review] Bug 8746 [Follow-up] Replace == by eq in string comparison
$indexmode == "dom" was returning always true even if $indexmode was not "dom" $indexmode eq "dom" works much better :)
Created attachment 16448 [details] [review] [SIGNED-OFF] Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML This avoid indexing failures due to "bad offset" or "bad length" error with ISO2709 format + minor improvements: - --length parameter is optional. If not given, it will execute the right sql query to find the number of records to index - new parameter --reset-index. If set, index is reset before indexing Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Comment: Work as described. No errors. Test: Edit record to make it longer than 9999. Without patch rebuild_sliced fails. With patches works.
Created attachment 16449 [details] [review] [SIGNED-OFF] Bug 8746: Fix indexation in DOM index mode When in DOM index mode, files exported by `rebuild_zebra.pl -x` are wrapped by '<collection></collection>' tag. This is a problem because splitting files produces invalid files. This is fixed by adding the missing <collection> tags in each generated file. Another problem was that the wrong zebra configuration file was used. The script now uses C4::Context->zebraconfig($server)->{config} to know which configuration file has to be used. Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 16450 [details] [review] [SIGNED-OFF] Bug 8746 [Follow-up] Replace == by eq in string comparison Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 17263 [details] [review] [PASSED QA] Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML This avoid indexing failures due to "bad offset" or "bad length" error with ISO2709 format + minor improvements: - --length parameter is optional. If not given, it will execute the right sql query to find the number of records to index - new parameter --reset-index. If set, index is reset before indexing Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Comment: Work as described. No errors. Test: Edit record to make it longer than 9999. Without patch rebuild_sliced fails. With patches works. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Created attachment 17264 [details] [review] [PASSED QA] Bug 8746: Fix indexation in DOM index mode When in DOM index mode, files exported by `rebuild_zebra.pl -x` are wrapped by '<collection></collection>' tag. This is a problem because splitting files produces invalid files. This is fixed by adding the missing <collection> tags in each generated file. Another problem was that the wrong zebra configuration file was used. The script now uses C4::Context->zebraconfig($server)->{config} to know which configuration file has to be used. Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Created attachment 17265 [details] [review] Bug 8746 [Follow-up] Replace == by eq in string comparison Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> All tests and QA script pass.
This patch has been pushed to master and 3.12.x. Out of curiosity, why is this script written in bash when all of the actual work is done by embedded Perl code?
> Out of curiosity, why is this script written in bash when all of the actual > work is done by embedded Perl code? My first intent was to use only existing script/programs to achieve this work, so I choose sh. Then I progressively realized that a little bit of Perl code was needed (for example, for retrieving the indexing mode). And now that I read the code again, I have to admit you're right. Script would have been much simpler if written in Perl from the beginning. But as such it works, so I think rewriting this script in Perl is not worth it. (But anyone who wants to do it is gracefully encouraged! ;-))