Bug 8746 - rebuild_zebra_sliced.sh don't work where Record length of 106041 is larger than the MARC spec allows
Summary: rebuild_zebra_sliced.sh don't work where Record length of 106041 is larger th...
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low major (vote)
Assignee: Julian Maurice
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-09-10 13:38 UTC by Stéphane Delaune
Modified: 2013-12-05 20:06 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML (7.84 KB, patch)
2012-09-20 12:29 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 8746: Fix indexation in DOM index mode (3.79 KB, patch)
2013-01-11 14:31 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 8746 [Follow-up] Replace == by eq in string comparison (1.31 KB, patch)
2013-03-19 13:55 UTC, Julian Maurice
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML (8.06 KB, patch)
2013-03-20 01:09 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 8746: Fix indexation in DOM index mode (3.86 KB, patch)
2013-03-20 01:09 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 8746 [Follow-up] Replace == by eq in string comparison (1.38 KB, patch)
2013-03-20 01:09 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[PASSED QA] Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML (8.11 KB, patch)
2013-04-07 19:01 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 8746: Fix indexation in DOM index mode (3.92 KB, patch)
2013-04-07 19:01 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 8746 [Follow-up] Replace == by eq in string comparison (1.45 KB, patch)
2013-04-07 19:01 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Stéphane Delaune 2012-09-10 13:38:19 UTC
rebuild_zebra_sliced.sh don't work where Record length of 106041 is larger than the MARC spec allows
Comment 1 Julian Maurice 2012-09-20 12:29:34 UTC Comment hidden (obsolete)
Comment 2 Mason James 2012-11-13 03:49:33 UTC
(In reply to comment #1)
> Created attachment 12386 [details] [review]
> Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML
> 
> This avoid indexing failures due to "bad offset" or "bad length" error
> with ISO2709 format
> 
> + minor improvements:
>   -  --length parameter is optional. If not given, it will execute the
>      right sql query to find the number of records to index
>   -  new parameter --reset-index. If set, index is reset before indexing
> 
> 

sorry, i could not get this script working correctly, using...
 $ perl ./misc/migration_tools/rebuild_zebra_sliced.sh --reset-index

using the normal 'rebuild_zebra.pl' worked OK, 

any ideas?
(an attached test .mrc file might be helpful for testing this patch?)
Comment 3 Julian Maurice 2012-11-22 10:29:05 UTC
Mason, did you get some error messages?
I just tried
  misc/migration_tools/rebuild_zebra_sliced.sh --reset-index
on a freshly updated install and it worked just fine.
Comment 4 Julian Maurice 2013-01-11 14:31:13 UTC Comment hidden (obsolete)
Comment 5 Julian Maurice 2013-01-11 14:32:16 UTC
Mason I think this patch should fix the problem you encountered.
Comment 6 Bernardo Gonzalez Kriegel 2013-02-23 04:50:48 UTC
I tried with small base (2K records).
rebuil_zebra.pl works ok, and don't have anomalous records.

Pre-patch with rebuild_zebra_sliced.sh -t biblios -s 1000 -f
...
Splitting file in chunks of 1000 records
Indexing /tmp/rebuild/export/biblio/exported_records_1000_00
Indexing /tmp/rebuild/export/biblio/exported_records_1000_01
Indexing /tmp/rebuild/export/biblio/exported_records_1000_02
Ok

Post-patch, same arguments
Splitting file in chunks of 1000 records
Indexing /tmp/rebuild/export/biblio/exported_records_1000_00
Indexing /tmp/rebuild/export/biblio/exported_records_1000_01
Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01.log
Split file and continue...
Splitting file in chunks of 500 records
Indexing /tmp/rebuild/export/biblio/exported_records_1000_01_500_00
Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01_500_00.log
...
Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01_500_00_250_00_125_00_62_00_31_00_15_01_7_00_3_00.log
Split file and continue...
Splitting file in chunks of 1 records
Indexing /tmp/rebuild/export/biblio/exported_records_1000_01_500_00_250_00_125_00_62_00_31_00_15_01_7_00_3_00_1_00
Indexing failed. See log file /tmp/rebuild/logs/zebraidx.exported_records_1000_01_500_00_250_00_125_00_62_00_31_00_15_01_7_00_3_00_1_00.log
...
Proceeds one by one.

Use grs1
Comment 7 Julian Maurice 2013-03-19 13:55:42 UTC Comment hidden (obsolete)
Comment 8 Julian Maurice 2013-03-19 13:57:28 UTC
$indexmode == "dom" was returning always true even if $indexmode was not "dom"
$indexmode eq "dom" works much better :)
Comment 9 Bernardo Gonzalez Kriegel 2013-03-20 01:09:16 UTC Comment hidden (obsolete)
Comment 10 Bernardo Gonzalez Kriegel 2013-03-20 01:09:24 UTC Comment hidden (obsolete)
Comment 11 Bernardo Gonzalez Kriegel 2013-03-20 01:09:30 UTC Comment hidden (obsolete)
Comment 12 Katrin Fischer 2013-04-07 19:01:06 UTC
Created attachment 17263 [details] [review]
[PASSED QA] Bug 8746: rebuild_zebra_sliced.sh now export/index records as MARCXML

This avoid indexing failures due to "bad offset" or "bad length" error
with ISO2709 format

+ minor improvements:
  -  --length parameter is optional. If not given, it will execute the
     right sql query to find the number of records to index
  -  new parameter --reset-index. If set, index is reset before indexing

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment: Work as described. No errors.

Test: Edit record to make it longer than 9999. Without patch rebuild_sliced
fails. With patches works.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Comment 13 Katrin Fischer 2013-04-07 19:01:17 UTC
Created attachment 17264 [details] [review]
[PASSED QA] Bug 8746: Fix indexation in DOM index mode

When in DOM index mode, files exported by `rebuild_zebra.pl -x` are
wrapped by '<collection></collection>' tag.
This is a problem because splitting files produces invalid files.
This is fixed by adding the missing <collection> tags in each generated
file.
Another problem was that the wrong zebra configuration file was used.
The script now uses C4::Context->zebraconfig($server)->{config} to know
which configuration file has to be used.

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Comment 14 Katrin Fischer 2013-04-07 19:01:56 UTC
Created attachment 17265 [details] [review]
Bug 8746 [Follow-up] Replace == by eq in string comparison

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
All tests and QA script pass.
Comment 15 Jared Camins-Esakov 2013-04-21 13:24:44 UTC
This patch has been pushed to master and 3.12.x.

Out of curiosity, why is this script written in bash when all of the actual work is done by embedded Perl code?
Comment 16 Julian Maurice 2013-04-29 13:23:27 UTC
> Out of curiosity, why is this script written in bash when all of the actual
> work is done by embedded Perl code?
My first intent was to use only existing script/programs to achieve this work,
so I choose sh.
Then I progressively realized that a little bit of Perl code was needed (for
example, for retrieving the indexing mode).
And now that I read the code again, I have to admit you're right. Script would
have been much simpler if written in Perl from the beginning.

But as such it works, so I think rewriting this script in Perl is not worth it.
(But anyone who wants to do it is gracefully encouraged! ;-))