Bug 13082 - tools/export.pl fix to forbid adding of all invalid records
Summary: tools/export.pl fix to forbid adding of all invalid records
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: master
Hardware: All All
: P3 normal (vote)
Assignee: Stéphane Delaune
QA Contact: Testopia
URL:
Keywords:
Depends on: 12229
Blocks:
  Show dependency treegraph
 
Reported: 2014-10-15 10:01 UTC by Stéphane Delaune
Modified: 2015-06-04 23:33 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Text to go in the release notes:
Version(s) released in:


Attachments
proposed patch (1.59 KB, patch)
2014-10-15 13:36 UTC, Stéphane Delaune
Details | Diff | Splinter Review
proposed patch (1.72 KB, patch)
2014-10-21 16:12 UTC, Stéphane Delaune
Details | Diff | Splinter Review
Bug 13082: fix to prevent adding of invalid records in marc file (2.19 KB, patch)
2014-10-25 10:34 UTC, Frédéric Demians
Details | Diff | Splinter Review
[PASSED QA] Bug 13082: fix to prevent adding of invalid records in marc file (2.37 KB, patch)
2014-10-25 10:57 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Stéphane Delaune 2014-10-15 10:01:50 UTC
records exported by Koha may be invalid for many reasons such as:
- records too large exceeding the 99999 bytes allowed by the leader
- the encoding problems that produce a deviation between the size defined in the leader and the real size of the record when reopening (it generates many character offsets)
- all other problems can generate warnings to the reopening of exported records (and the crash of most of the reimport's scripts)

bz12229 only fixes the problem with records too large, but it does not exclude all the invalid records from the exported file : a more complete solution must replace it.
Comment 1 Stéphane Delaune 2014-10-15 13:36:15 UTC Comment hidden (obsolete)
Comment 2 Stéphane Delaune 2014-10-21 16:12:59 UTC Comment hidden (obsolete)
Comment 3 Frédéric Demians 2014-10-25 10:34:19 UTC Comment hidden (obsolete)
Comment 4 Katrin Fischer 2014-10-25 10:57:18 UTC
Created attachment 32718 [details] [review]
[PASSED QA] Bug 13082: fix to prevent adding of invalid records in marc file

Test:

1. Edit record, add 100.000 chars text to 500a

2. xml export produce the record,

3. mrc export do not produce the record, warning on log export.pl:
   Record length of 111000 is larger than the MARC spec allows (99999
   bytes). at /usr/share/perl5/MARC/File/USMARC.pm line 314.  record
   (number 139489) is invalid and therefore not exported because its
   reopening generates warnings above at...

Signed-off-by: Frederic Demians <f.demians@tamil.fr>

I confirm that exporting biblio records larger than 10000 characters in
ISO2709 produces invalid files. After applying this patch, the culprit
record (too large, but also other inconsistencies preventing record
parsing with MARC::File::USMARC) is not exported anymore. A warning is
produced in Koha Apache log file. Warnings to the user on WUI would be
better, but it isn't the case yet, so it isn't a regression.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>

I agree that a visible warning/result message in the staff interface
would be nice, but this works as described.
Comment 5 Tomás Cohen Arazi 2014-11-14 14:50:48 UTC
Patch pushed to master.

Thanks Stéphane!