Bug 18757 - Problem when importing only items in MARC records
Summary: Problem when importing only items in MARC records
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Fridolin Somers
QA Contact: Testopia
URL:
Keywords:
: 23684 (view as bug list)
Depends on:
Blocks:
 
Reported: 2017-06-08 14:21 UTC by Fridolin Somers
Modified: 2020-11-30 21:48 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
19.11.00


Attachments
Bug 18757 - Problem when importing only items in MARC records (1.94 KB, patch)
2017-06-08 14:35 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 18757 - Problem when importing only items in MARC records (1.99 KB, patch)
2019-10-01 12:50 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 18757 - Problem when importing only items in MARC records (2.06 KB, patch)
2019-10-29 11:19 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin Somers 2017-06-08 14:21:44 UTC
When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records.
Those imported items are stored as XML to be staged.

The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist.
Comment 1 Fridolin Somers 2017-06-08 14:35:45 UTC
Created attachment 64104 [details] [review]
Bug 18757 - Problem when importing only items in MARC records

When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records.
Those imported items are stored as XML to be staged.

The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist.
You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a

This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml

Test plan :
- On a UNIMARC database
- Define a maching rule on title 200$a
- Select a record with items
- Export it using : Save as > MARC (Unicode/UTF-8)
- Delete all items
- Go to Tools > Stage MARC records for import
- Upload exported file
- Select title matching rule
- Select "Ingore incoming record" in "Action if matching record found :"
- Select Yes and "Always add items" in "Check for embedded item record data?"
- Click Stage for import
=> Without patch you get the error
=> With patch the import is staged
- Import into the catalog and check item is well recreated
Comment 2 Alex Buckley 2017-06-25 17:54:03 UTC
Hi Fridolin

When applying your patch on a clean branch up to date with master, I find that I am still getting the error message 'Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a' as I do without the patch applied. 

I am getting this error when trying to both create and import records.

I have tried setting the matching record rule to both 100$a and 200$a but continue to get the same error.
Comment 3 Fridolin Somers 2017-06-28 09:11:01 UTC
Looks like this only occurs with Plack.
One must test with plack on tools/manage-marc-import.pl.
Since there are problems with background jobs with plack, test with Bug 17240.
Comment 4 Fridolin Somers 2017-06-28 09:12:25 UTC
But this patch is I think logic since it does XML serialization like in C4::Items::_get_unlinked_subfields_xml
Comment 5 Tomás Cohen Arazi 2017-06-28 10:14:59 UTC
(In reply to Alex Buckley from comment #2)
> Hi Fridolin
> 
> When applying your patch on a clean branch up to date with master, I find
> that I am still getting the error message 'Unsupported UNIMARC character
> encoding [] for XML output for UNIMARC; 100$a' as I do without the patch
> applied. 
> 
> I am getting this error when trying to both create and import records.
> 
> I have tried setting the matching record rule to both 100$a and 200$a but
> continue to get the same error.

Make sure you restart Plack after applying the patch.
Comment 6 Fridolin Somers 2018-01-10 10:31:18 UTC
I set back to NSO
Comment 7 Mark Tompsett 2018-04-02 15:38:46 UTC
https://archive.ifla.org/VI/8/projects/UNIMARC-HoldingsFormat.pdf
-- but isn't there supposed to be a 100?!

12-15 Character Set (Mandatory)
These four character positions indicate the principal graphic character sets used in the record.
Positions 12-13 designate the G0 set and positions 14-15 designate the G1 set. If a G1 set is
not needed, positions 14-15 contain blanks.
01 = ISO 646, IRV version (basic Latin set)
02 = ISO Registration # 37 (basic Cyrillic set)
03 = ISO 5426 (extended Latin set)
04 = ISO DIS 5427 (extended Cyrillic set)
05 = ISO 5428 (Greek set)
06 = ISO 6438 (African coded character set)
07 = ISO 10586 (Georgian set)
08 = ISO 8957 (Hebrew set) Table 1
09 = ISO 8957 (Hebrew set) Table 2
10 = [Reserved]
11 = ISO 5426-2 (Latin characters used in minor European languages and obsolete
typography)
50 = ISO 10646 Level 3 (Unicode)

Oh yuck... reading the MARC::File::XML library... it basically fakes it.

    # Does the record think it is already Unicode?
    if ($original_encoding ne 'a' && lc($format) !~ /^unimarc/o) {
        # If not, we'll make it so
        $_transcode++;
        substr($ldr,9,1,'a');
        $record->leader( $ldr );
    }

which is why the as_xml in elsewhere does the "USMARC" thing. Okay. I'll test this, since I'm assuming Fridolin has already tested in a UNIMARC context before submitting this patch.
Comment 8 Mark Tompsett 2018-04-02 16:55:29 UTC
BTW, it should be noted that this same problem shows itself as an "Internal Server Error" if one tries to manually catalogue a record in UNIMARC, and uses something clearly bad, like "KABOOM!" in the 100$a field. :)

substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 561 (#2)
    (W substr)(F) You tried to reference a substr() that pointed outside of
    a string.  That is, the absolute value of the offset was larger than the
    length of the string.  See "substr" in perlfunc.  This warning is fatal if
    substr is used in an lvalue context (as the left hand side of an
    assignment or as a subroutine argument for example).
    
Use of uninitialized value $enc in string eq at
	/usr/share/perl5/MARC/File/XML.pm line 563 (#1)
Use of uninitialized value $enc in string eq at
	/usr/share/perl5/MARC/File/XML.pm line 565 (#1)
Use of uninitialized value $enc in concatenation (.) or string at
	/usr/share/perl5/MARC/File/XML.pm line 568 (#1)
Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> kaboom at /usr/share/perl5/MARC/File/XML.pm line 568.

No need to make any 200 rules.
Comment 9 Fridolin Somers 2018-04-25 12:10:17 UTC
(In reply to M. Tompsett from comment #7)

> which is why the as_xml in elsewhere does the "USMARC" thing. Okay. I'll
> test this, since I'm assuming Fridolin has already tested in a UNIMARC
> context before submitting this patch.

Indeed.
Several of our Koha UNIMARC have already this patch.

UNIMARC 100$a is a spaghetti dish ;-)
Comment 10 Biblibre Sandboxes 2019-10-01 12:50:14 UTC
Patch tested with a sandbox, by Amandine Zocca <azocca@ville-montauban.fr>
Comment 11 Biblibre Sandboxes 2019-10-01 12:50:38 UTC
Created attachment 93363 [details] [review]
Bug 18757 - Problem when importing only items in MARC records

When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records.
Those imported items are stored as XML to be staged.

The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist.
You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a

This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml

Test plan :
- On a UNIMARC database
- Define a maching rule on title 200$a
- Select a record with items
- Export it using : Save as > MARC (Unicode/UTF-8)
- Delete all items
- Go to Tools > Stage MARC records for import
- Upload exported file
- Select title matching rule
- Select "Ingore incoming record" in "Action if matching record found :"
- Select Yes and "Always add items" in "Check for embedded item record data?"
- Click Stage for import
=> Without patch you get the error
=> With patch the import is staged
- Import into the catalog and check item is well recreated

Signed-off-by: Amandine Zocca <azocca@ville-montauban.fr>
Comment 12 Martin Renvoize 2019-10-29 11:19:04 UTC
Created attachment 94826 [details] [review]
Bug 18757 - Problem when importing only items in MARC records

When importing records with Stage MARC records for import, one can use matching rules to only import items into existing records.
Those imported items are stored as XML to be staged.

The bug is that when MARC Flavour is UNIMARC the XML serialization fails because its is looking in field 100$a which does not exist.
You see in logs the error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a

This patch adds the format "USMARC" to XML serialization, like in C4::Items::_get_unlinked_subfields_xml

Test plan :
- On a UNIMARC database
- Define a maching rule on title 200$a
- Select a record with items
- Export it using : Save as > MARC (Unicode/UTF-8)
- Delete all items
- Go to Tools > Stage MARC records for import
- Upload exported file
- Select title matching rule
- Select "Ingore incoming record" in "Action if matching record found :"
- Select Yes and "Always add items" in "Check for embedded item record data?"
- Click Stage for import
=> Without patch you get the error
=> With patch the import is staged
- Import into the catalog and check item is well recreated

Signed-off-by: Amandine Zocca <azocca@ville-montauban.fr>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 13 Martin Renvoize 2019-10-29 11:19:35 UTC
Works as expected, passes qa scripts and code looks sound.. Passing QA
Comment 14 Martin Renvoize 2019-10-29 12:16:56 UTC
Nice work!

Pushed to master for 19.11.00
Comment 15 Katrin Fischer 2020-03-28 22:06:07 UTC
*** Bug 23684 has been marked as a duplicate of this bug. ***