Bug 17791 - On UNIMARC system, USMARC field 100$a gets overwritten when importing from MARC21 target
Summary: On UNIMARC system, USMARC field 100$a gets overwritten when importing from MA...
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Z39.50 / SRU / OpenSearch Servers (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Oleg Vasylenko
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-12-19 21:42 UTC by Oleg Vasylenko
Modified: 2017-10-16 08:08 UTC (History)
5 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Patch changes the context of Z39.50 search from local to remote server marc flavour (1.41 KB, patch)
2016-12-19 21:52 UTC, Oleg Vasylenko
Details | Diff | Splinter Review
Bug 17791 - On UNIMARC system, USMARC field 100$a gets overwritten Patch changes the context of Z39.50 search from local to remote server (1.44 KB, patch)
2017-10-15 12:58 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Oleg Vasylenko 2016-12-19 21:42:47 UTC
Overview: 
On Z39.50 search some functions depend not on the remote server marc flavour, but on current server setting. This bug is present on versions 3.22+

Steps to Reproduce: 
1) On UNIMARC system, go to Cataloguing → New from Z39.50/SRU
2) Search USMARC server (Library of Congress, for example) for a book, that has field 100 (for example, isbn 0816310653)
3) On results list, press «MARC» to see MARC fields

Actual Results: 
Instead of seeing the author name in subfield A, you'll see something like «20161219              frey50        », yet you could see other subfields properly

Expected Results:
Name of the author in 100$a

Additional Information: 
LC Z39.50/SRW/SRU Server Configuration Guidelines: https://www.loc.gov/z3950/lcserver.html
Comment 1 Oleg Vasylenko 2016-12-19 21:52:28 UTC
Created attachment 58276 [details] [review]
Patch changes the context of Z39.50 search from local to remote  server marc flavour
Comment 2 Oleg Vasylenko 2016-12-19 21:57:17 UTC
Patch prepared for current master
Comment 3 Zeno Tajoli 2016-12-20 08:32:29 UTC
Hi Oleg,

I'm not agree with this patch.
Why?
Now «20161219              frey50        » is correct default for Unimarc 100 field
If you have a Unimarc server and you want to use LOC [a MARC21 z39.50] you need to use the field "XSLT file(s)" in z39.50 definiton.
See: http://manual.koha-community.org/3.22/en/additionaladmin.html#addztarget
In bug 6536 there is an example of XSLT for MARC21 -> Unimarc conversion.
See: https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37264

Bye
Comment 4 Oleg Vasylenko 2016-12-20 08:58:14 UTC
Hi Zeno,

This change is normal if you access UNIMARC record on Unimarc server. But the problem is that this is changed before XSLT conversion. 

I do xslt conversion, but it converts this already changed string. You can check by yourself https://github.com/Gelow/unimarc/blob/master/marc21xml2unimarc.xsl
MARC view in Z39.50 search results shows the record as it is.

Koha code works the way, that it first converts codepage, and after that changes the subfield where it is stated, that it is UTF-8. But the problem is that it thinks that the record is in unimarc format, so it changes 100$a.

The patch uses server flavour, that is stated in Z39.50 servers table. That way everythig works fine. Function, that do codepage conversion understands that this record is USMARC (as stated in Z39.50 server table), and it changes leader, not 100$a. Because information about codepage is located in different places in these flavours.

You can see that this problem is present in different koha versions. But the patch is prepared for current master
Comment 5 Zeno Tajoli 2016-12-20 09:13:58 UTC
Ok Oleg, you are correct.
Comment 6 Oleg Vasylenko 2016-12-20 09:20:40 UTC
When you use XSLT conversion, you'll see the same problem

USMARC 100 is converted to UNIMARC 700. But the conversion happens after codepage is changed. So you'll see
700$a 20161219              frey50        
700$b 1933-

but not the author name in 700$a
Comment 7 Marcel de Rooy 2016-12-20 13:14:41 UTC
Oleg,
This seems to me an tricky change.
What you are trying to do, is currently not fully supported in Koha.
A conversion between MARC21 and UNIMARC did not make it until now.
If you do a XSLT conversion after this statement, it will probably be fine, but in many other cases it will not be followed by such a conversion and new problems will probably arise.
To make this foolproof, we need more changes (and checks)?
Comment 8 Oleg Vasylenko 2016-12-20 16:40:52 UTC
Marcel,
I am working on the MARC21 to UNIMARC conversion right now, and now it works good enough. I did many changes to original XSLT file published by Terry Reese on github (bug 16488). See https://github.com/Gelow/unimarc/blob/master/marc21xml2unimarc.xsl

The only problem is that field 100$a is corrupted on Z39.50 search. I added LOC server as SRU and field 100$a is not corrupted that way. This is because in sub _handle_one_result() right now SRU server is processed with it's flavour in mind. The same should be with Z39.50

What problems will we face after this patch?

The same as we have with SRU servers right now.
When UNIMARC system connects to MARC21 server, it will check encoding stated in 100$a (thanks to checks in XML.pm called from function AddBiblioToBatch()).If it's not set, you'll see XML encoding error («Unsupported UNIMARC character encoding [$enc] for XML output»).

In other cases everything should be fine

If we do not apply this patch
You can see the results table, and see sourse MARC with corrupted 100$a. 
But the table has empty columns because of different format, so who does need it?

To overcome stated problems it is possible to add some check in the code for UNIMARC and process the record some way (but record could become corrupted again). Or add XSLT file for conversion
Comment 9 Chris Cormack 2017-05-11 01:33:51 UTC
Changing to a new feature as this is adding support for on the fly conversion of MARC records from Z39.50
Comment 10 Katrin Fischer 2017-10-15 12:58:00 UTC
Changing to enh as this is just the starting point for adding the ability of the conversion (and should then change bug title also). After this made it in we could maybe add Oleg's XSLT file to the sample xsl files available and mark that 'new feature'?

Signing off, was not able to detect a regression.
Comment 11 Katrin Fischer 2017-10-15 12:58:56 UTC
Created attachment 68165 [details] [review]
Bug 17791 - On UNIMARC system, USMARC field 100$a gets overwritten Patch changes the context of Z39.50 search from local to remote server

Steps to Reproduce:
1) On UNIMARC system, go to Cataloguing → New from Z39.50/SRU
2) Search USMARC server (Library of Congress, for example) for a book, that has field 100 (for example, isbn 0816310653)
3) On results list, press «MARC» to see MARC fields

Expected Results:
Name of the author in 100$a

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 12 Marcel de Rooy 2017-10-16 08:06:58 UTC
This does not work. As said earlier, we need more checks. (The records cannot be imported without conversion. So we need more checks. For instance, only show import when the flavours match.)

Currently btw this results in an internal server error.
I started on a clean master, chose UNIMARC and added LOC as a search target (MARC21/USMARC).
Without this patch, I can search, but of course I do not see any titles.

With this patch, it crashes.
Unrecognized marcflavour: USMARC at /usr/share/koha/masterclone/C4/Charset.pm line 358.
Use of uninitialized value in substr at /usr/share/perl5/MARC/File/XML.pm line 561.
substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 561.
Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 563.
Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 563.
Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565.
Use of uninitialized value $enc in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 568.
Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 568.
Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a ->  at /usr/share/perl5/MARC/File/XML.pm line 568.

Script passes USMARC as marc flavour to MarcToUTF8Record.
This routine passes it to SetMarcUnicodeFlag which expects MARC21 or UNIMARC as Koha does in other places too.