Overview: On Z39.50 search some functions depend not on the remote server marc flavour, but on current server setting. This bug is present on versions 3.22+ Steps to Reproduce: 1) On UNIMARC system, go to Cataloguing → New from Z39.50/SRU 2) Search USMARC server (Library of Congress, for example) for a book, that has field 100 (for example, isbn 0816310653) 3) On results list, press «MARC» to see MARC fields Actual Results: Instead of seeing the author name in subfield A, you'll see something like «20161219 frey50 », yet you could see other subfields properly Expected Results: Name of the author in 100$a Additional Information: LC Z39.50/SRW/SRU Server Configuration Guidelines: https://www.loc.gov/z3950/lcserver.html
Created attachment 58276 [details] [review] Patch changes the context of Z39.50 search from local to remote server marc flavour
Patch prepared for current master
Hi Oleg, I'm not agree with this patch. Why? Now «20161219 frey50 » is correct default for Unimarc 100 field If you have a Unimarc server and you want to use LOC [a MARC21 z39.50] you need to use the field "XSLT file(s)" in z39.50 definiton. See: http://manual.koha-community.org/3.22/en/additionaladmin.html#addztarget In bug 6536 there is an example of XSLT for MARC21 -> Unimarc conversion. See: https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=37264 Bye
Hi Zeno, This change is normal if you access UNIMARC record on Unimarc server. But the problem is that this is changed before XSLT conversion. I do xslt conversion, but it converts this already changed string. You can check by yourself https://github.com/Gelow/unimarc/blob/master/marc21xml2unimarc.xsl MARC view in Z39.50 search results shows the record as it is. Koha code works the way, that it first converts codepage, and after that changes the subfield where it is stated, that it is UTF-8. But the problem is that it thinks that the record is in unimarc format, so it changes 100$a. The patch uses server flavour, that is stated in Z39.50 servers table. That way everythig works fine. Function, that do codepage conversion understands that this record is USMARC (as stated in Z39.50 server table), and it changes leader, not 100$a. Because information about codepage is located in different places in these flavours. You can see that this problem is present in different koha versions. But the patch is prepared for current master
Ok Oleg, you are correct.
When you use XSLT conversion, you'll see the same problem USMARC 100 is converted to UNIMARC 700. But the conversion happens after codepage is changed. So you'll see 700$a 20161219 frey50 700$b 1933- but not the author name in 700$a
Oleg, This seems to me an tricky change. What you are trying to do, is currently not fully supported in Koha. A conversion between MARC21 and UNIMARC did not make it until now. If you do a XSLT conversion after this statement, it will probably be fine, but in many other cases it will not be followed by such a conversion and new problems will probably arise. To make this foolproof, we need more changes (and checks)?
Marcel, I am working on the MARC21 to UNIMARC conversion right now, and now it works good enough. I did many changes to original XSLT file published by Terry Reese on github (bug 16488). See https://github.com/Gelow/unimarc/blob/master/marc21xml2unimarc.xsl The only problem is that field 100$a is corrupted on Z39.50 search. I added LOC server as SRU and field 100$a is not corrupted that way. This is because in sub _handle_one_result() right now SRU server is processed with it's flavour in mind. The same should be with Z39.50 What problems will we face after this patch? The same as we have with SRU servers right now. When UNIMARC system connects to MARC21 server, it will check encoding stated in 100$a (thanks to checks in XML.pm called from function AddBiblioToBatch()).If it's not set, you'll see XML encoding error («Unsupported UNIMARC character encoding [$enc] for XML output»). In other cases everything should be fine If we do not apply this patch You can see the results table, and see sourse MARC with corrupted 100$a. But the table has empty columns because of different format, so who does need it? To overcome stated problems it is possible to add some check in the code for UNIMARC and process the record some way (but record could become corrupted again). Or add XSLT file for conversion
Changing to a new feature as this is adding support for on the fly conversion of MARC records from Z39.50
Changing to enh as this is just the starting point for adding the ability of the conversion (and should then change bug title also). After this made it in we could maybe add Oleg's XSLT file to the sample xsl files available and mark that 'new feature'? Signing off, was not able to detect a regression.
Created attachment 68165 [details] [review] Bug 17791 - On UNIMARC system, USMARC field 100$a gets overwritten Patch changes the context of Z39.50 search from local to remote server Steps to Reproduce: 1) On UNIMARC system, go to Cataloguing → New from Z39.50/SRU 2) Search USMARC server (Library of Congress, for example) for a book, that has field 100 (for example, isbn 0816310653) 3) On results list, press «MARC» to see MARC fields Expected Results: Name of the author in 100$a Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
This does not work. As said earlier, we need more checks. (The records cannot be imported without conversion. So we need more checks. For instance, only show import when the flavours match.) Currently btw this results in an internal server error. I started on a clean master, chose UNIMARC and added LOC as a search target (MARC21/USMARC). Without this patch, I can search, but of course I do not see any titles. With this patch, it crashes. Unrecognized marcflavour: USMARC at /usr/share/koha/masterclone/C4/Charset.pm line 358. Use of uninitialized value in substr at /usr/share/perl5/MARC/File/XML.pm line 561. substr outside of string at /usr/share/perl5/MARC/File/XML.pm line 561. Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 563. Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 563. Use of uninitialized value $enc in string eq at /usr/share/perl5/MARC/File/XML.pm line 565. Use of uninitialized value $enc in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 568. Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/MARC/File/XML.pm line 568. Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> at /usr/share/perl5/MARC/File/XML.pm line 568. Script passes USMARC as marc flavour to MarcToUTF8Record. This routine passes it to SetMarcUnicodeFlag which expects MARC21 or UNIMARC as Koha does in other places too.
Created attachment 176866 [details] [review] Bug 17791: Fix Z39.50/XSLT import encoding issue Test plan: 1. On UNIMARC system, create a Z39.50 server configuration: Name: LOC_NO_XSLT Hostname: lx2.loc.gov Port: 210 Database: LCDB Preselected: yes Syntax: MARC21/USMARC Encoding: utf8 Record Type: bibliographic 2. Duplicate this Z39.50 server configuration, change its name to "LOC_WITH_XSLT" and set "XSLT File(s) for transforming results" to the absolute path of a MARC21 to UNIMARC XSL file. You can download one from here: https://raw.githubusercontent.com/Gelow/unimarc/refs/heads/master/marc21xml2unimarc.xsl 3. Go to Cataloguing → New from Z39.50/SRU 4. Search both servers for a book entitled "comprendre un pays". That should give you only one result for each server 5. On results list, press «MARC» to see MARC fields. The record from LOC_NO_XSLT should have its 100$a replaced by something like "20250121 frey50" and the rest of the record should be a valid MARC21 record The record from LOC_WITH_XSLT should also have a 100$a looking like "20250121 frey50" but it might be different depending on the XSLT used (the important part is the "50" at positions 26-27) It should also have a 700 field that look like this: $a Simon $b Jean-François but again, it depends on the XSLT used. The 700$a should NOT contain "20250121 frey50"
I tried in k-t-d, the results seem to vary, I don't understand why (instability of the connection to the Z3950 server?), and the version “with xslt” generates an error (I used the xslt sheet given in the test plan). Without XSLT, one time I get a record with only a leader a 100$a field LDR 00079 a2200037 4500 100 _a20250209 frey50 A few minutes later I get a MARC21 record : 100 1 _a20250209 frey50 _d1950- _eauthor. 245 10 _aComprendre un pays : _bethnologie de la Basse-Bretagne / _cJean-François Simon. With XSLT, one time I have that error Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> at /usr/share/perl5/MARC/File/XML.pm line 570 in MARC::File::XML::_unimarc_encoding at /usr/share/perl5/MARC/File/XML.pm line 570 567: } elsif ($enc eq '50') { 568: return 'UTF-8'; 569: } else { 570: die "Unsupported UNIMARC character encoding [$enc] for XML output for $f; 100\$a -> " . $r->subfield(100 => 'a'); 571: } 572: } 573: Show function arguments An other time that error : Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> Simon, Jean-Fran�ois, at /usr/share/perl5/MARC/File/XML.pm line 570 in MARC::File::XML::_unimarc_encoding at /usr/share/perl5/MARC/File/XML.pm line 570 567: } elsif ($enc eq '50') { 568: return 'UTF-8'; 569: } else { 570: die "Unsupported UNIMARC character encoding [$enc] for XML output for $f; 100\$a -> " . $r->subfield(100 => 'a'); 571: } 572: } 573: Show function arguments