---- Reported by frederic@tamil.fr 2009-04-03 12:07:04 ---- When marcflavour is UNIMARC, Koha Z39.50 returns biblio records in UNIMARC format. But it also returns that biblio record format is 'usmarc'. There is an incompatibilty. Z39.50 clients who interpret this value won't display correctly biblio records. --- Bug imported by chris@bigballofwax.co.nz 2010-05-21 01:06 UTC --- This bug was previously known as _bug_ 3087 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=3087 Actual time not defined. Setting to 0.0
Should zebra return that biblio record format is 'unimarc'?
> Should zebra return that biblio record format is 'unimarc'? Yes. It's not an issue for some z39.50 client, but for some other it's a blocker. For example, Koha z39.50 client itself uses this information. It means that if you have a Koha UNIMARC instance and that you want to query another Koha UNIMARC instance z39.50 server, you have to define this target as a USMARC/MARC21 one. It will work that way. But for other z39.50 client, the interpretation of this info is automatic and since Koha z39.50 says it send back MARC21 records when it send in fact UNIMARC, there will be display issues. It's very possible to modify the various Zebra config files and replace 'usmarc' by 'unimarc' in order to have a proper format. But you have to do it by hand. A better solution would install different zebra config files depending on marc flavor selected in the Koha web installer.
I think the zebra files are generated in the perl Makefile.PL stage, there you choose your marc flavour for them. I've just sent a patch against koha-conf.xml that adds the settings for the networked SRU server, and found that it is possible to add a conditional for that and solve it in that stage. I asked if 'unimarc' was the ACTUAL text the SRU server should populate to the clients, as it looks pretty simple to fix. I got confused with the fact that the 'biblioserver' is configured as 'usmarc' even if one choses to use UNIMARC during install.
Frederic, you've proposed a patch a long time ago ( http://www.mail-archive.com/koha-patches@lists.koha.org/msg01881.html) . Do you have a rebased version available somewhere that could be attached to this patch ?
(In reply to comment #4) > Frederic, you've proposed a patch a long time ago ( > http://www.mail-archive.com/koha-patches@lists.koha.org/msg01881.html) . Do you > have a rebased version available somewhere that could be attached to this patch > ? Please take a look also at bug 6536. If there is a newer version of the MARC to UNIMARC xslt conversion file from Biblibre, we could get it further.. I asked Jonathan this some time ago already.
(In reply to comment #4) > Frederic, you've proposed a patch a long time ago ( > http://www.mail-archive.com/koha-patches@lists.koha.org/msg01881.html) > . Do you have a rebased version available somewhere that could be > attached to this patch ? No. The patch should be revamped now. In the patch to bug 3216, I've proposed a new system installation parameter. I think that UNIMARC Z39.50 support must be done this way, with installer modifying on the fly koha-conf.xml depending on marcflavour choice.
Created attachment 9122 [details] [review] Proposed patch Modify Makefile.PL and Zebra configuration files in order to parametrized biblio record type returned by Zebra Z39.50 server. How to test: - Test with a MARC21 and a UNIMARC DB - Do a new installation - Search from OPAC - Search from a Z39.50 client like yaz-client: syntax = MARC21/UNIMARC must be choosed - It was working for MARC21: it continues to work - It wasn't working for UNIMARC: it works now, both in OPAC and from a Z39.50 client
Created attachment 10579 [details] [review] Bug 3087 Fix Z39.50 server to return the correct record syntax Modify Makefile.PL and Zebra configuration files in order to parametrized biblio record type returned by Zebra Z39.50 server. How to test: - Test with a MARC21 and a UNIMARC DB - Do a new installation - Search from OPAC - Search from a Z39.50 client like yaz-client: syntax = MARC21/UNIMARC must be choosed - It was working for MARC21: it continues to work - It wasn't working for UNIMARC: it works now, both in OPAC and from a Z39.50 client
Rebased 1/7/2012, still needs testing and if it is accepted will need a patch to update the packages also
I am willing to sign off on this one. I did a new install and it still works for MARC21. Did not test UNIMARC. But I am not fully sure if all details are correct now. Name and reference are now marc21. If I understand the YAZ manual correctly, the value MARC21 is no OID (object identifier) and cannot be used for the reference field (in usmarc.mar). I think it should just be USMARC (what it was). Note that UNIMARC and NORMARC e.g. are valid OIDs. (But maybe I did not find the latest version..) In yaz-client I tested format marc21, format xml and format usmarc. Format marc21 and usmarc, both say to me: Recordtype USmarc. Could that be a yaz version problem? (4.2.4)
From the log: [Mon Aug 20 10:31:51 2012] [error] [client 192.87.126.61] [Mon Aug 20 10:31:51 2012] z3950_search.pl: defined(@array) is deprecated at /usr/share/koha/testclone/cataloguing/z3950_search.pl line 107., referer: http://libdevelop.rijksmuseum.nl:8009/cgi-bin/koha/cataloguing/z3950_search.pl?frameworkcode= [Mon Aug 20 10:31:51 2012] [error] [client 192.87.126.61] [Mon Aug 20 10:31:51 2012] z3950_search.pl: \t(Maybe you should just omit the defined()?), referer: http://libdevelop.rijksmuseum.nl:8009/cgi-bin/koha/cataloguing/z3950_search.pl?frameworkcode=
(En réponse au commentaire 11) > From the log: > [Mon Aug 20 10:31:51 2012] [error] [client 192.87.126.61] [Mon Aug 20 > 10:31:51 2012] z3950_search.pl: defined(@array) is deprecated at > /usr/share/koha/testclone/cataloguing/z3950_search.pl line 107., > referer: > http://libdevelop.rijksmuseum.nl:8009/cgi-bin/koha/cataloguing/z3950_search. > pl?frameworkcode= This line hasn't been introduced by this patch.
> Name and reference are now marc21. If I understand the YAZ manual correctly, > the value MARC21 is no OID (object identifier) and cannot be used for the > reference field (in usmarc.mar). I think it should just be USMARC (what it > was). I think Marc21 and USMarc OID are equivalents. Have you seen that? http://www.loc.gov/z3950/agency/defns/oids.html#5
(In reply to comment #13) > I think Marc21 and USMarc OID are equivalents. Have you seen that? > > http://www.loc.gov/z3950/agency/defns/oids.html#5 OK
Frederic, It took me some time to find this out too. But the retrieval section in koha-conf.xml is now just fallback information. So you should also change the new retrieval files in etc/zebradb for biblios and authorities on dom and grs1. Just found this out by testing the 9998 port with some recordSchema's. Marcel
Thanks Marcel for testing so thoroughly! It is invaluable. > It took me some time to find this out too. But the retrieval section > in koha-conf.xml is now just fallback information. So you should also > change the new retrieval files in etc/zebradb for biblios and > authorities on dom and grs1. I'm not sure to understand. Could you elaborate? Do you mean that for UNIMARC there isn't XSL converter to Dublin Core for example? In this case, this was already the case. This isn't a regression. We just lack that currently in Koha, and we should plan to fix it separately.
(In reply to comment #16) > I'm not sure to understand. Could you elaborate? Do you mean that for > UNIMARC there isn't XSL converter to Dublin Core for example? In this > case, this was already the case. This isn't a regression. We just lack > that currently in Koha, and we should plan to fix it separately. No, I tested only MARC21 actually. But it is more trivial. The retrieval schemes are now in separate xml files in etc/zebradb like retrieval-info-auth-dom.xml etc. (4 files). Your patch only updates etc/koha-conf.xml. The section you update, is only used if the separate retrieval file should not be found. Actually, I would recommend to just delete the retrieval fallback section in koha-conf.xml. Now we have a confusing situation; fallback within the same etc directory does not make much sense to me. If you look at those files, you will certainly understand it.
> No, I tested only MARC21 actually. But it is more trivial. The > retrieval schemes are now in separate xml files in etc/zebradb like > retrieval-info-auth-dom.xml etc. (4 files). Thanks for the explanation. I wasn't aware of this. It seems that what you're describing has been introduced by bug 7818. > Your patch only updates etc/koha-conf.xml. The section you update, is > only used if the separate retrieval file should not be found. I will send an updated patch to fix those 4 files. > Actually, I would recommend to just delete the retrieval fallback > section in koha-conf.xml. Now we have a confusing situation; fallback > within the same etc directory does not make much sense to me. Good idea. I think that this simplification should be asked in 7818 bug directly, so Galen could review it.
Created attachment 11886 [details] [review] Bug 3087 Fix Z39.50 server to return the correct record syntax
I don't see what prevent this patch being applied on 3.10.
Frederic, I was close to signing off on this one, but still have one concern. This patch may probably break things for the NORMARC flavour. Could you make a small adjustment that makes NORMARC behave like it was MARC21? Magnuse will test BTW. A small point additionally: I see the lines <retrieval syntax="xml" name="F"/> <retrieval syntax="xml" name="B"/> in koha-conf.xml but not in retrieval-... files (only syntax=marc21). If we can add them here, these sections are more in line. It will give you recordSchema F and B as well on the sru output.
I had a look at this from a NORMARC perspective, and as far as I can tell (without actually testing it) it will be OK for NORMARC. I'll take the blame if it isn't... :-) Please remember that debian/templates/koha-conf-site.xml needs to be changed too, so installations running off the packages behave the same way as the others.
Marcel: Thanks for your patience! Magnuse: Is it a problem if with this patch, Koha Z39.50 server says that it returns NORMAC records to clients? It used to say record were in MARC21 (which was the problem for UNIMARC biblios).
(In reply to comment #23) > Magnuse: Is it a problem if with this patch, Koha Z39.50 server says that it > returns NORMAC records to clients? It used to say record were in MARC21 > (which was the problem for UNIMARC biblios). Nope, that definitely sounds like an improvement!
> A small point additionally: I see the lines > <retrieval syntax="xml" name="F"/> > <retrieval syntax="xml" name="B"/> > in koha-conf.xml but not in retrieval-... files (only syntax=marc21). > If we can add them here, these sections are more in line. > It will give you recordSchema F and B as well on the sru output. It is like that since patch for bug 7818. I really don't know if it is necessary or not. Don't you thing that this issue, if it is one, should be covered by a new bug or as follow-up for bug 7818?
(In reply to comment #25) > It is like that since patch for bug 7818. I really don't know if it is > necessary or not. Don't you thing that this issue, if it is one, should > be covered by a new bug or as follow-up for bug 7818? This is just an minor issue. As you say, probably for another report. I will signoff.
Created attachment 12903 [details] [review] Signed patch for 3087
QA Comment: Looks good to me. Mainly improving details in Zebra config files. Passing QA with the note that a followup is still needed for packaging (debian/templates). Please contact packaging manager for details. RM may decide if this can be pushed already or should wait for its packaging followup.
Testing this feature before pushing, I tried what all the existing UNIMARC libraries will do: just update the code. If you do that, and don't update your koha-zebra configuration files, you get a very nasty: Error: Can't call method "raw" on an undefined value at /home/paul/koha.dev/koha-community/C4/Search.pm line 461. (on staff as well as on OPAC) I think it's what I should expect, because the preferredRecordSyntax is now: $context->{ Zconn }->{ $server }->option( preferredRecordSyntax => C4::Context->preference("marcflavour") ); so UNIMARC in my case, and, of course, the retrieval syntax is still usmarc :(( * grepping zebra conf: grep -R "syntax=\"usmarc\"" * show that 4 files are concerned: Using sed to update the configuration: sed -i "s/syntax=\"usmarc\"/syntax=\"unimarc\"/" koha-conf.xml sed -i "s/syntax=\"usmarc\"/syntax=\"unimarc\"/" zebradb/retrieval-info-bib-dom.xml sed -i "s/syntax=\"usmarc\"/syntax=\"unimarc\"/" zebradb/retrieval-info-auth-grs1.xml sed -i "s/syntax=\"usmarc\"/syntax=\"unimarc\"/" zebradb/retrieval-info-auth-dom.xml * grep -R "syntax=\"usmarc\"" * tells me that there is more syntax="unimarc" in my configuration files * restarting zebra * restarting plack * searching ... strike ! it works again QUESTION BEFORE PUSHING = how to announce as widely as possible the configuration change in 3.10 ? just in the Release Notes ? a specific entry on koha-community.org ? I plan to announce that widely to the french community, of course but Unimarc is not only france ;-) waiting for your ideas before pushing (and, once pushed, I'll have to send a mail in // to koha-devel & tweet for unimarc developers !
> If you do that, and don't update your koha-zebra configuration files, > you get a very nasty: That's very bad! Is there a way to modify automatically Koha configuration files during the update process? like for the DB with updatedatabase.pl. > QUESTION BEFORE PUSHING = how to announce as widely as possible the > configuration change in 3.10 ? just in the Release Notes ? a specific entry > on koha-community.org ? I plan to announce that widely to the french > community, of course but Unimarc is not only france ;-) Your 'sed' commands seem easy to do for a sysadmin. But an automatic update would be better of course.
(In reply to comment #30) > > If you do that, and don't update your koha-zebra configuration files, > > you get a very nasty: > > That's very bad! Is there a way to modify automatically Koha > configuration files during the update process? like for the DB with > updatedatabase.pl. hélas, updatedatabase can be run from web interface, and, in this case, can't access zebra configuration files. > Your 'sed' commands seem easy to do for a sysadmin. But an automatic > update would be better of course. Agreed, but I don't see which one we could propose.
This is a problem with any Zebra update, and I don't think there's a solution, other than to tell users to run the upgrade process.
Here is a shortcut of your sed commands, if it can help: sed 's/syntax="usmarc"/syntax="unimarc"/' `grep -rl 'syntax="usmarc"' *` We may have the same issue, and even worse, with upgrading and GRS-1/DOM support. Enabling DOM means reinstalling in order to have an updated koha-conf.xml. Or is Makefile.PL able to handle that? I doubt.
So the needed parts would not be added when running Makefile with the link to existing conf file? Is that the problem?
For me: -- a specific entry in koha-community.org -- a FAQ in the manual -- an article on Koha new -- a mail on koha-dev and general Koha list -- Release Note -- a wiki page -- In the upgrade section of INSTALL files I will send the new to every italian user of Koha with Unimarc that I know.
If people make run upgrade it will work. The sed is only used for people using a git install. This should be made clear. Even git installs can run the upgrade process and it will. Running upgrade when you upgrade..... What will they think of next?
Patch pushed to master Switched to "ENH" to be sure it's highlighted in the release notes of 3.10 Sending a mail immediately to mailing list for users running master & git. ( Thanks chris for the hint of make upgrade )
ERRATUM: To update all your Zebra configuration files: cd path/to/your/zebra/config/files sed -i 's/syntax="usmarc"/syntax="unimarc"/' `grep -rl 'syntax="usmarc"' *`
Addition to the previous erratum: the sed is not enough... it does not change usmarc.mar, that is required for Authorities in grs1 The authority search badly fails until you've updated usmarc.mar to have: name unimarc reference unimarc (instead of name usmarc reference usmarc ) So upgrading requires 3 steps: cd path/to/your/zebra/config/files sed -i 's/syntax="usmarc"/syntax="unimarc"/' `grep -rl 'syntax="usmarc"' *` vi etc/usmarc.mar
4th step : restart Zebra ! Configuration files are only read at startup, so you need to restart zebra for the changes to take effect.
To update all your Zebra configuration files with a single command after loading the environment variables ($KOHA_CONF) : cd `echo $KOHA_CONF | sed 's/koha-conf.xml$//'` && sed -i 's/syntax="usmarc"/syntax="unimarc"/' `grep -rl 'syntax="usmarc"' *` && sed -i 's/usmarc/unimarc/' ./zebradb/etc/usmarc.mar && sed -i 's/USmarc/unimarc/' ./zebradb/etc/usmarc.mar don't forget to restart zebra after
See bug 9256 for some unintended consequences of this bug.
(In reply to comment #42) > See bug 9256 for some unintended consequences of this bug. The needed followup for packages was already mentioned on October 18. Looks like it never made it?
(In reply to comment #43) > (In reply to comment #42) > > See bug 9256 for some unintended consequences of this bug. > > The needed followup for packages was already mentioned on October 18. Looks > like it never made it? Yup. Attachment 12903 [details] should never have made it into Koha without that followup for the packages. The need for a followup was first mentioned by Chris C. on July 1st, in comment 9. (BTW, I just upgraded a git-install to current master yesterday and that worked nicely after i ran through the upgrade process, even for NORMARC.)
(En réponse au commentaire 44) > (In reply to comment #43) > > (In reply to comment #42) > > > See bug 9256 for some unintended consequences of this bug. > > > > The needed followup for packages was already mentioned on October 18. Looks > > like it never made it? > > Yup. Attachment 12903 [details] should never have made it into Koha without > that followup for the packages. The need for a followup was first mentioned > by Chris C. on July 1st, in comment 9. > > (BTW, I just upgraded a git-install to current master yesterday and that > worked nicely after i ran through the upgrade process, even for NORMARC.) Koha Z39.50 saying
> Yup. Attachment 12903 [details] should never have made it into Koha > without that followup for the packages. The need for a followup was > first mentioned Koha Z39.50 server saying it returns MARC21 records when it returns UNIMARC or NORMARC is a bug. Of course, it can be decided that it's a marginal bug since it 'only' affects French UNIMARC Koha users exposing their catalog via Z39.50... And so let revert this patch. Otherwise, Debian packages have to be updated. I can help. But I don't use packages, and so would need Robin assistance.
(In reply to comment #46) > > Yup. Attachment 12903 [details] should never have made it into Koha > > without that followup for the packages. The need for a followup was > > first mentioned > > Koha Z39.50 server saying it returns MARC21 records when it returns > UNIMARC or NORMARC is a bug. Of course, it can be decided that it's a > marginal bug since it 'only' affects French UNIMARC Koha users exposing > their catalog via Z39.50... And so let revert this patch. > > Otherwise, Debian packages have to be updated. I can help. But I don't > use packages, and so would need Robin assistance. I am in favour of reverting this until it is fixed, if it is not fixed by 3.10.3 I will revert it out of 3.10.x
(In reply to comment #47) > I am in favour of reverting this until it is fixed, if it is not fixed by > 3.10.3 I will revert it out of 3.10.x I think Jared thought it was too late to revert it, because it would break things for people who already upgraded to 3.10.x. But I'll let him speak for himself, of course.
> I am in favour of reverting this until it is fixed, if it is not fixed > by 3.10.3 I will revert it out of 3.10.x I'm in favour of reverting Debian packages from Koha. It may just be me, but I don't use packages :-) Who exactly is affected by this bug? Neither my UNIMARC nor my MARC21 libraries are. Who will fix that? Since I don't have a clue how packages work, I can't do it alone. Nobody having interest in this, does it mean that I just have to wait, time passes, and see the reversion of this patch?
(In reply to comment #49) > > I am in favour of reverting this until it is fixed, if it is not fixed > > by 3.10.3 I will revert it out of 3.10.x > > I'm in favour of reverting Debian packages from Koha. It may just be me, > but I don't use packages :-) It's just as well you are not release maintainer then isn't it :) > > Who exactly is affected by this bug? Neither my UNIMARC nor my MARC21 > libraries are. > Any library that uses the packages and does not use MARC21 will have a broken search, that is UNIMARC and NORMARC package users. This doesn't affect any of my libraries, but obviously thinking about only our libraries is a bad idea, and how we got into this mess in the first place > Who will fix that? Since I don't have a clue how packages work, I can't > do it alone. Nobody having interest in this, does it mean that I just > have to wait, time passes, and see the reversion of this patch? I will work with Robin and Magnuse to try to undo this major regression with patches to make it so you can have more than one flavour of MARC on a server, and allow search to work with non MARC21 package installs also. I will also try to be more vocal next time so this doesn't happen again.
> Any library that uses the packages and does not use MARC21 will have a > broken search, that is UNIMARC and NORMARC package users. This doesn't > affect any of my libraries, but obviously thinking about only our > libraries is a bad idea, and how we got into this mess in the first > place You're judging on mere intent. In the first place, this mess occurs because the intricacy of standard installation and package installation is complex, and well known by few. Let's stay positive, and be assure my motives are not exempt of altruism... "It is not from the benevolence of the butcher, the brewer, or the baker, that we expect our dinner, but from their regard to their own interest", as said Adam Smith. The question is to find out how to conciliate our own interest with Koha public good. > I will work with Robin and Magnuse to try to undo this major > regression with patches to make it so you can have more than one > flavour of MARC on a server, and allow search to work with non MARC21 > package installs also. I will also try to be more vocal next time so > this doesn't happen again. Thanks.
Adam Smith also said "To feel much for others and little for ourselves; to restrain our selfishness and exercise our benevolent affections, constitute the perfection of human nature. " :-)
(In reply to comment #48) > (In reply to comment #47) > > I am in favour of reverting this until it is fixed, if it is not fixed by > > 3.10.3 I will revert it out of 3.10.x > > I think Jared thought it was too late to revert it, because it would break > things for people who already upgraded to 3.10.x. But I'll let him speak for > himself, of course. Yeah, reverting it would break existing standard installs.
(In reply to comment #53) > (In reply to comment #48) > > (In reply to comment #47) > > > I am in favour of reverting this until it is fixed, if it is not fixed by > > > 3.10.3 I will revert it out of 3.10.x > > > > I think Jared thought it was too late to revert it, because it would break > > things for people who already upgraded to 3.10.x. But I'll let him speak for > > himself, of course. > > Yeah, reverting it would break existing standard installs. From BibLibre POV (that is managing 70+ french libraries in UNIMARC), we don't use the packages, so we didn't saw this problem & don't face it. Reverting the patch will cause trouble to us though. Once i've said that, I can't help with a patch to fix the problem, because I know nothing about packaging. I fear it will be tricky, because it requires the sysop to know which MARC flavour must be choosen, and that's not something well known by common sysop, so we must carefully explain the possible choices...