Bug 9256 - Search is broken for 3.10 package-installs when marcflavour != marc21
Summary: Search is broken for 3.10 package-installs when marcflavour != marc21
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Packaging (show other bugs)
Version: master
Hardware: All All
: P5 - low critical (vote)
Target Milestone: Oldversions
Assignee: Magnus Enger
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 3087 9996
  Show dependency treegraph
 
Reported: 2012-12-10 17:32 UTC by Magnus Enger
Modified: 2018-12-16 19:34 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 9256 - Fix search for the packages (33.80 KB, patch)
2013-03-06 13:19 UTC, Magnus Enger
Details | Diff | Splinter Review
Bug 9256 - Fix search for the packages (34.00 KB, patch)
2013-03-12 16:45 UTC, Mirko Tietgen
Details | Diff | Splinter Review
[PASSED QA] Bug 9256 - Fix search for the packages (34.22 KB, patch)
2013-03-19 15:27 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Enger 2012-12-10 17:32:33 UTC
Bug 3087 updated koha-conf.xml so the MARC flavour is configurable:

-         <backend syntax="usmarc" name="F">
+         <backend syntax="__ZEBRA_MARC_FORMAT__" name="F">

But debian/templates/koha-conf-site.xml.in was not updated, so it is still hardcoded to usmarc:

        <retrieval syntax="usmarc" name="F"/>
        <retrieval syntax="usmarc" name="B"/>
Comment 1 Magnus Enger 2012-12-10 17:35:26 UTC
There might be similar issues with the other config files affected by bug 3087:

 C4/Context.pm                            |    2 +
 Makefile.PL                              |    1 +
 etc/koha-conf.xml                        |   46 +++++++++++++++---------------
 etc/zebradb/etc/usmarc.mar               |    4 +-
 etc/zebradb/retrieval-info-auth-dom.xml  |    4 +-
 etc/zebradb/retrieval-info-auth-grs1.xml |   14 ++++----
 etc/zebradb/retrieval-info-bib-dom.xml   |    4 +-
 etc/zebradb/retrieval-info-bib-grs1.xml  |   14 ++++----
 8 files changed, 46 insertions(+), 43 deletions(-)
Comment 2 Magnus Enger 2012-12-10 17:58:36 UTC
I'm increasing the severity: as far as I can tell the packages are broken for marcflavour != marc21 now.
Comment 3 Chris Cormack 2012-12-10 21:38:21 UTC
Magnus, by changing that file manually on your 3.10.0 install did it fix the issue?
Comment 4 Robin Sheat 2012-12-10 21:48:55 UTC
I'm a little confused, but it might be my lack of understanding of how this works.

My assumption is:

__ZEBRA_MARC_FORMAT__ comes out as marc21, normarc, whathaveyou. However usmarc is the binary MARC format, not the schema that lies on top of that. So, wouldn't changing "usmarc" to "normarc", or even "marc21" break things?

However, clearly this is wrong, I just don't know what the right version is.
Comment 5 Magnus Enger 2012-12-10 22:50:05 UTC
I'm afraid I don't completely understand this either, and I have not had the time to investigate thoroughly enough, but it seems pretty clear to me that the changes brought on by bug 3087 needs to be reflected in the koha-conf.xml and zebra config files created by the packages. 

What I did find out was that if I changed marcflavour from normarc to marc21 and then reindexed, searching worked. If I changed it back to normarc again and reindexed, it did not work.
Comment 6 Magnus Enger 2012-12-11 13:36:13 UTC
I got it to work now, by reproducing the changes introduced by bug 3087 on the relevant config files. According to my notes, I did something like this:

sed -i "s/syntax=\"usmarc\"/syntax=\"normarc\"/" /etc/koha/sites/esme/koha-conf.xml
sed -i "s/marc21/normarc/" /etc/koha/zebradb/etc/usmarc.mar
sed -i "s/syntax=\"marc21\"/syntax=\"normarc\"/" /etc/koha/zebradb/retrieval-info-bib-dom.xml

But my notes might be missing things...

So it looks to me like the packages will need to do the same kinds of substitutions that bug 3087 introduced. 

There is one big question, though: Since I had to edit non-instance specific files like /etc/koha/zebradb/retrieval-info-bib-dom.xml - does that mean we will only be able to support one MARC dialect on one server?
Comment 7 Magnus Enger 2012-12-28 12:39:34 UTC
Trying to find a more descriptive (and fetching) title.
Comment 8 Chris Cormack 2012-12-28 12:50:41 UTC
> 
> There is one big question, though: Since I had to edit non-instance specific
> files like /etc/koha/zebradb/retrieval-info-bib-dom.xml - does that mean we
> will only be able to support one MARC dialect on one server?

If that is true bug 3087 should be reverted. We have broken Koha to get an z3950 server going for external use. We should not break core like that.
Comment 9 Magnus Enger 2013-03-01 14:40:12 UTC
So I finally got around to spending some more time on this, and looks like there are three problems here, from the point of view of a package install:

* 1. usmarc.mar

After bug 3087 this file gets the marcflavour of the installation. I think the solution is to have one file for each marcflavour, e.g. marc21.mar, normarc.mar and unimarc.mar. 

usmarc.mar is referenced by the different record.abs files for the different marcflavours, so updating them to point to the right .mar file for their flavour should be trivial. 

Side note: I'm not sure what role the .mar file actually plays in Zebra/Koha. I have tried changing the contents around a bit without any apparent influence on searching. The only documentation I have been able to find is this:

3.9. The MARC (ISO2709) Representation (.mar) Files
This file provides rules for representing a record in the ISO2709 format. The rules pertain mostly to the values of the constant-length header of the record. 
http://www.indexdata.com/zebra/doc/grs-conf.html#grs-mar-files

Perhaps Robin is right too, in that it could have been left as it was before Bug 3087? 

* 2. koha-conf.xml

Occurences of hardcoded "usmarc" in debian/templates/koha-conf-site.xml.in need to be replaced with the __ZEBRA_MARC_FORMAT__ placeholder, so it can get the right values when a new site is created:

From <retrieval syntax="usmarc" name="F"/>
To <retrieval syntax="__ZEBRA_MARC_FORMAT__" name="F"/>

* 3. retrieval-info-*-*.xml

To be exact:
retrieval-info-auth-dom.xml
retrieval-info-auth-grs1.xml
retrieval-info-bib-dom.xml
retrieval-info-bib-grs1.xml

Similar to koha-conf.xml, the __ZEBRA_MARC_FORMAT__ placeholders in these files now get replaced with the actual marcflavour in a regular installation. In a dev install you will have one copy of each of these files for each of your installations, so verything works as expected. Not so for package installs, where these files are not genereated from templates and so continue to have the hardcoded "usmarc". Another major problem is that these files are common to all the instances on a server (in /etc/koha/zebradb). So it looks like we either have to

- make the packages generate these files from templates whenever a new instance is added, and find a way to make them instance-sepecific (not shared), or
- create one file for each marcflavour and make whatever references these files reference the files for the right marcflavour

As far as I can tell, the files that reference these files are the koha-conf.xml files. It looks like it should not be too hard to make the retrieval-info-*.xml files instance specific and generate them from a template. That way we just have two sets of template files to keep updated, and not 3 versions of each of the files, for different marcflavours.
Comment 10 Robin Sheat 2013-03-03 22:46:11 UTC
What you're saying doesn't seem bad or too difficult. One point though: where possible, I'd like to centralise the files as much as reasonable. That is, rather than create an instance-specific version, it'd be preferable to have a file for all the combinations. We could generated these as part of the build process, if need be.

My reasoning for this is simply that this means they'll get upgraded as koha gets upgraded. There will be situations where this really isn't feasible, but if it's just a handful more files, sensibly named, it should be OK. If we start having dozens of them (especially if they have to be maintained manually) then it is probably time to template a per-instance one.
Comment 11 Magnus Enger 2013-03-06 13:19:48 UTC Comment hidden (obsolete)
Comment 12 Magnus Enger 2013-03-06 13:25:31 UTC
Oh, testing that this does not bring back the problems described by Bug 3087 is probably a good idea too...
Comment 13 Mirko Tietgen 2013-03-08 14:17:56 UTC
Installed Koha-common 3.10.3-1
created instances test1 and test2 with Marc21 and Normarc respectively
installed Norwegian language for test2 so I can choose Normarc in the web installer
ran web installer
created a staff user
changed marcflavor to Normarc for test2
added some biblios, reindexed
searched test1 (marc21), all looks ok
searched test2 (normarc), got message »Feil: Can't call method "raw" on an undefined value at /usr/share/koha/lib/C4/Search.pm line 471.«

Upgraded to my own package of master+9256
created instances test3 and test4 with Marc21 and Normarc respectively
installed Norwegian language for test4 so I can choose Normarc in the web installer (needs to be done again or the layouts are broken!)
ran web installer
created a staff user
changed marcflavor to Normarc for test4

Unicode is broken in the patron type pulldown when you try to create a new user. I have not looked around more, instead I tried searching.

added some biblios, reindexed
searched test3 (marc21), all looks ok
searched test4 (normarc), all looks ok.
(if you forget to change marcflavor syspref to Normarc, you will get the same error as described for test2 instance)

So the patch seems to do what it is intended to do. I do not recall having seen broken Unicode when creating a patron in the unpatched version (maybe I didn't pay attention) and I want to have another look at it later but so far it's looking good.

I have a repository with the package if somebody else would like to try it but does not feel like going through the pain of building it themselves. I won't be able to test for Unimarc problems for instance.
Comment 14 Magnus Enger 2013-03-12 09:09:42 UTC
(In reply to comment #13)
> Unicode is broken in the patron type pulldown when you try to create a 
> new user. I have not looked around more, instead I tried searching.
...
> So the patch seems to do what it is intended to do. I do not recall having
> seen broken Unicode when creating a patron in the unpatched version (maybe I
> didn't pay attention) and I want to have another look at it later but so far
> it's looking good.

There is a signed off patch for the Unicode problem on Bug 9772.
Comment 15 Mirko Tietgen 2013-03-12 16:45:59 UTC Comment hidden (obsolete)
Comment 16 Katrin Fischer 2013-03-19 15:27:13 UTC
Created attachment 16410 [details] [review]
[PASSED QA] Bug 9256 - Fix search for the packages

See the bug for a description of the problem.

This patch tries to restore searching for marcflavour != MARC21 as well as
allowing instances with different marcflavors to co-exist on the same server.

To test:
- Do a package install with e.g. the official squeeze-dev packages and create at
  least two instances, with different marcflavours, e.g.:
  sudo koha-create --create-db --marcflavor marc21 test1
  sudo koha-create --create-db --marcflavor normarc test2
- Run through the web installers for both instances and add a couple of
  records to each. Wait for the records to be indexed or run indexing manually
  with
  sudo koha-rebuild-zebra -f test1
  sudo koha-rebuild-zebra -f test2
- Try searching for the records you added. It should work in test1 but not in
  test2.
- Apply the patch and build packages with the build-git-snapshot script
- Install the new koha-common package
- Create two instances (because of Bug 9754 it is probably best to give the
  instances different names than the ones you created above, or to do this on
  a fresh VM or similar) and add records, as described above. Searching should
  now work equally well for both instances.

Please note: Because of Bug 9752 you will have to set marcflavour = NORMARC
by hand before you do the searching, if you choose NORMARC as the marc flavour
on one of the instances you create.

Please note too: I am not confident that this is the perfect solution, so
merciless and thorough testing is necessary! ;-)

Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Works for me for GRS-1 (package installation out of the box). Could not figure out how to set up DOM indexing and eventually stopped caring about it.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Build packages with the patch and checked that creating
instances and search within them works for both MARC21 and NORMARC.
All tests and QA script pass.
Comment 17 Jared Camins-Esakov 2013-03-19 23:34:54 UTC
This patch has been pushed to master.
Comment 18 Chris Cormack 2013-03-21 00:26:39 UTC
Pushed to 3.10.x will be in 3.10.4
Comment 19 Magnus Enger 2013-04-03 19:03:15 UTC
Search works now! (Tested with packages built off current master.)