Bug 9972 - Add/ change some zebra indexes (MARC21) (GRS1+DOM)
Summary: Add/ change some zebra indexes (MARC21) (GRS1+DOM)
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Mirko Tietgen
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-04-02 13:03 UTC by Mirko Tietgen
Modified: 2015-06-04 23:30 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
9972 Add/ change some zebra indexes (MARC21) (GRS1) (3.63 KB, patch)
2013-04-02 13:05 UTC, Mirko Tietgen
Details | Diff | Splinter Review
9972 Follow-up: DOM version (4.61 KB, patch)
2013-04-02 14:39 UTC, Mirko Tietgen
Details | Diff | Splinter Review
9972 Add/ change some zebra indexes (MARC21) (GRS1) (8.05 KB, patch)
2013-04-02 14:43 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 9972 Follow-up (3.47 KB, patch)
2013-07-18 14:19 UTC, Mirko Tietgen
Details | Diff | Splinter Review
9972 Add/ change some zebra indexes (MARC21) (GRS1) (8.14 KB, patch)
2013-09-19 11:52 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 9972 Add/ change some zebra indexes (MARC21) (10.29 KB, patch)
2013-10-17 15:11 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 9972 Add/ change some zebra indexes (MARC21) (10.32 KB, patch)
2014-03-03 12:53 UTC, Mirko Tietgen
Details | Diff | Splinter Review
Bug 9972 Add/ change some zebra indexes (MARC21) (10.38 KB, patch)
2014-03-03 20:55 UTC, Chris Cormack
Details | Diff | Splinter Review
[PASSED QA] Bug 9972 Add/ change some zebra indexes (MARC21) (10.45 KB, patch)
2014-03-25 08:56 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mirko Tietgen 2013-04-02 13:03:32 UTC
This patch adds :w and :p versions to the index for »Lexile number« (it has only :n so far) and adds indexes for 653$a (Index term uncontrolled) and 655$a (Index term Genre/Form). It also adds the »curriculum«-index to Search.pm.
Comment 1 Mirko Tietgen 2013-04-02 13:05:00 UTC Comment hidden (obsolete)
Comment 2 Mirko Tietgen 2013-04-02 13:16:46 UTC
(this patch is for GRS1 indexing, not DOM)

To test:

- apply patch

A) prepare a record with data in
-- 521$a (VALUE1) (please use a non-numerical value here)
-- 653$a (VALUE2)
-- 655$a (VALUE3)

- reindex

B) Try to find the record in your OPAC:

URL_to_your_OPAC/opac-search.pl?idx=lex&q=VALUE1
URL_to_your_OPAC/opac-search.pl?idx=Index-term-uncontrolled&q=VALUE2
URL_to_your_OPAC/opac-search.pl?idx=Index-term-genre&q=VALUE3
Comment 3 Mirko Tietgen 2013-04-02 14:39:25 UTC Comment hidden (obsolete)
Comment 4 Mirko Tietgen 2013-04-02 14:43:15 UTC Comment hidden (obsolete)
Comment 5 Kyle M Hall 2013-04-08 11:36:08 UTC
I tried to test this patch and was only able to get URL_to_your_OPAC/opac-search.pl?idx=lex&q=VALUE1 to work. I symlinked all the files to my koha-dev dir like this:
lrwxrwxrwx 1 koha koha 47 Apr  5 11:03 ../koha-dev/etc/zebradb/ccl.properties -> /home/koha/kohaclone/etc/zebradb/ccl.properties

lrwxrwxrwx 1 koha koha 53 Apr  5 12:25 ../koha-dev/etc/zebradb/biblios/etc/bib1.att -> /home/koha/kohaclone/etc/zebradb/biblios/etc/bib1.att

lrwxrwxrwx 1 koha koha 83 Apr  5 11:05 ../koha-dev/etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml -> /home/koha/kohaclone/etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml

lrwxrwxrwx 1 koha koha 84 Apr  5 12:22 ../koha-dev/etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl -> /home/koha/kohaclone/etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl

lrwxrwxrwx 1 koha koha 68 Apr  5 12:23 ../koha-dev/etc/zebradb/marc_defs/marc21/biblios/record.abs -> /home/koha/kohaclone/etc/zebradb/marc_defs/marc21/biblios/record.abs

Did I miss anything?


(In reply to comment #4)
> Created attachment 17137 [details] [review] [review]
> 9972 Add/ change some zebra indexes (MARC21) (GRS1)
> 
> This patch adds :w and :p versions to the index for »Lexile number« (it has
> only :n so far) and adds indexes for 653$a (Index term uncontrolled) and
> 655$a (Index term Genre/Form). It also adds the »curriculum«-index to
> Search.pm.
> 
> 9972 Follow-up: DOM version
> 
> A DOM version of the same indexes
> 
> Squashed both patches together.
Comment 6 Mirko Tietgen 2013-04-15 11:14:19 UTC
Kyle, thanks for trying. I built a package with master+patch to test it and that worked fine for me.

I tried symlinking and ran into problems too. One is that there is a reference to a marc.mar file that seems to be missing in my dev installation. I still have errors after symlinking that. I have to take another look at this later.
Comment 7 Mirko Tietgen 2013-04-16 16:28:28 UTC
Revised test plan based on Jared's comment at http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8750#c5. I used a new db and file path.

1) Apply patch.
2) Create test DB (cf. INSTALL.debian, change <user> and <pass>)
mysqladmin -uroot -p<pass> create bug9972
mysql -uroot -p<pass>
grant all on bug9972.* to '<user>'@'localhost' identified by '<pass>';
flush privileges;
3) Run Makefile.PL, choose some local path for test installation, choose GRS-1 for biblio indexing.
4) Run make, make install.
5) symlink /etc/instancename/koha-httpd.conf to /etc/apache2/sites-available/instancename, a2ensite instancename. disable other koha instance if neccessary, 
5) Run webinstaller.
6) Catalogue a record with data in
-- 521$a (eg fivetwoone)
-- 653$a (eg sixfivethree)
-- 655$a (eg sixfivefive)
-- 658$a (eg sixfiveeight)
7) Reindex. Also check if zebrasrv for that instance is running, mine wasn't.
8) Try searches, change q= at the end to match your test record data.
http://YourOPAC/opac-search.pl?idx=lex&q=fivetwoone
http://YourOPAC/opac-search.pl?idx=Index-term-genre&q=sixfivefive
http://YourOPAC/opac-search.pl?idx=Index-term-uncontrolled&q=sixfivethree
http://YourOPAC/opac-search.pl?idx=curriculum&q=sixfivethree
You should reach the same record with each search.
9) Run Makefile.PL, choose the same local path as last time, choose DOM for biblio indexing.
10) Run make, make install.
11) Reindex.
12) Try searches described in 8 again. You should find your test record for all queries.
Comment 8 Mirko Tietgen 2013-04-17 09:05:29 UTC
There is an error in the last query, it should be
http://YourOPAC/opac-search.pl?idx=curriculum&q=sixfiveeight
Comment 9 Marcel de Rooy 2013-07-18 10:03:12 UTC
Hi Mirko,
You seem to mix up 653 [uncontrolled] and 655 [genre] here:

@@ -587,6 +587,13 @@ definition file (probably something like {biblio,authority}-koha-indexdefs.xml)
   <xslo:template mode="index_subfields" match="marc:datafield[@tag='653']">
     <xslo:for-each select="marc:subfield">
+      <xslo:if test="contains('a', @code)">
+        <z:index name="Index-term-genre:w Index-term-genre:p Subject:w Subject:p">
+          <xslo:value-of select="."/>
+        </z:index>
+      </xslo:if>
+    </xslo:for-each>
+    <xslo:for-each select="marc:subfield">

Also a question: You modify both files for DOM. But is that how it should be done? Or is the change you submit a result of generating the other file again with the script?
Comment 10 Mirko Tietgen 2013-07-18 13:47:46 UTC
Marcel, thanks a lot for spotting the mix-up! You are right of course! I will provide a revised patch.

About the DOM files, it was my understanding that they should both be changed to have the indexes as part of the standard setup and in case someone rebuilds the DOM indexes later.

I just asked Jared to be on the safe side and the correct way is to change one file, generate the other and have both changes in the patch. http://irc.koha-community.org/koha/2013-07-18#i_1349220
Comment 11 Mirko Tietgen 2013-07-18 14:19:30 UTC Comment hidden (obsolete)
Comment 12 Mirko Tietgen 2013-07-18 14:22:25 UTC
I added a follow-up for easier testing. I can squash the patches if that is preferred.
Comment 13 I'm just a bot 2013-09-18 11:27:14 UTC
Applying: Bug 9972 Follow-up
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 9972 Follow-up
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 14 Mirko Tietgen 2013-09-19 11:52:08 UTC Comment hidden (obsolete)
Comment 15 I'm just a bot 2013-09-24 05:46:55 UTC
Patch applied cleanly, go forth and signoff
Comment 16 Mirko Tietgen 2013-09-24 10:46:08 UTC
Reposting the test plan as it got lost in the comments:

1) Apply patch.
2) Create test DB (cf. INSTALL.debian, change <user> and <pass>)
mysqladmin -uroot -p<pass> create bug9972
mysql -uroot -p<pass>
grant all on bug9972.* to '<user>'@'localhost' identified by '<pass>';
flush privileges;
3) Run Makefile.PL, choose some local path for test installation, choose GRS-1 for biblio indexing.
4) Run make, make install.
5) symlink /etc/instancename/koha-httpd.conf to /etc/apache2/sites-available/instancename, a2ensite instancename. disable other koha instance if neccessary, 
5) Run webinstaller.
6) Catalogue a record with data in
-- 521$a (eg fivetwoone)
-- 653$a (eg sixfivethree)
-- 655$a (eg sixfivefive)
-- 658$a (eg sixfiveeight)
7) Reindex. Also check if zebrasrv for that instance is running, mine wasn't.
8) Try searches, change q= at the end to match your test record data.
http://YourOPAC/opac-search.pl?idx=lex&q=fivetwoone
http://YourOPAC/opac-search.pl?idx=Index-term-genre&q=sixfivefive
http://YourOPAC/opac-search.pl?idx=Index-term-uncontrolled&q=sixfivethree
http://YourOPAC/opac-search.pl?idx=curriculum&q=sixfiveeight
You should reach the same record with each search.
9) Run Makefile.PL, choose the same local path as last time, choose DOM for biblio indexing.
10) Run make, make install.
11) Reindex.
12) Try searches described in 8 again. You should find your test record for all queries.
Comment 17 Mirko Tietgen 2013-10-17 15:11:59 UTC Comment hidden (obsolete)
Comment 18 Mirko Tietgen 2013-10-17 15:14:08 UTC
To test:

1) Apply patch.
2) Create test DB (cf. INSTALL.debian, change <user> and <pass>)
mysqladmin -uroot -p<pass> create bug9972
mysql -uroot -p<pass>
grant all on bug9972.* to '<user>'@'localhost' identified by '<pass>';
flush privileges;
3) Run Makefile.PL, choose some local path for test installation, choose GRS-1 for biblio indexing.
4) Run make, make install.
5) symlink /etc/instancename/koha-httpd.conf to /etc/apache2/sites-available/instancename, a2ensite instancename. disable other koha instance if neccessary, 
5) Run webinstaller.
6) Catalogue a record with data in
-- 041$a (eg audio)
-- 041$j (eg subtitle)
-- 521$a (eg fivetwoone)
-- 653$a (eg sixfivethree)
-- 655$a (eg sixfivefive)
-- 658$a (eg sixfiveeight)
7) Reindex. Also check if zebrasrv for that instance is running, mine wasn't.
8) Try searches, change q= at the end to match your test record data.
http://YourOPAC/opac-search.pl?idx=ln-audio&q=audio
http://YourOPAC/opac-search.pl?idx=ln-subtitle&q=subtitle
http://YourOPAC/opac-search.pl?idx=lex&q=fivetwoone
http://YourOPAC/opac-search.pl?idx=Index-term-genre&q=sixfivefive
http://YourOPAC/opac-search.pl?idx=Index-term-uncontrolled&q=sixfivethree
http://YourOPAC/opac-search.pl?idx=curriculum&q=sixfiveeight
You should reach the correct record with each search.
9) Run Makefile.PL, choose the same local path as last time, choose DOM for biblio indexing.
10) Run make, make install.
11) Reindex.
12) Try searches described in 8 again. You should find your test record for all queries.
Comment 19 I'm just a bot 2013-10-30 07:16:22 UTC
Patch applied cleanly, go forth and signoff
Comment 20 Mirko Tietgen 2014-03-03 12:53:57 UTC Comment hidden (obsolete)
Comment 21 Chris Cormack 2014-03-03 20:55:36 UTC Comment hidden (obsolete)
Comment 22 Martin Renvoize 2014-03-12 13:58:18 UTC
Hi Mirko, 

I'm pretty happy that everything here works, and the code looks good. However, I would like to have some clear reasoning behind why your adding these specific indexes?  Mostly because, to me it's not clear how they are usful (I'm sure they are.. but some context is always good).

Could you comment on this, and perhaps ammend your commit message to include some examples of where the indexes are usefull?

Cheers
Comment 23 Mirko Tietgen 2014-03-20 12:04:59 UTC
Hi Martin, thanks for taking a look!

The change for 521 is necessary in our case for levels of language learners that come in not-only numeric values (like A1, A2, B1…). We use 658 for language certificates (like TOEFL).

653 in our case is for a classification by language that we need to seperate form the other subject fields, but it could be used in very different ways in other libraries. We use 655 for another internal classification.

I did a talk about it in Edinburgh, maybe that is of interest to you (around page 16 http://amor.cms.hu-berlin.de/~tietgemi/koha/tietgen-kohacon2012-koha.in.language.centres.pdf).

Does that answer your question?
Comment 24 Martin Renvoize 2014-03-25 08:51:34 UTC
Ah yes, I remember the talk now, I was there.

Happy to pass QA on this now then.
Comment 25 Martin Renvoize 2014-03-25 08:56:43 UTC
Created attachment 26544 [details] [review]
[PASSED QA] Bug 9972 Add/ change some zebra indexes (MARC21)

This patch adds :w and :p versions to the index for »Lexile number« (it has only :n so far) and adds indexes for 653 (Index term uncontrolled), 655 (Index term Genre/Form), 041 (language-audio) and 041 (language-subtitle). It also adds the »curriculum«-index to Search.pm.

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 26 Galen Charlton 2014-04-20 16:55:02 UTC
Pushed to master, along with follow-ups to enable QueryParser to use the new indexes.

Thanks, Mirko!
Comment 27 Fridolin Somers 2014-08-13 08:32:12 UTC
Pushed to 3.14.x, will be in 3.14.10