Bug 9352 - Zebra indexes useless subfields in UNIMARC 7XX
Summary: Zebra indexes useless subfields in UNIMARC 7XX
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Bugs List
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-01-05 12:18 UTC by Mathieu Saby
Modified: 2022-06-23 07:13 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
More specific indexing of UNIMARC 7XX fields in Zebra (11.52 KB, patch)
2013-01-05 12:21 UTC, Mathieu Saby
Details | Diff | Splinter Review
More specific indexing of UNIMARC 7XX fields in Zebra (better test plan) (11.86 KB, patch)
2013-01-05 14:13 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 1/2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) (11.87 KB, patch)
2013-02-11 09:57 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 2/2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) (58.23 KB, patch)
2013-02-11 09:57 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 1/2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) (11.75 KB, patch)
2013-04-03 13:04 UTC, Mathieu Saby
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) (11.88 KB, patch)
2013-08-28 01:10 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) (59.14 KB, patch)
2013-10-03 16:25 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) (33.43 KB, patch)
2014-08-20 16:13 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) (34.16 KB, patch)
2014-08-25 14:41 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) (34.17 KB, patch)
2014-08-25 19:20 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3/3] Add a new XSLT template for indexing goup of subfields as a whole (8.11 KB, patch)
2014-09-01 21:48 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3/3] Bug 9352: Add a new XSLT template for indexing goup of subfields as a whole (25.04 KB, patch)
2014-09-04 17:43 UTC, Mathieu Saby
Details | Diff | Splinter Review
[SIGNED OFF] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) (11.95 KB, patch)
2014-09-05 08:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
[SIGNED OFF] Bug 9352 : More specific indexing of UNIMARC 7XX fields and 511 (DOM) (34.21 KB, patch)
2014-09-05 08:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
[SIGNED OFF] Bug 9352: Add a new XSLT template for indexing goup of subfields as a whole (25.07 KB, patch)
2014-09-05 08:49 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) (11.99 KB, patch)
2014-10-09 15:47 UTC, Paul Poulain
Details | Diff | Splinter Review
Bug 9352 : More specific indexing of UNIMARC 7XX fields and 511 (DOM) (34.25 KB, patch)
2014-10-09 15:49 UTC, Paul Poulain
Details | Diff | Splinter Review
Bug 9352: Add a new XSLT template for indexing goup of subfields as a whole (25.17 KB, patch)
2014-10-09 15:49 UTC, Paul Poulain
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Saby 2013-01-05 12:18:05 UTC
Zebra indexes all subfields of UNIMARC 7XX fields in author index, including $9, $3, $4 (function code), $f (dates of authors), $c (additions other than dates), $p (address).
It causes Koha to give too much results.
For example, if an author is born in 1984 and is a teacher, searching "1984" or "teacher" in simple search will return all work by this author if these pieces of informations are in 7XX field.
This is not how most ILS work, and it should be corrected.
Comment 1 Mathieu Saby 2013-01-05 12:21:40 UTC Comment hidden (obsolete)
Comment 2 Mathieu Saby 2013-01-05 12:22:35 UTC
Solution :
This patch makes indexing of unimarc 7XX fields more specific. For each field, useless subfields are not indexed.
70X : Do not index $f (dates),$c (additions other than dates),$p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number.
710-712 : Do not index $p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. (I kept all other subfields : even if some may be useless I am not sure of it)
716 : Do not index $f (dates),$c (additions other than dates),$3,$4. Index $9 only in Koha-Auth-Number.
72X : Do not index $f (dates),$3,$4. Index $9 only in Koha-Auth-Number.
730 : Do not index $4. Index $9 only in Koha-Auth-Number.
Additionnaly, this patch indexes 205$f/$g in Author index (author of the edition of the work)
Testing :
a/ Create a record with
700$a Doe $b John $f1950 $cteacher $4070
716$a Trademark $f1960 $cgreat $4340
720$a Family $f1980 $4651
205$a 1st edition $fBy some guy $gAnd other guys

a/ Before applying patch :
Search in simple search and author search : "teacher", "great", "1950", "070", "340", "651" => you will see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family"        => you will see the record among the results
Search in simple search and author search : "guy", "guys"                                   => you will not see the record among the results


b/ Before applying patch :
Search in simple search and author search : "teacher", "great", "1950", "070", "340", "651" => you will not see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family"        => you will see the record among the results
Search in simple search and author search : "guy", "guys"                                   => you will see the record among the results
Comment 3 Mathieu Saby 2013-01-05 12:30:01 UTC
I forgot to say that 
- after applying the patch you need to rebuild Zebra (so you cannot test it on a sandbox)
- for the moment, it works only for GRS1 indexing, not DOM indexing

M. Saby
Rennes 2 university
Comment 4 Mathieu Saby 2013-01-05 12:32:43 UTC
Grrr, I made typos on testing plan. 
So it is : 

a/ Create a record
b/ Before applying patch, try some searches
c/ Apply
d/ Rebuild zebra
e/ Try the same searches as before


M. Saby
Comment 5 Mathieu Saby 2013-01-05 14:13:29 UTC Comment hidden (obsolete)
Comment 6 Mathieu Saby 2013-01-05 14:14:13 UTC
Testing :
a/ Create a record with :
700$a Doe $b John $f1950 $cteacher $4070
710$a Corporate $b Meeting $q Address
716$a Trademark $f1960 $cgreat $4340
720$a Family $f1980 $4651
205$a 1st edition $fBy some guy $gAnd other guys

b/ Before applying patch, make several searches :
Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in simple search and author search : "guy", "guys"
=> you will see the record among the results with simple search (because 205 is indexed in title)
=> you will not see the record among the results with author search

c/ Apply patch
d/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset

e/ Try the same searches as before :
Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will not see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in simple search and author search : "guy", "guys"
=> you will see the record among the results in both simple search and author search
Comment 7 Frédéric Demians 2013-02-10 07:06:36 UTC
Could you add DOM support?
Comment 8 Mathieu Saby 2013-02-11 08:39:00 UTC
OK Frederic.
I am not sure the patch will have visible effects if you have authorities indexed. Maybe I should try to do the same kind of patch for authorities?

Mathieu
Comment 9 Frédéric Demians 2013-02-11 08:47:45 UTC
> I am not sure the patch will have visible effects if you have
> authorities indexed. Maybe I should try to do the same kind of patch
> for authorities?

In this occurrence, biblio and authorities are unrelated. My point is
that there are now two indexing modes, GRS-1 and DOM. Two categories of
files handle the indexing: records.abs for GRS-1, and
biblio-koha-indexdefs.xml, biblio-zebra-indexdefs.xsl for DOM. So if you
modify records.abs, you have similarly to modify
biblio-koha-indexdefs.xml.
Comment 10 Mathieu Saby 2013-02-11 09:57:22 UTC Comment hidden (obsolete)
Comment 11 Mathieu Saby 2013-02-11 09:57:55 UTC Comment hidden (obsolete)
Comment 12 Mathieu Saby 2013-02-11 09:58:40 UTC
2 patches, one for GRS1 and one for DOM indexing. 
Each need to be tested separately.

M. Saby
Comment 13 Mathieu Saby 2013-04-03 13:04:46 UTC Comment hidden (obsolete)
Comment 14 Bernardo Gonzalez Kriegel 2013-08-28 01:10:55 UTC Comment hidden (obsolete)
Comment 15 Bernardo Gonzalez Kriegel 2013-08-28 01:11:57 UTC
Half signed, works well with GRS1
Comment 16 Mathieu Saby 2013-08-28 07:18:21 UTC
Thank you for testing.
And do you agree that the corrected behavior is better for users?

M. Saby
Comment 17 I'm just a bot 2013-09-29 05:40:10 UTC
Patch applied cleanly, go forth and signoff
Comment 18 Mathieu Saby 2013-10-02 15:59:29 UTC
I don't know if it need to be tested on DOM, as DOM is not working well for UNIMARC.
Maybe we should consider it as signed off?

Mathieu
Comment 19 Mathieu Saby 2013-10-02 16:01:16 UTC
I would like to see this patch and the other ones about unimarc pushed before making my patch for configuring facets. Otherwise there would be too many dependancies.

Mathieu
Comment 20 Bernardo Gonzalez Kriegel 2013-10-02 16:15:13 UTC
(In reply to mathieu saby from comment #18)
> I don't know if it need to be tested on DOM, as DOM is not working well for
> UNIMARC.
> Maybe we should consider it as signed off?
> 
> Mathieu

Only way I think is obsoleting the other patch.

Thinking on your other comment, what's is needed?
I could try to help.
Comment 21 Mathieu Saby 2013-10-02 18:29:32 UTC
Hi
Not sure what comment you refer to ;-)

What is needed is applying attachment 15229 [details] [review] on a DOM Unimarc Koha, and testing it (copying modified configuration files into {installdir}/etc/zebra.... and rebuild_zebra.pl)

As it is pure copy of the patch for GRS1, I'd say there is no need to test ALL the items of the test plan. If it was working for GRS1, it *must* work with DOM.
And if there is something wrong, the QA team will tell us.

Maybe just check you can still search for authors after the pach is applied, and that the search is limited to the fistname + family name of the author (not his birthdate, nor his function code in $4 subfield)

You're very kind for proposing your help.

Mathieu
Comment 22 Mathieu Saby 2013-10-03 16:25:52 UTC Comment hidden (obsolete)
Comment 23 I'm just a bot 2013-10-28 06:23:34 UTC
Patch applied cleanly, go forth and signoff
Comment 24 I'm just a bot 2013-11-22 22:48:56 UTC
Patch applied cleanly, go forth and signoff
Comment 25 Nick Clemens (kidclamp) 2014-08-19 02:43:28 UTC
Since I am all setup with a unimarc do you want to rebase this and just do DOM?  Happy to test if so
Comment 26 Mathieu Saby 2014-08-19 14:34:03 UTC
Hello
I was about to suggest it to you ;-)
I think I will have time to do that tomorrow.

Mathieu
Comment 27 Mathieu Saby 2014-08-20 16:13:59 UTC Comment hidden (obsolete)
Comment 28 Mathieu Saby 2014-08-20 16:14:40 UTC
2d patch, for DOM indexing
Rebased on 20 Aug 2014

Same changes as in GRS-1 patch, with 2 small additions :
- I take the occasion for indexing the 511 field (I don't know why it was omitted)
- index 710$a and 720$a in Author:s as well as 700$a (seems logical)

(biblio-zebra-indexdefs.xsl is made from biblio-koha-indexdefs.xml with xsltproc)

Same test as first patch :
Testing :
a/ Create a record with some values. for example:
700$a Doe $b John $f1950 $cteacher $4070
710$a Corporate $b Meeting $q Address
716$a Trademark $f1960 $cgreat $4340
720$a Family $f1980 $4651
205$a 1st edition $fBy some guy $gAnd other guys
511$a falsetitle

b/ Before applying patch, make several searches :
Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in simple search and author search : "guy", "guys"
=> you will see the record among the results with simple search (because 205 is indexed in title)
=> you will not see the record among the results with author search
Search ti:falsetitle
=> no results

c/ Apply the patch
d/ copy the 2 modified files into your zebra configuration directory
e/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset

f/ Try the same searches as before :
Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will not see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in simple search and author search : "guy", "guys"
=> you will see the record among the results in both simple search and author search
Search ti:falsetitle
=> 1 result
Comment 29 Nick Clemens (kidclamp) 2014-08-23 14:51:48 UTC
Applied on patch 2 for testing.

Test plan says to add Address to subfield $q - I can't find that field, did you mean $p? - I assumed yes

Before patch all worked as expected

After applying results were as below, looks like 716 is still indexed:
(ss=simple search au:author search)
teacher: ss-y au-n
great: ss-y au-y
Address: ss-y au-n
1950: ss-y au-n
1960: ss-y au-y
070: ss-y au-n
340: ss-y au-y
651: ss-y au-n

Doe: ss-y au-y
John Doe: ss-y au-n (though yes as:Doe, John)
Trademark: ss-y au-y
Family: ss-y au-y
corporate: ss-y au-y
meeting: ss-y au-y
guy: ss-y au-y
guys: ss-y au-n

ti:falsetitle - 1 result
Comment 30 Mathieu Saby 2014-08-23 16:45:19 UTC
(In reply to Nick Clemens from comment #29)
> Applied on patch 2 for testing.
> 
> Test plan says to add Address to subfield $q - I can't find that field, did
> you mean $p? - I assumed yes
> 
Yes, it is $p. sorry for that
Comment 31 Mathieu Saby 2014-08-23 16:49:36 UTC
I'm not sure to understand your test. You mean it is ok for 7XX in general but not for 716?

M. Saby
Comment 32 Nick Clemens (kidclamp) 2014-08-23 18:08:36 UTC
Sorry, I could have been more clear.  
Short answer to your question is yes.

Longer explanation:
716$a Trademark $f1960 $cgreat $4340 <--All of these terms still show in both simple and author search after applying the patch (and copying files and rebuilding)

It looks like all fields of 716 are still being indexed.

Everything else seems correct as defined in the test plan


In my list of search/results:
y = Test record appears in search results
n = Test record does not appear in search results

(In reply to mathieu saby from comment #31)
> I'm not sure to understand your test. You mean it is ok for 7XX in general
> but not for 716?
> 
> M. Saby
Comment 33 Mathieu Saby 2014-08-23 20:42:48 UTC
I understand:
I wrote 
<index_data_field xmlns="http://www.koha-community.org/schemas/index-defs" tag="716" subfields="a">
this should be
<index_subfields xmlns="http://www.koha-community.org/schemas/index-defs" tag="716" subfields="a">

I will fix that tomorrow

M. Saby
Comment 34 Mathieu Saby 2014-08-25 14:41:35 UTC Comment hidden (obsolete)
Comment 35 Mathieu Saby 2014-08-25 14:42:35 UTC
I rewrote the 2d patch for fixing the issue with 716 field. You can test again, only 716 field.

M. Saby
Comment 36 Nick Clemens (kidclamp) 2014-08-25 16:00:20 UTC
716 is working as expected now:  trademark is searchable as simple/author / 1960,great,340 are searchable as simple search but not author search

I apologize though, reviewing my test results I missed one other mistake:
205$g is not being indexed as author

author search for 'guys' or 'and other guys' does not return the test record.

Sorry for my mistake, I had it written in my test results but did not remember it when I replied

(In reply to mathieu saby from comment #35)
> I rewrote the 2d patch for fixing the issue with 716 field. You can test
> again, only 716 field.
> 
> M. Saby
Comment 37 Mathieu Saby 2014-08-25 19:20:22 UTC Comment hidden (obsolete)
Comment 38 Mathieu Saby 2014-08-25 19:22:46 UTC
I rewrote it quickly witout regenerating the patch, but I suppose it will work, as the count of lines will be the same. If it does not apply, I will take more time to rewrite it (not the good machine this evening)
Comment 39 Nick Clemens (kidclamp) 2014-08-25 22:31:55 UTC
Patch applied cleanly,guys and 'other guys' now both indexed.

I was feeling peppy though, so I retested all searches and came across one small issue searching for 'John Doe'

Simple search works as both 'doe, john' or 'john doe'

Author search from dropdown (in advanced search or OPAC) works as both 'doe, john' or 'john doe'

CCL search DOES NOT work foir either 'au:doe, john'  or 'au:john doe' or for any other records/authors

Any cause you could see for that?
Comment 40 Mathieu Saby 2014-08-25 22:49:32 UTC
(In reply to Nick Clemens from comment #39)
> Patch applied cleanly,guys and 'other guys' now both indexed.
> 
> I was feeling peppy though, so I retested all searches and came across one
> small issue searching for 'John Doe'
> 
> Simple search works as both 'doe, john' or 'john doe'
> 
> Author search from dropdown (in advanced search or OPAC) works as both 'doe,
> john' or 'john doe'
> 
> CCL search DOES NOT work foir either 'au:doe, john'  or 'au:john doe' or for
> any other records/authors
> 
> Any cause you could see for that?

Weird. 
And did it work for "au:john doe" before applying the patch?
With the patch, does it work for "au:doe" or "au:john"?
and for other indexes (ti, etc.) ?
did you try with and without Queryparser activated?
Comment 41 Mathieu Saby 2014-08-25 23:03:00 UTC
I tested in a running Koha:
au is making an exact search on the phrase. I did not remember that... Maybe it depends on syspref

For ex:
I just changed the order of the first name and surname:
http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=au%3AProust+Marcel&branch_group_limit=
does not gives the same results as
http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=au%3AProust+Marcel&branch_group_limit=

And : 
http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=+Mijolla%2C+Alain+de&branch_group_limit=
-> 1 result (from 700 field)
http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=+Alain+de+Mijolla&branch_group_limit=
-> 1 result (from 200$f field)
http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=+Alain+Mijolla&branch_group_limit=
-> 0 result

So, to test 700$aDoe$bJohn
au:Doe John should give you 1 result

Is that the case?
Mathieu
Comment 42 Nick Clemens (kidclamp) 2014-08-26 00:29:08 UTC
No, with patches applied, files copied, zebra rebuilt:
"au:doe john" gives no results

I switched to master, recopied files, rebuilt zebra:
"au:doe john" gives 1 result

"au:john doe" gives no results in either

Quotes are not part of search, just to indicate exactly what I typed into the search bar.

With queryparser on the order of the terms doesn't matter, 1 result is returned either way, both in master and with patch applied.

Sorry, didn't mean to find a new problem, I am just in the habit of always using the CCL terms
Comment 43 Mathieu Saby 2014-08-26 07:52:00 UTC
In fact, the issue is obvious ;-)
Before, all the content of the field was indexed in "au:phrase" index. like that:
"Doe John (1945 ; writer)"
Now, a and b subfields are indexed separately in "au:phrase" (and also au:word):
"Doe"
"John"
But no "Doe John"

To do that, I need probably need to change the behavior of the xsl file generating zebra-biblio-indexdef from koha-biblio-indexdef. Or maybe there is already a template for that in the file.

I'm going to ask the lists.

Setting the patch in Fail QA for the moment
Comment 44 Mathieu Saby 2014-09-01 21:48:07 UTC Comment hidden (obsolete)
Comment 45 Mathieu Saby 2014-09-01 21:48:43 UTC
With the 3rd patch, everything should work

Mathieu
Comment 46 Mathieu Saby 2014-09-02 23:06:19 UTC
The last patch is broken somewhere, please don't test
Comment 47 Mathieu Saby 2014-09-04 17:43:11 UTC Comment hidden (obsolete)
Comment 48 Mathieu Saby 2014-09-04 17:43:32 UTC
NEW 3RD PATCH:


This patch adds a new template in koha-indexdefs-to-zebra.xsl in order to index a specific goup of subfields as a whole.
It can be used for indexing as a "phrase" a string resulting from the concatenation of some indexes (with a space between them)
To use this new template, use the new "index_group_of_subfields" element in biblio-koha-indexdefs.xml, in the same way you used index_subfields

Ex : to index the concatenation of "$a $b $d $g" subfields of 700 UNIMARC field in Author:p
  <index_group_of_subfields xmlns="http://www.koha-community.org/schemas/index-defs" tag="700" subfields="abdg">
    <target_index>Author:p</target_index>
  </index_group_of_subfields>

This patch also change the biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl for UNIMARC, in order to use this new template for 700-712 fields
For 700 and 710, it also indexes in "Author:s" (used for sorting) the concatenation of subfields instead of only the $a subfield

To test : in a DOM UNIMARC Koha, apply all the patches, copy the modified files in your Zebra configuration directory, rebuild zebra, and make some tests

Everything should be the same as when only the first 2 patches were applied, EXCEPT this behavior:
- create a record with 700$aDoe$bJohn (and nothing in 200$f or $g)
- Go in Advanced Search, and select the "Author as phrase" index
- with only the first 2 patches, you will have 0 result if you enter "Doe John" or "Doe John"
- with the 3 patches, you will have 1 result if you enter "Doe John" or "Doe John"
- same kind of test for 710 field
- the logic is the same for 701, 702,711, 712. So, no need to test
- make some searches and test the sorting of your results by author name ("Author A-Z" and "Author Z-A"). The complete name of main author (700 and 710) sould be used for sorting the results, instead of only $&

Note that this it better to test it in advanced search, because the behavior of "au:" CCL index in simple search is maybe bugged
(it searches the phrase index, while it should probably searches the word index)
Comment 49 Marcel de Rooy 2014-09-04 20:16:03 UTC
Last patch has no signoff?
Comment 50 Mathieu Saby 2014-09-04 21:05:15 UTC
(In reply to M. de Rooy from comment #49)
> Last patch has no signoff?

Nick Clemens tested patch 2. For him it was working, except the issue raised in comment 39.
So he did not sign it off, and I wrote a follow-up, which is the 3rd patch. 

I'm not completely sure of that, but I think the ability to index a group of subfields in the same index could be usefull for other MARC fields, and not only for UNIMARC.

Mathieu
Comment 51 Mathieu Saby 2014-09-04 21:06:04 UTC
Oh, sorry, patch 2 and 3 need sign off, I put a wrong status
Comment 52 Martin Renvoize 2014-09-05 08:49:24 UTC Comment hidden (obsolete)
Comment 53 Martin Renvoize 2014-09-05 08:49:35 UTC Comment hidden (obsolete)
Comment 54 Martin Renvoize 2014-09-05 08:49:43 UTC Comment hidden (obsolete)
Comment 55 Martin Renvoize 2014-09-05 08:53:31 UTC
Happy to sign off...

I'm particularly interested in patch 3, which introduces a new feature to the koha-indexdefs-to-zebra.xsl allowing subfield grouping for indexing purposes.

It looks all good from my testing.
Comment 56 Marcel de Rooy 2014-09-08 12:59:04 UTC
I would like to QA this one soon. Put it on my list now.
Comment 57 Paul Poulain 2014-10-09 15:47:54 UTC
Created attachment 32097 [details] [review]
Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1)

1st patch : for GRS1 indexing
Problem :
Zebra indexes all subfields of UNIMARC 7XX fields in author index, including $9, $3, $4 (function code), $f (dates of authors), $c (additions other than dates), $p (address).
It causes Koha to give too much results.
For example, if an author is born in 1984 and is a teacher, searching "1984" or "teacher" in simple search will return all work by this author if these pieces of informations are in 7XX field. This is not how most ILS work, and it should be corrected.

Solution :
This patch makes indexing of unimarc 7XX fields more specific. For each field, useless subfields are not indexed.
70X : Do not index $f (dates),$c (additions other than dates),$p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number.
710-712 : Do not index $p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. (I kept all other subfields : even if some may be useless I am not sure of it)
716 : Do not index $f (dates),$c (additions other than dates),$3,$4. Index $9 only in Koha-Auth-Number.
72X : Do not index $f (dates),$3,$4. Index $9 only in Koha-Auth-Number.
730 : Do not index $4. Index $9 only in Koha-Auth-Number.
Additionnaly, this patch indexes 205$f/$g in Author index (author of the edition of the work)
Testing :
a/ Create a record with :
700$a Doe $b John $f1950 $cteacher $4070
710$a Corporate $b Meeting $p Address
716$a Trademark $f1960 $cgreat $4340
720$a Family $f1980 $4651
205$a 1st edition $fBy some guy $gAnd other guys

b/ Before applying patch, make several searches :
Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will see the record among the results
Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in author index : "guy", "guys"
=> you will see the record among the results with simple search (because 205 is indexed in title)
=> you will not see the record among the results with author search

c/ Apply patch
d/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset

e/ Try the same searches as before :
Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will not see the record among the results
Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in author index : "guy", "guys"
=> you will see the record among the results in both simple search and author search

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment: Work as described with GRS1.
No koha-qa errors.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Comment 58 Paul Poulain 2014-10-09 15:49:14 UTC
Created attachment 32098 [details] [review]
Bug 9352 : More specific indexing of UNIMARC 7XX fields and 511 (DOM)

2d patch, for DOM indexing
Rebased on 20 Aug 2014, Fixed on 25 Aug 2014 (716 field and 205$g)

Same changes as in GRS-1 patch, with 2 small additions :
- I take the occasion for indexing the 511 field (I don't know why it was omitted)
- index 710$a and 720$a in Author:s as well as 700$a (seems logical)

(biblio-zebra-indexdefs.xsl is made from biblio-koha-indexdefs.xml with xsltproc)

Same test as first patch :
Testing :
a/ Create a record with some values. for example:
700$a Doe $b John $f1950 $cteacher $4070
710$a Corporate $b Meeting $q Address
716$a Trademark $f1960 $cgreat $4340
720$a Family $f1980 $4651
205$a 1st edition $fBy some guy $gAnd other guys
511$a falsetitle

b/ Before applying patch, make several searches :
Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in simple search and author search : "guy", "guys"
=> you will see the record among the results with simple search (because 205 is indexed in title)
=> you will not see the record among the results with author search
Search ti:falsetitle
=> no results

c/ Apply the patch
d/ copy the 2 modified files into your zebra configuration directory
e/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset

f/ Try the same searches as before :
Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651"
=> you will not see the record among the results
Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting"
=> you will see the record among the results
Search in simple search and author search : "guy", "guys"
=> you will see the record among the results in both simple search and author search
Search ti:falsetitle
=> 1 result

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Comment 59 Paul Poulain 2014-10-09 15:49:44 UTC
Created attachment 32099 [details] [review]
Bug 9352: Add a new XSLT template for indexing goup of subfields as a whole

This patch adds a new template in koha-indexdefs-to-zebra.xsl in order to index a specific goup of subfields as a whole.
It can be used for indexing as a "phrase" a string resulting from the concatenation of some indexes (with a space between them)
To use this new template, use the new "index_group_of_subfields" element in biblio-koha-indexdefs.xml, in the same way you used index_subfields

Ex : to index the concatenation of "$a $b $d $g" subfields of 700 UNIMARC field in Author:p
  <index_group_of_subfields xmlns="http://www.koha-community.org/schemas/index-defs" tag="700" subfields="abdg">
    <target_index>Author:p</target_index>
  </index_group_of_subfields>

This patch also change the biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl for UNIMARC, in order to use this new template for 700-712 fields
For 700 and 710, it also indexes in "Author:s" (used for sorting) the concatenation of subfields instead of only the $a subfield

To test : in a DOM UNIMARC Koha, apply all the patches, copy the modified files in your Zebra configuration directory, rebuild zebra, and make some tests

Everything should be the same as when only the first 2 patches were applied, EXCEPT this behavior:
- create a record with 700$aDoe$bJohn (and nothing in 200$f or $g)
- Go in Advanced Search, and select the "Author as phrase" index
- with only the first 2 patches, you will have 0 result if you enter "Doe John" or "Doe John"
- with the 3 patches, you will have 1 result if you enter "Doe John" or "Doe John"
- same kind of test for 710 field
- the logic is the same for 701, 702,711, 712. So, no need to test
- make some searches and test the sorting of your results by author name ("Author A-Z" and "Author Z-A"). The complete name of main author (700 and 710) sould be used for sorting the results, instead of only $&

Note that this it better to test it in advanced search, because the behavior of "au:" CCL index in simple search is maybe bugged
(it searches the phrase index, while it should probably searches the word index)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Comment 60 Tomás Cohen Arazi 2014-10-15 14:14:43 UTC
I like to improve UNIMARC support, so will be available to hangout and/or comment on whatever you need. But

- GRS-1 is now deprecated (no problem with that, just obsolete that patch)
- The rest of the patches don't apply
- It seems you are re-implementing on koha-indexdefs-to-zebra.xsl something already existed, i might be wrong, but look at 'handle-index-subfields'
Comment 61 Mathieu Saby 2014-10-15 21:55:41 UTC
(In reply to Tomás Cohen Arazi from comment #60)
> I like to improve UNIMARC support, so will be available to hangout and/or
> comment on whatever you need. But
> 
> - GRS-1 is now deprecated (no problem with that, just obsolete that patch)
> - The rest of the patches don't apply
> - It seems you are re-implementing on koha-indexdefs-to-zebra.xsl something
> already existed, i might be wrong, but look at 'handle-index-subfields'

I will rebase it later.
But for your 3rd point : the idea is to create a new behavior : indexing as a single phrase, in a single index, a string resulting from the concatenation of some subfieds.

I don't think you can do that with 'handle-index-subfields'.

Mathieu
Comment 62 Mathieu Saby 2014-10-25 12:23:25 UTC
Hi Thomas
My changes for 7XX are a bit complex, and not very urgent.
So could you please push the patch for 6XX, and any other patch related to Zebra, and I will take an other look next month, and rebase what need to be rebased?

Mathieu