Zebra indexes all subfields of UNIMARC 7XX fields in author index, including $9, $3, $4 (function code), $f (dates of authors), $c (additions other than dates), $p (address). It causes Koha to give too much results. For example, if an author is born in 1984 and is a teacher, searching "1984" or "teacher" in simple search will return all work by this author if these pieces of informations are in 7XX field. This is not how most ILS work, and it should be corrected.
Created attachment 14438 [details] [review] More specific indexing of UNIMARC 7XX fields in Zebra
Solution : This patch makes indexing of unimarc 7XX fields more specific. For each field, useless subfields are not indexed. 70X : Do not index $f (dates),$c (additions other than dates),$p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. 710-712 : Do not index $p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. (I kept all other subfields : even if some may be useless I am not sure of it) 716 : Do not index $f (dates),$c (additions other than dates),$3,$4. Index $9 only in Koha-Auth-Number. 72X : Do not index $f (dates),$3,$4. Index $9 only in Koha-Auth-Number. 730 : Do not index $4. Index $9 only in Koha-Auth-Number. Additionnaly, this patch indexes 205$f/$g in Author index (author of the edition of the work) Testing : a/ Create a record with 700$a Doe $b John $f1950 $cteacher $4070 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys a/ Before applying patch : Search in simple search and author search : "teacher", "great", "1950", "070", "340", "651" => you will see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will not see the record among the results b/ Before applying patch : Search in simple search and author search : "teacher", "great", "1950", "070", "340", "651" => you will not see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results
I forgot to say that - after applying the patch you need to rebuild Zebra (so you cannot test it on a sandbox) - for the moment, it works only for GRS1 indexing, not DOM indexing M. Saby Rennes 2 university
Grrr, I made typos on testing plan. So it is : a/ Create a record b/ Before applying patch, try some searches c/ Apply d/ Rebuild zebra e/ Try the same searches as before M. Saby
Created attachment 14439 [details] [review] More specific indexing of UNIMARC 7XX fields in Zebra (better test plan) I corrected the test plan. M. Saby
Testing : a/ Create a record with : 700$a Doe $b John $f1950 $cteacher $4070 710$a Corporate $b Meeting $q Address 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys b/ Before applying patch, make several searches : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results with simple search (because 205 is indexed in title) => you will not see the record among the results with author search c/ Apply patch d/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset e/ Try the same searches as before : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will not see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results in both simple search and author search
Could you add DOM support?
OK Frederic. I am not sure the patch will have visible effects if you have authorities indexed. Maybe I should try to do the same kind of patch for authorities? Mathieu
> I am not sure the patch will have visible effects if you have > authorities indexed. Maybe I should try to do the same kind of patch > for authorities? In this occurrence, biblio and authorities are unrelated. My point is that there are now two indexing modes, GRS-1 and DOM. Two categories of files handle the indexing: records.abs for GRS-1, and biblio-koha-indexdefs.xml, biblio-zebra-indexdefs.xsl for DOM. So if you modify records.abs, you have similarly to modify biblio-koha-indexdefs.xml.
Created attachment 15228 [details] [review] [PATCH 1/2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1)
Created attachment 15229 [details] [review] [PATCH 2/2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) 2d patch, for DOM indexing.
2 patches, one for GRS1 and one for DOM indexing. Each need to be tested separately. M. Saby
Created attachment 17165 [details] [review] [PATCH 1/2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) fixing typo in test plan
Created attachment 20677 [details] [review] [SIGNED-OFF] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) 1st patch : for GRS1 indexing Problem : Zebra indexes all subfields of UNIMARC 7XX fields in author index, including $9, $3, $4 (function code), $f (dates of authors), $c (additions other than dates), $p (address). It causes Koha to give too much results. For example, if an author is born in 1984 and is a teacher, searching "1984" or "teacher" in simple search will return all work by this author if these pieces of informations are in 7XX field. This is not how most ILS work, and it should be corrected. Solution : This patch makes indexing of unimarc 7XX fields more specific. For each field, useless subfields are not indexed. 70X : Do not index $f (dates),$c (additions other than dates),$p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. 710-712 : Do not index $p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. (I kept all other subfields : even if some may be useless I am not sure of it) 716 : Do not index $f (dates),$c (additions other than dates),$3,$4. Index $9 only in Koha-Auth-Number. 72X : Do not index $f (dates),$3,$4. Index $9 only in Koha-Auth-Number. 730 : Do not index $4. Index $9 only in Koha-Auth-Number. Additionnaly, this patch indexes 205$f/$g in Author index (author of the edition of the work) Testing : a/ Create a record with : 700$a Doe $b John $f1950 $cteacher $4070 710$a Corporate $b Meeting $p Address 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys b/ Before applying patch, make several searches : Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will see the record among the results Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in author index : "guy", "guys" => you will see the record among the results with simple search (because 205 is indexed in title) => you will not see the record among the results with author search c/ Apply patch d/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset e/ Try the same searches as before : Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will not see the record among the results Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in author index : "guy", "guys" => you will see the record among the results in both simple search and author search Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Comment: Work as described with GRS1. No koha-qa errors.
Half signed, works well with GRS1
Thank you for testing. And do you agree that the corrected behavior is better for users? M. Saby
Patch applied cleanly, go forth and signoff
I don't know if it need to be tested on DOM, as DOM is not working well for UNIMARC. Maybe we should consider it as signed off? Mathieu
I would like to see this patch and the other ones about unimarc pushed before making my patch for configuring facets. Otherwise there would be too many dependancies. Mathieu
(In reply to mathieu saby from comment #18) > I don't know if it need to be tested on DOM, as DOM is not working well for > UNIMARC. > Maybe we should consider it as signed off? > > Mathieu Only way I think is obsoleting the other patch. Thinking on your other comment, what's is needed? I could try to help.
Hi Not sure what comment you refer to ;-) What is needed is applying attachment 15229 [details] [review] on a DOM Unimarc Koha, and testing it (copying modified configuration files into {installdir}/etc/zebra.... and rebuild_zebra.pl) As it is pure copy of the patch for GRS1, I'd say there is no need to test ALL the items of the test plan. If it was working for GRS1, it *must* work with DOM. And if there is something wrong, the QA team will tell us. Maybe just check you can still search for authors after the pach is applied, and that the search is limited to the fistname + family name of the author (not his birthdate, nor his function code in $4 subfield) You're very kind for proposing your help. Mathieu
Created attachment 21788 [details] [review] [PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) little correction : add 2 lines for "index_all" in the begining and the end of biblio-zebra-indexdefs.xsl (biblio-zebra-indexdefs.xsl is made from biblio-koha-indexdefs.xml with xsltproc) M. Saby
Since I am all setup with a unimarc do you want to rebase this and just do DOM? Happy to test if so
Hello I was about to suggest it to you ;-) I think I will have time to do that tomorrow. Mathieu
Created attachment 31031 [details] [review] [PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM)
2d patch, for DOM indexing Rebased on 20 Aug 2014 Same changes as in GRS-1 patch, with 2 small additions : - I take the occasion for indexing the 511 field (I don't know why it was omitted) - index 710$a and 720$a in Author:s as well as 700$a (seems logical) (biblio-zebra-indexdefs.xsl is made from biblio-koha-indexdefs.xml with xsltproc) Same test as first patch : Testing : a/ Create a record with some values. for example: 700$a Doe $b John $f1950 $cteacher $4070 710$a Corporate $b Meeting $q Address 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys 511$a falsetitle b/ Before applying patch, make several searches : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results with simple search (because 205 is indexed in title) => you will not see the record among the results with author search Search ti:falsetitle => no results c/ Apply the patch d/ copy the 2 modified files into your zebra configuration directory e/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset f/ Try the same searches as before : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will not see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results in both simple search and author search Search ti:falsetitle => 1 result
Applied on patch 2 for testing. Test plan says to add Address to subfield $q - I can't find that field, did you mean $p? - I assumed yes Before patch all worked as expected After applying results were as below, looks like 716 is still indexed: (ss=simple search au:author search) teacher: ss-y au-n great: ss-y au-y Address: ss-y au-n 1950: ss-y au-n 1960: ss-y au-y 070: ss-y au-n 340: ss-y au-y 651: ss-y au-n Doe: ss-y au-y John Doe: ss-y au-n (though yes as:Doe, John) Trademark: ss-y au-y Family: ss-y au-y corporate: ss-y au-y meeting: ss-y au-y guy: ss-y au-y guys: ss-y au-n ti:falsetitle - 1 result
(In reply to Nick Clemens from comment #29) > Applied on patch 2 for testing. > > Test plan says to add Address to subfield $q - I can't find that field, did > you mean $p? - I assumed yes > Yes, it is $p. sorry for that
I'm not sure to understand your test. You mean it is ok for 7XX in general but not for 716? M. Saby
Sorry, I could have been more clear. Short answer to your question is yes. Longer explanation: 716$a Trademark $f1960 $cgreat $4340 <--All of these terms still show in both simple and author search after applying the patch (and copying files and rebuilding) It looks like all fields of 716 are still being indexed. Everything else seems correct as defined in the test plan In my list of search/results: y = Test record appears in search results n = Test record does not appear in search results (In reply to mathieu saby from comment #31) > I'm not sure to understand your test. You mean it is ok for 7XX in general > but not for 716? > > M. Saby
I understand: I wrote <index_data_field xmlns="http://www.koha-community.org/schemas/index-defs" tag="716" subfields="a"> this should be <index_subfields xmlns="http://www.koha-community.org/schemas/index-defs" tag="716" subfields="a"> I will fix that tomorrow M. Saby
Created attachment 31143 [details] [review] [PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM)
I rewrote the 2d patch for fixing the issue with 716 field. You can test again, only 716 field. M. Saby
716 is working as expected now: trademark is searchable as simple/author / 1960,great,340 are searchable as simple search but not author search I apologize though, reviewing my test results I missed one other mistake: 205$g is not being indexed as author author search for 'guys' or 'and other guys' does not return the test record. Sorry for my mistake, I had it written in my test results but did not remember it when I replied (In reply to mathieu saby from comment #35) > I rewrote the 2d patch for fixing the issue with 716 field. You can test > again, only 716 field. > > M. Saby
Created attachment 31151 [details] [review] [PATCH 2] Bug 9352 : More specific indexing of UNIMARC 7XX fields (DOM) My fault! Try that.
I rewrote it quickly witout regenerating the patch, but I suppose it will work, as the count of lines will be the same. If it does not apply, I will take more time to rewrite it (not the good machine this evening)
Patch applied cleanly,guys and 'other guys' now both indexed. I was feeling peppy though, so I retested all searches and came across one small issue searching for 'John Doe' Simple search works as both 'doe, john' or 'john doe' Author search from dropdown (in advanced search or OPAC) works as both 'doe, john' or 'john doe' CCL search DOES NOT work foir either 'au:doe, john' or 'au:john doe' or for any other records/authors Any cause you could see for that?
(In reply to Nick Clemens from comment #39) > Patch applied cleanly,guys and 'other guys' now both indexed. > > I was feeling peppy though, so I retested all searches and came across one > small issue searching for 'John Doe' > > Simple search works as both 'doe, john' or 'john doe' > > Author search from dropdown (in advanced search or OPAC) works as both 'doe, > john' or 'john doe' > > CCL search DOES NOT work foir either 'au:doe, john' or 'au:john doe' or for > any other records/authors > > Any cause you could see for that? Weird. And did it work for "au:john doe" before applying the patch? With the patch, does it work for "au:doe" or "au:john"? and for other indexes (ti, etc.) ? did you try with and without Queryparser activated?
I tested in a running Koha: au is making an exact search on the phrase. I did not remember that... Maybe it depends on syspref For ex: I just changed the order of the first name and surname: http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=au%3AProust+Marcel&branch_group_limit= does not gives the same results as http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=au%3AProust+Marcel&branch_group_limit= And : http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=+Mijolla%2C+Alain+de&branch_group_limit= -> 1 result (from 700 field) http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=+Alain+de+Mijolla&branch_group_limit= -> 1 result (from 200$f field) http://catalogue.univ-aix-marseille.fr/cgi-bin/koha/opac-search.pl?idx=&q=+Alain+Mijolla&branch_group_limit= -> 0 result So, to test 700$aDoe$bJohn au:Doe John should give you 1 result Is that the case? Mathieu
No, with patches applied, files copied, zebra rebuilt: "au:doe john" gives no results I switched to master, recopied files, rebuilt zebra: "au:doe john" gives 1 result "au:john doe" gives no results in either Quotes are not part of search, just to indicate exactly what I typed into the search bar. With queryparser on the order of the terms doesn't matter, 1 result is returned either way, both in master and with patch applied. Sorry, didn't mean to find a new problem, I am just in the habit of always using the CCL terms
In fact, the issue is obvious ;-) Before, all the content of the field was indexed in "au:phrase" index. like that: "Doe John (1945 ; writer)" Now, a and b subfields are indexed separately in "au:phrase" (and also au:word): "Doe" "John" But no "Doe John" To do that, I need probably need to change the behavior of the xsl file generating zebra-biblio-indexdef from koha-biblio-indexdef. Or maybe there is already a template for that in the file. I'm going to ask the lists. Setting the patch in Fail QA for the moment
Created attachment 31314 [details] [review] [PATCH 3/3] Add a new XSLT template for indexing goup of subfields as a whole
With the 3rd patch, everything should work Mathieu
The last patch is broken somewhere, please don't test
Created attachment 31380 [details] [review] [PATCH 3/3] Bug 9352: Add a new XSLT template for indexing goup of subfields as a whole
NEW 3RD PATCH: This patch adds a new template in koha-indexdefs-to-zebra.xsl in order to index a specific goup of subfields as a whole. It can be used for indexing as a "phrase" a string resulting from the concatenation of some indexes (with a space between them) To use this new template, use the new "index_group_of_subfields" element in biblio-koha-indexdefs.xml, in the same way you used index_subfields Ex : to index the concatenation of "$a $b $d $g" subfields of 700 UNIMARC field in Author:p <index_group_of_subfields xmlns="http://www.koha-community.org/schemas/index-defs" tag="700" subfields="abdg"> <target_index>Author:p</target_index> </index_group_of_subfields> This patch also change the biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl for UNIMARC, in order to use this new template for 700-712 fields For 700 and 710, it also indexes in "Author:s" (used for sorting) the concatenation of subfields instead of only the $a subfield To test : in a DOM UNIMARC Koha, apply all the patches, copy the modified files in your Zebra configuration directory, rebuild zebra, and make some tests Everything should be the same as when only the first 2 patches were applied, EXCEPT this behavior: - create a record with 700$aDoe$bJohn (and nothing in 200$f or $g) - Go in Advanced Search, and select the "Author as phrase" index - with only the first 2 patches, you will have 0 result if you enter "Doe John" or "Doe John" - with the 3 patches, you will have 1 result if you enter "Doe John" or "Doe John" - same kind of test for 710 field - the logic is the same for 701, 702,711, 712. So, no need to test - make some searches and test the sorting of your results by author name ("Author A-Z" and "Author Z-A"). The complete name of main author (700 and 710) sould be used for sorting the results, instead of only $& Note that this it better to test it in advanced search, because the behavior of "au:" CCL index in simple search is maybe bugged (it searches the phrase index, while it should probably searches the word index)
Last patch has no signoff?
(In reply to M. de Rooy from comment #49) > Last patch has no signoff? Nick Clemens tested patch 2. For him it was working, except the issue raised in comment 39. So he did not sign it off, and I wrote a follow-up, which is the 3rd patch. I'm not completely sure of that, but I think the ability to index a group of subfields in the same index could be usefull for other MARC fields, and not only for UNIMARC. Mathieu
Oh, sorry, patch 2 and 3 need sign off, I put a wrong status
Created attachment 31385 [details] [review] [SIGNED OFF] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) 1st patch : for GRS1 indexing Problem : Zebra indexes all subfields of UNIMARC 7XX fields in author index, including $9, $3, $4 (function code), $f (dates of authors), $c (additions other than dates), $p (address). It causes Koha to give too much results. For example, if an author is born in 1984 and is a teacher, searching "1984" or "teacher" in simple search will return all work by this author if these pieces of informations are in 7XX field. This is not how most ILS work, and it should be corrected. Solution : This patch makes indexing of unimarc 7XX fields more specific. For each field, useless subfields are not indexed. 70X : Do not index $f (dates),$c (additions other than dates),$p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. 710-712 : Do not index $p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. (I kept all other subfields : even if some may be useless I am not sure of it) 716 : Do not index $f (dates),$c (additions other than dates),$3,$4. Index $9 only in Koha-Auth-Number. 72X : Do not index $f (dates),$3,$4. Index $9 only in Koha-Auth-Number. 730 : Do not index $4. Index $9 only in Koha-Auth-Number. Additionnaly, this patch indexes 205$f/$g in Author index (author of the edition of the work) Testing : a/ Create a record with : 700$a Doe $b John $f1950 $cteacher $4070 710$a Corporate $b Meeting $p Address 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys b/ Before applying patch, make several searches : Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will see the record among the results Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in author index : "guy", "guys" => you will see the record among the results with simple search (because 205 is indexed in title) => you will not see the record among the results with author search c/ Apply patch d/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset e/ Try the same searches as before : Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will not see the record among the results Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in author index : "guy", "guys" => you will see the record among the results in both simple search and author search Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Comment: Work as described with GRS1. No koha-qa errors. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 31386 [details] [review] [SIGNED OFF] Bug 9352 : More specific indexing of UNIMARC 7XX fields and 511 (DOM) 2d patch, for DOM indexing Rebased on 20 Aug 2014, Fixed on 25 Aug 2014 (716 field and 205$g) Same changes as in GRS-1 patch, with 2 small additions : - I take the occasion for indexing the 511 field (I don't know why it was omitted) - index 710$a and 720$a in Author:s as well as 700$a (seems logical) (biblio-zebra-indexdefs.xsl is made from biblio-koha-indexdefs.xml with xsltproc) Same test as first patch : Testing : a/ Create a record with some values. for example: 700$a Doe $b John $f1950 $cteacher $4070 710$a Corporate $b Meeting $q Address 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys 511$a falsetitle b/ Before applying patch, make several searches : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results with simple search (because 205 is indexed in title) => you will not see the record among the results with author search Search ti:falsetitle => no results c/ Apply the patch d/ copy the 2 modified files into your zebra configuration directory e/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset f/ Try the same searches as before : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will not see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results in both simple search and author search Search ti:falsetitle => 1 result Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 31387 [details] [review] [SIGNED OFF] Bug 9352: Add a new XSLT template for indexing goup of subfields as a whole This patch adds a new template in koha-indexdefs-to-zebra.xsl in order to index a specific goup of subfields as a whole. It can be used for indexing as a "phrase" a string resulting from the concatenation of some indexes (with a space between them) To use this new template, use the new "index_group_of_subfields" element in biblio-koha-indexdefs.xml, in the same way you used index_subfields Ex : to index the concatenation of "$a $b $d $g" subfields of 700 UNIMARC field in Author:p <index_group_of_subfields xmlns="http://www.koha-community.org/schemas/index-defs" tag="700" subfields="abdg"> <target_index>Author:p</target_index> </index_group_of_subfields> This patch also change the biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl for UNIMARC, in order to use this new template for 700-712 fields For 700 and 710, it also indexes in "Author:s" (used for sorting) the concatenation of subfields instead of only the $a subfield To test : in a DOM UNIMARC Koha, apply all the patches, copy the modified files in your Zebra configuration directory, rebuild zebra, and make some tests Everything should be the same as when only the first 2 patches were applied, EXCEPT this behavior: - create a record with 700$aDoe$bJohn (and nothing in 200$f or $g) - Go in Advanced Search, and select the "Author as phrase" index - with only the first 2 patches, you will have 0 result if you enter "Doe John" or "Doe John" - with the 3 patches, you will have 1 result if you enter "Doe John" or "Doe John" - same kind of test for 710 field - the logic is the same for 701, 702,711, 712. So, no need to test - make some searches and test the sorting of your results by author name ("Author A-Z" and "Author Z-A"). The complete name of main author (700 and 710) sould be used for sorting the results, instead of only $& Note that this it better to test it in advanced search, because the behavior of "au:" CCL index in simple search is maybe bugged (it searches the phrase index, while it should probably searches the word index) Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Happy to sign off... I'm particularly interested in patch 3, which introduces a new feature to the koha-indexdefs-to-zebra.xsl allowing subfield grouping for indexing purposes. It looks all good from my testing.
I would like to QA this one soon. Put it on my list now.
Created attachment 32097 [details] [review] Bug 9352 : More specific indexing of UNIMARC 7XX fields (GRS1) 1st patch : for GRS1 indexing Problem : Zebra indexes all subfields of UNIMARC 7XX fields in author index, including $9, $3, $4 (function code), $f (dates of authors), $c (additions other than dates), $p (address). It causes Koha to give too much results. For example, if an author is born in 1984 and is a teacher, searching "1984" or "teacher" in simple search will return all work by this author if these pieces of informations are in 7XX field. This is not how most ILS work, and it should be corrected. Solution : This patch makes indexing of unimarc 7XX fields more specific. For each field, useless subfields are not indexed. 70X : Do not index $f (dates),$c (additions other than dates),$p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. 710-712 : Do not index $p (affiliation/address),$3,$4. Index $9 only in Koha-Auth-Number. (I kept all other subfields : even if some may be useless I am not sure of it) 716 : Do not index $f (dates),$c (additions other than dates),$3,$4. Index $9 only in Koha-Auth-Number. 72X : Do not index $f (dates),$3,$4. Index $9 only in Koha-Auth-Number. 730 : Do not index $4. Index $9 only in Koha-Auth-Number. Additionnaly, this patch indexes 205$f/$g in Author index (author of the edition of the work) Testing : a/ Create a record with : 700$a Doe $b John $f1950 $cteacher $4070 710$a Corporate $b Meeting $p Address 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys b/ Before applying patch, make several searches : Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will see the record among the results Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in author index : "guy", "guys" => you will see the record among the results with simple search (because 205 is indexed in title) => you will not see the record among the results with author search c/ Apply patch d/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset e/ Try the same searches as before : Search in author index : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will not see the record among the results Search in author index : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in author index : "guy", "guys" => you will see the record among the results in both simple search and author search Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Comment: Work as described with GRS1. No koha-qa errors. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Created attachment 32098 [details] [review] Bug 9352 : More specific indexing of UNIMARC 7XX fields and 511 (DOM) 2d patch, for DOM indexing Rebased on 20 Aug 2014, Fixed on 25 Aug 2014 (716 field and 205$g) Same changes as in GRS-1 patch, with 2 small additions : - I take the occasion for indexing the 511 field (I don't know why it was omitted) - index 710$a and 720$a in Author:s as well as 700$a (seems logical) (biblio-zebra-indexdefs.xsl is made from biblio-koha-indexdefs.xml with xsltproc) Same test as first patch : Testing : a/ Create a record with some values. for example: 700$a Doe $b John $f1950 $cteacher $4070 710$a Corporate $b Meeting $q Address 716$a Trademark $f1960 $cgreat $4340 720$a Family $f1980 $4651 205$a 1st edition $fBy some guy $gAnd other guys 511$a falsetitle b/ Before applying patch, make several searches : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results with simple search (because 205 is indexed in title) => you will not see the record among the results with author search Search ti:falsetitle => no results c/ Apply the patch d/ copy the 2 modified files into your zebra configuration directory e/ Rebuild Zebra with ./misc/migration_tools/rebuild_zebra.pl -b -x -v -reset f/ Try the same searches as before : Search in simple search and author search : "teacher", "great", "Address", "1950", "1960", "070", "340", "651" => you will not see the record among the results Search in simple search and author search : "Doe", "John Doe", "Trademark", "Family","corporate", "meeting" => you will see the record among the results Search in simple search and author search : "guy", "guys" => you will see the record among the results in both simple search and author search Search ti:falsetitle => 1 result Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Created attachment 32099 [details] [review] Bug 9352: Add a new XSLT template for indexing goup of subfields as a whole This patch adds a new template in koha-indexdefs-to-zebra.xsl in order to index a specific goup of subfields as a whole. It can be used for indexing as a "phrase" a string resulting from the concatenation of some indexes (with a space between them) To use this new template, use the new "index_group_of_subfields" element in biblio-koha-indexdefs.xml, in the same way you used index_subfields Ex : to index the concatenation of "$a $b $d $g" subfields of 700 UNIMARC field in Author:p <index_group_of_subfields xmlns="http://www.koha-community.org/schemas/index-defs" tag="700" subfields="abdg"> <target_index>Author:p</target_index> </index_group_of_subfields> This patch also change the biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl for UNIMARC, in order to use this new template for 700-712 fields For 700 and 710, it also indexes in "Author:s" (used for sorting) the concatenation of subfields instead of only the $a subfield To test : in a DOM UNIMARC Koha, apply all the patches, copy the modified files in your Zebra configuration directory, rebuild zebra, and make some tests Everything should be the same as when only the first 2 patches were applied, EXCEPT this behavior: - create a record with 700$aDoe$bJohn (and nothing in 200$f or $g) - Go in Advanced Search, and select the "Author as phrase" index - with only the first 2 patches, you will have 0 result if you enter "Doe John" or "Doe John" - with the 3 patches, you will have 1 result if you enter "Doe John" or "Doe John" - same kind of test for 710 field - the logic is the same for 701, 702,711, 712. So, no need to test - make some searches and test the sorting of your results by author name ("Author A-Z" and "Author Z-A"). The complete name of main author (700 and 710) sould be used for sorting the results, instead of only $& Note that this it better to test it in advanced search, because the behavior of "au:" CCL index in simple search is maybe bugged (it searches the phrase index, while it should probably searches the word index) Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
I like to improve UNIMARC support, so will be available to hangout and/or comment on whatever you need. But - GRS-1 is now deprecated (no problem with that, just obsolete that patch) - The rest of the patches don't apply - It seems you are re-implementing on koha-indexdefs-to-zebra.xsl something already existed, i might be wrong, but look at 'handle-index-subfields'
(In reply to Tomás Cohen Arazi from comment #60) > I like to improve UNIMARC support, so will be available to hangout and/or > comment on whatever you need. But > > - GRS-1 is now deprecated (no problem with that, just obsolete that patch) > - The rest of the patches don't apply > - It seems you are re-implementing on koha-indexdefs-to-zebra.xsl something > already existed, i might be wrong, but look at 'handle-index-subfields' I will rebase it later. But for your 3rd point : the idea is to create a new behavior : indexing as a single phrase, in a single index, a string resulting from the concatenation of some subfieds. I don't think you can do that with 'handle-index-subfields'. Mathieu
Hi Thomas My changes for 7XX are a bit complex, and not very urgent. So could you please push the patch for 6XX, and any other patch related to Zebra, and I will take an other look next month, and rebase what need to be rebased? Mathieu