Most of 6XX may contain a $2 that identifies the system used for indexing. It should not be indexed. In French libraries, $2 contains "rameau". So searching books about the music composer "Rameau" retreive thousands of records! For some 6XX fiels, other subfields should not be indexed, for example 600$t (dates of persons). And indexing of $j,$x,$y,$z subdivision could be improved by indexing them in specific indexes M. Saby
Additionnally, subject indexing could be improved by using specific indexes for each 6XX if possible. In ccl.properties, a specific index is defined, but not used in record.abs : Subject-name-personal, alias su-na Other aliases are defined, but as synonyms to Subject index : su-to, su-geo and su-ut. We can use index Subject-name-personal and create new specific indexes by using existing bib1 attributes. M. Saby
Created attachment 16179 [details] [review] [PATCH] Bug 9828: More specific indexing of UNIMARC 6XX fields Most of 6XX may contain a $2 that identifies the system used for indexing. It should not be indexed. In French libraries, $2 contains "rameau". So searching books about the music composer "Rameau" retreive thousands of records! For some 6XX fiels, other subfields should not be indexed, for example dates of persons and familiy, or adresses. In unimarc guide, 600$t,601$t,602$t are said to exist but to be "not used". I keep them indexed. Additionnally, subject indexing could be improved by using specific indexes for each 6XX if possible : In ccl.properties : - su-to, su-geo and su-ut are defined as aliases of Subject. - a specific index are defined, but not used in record.abs : Subject-name-personal, alis su-na We can use this index and create new specific indexes by using existing bib1 attributes. We could also index $j,$x,$y,$z subdivision in specific indexes. This patch does the following changes : 1) Add comments in record.abs 2) For all 6XX : Not indexing $2 (LSCH, Rameau...), $3 and $5 3) Not indexing specific subfields 600 : Personal name used as a subject // see Marc21 600 not indexing c (additional elements),f (dates),p (address/affiliation) 602 : Family name used as a subject // see Marc21 600 3X not indexing f (dates) 616 : Trademark not indexing c,f 3) For all 6XX : index $j,$x,$y,$z in several indexes in addition to the specfific index for their 6XX field: # 6XX$j : Genre/form : indexed in Subject, Subject-subdivision, Subject-genre-form # 6XX$x : Subject : indexed in Subject, Subject-subdivision # (could be topical subject or genre/form subject, so don't index in Subject-topical) # 6XX$y : Geographical subject : indexed in Subject, Subject-subdivision, Subject-name-geographical # 6XX$z : Chronological subject : indexed in Subject, Subject-subdivision, Subject-chronological 4) Define in ccl.properties some specific indexes : Subject-name-conference 1=1073 => alias su-conf Subject-name-corporate 1=1074 => alias su-corp Subject-genre-form 1=1075 => alias su-genre and su-form Subject-geographical 1=1076 => alias su-geo Subject-chronological 1=1077 => alias su-chrono Subject-title 1=1078 => alias su-ut and su-ti Subject-topical 1=1079 => alias su-to 5) Adding new aliases in Search.pm : su-chrono, su-form, su-genre, su-corp, su-conf, su-ti 6) Using these new indexes in record.abs for 600 : all field in Subject and Subject-Personal-Name all subfields except subdivisions in Personal-name 601 : all field in Subject, Subject-name-conference and Subject-name-corporate and Subject-name-conf all subfields except subdivisions in Corporate-name and Conference-name 602 : same as 600 but could be improved later 604 : all field in Subject and Subject-title ; $a in Subject-Personal-Name all subfields except subdivisions in Name-and-Title 605 : all field in Subject and Subject-title 606 : all field in Subject and Subject-topical 607 : all field in Subject and Subject-geographical all subfields except subdivisions in Name-geographic 608 : all field in Subject and Subject-genre-form To test : A. In a GRS 1 environment 1) Apply the patch 2) Rebuild zebra 3) Create a record A with - the string "bz9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f - the string "subform" in 602$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z 6) try to search "su:bz9828". You should have no results 7) try to search "su-geo:subgeo". You should have 1 result : record A 8) try to search "su-genre:subform". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant B. In a DOM environment same operations M. Saby
Created attachment 17164 [details] [review] [PATCH] Bug 9828: More specific indexing of UNIMARC 6XX fields Fixing typos in test plan
Patch applied cleanly, go forth and signoff
Applying: Bug 9828: More specific indexing of UNIMARC 6XX fields Using index info to reconstruct a base tree... M C4/Search.pm M etc/zebradb/biblios/etc/bib1.att M etc/zebradb/ccl.properties M etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml M etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl M etc/zebradb/marc_defs/unimarc/biblios/record.abs Falling back to patching base and 3-way merge... Auto-merging etc/zebradb/marc_defs/unimarc/biblios/record.abs Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml Auto-merging etc/zebradb/ccl.properties CONFLICT (content): Merge conflict in etc/zebradb/ccl.properties Auto-merging etc/zebradb/biblios/etc/bib1.att Auto-merging C4/Search.pm Patch failed at 0001 Bug 9828: More specific indexing of UNIMARC 6XX fields The copy of the patch that failed is found in: /home/christopher/git/koha/.git/rebase-apply/patch When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Created attachment 22379 [details] [review] [PATCH] Bug 9828: More specific indexing of UNIMARC 6XX fields Rebased patch
A followup is probably needed for QueryParser. I don't know how to do that. Mathieu
The sandbox you've requested is not ready. Some problems occurred applying patches from bug 9828: <h1>Something went wrong !</h1>Applying: Bug 9828: More specific indexing of UNIMARC 6XX fields Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging C4/Search.pm Auto-merging etc/zebradb/biblios/etc/bib1.att Auto-merging etc/zebradb/ccl.properties Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl Auto-merging etc/zebradb/marc_defs/unimarc/biblios/record.abs Failed to merge in the changes. Patch failed at 0001 Bug 9828: More specific indexing of UNIMARC 6XX fields When you have resolved this problem run git bz apply --continue. If you would prefer to skip this patch, instead run git bz apply --skip. To restore the original branch and stop patching run git bz apply --abort. Bug 9828 - Zebra indexes useless subfields in UNIMARC 6XX
Hello You cannot test this patch on a sandbox, because you need to rebuild zebra. If you want to test it, tell me, I will rebase the patch M. Saby
Ah, if can't do sandbox I will have to setup a koha instance with UNIMARC, once I have one ready to go I will let you know I am ready for rebase
I have a UNIMARC instance setup with dom indexing. I can confirm that before patch: - 6XX $2 is indexed - su-to, su-geo and su-ut return the same results as su - su-na does not return results - su-chrono, su-form, su-genre, su-corp, su-conf, su-ti return incorrect results if any Happy to test patch in dom if you rebase, I don't have a grs1 setup, but it is now deprecated so maybe is not needed for this patch?
Created attachment 30954 [details] [review] [PATCH 1/2] Bug 9828: More specific indexing of UNIMARC 6XX fields
[New commit on 18 Aug 2014 : rebased, and DOM indexing only] Issues to fix : Most of 6XX may contain a $2 that identifies the system used for indexing. It should not be indexed. In French libraries, $2 contains "rameau". So searching books about the music composer "Rameau" retreive thousands of records! For some 6XX fiels, other subfields should not be indexed, for example dates of persons and family, or adresses. In Unimarc guide, 600$t,601$t,602$t are said to exist but to be "not used". I keep them indexed. Additionnally, subject indexing could be improved by using specific indexes for each 6XX if possible : In ccl.properties : - su-to, su-geo and su-ut are defined as aliases of Subject. - a specific index is defined, but not used in record.abs : Subject-name-personal, alias su-na We can use these indexes and create new specific indexes by using existing bib1 attributes. We could also index $j,$x,$y,$z subdivision in specific indexes. This patch does the following changes : 1) For all 6XX : Not indexing $2 (LSCH, Rameau...), $3 and $5 2) Suppressing the indexing of some specific subfields, depending on the field: 600 : Personal name used as a subject // see Marc21 600 not indexing c (additional elements),f (dates),p (address/affiliation) 602 : Family name used as a subject // see Marc21 600 3X not indexing f (dates) 616 : Trademark not indexing c,f 3) For all 6XX : index $j,$x,$y,$z in several indexes in addition to the specfific index for their 6XX field: 4) Define in ccl.properties some specific indexes : Subject-name-conference 1=1073 => alias su-conf Subject-name-corporate 1=1074 => alias su-corp Subject-genre-form 1=1075 => alias su-genre and su-form Subject-geographical 1=1076 => alias su-geo Subject-chronological 1=1077 => alias su-chrono Subject-title 1=1078 => alias su-ut and su-ti Subject-topical 1=1079 => alias su-to 5) Adding new aliases in Search.pm : su-chrono, su-form, su-genre, su-corp, su-conf, su-ti 6) Using these new indexes in for 600 : Subject and Subject-Personal-Name ; all subfields except subdivisions in Personal-name 601 : Subject, Subject-name-conference and Subject-name-corporate and Subject-name-conf ; all subfields except subdivisions in Corporate-name and Conference-name 602 : same as 600 but could be improved later 604 : Subject and Subject-title ; $a in Subject-Personal-Name ; all subfields except subdivisions in Name-and-Title 605 : Subject and Subject-title 606 : Subject and Subject-topical 607 : Subject and Subject-geographical ; all subfields except subdivisions in Name-geographic 608 : Subject and Subject-genre-form To test : A. In a UNIMARC-DOM indexing environment 1) Apply the patch 2) Rebuild zebra 3) Create a record A with some values in critical fields, for example: - the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2 - the string "subform" in 600$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z 6) try to search "su:test9828". You should have no results 7) try to search "su-genre:subform". You should have 1 result : record A 8) try to search "su-geo:subgeo". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant Indexing of subjects could maybe be improved later
Created attachment 30955 [details] [review] [PATCH 2/2] Bug 9828 : Add and fix comments in UNIMARC biblio-koha-indexdefs.xml 2d patch : Only cosmetic : - the references to lines record.abs are now useless and outdated - some comments added in record.abs could be usefull in biblio-koha-indexdefs.xml No change expected, only comments
For testing, I forgot a step : between "apply the patch" and "rebuild zebra", of course you need to copy the modified files from your source directory to the actual zebra configuration directory of your machine. But I suppose you know how to do that. Mathieu
And like Nick suggested, I removed the part of code related to GRS-1, as it is now more or less deprecated. So, please test the patch in a DOM UNIMARC environment. Mathieu
Supposing I know how to do things: generally not a good idea :-) I figured it out it pretty quick though
Created attachment 30960 [details] [review] [SIGNED OFF] Bug 9828: More specific indexing of UNIMARC 6XX fields [New commit on 18 Aug 2014 : rebased, and DOM indexing only] Issues to fix : Most of 6XX may contain a $2 that identifies the system used for indexing. It should not be indexed. In French libraries, $2 contains "rameau". So searching books about the music composer "Rameau" retreive thousands of records! For some 6XX fiels, other subfields should not be indexed, for example dates of persons and family, or adresses. In Unimarc guide, 600$t,601$t,602$t are said to exist but to be "not used". I keep them indexed. Additionnally, subject indexing could be improved by using specific indexes for each 6XX if possible : In ccl.properties : - su-to, su-geo and su-ut are defined as aliases of Subject. - a specific index is defined, but not used in record.abs : Subject-name-personal, alias su-na We can use these indexes and create new specific indexes by using existing bib1 attributes. We could also index $j,$x,$y,$z subdivision in specific indexes. This patch does the following changes : 1) For all 6XX : Not indexing $2 (LSCH, Rameau...), $3 and $5 2) Suppressing the indexing of some specific subfields, depending on the field: 600 : Personal name used as a subject // see Marc21 600 not indexing c (additional elements),f (dates),p (address/affiliation) 602 : Family name used as a subject // see Marc21 600 3X not indexing f (dates) 616 : Trademark not indexing c,f 3) For all 6XX : index $j,$x,$y,$z in several indexes in addition to the specfific index for their 6XX field: 4) Define in ccl.properties some specific indexes : Subject-name-conference 1=1073 => alias su-conf Subject-name-corporate 1=1074 => alias su-corp Subject-genre-form 1=1075 => alias su-genre and su-form Subject-geographical 1=1076 => alias su-geo Subject-chronological 1=1077 => alias su-chrono Subject-title 1=1078 => alias su-ut and su-ti Subject-topical 1=1079 => alias su-to 5) Adding new aliases in Search.pm : su-chrono, su-form, su-genre, su-corp, su-conf, su-ti 6) Using these new indexes in for 600 : Subject and Subject-Personal-Name ; all subfields except subdivisions in Personal-name 601 : Subject, Subject-name-conference and Subject-name-corporate and Subject-name-conf ; all subfields except subdivisions in Corporate-name and Conference-name 602 : same as 600 but could be improved later 604 : Subject and Subject-title ; $a in Subject-Personal-Name ; all subfields except subdivisions in Name-and-Title 605 : Subject and Subject-title 606 : Subject and Subject-topical 607 : Subject and Subject-geographical ; all subfields except subdivisions in Name-geographic 608 : Subject and Subject-genre-form To test : A. In a UNIMARC-DOM indexing environment 1) Apply the patch 2) Rebuild zebra 3) Create a record A with some values in critical fields, for example: - the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2 - the string "subform" in 600$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z 6) try to search "su:test9828". You should have no results 7) try to search "su-genre:subform". You should have 1 result : record A 8) try to search "su-geo:subgeo". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant Indexing of subjects could maybe be improved later Signed-off-by: Nick Clemens <nick@quecheelibrary.org> All seems to work as expected, I am not super-familiar with UNIMARC but I wonder if in su-corp and su-conf the subdivisions might be useful (e.g. France-Gendarmie / Staatsbibliothek-Berlin)
Created attachment 30961 [details] [review] [SIGNED OFF] Bug 9828: More specific indexing of UNIMARC 6XX fields [New commit on 18 Aug 2014 : rebased, and DOM indexing only] Issues to fix : Most of 6XX may contain a $2 that identifies the system used for indexing. It should not be indexed. In French libraries, $2 contains "rameau". So searching books about the music composer "Rameau" retreive thousands of records! For some 6XX fiels, other subfields should not be indexed, for example dates of persons and family, or adresses. In Unimarc guide, 600$t,601$t,602$t are said to exist but to be "not used". I keep them indexed. Additionnally, subject indexing could be improved by using specific indexes for each 6XX if possible : In ccl.properties : - su-to, su-geo and su-ut are defined as aliases of Subject. - a specific index is defined, but not used in record.abs : Subject-name-personal, alias su-na We can use these indexes and create new specific indexes by using existing bib1 attributes. We could also index $j,$x,$y,$z subdivision in specific indexes. This patch does the following changes : 1) For all 6XX : Not indexing $2 (LSCH, Rameau...), $3 and $5 2) Suppressing the indexing of some specific subfields, depending on the field: 600 : Personal name used as a subject // see Marc21 600 not indexing c (additional elements),f (dates),p (address/affiliation) 602 : Family name used as a subject // see Marc21 600 3X not indexing f (dates) 616 : Trademark not indexing c,f 3) For all 6XX : index $j,$x,$y,$z in several indexes in addition to the specfific index for their 6XX field: 4) Define in ccl.properties some specific indexes : Subject-name-conference 1=1073 => alias su-conf Subject-name-corporate 1=1074 => alias su-corp Subject-genre-form 1=1075 => alias su-genre and su-form Subject-geographical 1=1076 => alias su-geo Subject-chronological 1=1077 => alias su-chrono Subject-title 1=1078 => alias su-ut and su-ti Subject-topical 1=1079 => alias su-to 5) Adding new aliases in Search.pm : su-chrono, su-form, su-genre, su-corp, su-conf, su-ti 6) Using these new indexes in for 600 : Subject and Subject-Personal-Name ; all subfields except subdivisions in Personal-name 601 : Subject, Subject-name-conference and Subject-name-corporate and Subject-name-conf ; all subfields except subdivisions in Corporate-name and Conference-name 602 : same as 600 but could be improved later 604 : Subject and Subject-title ; $a in Subject-Personal-Name ; all subfields except subdivisions in Name-and-Title 605 : Subject and Subject-title 606 : Subject and Subject-topical 607 : Subject and Subject-geographical ; all subfields except subdivisions in Name-geographic 608 : Subject and Subject-genre-form To test : A. In a UNIMARC-DOM indexing environment 1) Apply the patch 2) Rebuild zebra 3) Create a record A with some values in critical fields, for example: - the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2 - the string "subform" in 600$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z 6) try to search "su:test9828". You should have no results 7) try to search "su-genre:subform". You should have 1 result : record A 8) try to search "su-geo:subgeo". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant Indexing of subjects could maybe be improved later Signed-off-by: Nick Clemens <nick@quecheelibrary.org> All seems to work as expected, I am not super-familiar with UNIMARC but I wonder if in su-corp and su-conf the subdivisions might be useful (e.g. France-Gendarmie / Staatsbibliothek-Berlin)
Created attachment 30962 [details] [review] [SIGNED OFF] Bug 9828 : Add and fix comments in UNIMARC biblio-koha-indexdefs.xml Only cosmetic : - the references to lines record.abs are now useless and outdated - some comments added in record.abs could be usefull in biblio-koha-indexdefs.xml No change expected, only comments Signed-off-by: Nick Clemens <nick@quecheelibrary.org>
> > All seems to work as expected, I am not super-familiar with UNIMARC but I > wonder if in su-corp and su-conf the subdivisions might be useful (e.g. > France-Gendarmie / Staatsbibliothek-Berlin) For 601 field, the j, x, y, z subfield are mentionned in UNIMARC standard. See http://multimedia.bnf.fr/unimarcb_trad/B601-6-2013.pdf The best would be to index separately meetings and organizations, which are both in 601 field, but with different "indicator" value. I'm trying to write a followup for that. M. Saby
I also need to add a followup for Queryparser
Created attachment 31029 [details] [review] [PATCH 3/3] Bug 9828 : Followup for Queryparser and deletion of useless 6XX$9 This followup - changes some indexes in Queryparser configuration file - supresses some clearly useless 6XX$9 in biblio-koha-indexdefs.xml and adds 2 new ones, probably useless (not sure of that) - change the name of index Subject-geographical to Subject-name-geographical in ccl.properties (to match bib1.att) the xsl file zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl was generated with the following command: xsltproc zebradb/xsl/koha-indexdefs-to-zebra.xsl zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml > zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl To test : 1) Apply the 3 patches 2) copy the modified files from the source directory to the directory where you store the config files for Zebra and Queryparser The files modified by the 3 patches and that need to be copied are: etc/zebradb/biblios/etc/bib1.att etc/zebradb/ccl.properties etc/searchengine/queryparser.yaml etc/zebradb/ccl.properties .../unimarc/biblios/biblio-koha-indexdefs.xml .../unimarc/biblios/biblio-zebra-indexdefs.xsl 3) Rebuild Zebra 4) Create a record A with some values in critical fields, for example: - the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2 - the string "subform" in 600$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z WITHOUT QP activated in sysprefs ("Don't try to use QP"): 6) try to search "su:test9828". You should have no results 7) try to search "su-genre:subform". You should have 1 result : record A 8) try to search "su-geo:subgeo". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant WITH QP activated in sysprefs: Same tests
Would you be kind enough to test it again, with the 3rd patch applied on top of the 2 first ones? You need to test without QP activated : it should work as in your previous test and with QP activated (it should work also). (Results with QP are sometimes slightly different, I cannot remember why) M. Saby
Note that it is currently impossible to make a more precise indexing of some fields depending on the value stored in the 2 indicators (for field 601 for example). This need an change in / etc / zebradb / xsl / koha-indexdefs-to-zebra.xsl M. Saby
Created attachment 31113 [details] [review] [SIGNED OFF] Bug 9828 : Followup for Queryparser and deletion of useless 6XX$9 This followup - changes some indexes in Queryparser configuration file - supresses some clearly useless 6XX$9 in biblio-koha-indexdefs.xml and adds 2 new ones, probably useless (not sure of that) - change the name of index Subject-geographical to Subject-name-geographical in ccl.properties (to match bib1.att) the xsl file zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl was generated with the following command: xsltproc zebradb/xsl/koha-indexdefs-to-zebra.xsl zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml > zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl To test : 1) Apply the 3 patches 2) copy the modified files from the source directory to the directory where you store the config files for Zebra and Queryparser The files modified by the 3 patches and that need to be copied are: etc/zebradb/biblios/etc/bib1.att etc/zebradb/ccl.properties etc/searchengine/queryparser.yaml etc/zebradb/ccl.properties .../unimarc/biblios/biblio-koha-indexdefs.xml .../unimarc/biblios/biblio-zebra-indexdefs.xsl 3) Rebuild Zebra 4) Create a record A with some values in critical fields, for example: - the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2 - the string "subform" in 600$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z WITHOUT QP activated in sysprefs ("Don't try to use QP"): 6) try to search "su:test9828". You should have no results 7) try to search "su-genre:subform". You should have 1 result : record A 8) try to search "su-geo:subgeo". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant WITH QP activated in sysprefs: Same tests Signed-off-by: Nick Clemens <nick@quecheelibrary.org>
New bug created for indicators: bug 12813
QA comments: * NOT UNIMARC-specific things: - this patch fixes 2 obvious mistakes in bib1.att. It means it also fixes a bug in MARC21 that no-one ever saw : -att 1075 Subject-genre/form -att 1076 Subject-name-gerographical +att 1075 Subject-genre-form +att 1076 Subject-name-geographical - this patch adds the following attributes: +su-corp Subject-name-corporate +su-ti Subject-title => those 2 additions are harmless, i've checked that there is no reference to that in marc21/normarc directories no other comment, test plan works.
Created attachment 32111 [details] [review] Bug 9828: More specific indexing of UNIMARC 6XX fields [New commit on 18 Aug 2014 : rebased, and DOM indexing only] Issues to fix : Most of 6XX may contain a $2 that identifies the system used for indexing. It should not be indexed. In French libraries, $2 contains "rameau". So searching books about the music composer "Rameau" retreive thousands of records! For some 6XX fiels, other subfields should not be indexed, for example dates of persons and family, or adresses. In Unimarc guide, 600$t,601$t,602$t are said to exist but to be "not used". I keep them indexed. Additionnally, subject indexing could be improved by using specific indexes for each 6XX if possible : In ccl.properties : - su-to, su-geo and su-ut are defined as aliases of Subject. - a specific index is defined, but not used in record.abs : Subject-name-personal, alias su-na We can use these indexes and create new specific indexes by using existing bib1 attributes. We could also index $j,$x,$y,$z subdivision in specific indexes. This patch does the following changes : 1) For all 6XX : Not indexing $2 (LSCH, Rameau...), $3 and $5 2) Suppressing the indexing of some specific subfields, depending on the field: 600 : Personal name used as a subject // see Marc21 600 not indexing c (additional elements),f (dates),p (address/affiliation) 602 : Family name used as a subject // see Marc21 600 3X not indexing f (dates) 616 : Trademark not indexing c,f 3) For all 6XX : index $j,$x,$y,$z in several indexes in addition to the specfific index for their 6XX field: 4) Define in ccl.properties some specific indexes : Subject-name-conference 1=1073 => alias su-conf Subject-name-corporate 1=1074 => alias su-corp Subject-genre-form 1=1075 => alias su-genre and su-form Subject-geographical 1=1076 => alias su-geo Subject-chronological 1=1077 => alias su-chrono Subject-title 1=1078 => alias su-ut and su-ti Subject-topical 1=1079 => alias su-to 5) Adding new aliases in Search.pm : su-chrono, su-form, su-genre, su-corp, su-conf, su-ti 6) Using these new indexes in for 600 : Subject and Subject-Personal-Name ; all subfields except subdivisions in Personal-name 601 : Subject, Subject-name-conference and Subject-name-corporate and Subject-name-conf ; all subfields except subdivisions in Corporate-name and Conference-name 602 : same as 600 but could be improved later 604 : Subject and Subject-title ; $a in Subject-Personal-Name ; all subfields except subdivisions in Name-and-Title 605 : Subject and Subject-title 606 : Subject and Subject-topical 607 : Subject and Subject-geographical ; all subfields except subdivisions in Name-geographic 608 : Subject and Subject-genre-form To test : A. In a UNIMARC-DOM indexing environment 1) Apply the patch 2) Rebuild zebra 3) Create a record A with some values in critical fields, for example: - the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2 - the string "subform" in 600$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z 6) try to search "su:test9828". You should have no results 7) try to search "su-genre:subform". You should have 1 result : record A 8) try to search "su-geo:subgeo". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant Indexing of subjects could maybe be improved later Signed-off-by: Nick Clemens <nick@quecheelibrary.org> All seems to work as expected, I am not super-familiar with UNIMARC but I wonder if in su-corp and su-conf the subdivisions might be useful (e.g. France-Gendarmie / Staatsbibliothek-Berlin) Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Created attachment 32112 [details] [review] Bug 9828 : Add and fix comments in UNIMARC biblio-koha-indexdefs.xml Only cosmetic : - the references to lines record.abs are now useless and outdated - some comments added in record.abs could be usefull in biblio-koha-indexdefs.xml No change expected, only comments Signed-off-by: Nick Clemens <nick@quecheelibrary.org> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Created attachment 32113 [details] [review] Bug 9828 : Followup for Queryparser and deletion of useless 6XX$9 This followup - changes some indexes in Queryparser configuration file - supresses some clearly useless 6XX$9 in biblio-koha-indexdefs.xml and adds 2 new ones, probably useless (not sure of that) - change the name of index Subject-geographical to Subject-name-geographical in ccl.properties (to match bib1.att) the xsl file zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl was generated with the following command: xsltproc zebradb/xsl/koha-indexdefs-to-zebra.xsl zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml > zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl To test : 1) Apply the 3 patches 2) copy the modified files from the source directory to the directory where you store the config files for Zebra and Queryparser The files modified by the 3 patches and that need to be copied are: etc/zebradb/biblios/etc/bib1.att etc/zebradb/ccl.properties etc/searchengine/queryparser.yaml etc/zebradb/ccl.properties .../unimarc/biblios/biblio-koha-indexdefs.xml .../unimarc/biblios/biblio-zebra-indexdefs.xsl 3) Rebuild Zebra 4) Create a record A with some values in critical fields, for example: - the string "test9828" in 600$c 600$f 600$p, 602$f, 616$c, 616$f, 606$2,600$2 - the string "subform" in 600$j 4) Create a record B with the string "subgeo" in 606$y 5) Create a record C with the string "subdate" in 606$z WITHOUT QP activated in sysprefs ("Don't try to use QP"): 6) try to search "su:test9828". You should have no results 7) try to search "su-genre:subform". You should have 1 result : record A 8) try to search "su-geo:subgeo". You should have 1 result : record B 9) try to search "su-chrono:subdate". You should have 1 result : record C 10) on existing records, try su-ut, su-to, su-na, su-form, su-corp, su-geo indexes, and see it results are relevant WITH QP activated in sysprefs: Same tests Signed-off-by: Nick Clemens <nick@quecheelibrary.org> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Note: France -- Gendarmerie is encoded as $aFrance $bGendarmerie http://multimedia.bnf.fr/unimarcb_trad/B601-6-2013.pdf M. Saby
Patches don't apply cleanly, please rebase.
Of course, because of the new code for facets ;-) I will rebase this week end. Mathieu
(In reply to mathieu saby from comment #36) > Of course, because of the new code for facets ;-) > I will rebase this week end. Of course, sorry for that Mathieu. Please feel free to contact me regarding this and the other UNIMARC bug, I'd like to have them pushed soon.
Created attachment 32720 [details] [review] [PATCH 1/3] Bug 9828: More specific indexing of UNIMARC 6XX fields
Created attachment 32725 [details] [review] [PATCH 1/3] Bug 9828: More specific indexing of UNIMARC 6XX fields
Created attachment 32726 [details] [review] [PATCH 2/3] Bug 9828 : Add and fix comments in UNIMARC biblio-koha-indexdefs.xml
Created attachment 32727 [details] [review] [PATCH 3/3] Bug 9828 : Followup for Queryparser and deletion of useless 6XX$9
Rebased I hope I did not break anything... I saw an issue in 995$b/c, which are not indexed in the same way in facets and in ordinary index. I will send a message to the french list, as it is not linked with this patch. Mathieu
Patch pushed to master. Thanks Mathieu!
Mathieu, I tried to email you but it bounced. Your patches seem to affect MARC21 (and probably NORMARC) search behaviour. I think it is because you removed: Subject 1=21 su Subject -su-to Subject and probably because you only patched the UNIMARC biblio-koha-indexdefs.xml file. I'm reverting it, but I want it pushed again soon. So, please take a look at the output of this with your patches applied: prove t/db_dependent/QueryParser.t t/db_dependent/Search.t to find out. I think this kind of problem is one of the reasons we need better regression tests, lacking a lot in UNIMARC.
I will try to have a look at that. But not obvious, because I don't have a MARC21 install... Mathieu
Could you precise how the MARC21 was affected? No result at all? To much results? Issue with the facets maybe? I suppressed those 3 aliases -su-to Subject -su-geo Subject -su-ut Subject But I redefined them later in ccl.properties. So, MARC21 should not be broken What I see from MARC21 config file: - 650 field is indexed in "Subject" index only - "su-to" is used only for building the facets on 650 So the facets will probably be more precise than before. Is this the behavior you noticed? If so, it is not a bug. Mathieu
By the way, the facet code for unimarc could slighty be improved, but it can be done in an other bug...
The test t/Queryparser fails because of a wrong "+ authority-number:"
Created attachment 32882 [details] [review] [PATCH 4/4] Bug 9828 - Delete a duplicate line in queryparser.yaml the line "authority-number:" was wrongly duplicated This change should make t/Queryparser.t happy
I added a followup to silent t/Queryparser Could you tell me if it is OK, and what is the issue for MARC21 ? Mathieu
(In reply to mathieu saby from comment #50) > I added a followup to silent t/Queryparser > > Could you tell me if it is OK, and what is the issue for MARC21 ? Mathieu, what I noticed is that $ prove t/db_dependent/Search.t was failing with your patches. Will try with the new ones.
It would be nice to see this get into 3.18. I can confirm that t/db_dependent/Search.t was failing - hope it can be fixed.
Search.t is not failing anymore with my new patches ;-) Mathieu
I'm failing this one so maybe any MARC21/NORMARC dev can do the changes needed for them. I'll try to, but not sure i'll have the time.
Are you sure there is still a problem? I thought I had fixed it... Mathieu
(In reply to Tomás Cohen Arazi from comment #54) > I'm failing this one so maybe any MARC21/NORMARC dev can do the changes > needed for them. > I'll try to, but not sure i'll have the time. Tomas, I don't understand what you mean here. In my comment 30, I pointed that this patch ALSO fixed some MARC21/NORMAC errors, not that it need some work to fix MARC21/NORMAC errors ! So ? (back to passed QA because this patch is really useful for UNIMARC !)
Hello Paul & Tomas - Paul, since your comment 30 the patch has been changed several times... - Tomas, see my comment 53, i rewrote the patch, so there should not be any problem in unit tests... So, in my opinion this patch has been signed off, but need a NEW look by QA, especially to check if UT are broken, and if everything works well with MARC21 install. So, I think the best status is "Signed off" Mathieu
Created attachment 34537 [details] [review] Bug 9828 - Delete a duplicate line in queryparser.yaml the line "authority-number:" was wrongly duplicated This change should make t/Queryparser.t happy Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> t/Queryparser.t and t/Search.t pass.
I'm sorry, but it does break the tests, still. What I did was: - a clean dev install choosing MARC21 - loaded the default DB structure with the default data - Run $ prove t/db_dependent/Search.t => BOOM. I fail it, but plan to help on the MARC21 side as soon as I get back to the office from vacation.
(In reply to Tomás Cohen Arazi from comment #59) > I'm sorry, but it does break the tests, still. > > What I did was: > - a clean dev install choosing MARC21 > - loaded the default DB structure with the default data > - Run > $ prove t/db_dependent/Search.t > => BOOM. > > I fail it, but plan to help on the MARC21 side as soon as I get back to the > office from vacation. Sorry for that :/ I don't think I will have to fix that before long. Mathieu
Still relevant?