Description
Mathieu Saby
2013-11-05 15:36:35 UTC
Created attachment 22733 [details] [review] Improve UNIMARC biblio indexing This patch makes the following changes to UNIMARC biblio indexing : A. Changes to UNIMARC conf files 1. add comments to biblio-koha-indexdefs.xml 2. make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg 3. suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in unimarc format) 4. unindex some fields unneeded by most users (328, 207,230,210a,215) 6. replace Title-host with Host-item -- see bug 11119 7. index 208 in Material-Type -- see bug 11119 8. index 100 pos 8-9 and 9-12 in pubdate, but not in pubdate:s This change will only index 2 new fields, but not change the current sorting (made on 210$d) More work is needed for improving date management. 9. Index all subfields of note 318 and 327 in note index 10. Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation 11. Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) 12. Index 328 note in Dissertation-information as well as note B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated. 1. Apply the patch in a UNIMARC Koha running with DOM and ICU 2. copy src/etc/searchengine/queryparser.yaml into the main config directory of QP 2. copy src/etc/zebradb/* (all the tree) into the main config directory of Zebra 3. reindex biblios (rebuild_zebra.pl -r -b -x -v) 4. test note index : make some searches on 318$b or 327$b 5. test author index : make some searches on 314 field 6. test title index : make some searches on 304 and 327 field 7. test Dissertation-information index : make some searches on 328 field 8. In a record, put in the dates of 100 fields the values "1000" and "1001" ; try to search a book written in year 1000, you should find the record ; idem for year 1001 9. Regression test (with and without QP activated) : make some searches and sort by date. It should work (well or bad...) as before 10. Regression test (with and without QP activated) : make some searches on EAN (073). It should work as before something went wrong... So, you prefer to refine the code ? (In reply to Zeno Tajoli from comment #3) > So, you prefer to refine the code ? What do you mean by that? I made a mistake, and forgot to join the zebra xsl file. I'm testing and correcting the patch in a few minutes. And I am considering making a date sort on 100 coded field instead of 210$d. Mathieu Well, in fact it is strange to see 'In discussion' after only two comment with a patch attached. So I guess that you want to refine the patch for same motivation. Only this. We will try to sign it when you have finish. (In reply to Zeno Tajoli from comment #5) > Well, in fact it is strange to see 'In discussion' after only two comment > with a patch attached. > So I guess that you want to refine the patch for same motivation. > Only this. > We will try to sign it when you have finish. Oh, ok. So, yes, refining and fixing ;-) By the way, could you tell me if 359 field is used in Italy? No, tag 359 is not wide used in Italy. Bye Created attachment 22737 [details] [review] Improve UNIMARC biblio indexing This patch makes the following changes to UNIMARC biblio indexing : A. Changes to UNIMARC conf files 1. add comments to biblio-koha-indexdefs.xml 2. make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg 3. suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in unimarc format) 4. unindex some (sub)fields unneeded by most users (318, 207,230,210a,215, 4XXd) 5. change the way 308 field is indexed (no visible changes) 6. replace Title-host with Host-item -- see bug 11119 7. index 208 in Material-Type -- see bug 11119 8. index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n 9. index 100 pos 8-9 in pubdate:s instead of 210$d 10. Index all subfields of note 334 and 327 in note index 11. Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation 12. Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) 13. Index 328 note in Dissertation-information as well as note 14. Index 328$t in Title B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated. 1. Apply the patch in a UNIMARC Koha running with DOM and ICU 2. copy src/etc/searchengine/queryparser.yaml into the main config directory of QP 3. copy src/etc/zebradb/ccl.properties into the main config directory of Zebra 4. copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config directory of Zebra 5. reindex biblios (rebuild_zebra.pl -r -b -x -v) 6. test note index : make some searches on 334$b or 327$b 7. test author index : make some searches on 314 field 8. test title index : make some searches on 304 and 327 field, make a search on 328$t subfield 9. test Dissertation-information index : make some searches on 328 field 10. In a record, put in the dates of 100 fields the values "1000" (1st date) and "1001" (2d date) ; try to search a book written in year 1000, you should find the record ; idem for year 1001 11. make some searches and sort by date. It should work better as before, especially if you have values like "c2009" or "impr. 2010" in 210 field 12. Regression test : make some searches on several indexes, like EAN, etc. It should work as before Test 10-12 with and without Queryparser activated. Of course, to test search and sort by dates, you need to have full records, with dates in 100 field as well as 210 field. I forgot to copy the number of the bug in the last attachment ;-) But it can be tested. Mathieu *** Bug 11119 has been marked as a duplicate of this bug. *** *** Bug 11026 has been marked as a duplicate of this bug. *** Hi, how could I test Dissertation-information index ? It is not clear for me which option of 'Advanced Search' I need to use to search on Dissertation-information index Not in advanced search, but in normal search, try "Dissertation-information:xxxxx" Mathieu Created attachment 22750 [details] [review] Improve UNIMARC biblio indexing New patch, 1 ligne changed (I had forgotten to replace a Title-host in record.abs). M. Saby Kind Mathieu Saby, I install master 3.13.00.042 choosing ICU, DOM, unimarc, as required, and the Italian version of koha. A problem appears at 5-th step of the test plan, with this warning messagge: ./rebuild_zebra.pl -r -b -x -v Zebra configuration information ================================ Zebra biblio directory = /home/koha/bug-11202/var/lib/zebradb/biblios Zebra authorities directory = /home/koha/bug-11202/var/lib/zebradb/authorities Koha directory = /var/root-koha/bug-11202 BIBLIONUMBER in : 001$@ BIBLIOITEMNUMBER in : 090$a ================================ skipping authorities ==================== exporting biblio ==================== 1 Records exported: 1 ==================== REINDEXING zebra ==================== 16:35:52-07/11 zebraidx(9494) [warn] /home/koha/bug-11202/etc/zebradb/biblios/etc/dom-config.xml: could not parse xslt stylesheet /home/koha/bug-11202/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl 16:35:52-07/11 zebraidx(9494) [warn] /home/koha/bug-11202/etc/zebradb/biblios/etc/dom-config.xml: could not parse xslt stylesheet /home/koha/bug-11202/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl ==================== CLEANING ==================== Then the search answers erroneously zero findings. Can you help me? I pass the patch to "Failed QA" status. thank you for testing. I suppose you don't have only 1 record in your catalog? Could you tell me if italian unimarc has something different from standard ifla unimarc? Mathieu Oups, and I forgot : if you run rebuild_zebra without applying my patch, do you have some errors too ? Mathieu (In reply to mathieu saby from comment #19) > [...] if you run rebuild_zebra without applying my patch, do > you have some errors too ? > > Mathieu Kind Mathieu, I tested on stable and on master on 2013-11-07, and I had no error in both. Thanks, Paola (In reply to Paola Rossi from comment #20) > (In reply to mathieu saby from comment #19) > > [...] if you run rebuild_zebra without applying my patch, do > > you have some errors too ? > > > > Mathieu > > Kind Mathieu, I tested on stable and on master on 2013-11-07, and I had no > error in both. > Thanks, Paola Grazie mille. So, I will take a look at my patch. Mathieu (In reply to mathieu saby from comment #18) > thank you for testing. > I suppose you don't have only 1 record in your catalog? > Could you tell me if italian unimarc has something different from standard > ifla unimarc? > > Mathieu The error occurs indexing only one of these bibliographic records: Nilo, antico Nilo / Folco Quilici 1989 Il Nilo / Pier Domenico Boccardi 1903 Verde Nilo / Cesare Brandi 2001 in target Italian Bibliographic Service (SBN): opac.sbn.it:2100, database nopac These are three standard records. (In reply to Paola Rossi from comment #22) > (In reply to mathieu saby from comment #18) > > thank you for testing. > > I suppose you don't have only 1 record in your catalog? > > Could you tell me if italian unimarc has something different from standard > > ifla unimarc? > > > > Mathieu > > The error occurs indexing only one of these bibliographic records: > Nilo, antico Nilo / Folco Quilici 1989 > Il Nilo / Pier Domenico Boccardi 1903 > Verde Nilo / Cesare Brandi 2001 > > in target Italian Bibliographic Service (SBN): opac.sbn.it:2100, database > nopac > > These are three standard records. Funny. Maybe it is the "nilo" word which breaks indexing, but I don't understand how... Did you use rebuild_zebra_sliced to know that? Could you run rebuild_zebra on one of this record, with "-v -v" option instead of just "-v", and paste the results here please? I'm trying to reapply my patch, and to import your record with z3950 Mathieu
> Could you run rebuild_zebra on one of this record, with "-v -v" option
> instead of just "-v", and paste the results here please?
>
I've just sent it to your personal email.
(In reply to mathieu saby from comment #23) [...] > Could you run rebuild_zebra on one of this record, with "-v -v" option > instead of just "-v", and paste the results here please? > output of the command rebuild_zebra : http://paste.koha-community.org/260 Thanks. indeed, it's looong ;-) I crashed my etc/ dir with a wrong manipulation, so, still a little work to do have an answer... I rebuilt biblio-zebra-indexdefs.xml. 2 lines were missing in previous patch, I don't know why. Patch coming Created attachment 22973 [details] [review] [PATCH 2] Bug 11202 : Fix biblio-zebra-indexdefs.xsl 2 lines were missing in biblio-zebra-indexdefs.xsl, so Zebra could not rebuild the index. This patch should fix that. Apply this patch with main patch, and follow the test plan of main patch. Thanks, Mathieu. Now Zebra rebuilds the index. Step 10: 1) In a record, I put in the dates of 100 fields the values "1000" (1st date, 9-12) and "1001" (2d date, 13-16), and nothing in 210$d. Searching (in advanced search) year 1000, I don't find the record; neither searching 1001. The answer is zero findings for 'yr,st-numeric: 1000'. Searching "yr:1000" in simple search, I don't find the record. The answer is zero findings for 'yr:1000'. 2) In a record, I put in the dates of 100 fields the values "1000" (1st date, 9-12) and "1001" (2d date, 13-16), and the value "1000" in 210$d. Searching (in advanced search) year 1000, I find the record. Searching "yr:1000" in simple search, I don't find the record. The answer is zero findings for 'yr:1000'. 3) In a record, I put in the dates of 100 fields the values "1903" (1st date, 9-12) and "1904" (2d date, 13-16), and nothing in 210$d. Searching (in advanced search or in simple search) year 1903, I don't find the record (like 1). So I pass the patch to "Failed QA" status. (In reply to Paola Rossi from comment #29) > Thanks, Mathieu. Now Zebra rebuilds the index. > > Step 10: > 1) In a record, I put in the dates of 100 fields the values "1000" (1st > date, 9-12) and "1001" (2d date, 13-16), and nothing in 210$d. > > Searching (in advanced search) year 1000, I don't find the record; neither > searching 1001. The answer is zero findings for 'yr,st-numeric: 1000'. > Searching "yr:1000" in simple search, I don't find the record. The answer is > zero findings for 'yr:1000'. > > 2) In a record, I put in the dates of 100 fields the values "1000" (1st > date, 9-12) and "1001" (2d date, 13-16), and the value "1000" in 210$d. > Searching (in advanced search) year 1000, I find the record. > Searching "yr:1000" in simple search, I don't find the record. The answer is > zero findings for 'yr:1000'. > > 3) In a record, I put in the dates of 100 fields the values "1903" (1st > date, 9-12) and "1904" (2d date, 13-16), and nothing in 210$d. > Searching (in advanced search or in simple search) year 1903, I don't find > the record (like 1). > > So I pass the patch to "Failed QA" status. Thank you for testing again. In fact searching dates with yr index in simple search does not work in master. What is strange is that you don't find the record in advanced search if the date is only in 100 and not in 210. I'm taking a look. Mathieu The year index in dropdown list does not function well. The request sent to zebra is broken (in master) I've made a ticket to our vendor for that. For searching by date, the only way is to use the "filters" that appears in advanced search. These filters can be used even if they are no other search criteria (so they are not really filters ;-) I've proposed to our vendor to suppress the year index from dropdown list. It's unusable on UX point of view, and it is broken... We should use the filters. Well, now, I'm getting back to the bug itself ;-) Mathieu I modified a record with : 100 _a20001109d10251026k||y0frey50 ba => 1st date : 1025 2d date : 1026 210 _aParis _cCahiers du cinéma _d2000 => date in 210 : 2000 I applied the patch, copy the conf files, and rebuild zebra. Search with limit=1025: ......./cgi-bin/koha/catalogue/search.pl?idx=kw&q=&op=and&idx=kw&q=&op=and&idx=kw&q=&limit-yr=1025&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&sort_by=relevance => 1 result Search with limit=1026: ......./cgi-bin/koha/catalogue/search.pl?idx=kw&q=&op=and&idx=kw&q=&op=and&idx=kw&q=&limit-yr=1026&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&sort_by=relevance => 1 result Search with limit=2000: http://pro.koha-msaby/cgi-bin/koha/catalogue/search.pl?idx=kw&q=&op=and&idx=kw&q=&op=and&idx=kw&q=&limit-yr=2000&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&sort_by=relevance => several results, and my record belongs to them. So the value of 210$d was searched. ==> For me, it works. Could you try to make a test with the same steps ? Mathieu oups, I made some mistakes. Fields 411 and others fields 4xx are missing. please do not test while it is fixed! Created attachment 23041 [details] [review] [PATCH 3] Fix indexing of 411, 412, 511 and index new fields 506,507,576,577 * Due to a mistake, 411, 412 and 511 fields were not indexed by the main patch. * This patch also index new fields added in UNIMARC format in 2012 to make it more FRBR compliant : 506, 507, 576, 577. These fields are experimental in most UNIMARC countries. They are not currently used in France but will probably be in some months or years. * This patch also reorganizes definitions of 5XX fields and adds comments, without functional changes To test: - take a record with some data in 411, 412 or 511 fields - reindex - check the record can be retrieved by searching the data of those fields - if you want, test 506,507,576,577 fields (as they are mostly experimental, and as index definition is trivial, this test can be skipped) To test this bug: apply the 3 patches. Look at test plan of 1st patch and 3rd patch (whitch only adds a few indexes), and comments 30 to 32 if you have trouble for testing the new way of indexing dates. Mathieu Created attachment 23108 [details] [review] [PATCH] Bug 11202 - Improve UNIMARC biblio indexing All previous patches (non signed off) squashed, with little changes Test plan updated This patch makes the following changes to UNIMARC biblio indexing : A. Changes to record.abs and biblio-koha*/biblio-zebra* conf files 1. Changes with no expected functional effects * add comments to biblio-koha-indexdefs.xml * make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg * reorganizes definitions of 5XX fields * suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in UNIMARC format) * change the way 308 field is indexed 2. Changes with functional effects * index new fields added in UNIMARC format in 2012 to make it more FRBR compliant : 506, 507, 576, 577. These fields are experimental in most UNIMARC countries. They are not currently used in France but will probably be in some months or years. * unindex some (sub)fields unneeded by users (207,230,215,4XXd) * index 208 in Material-Type -- see bug 11119 * index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n * index 100 pos 8-9 in pubdate:s instead of 210$d * Index all subfields of note 334 and 327 in note index * Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation * Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) * Index 328 note in Dissertation-information as well as note * Index 328$t in Title * replace Title-host with Host-item -- see bug 11119 B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated... 1. Prepare Koha : * Apply the patch in a UNIMARC Koha running with DOM and ICU * copy src/etc/searchengine/queryparser.yaml into the main config directory of QP * copy src/etc/zebradb/ccl.properties into the main config directory of Zebra * copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config directory of Zebra * reindex biblios (rebuild_zebra.pl -r -b -x -v) 2. test note index : make some searches on 334$b or 327$b 3. test author index : make some searches on 314 field 4. test title index : make some searches on 304 and 327 field, make a search on 328$t subfield 5. test Dissertation-information index : make some searches on 328 field 6. test 506,507,576,577 fields (as they are mostly experimental, and as index definition is trivial, this test can be skipped) 7. In a record, put in the dates of 100 fields the values "1000" (1st date) and "1001" (2d date) ; try to search a book written in year 1000, you should find the record ; idem for year 1001 8. make some searches and sort by date. It should work better as before, especially if you have values like "c2009" or "impr. 2010" in 210 field 9. Regression test : make some searches on several indexes, like EAN, etc. It should work as before Test 2-9 with and without Queryparser activated. Of course, to test search and sort by dates, you need to have full records, with dates in 100 field as well as 210 field. To search by date, you need to use advanced search, click on "More" and use date filter. Do not use "yr" index in simple search, it does not work. Do not use "date" index in dropdown list, it does not work either. Patch applied cleanly, go forth and signoff Hello, Mathieu. I apply the patch against master 3.15.00.002. Rebuilding zebra indexes, an error appears: <http://paste.koha-community.org/10> Rebuilding zebra indexes against master without patching, the error doesn't appear. The two notices, coming from opac.sbn.it:2100, DB nopac, are: Nilo, antico Nilo / Folco Quilici Il Nilo / Pier Domenico Boccardi So I pass the patch to "Failed QA" status. Created attachment 23238 [details] [review] Bug 11202 - Improve UNIMARC biblio indexing Sorry :( It's fixed. Mathieu Thanks, Mathieu. Now rebuilding zebra indexes is OK. I turn "UseQueryParser" preference to the second option "Try". As required at step 5, I submit: Dissertation-information: XXXX in homepage, selecting tab "Search the catalog". Koha answers: Home › Catalogo › Ricerca di 'pqf=@and @and @or @or @attr 1=1016 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=20 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=34 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc"' Nessun risultato trovato Nessun risultato rispondente alla tua ricerca per 'pqf=@and @and @or @or @attr 1=1016 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=20 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=34 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc"'. So I pass the patch to "Failed QA" status. Oh! I did not found documentation on how to update Queryparser conf, so maybe I did the things wrongly. I should maybe ask Jared... Did you try to use this new index without the QueryParser? And can you say me if searches on other indexes with the Queryparser activated are working well on your instance without my patch? Mathieu Hello Paola It's a bug of the queryparser. I've created a new bug: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11353 So, you must use only lowercases in indexes : "dissertation-information" and NOT "Dissertation-information". I update test plan. Mathieu Created attachment 23348 [details] [review] Bug 11202 - Improve UNIMARC biblio indexing Fixing test plan Back to need sign off Created attachment 23579 [details] [review] Improve UNIMARC biblio indexing I test the patch against master 3.15.00.004. Simple sort is ok when default sort is not set to relevance-asc (see bug 11255). I sign off the patch. Thank you Paola! Created attachment 23830 [details] [review] [PATCH] Bug 11202 - Improve UNIMARC biblio indexing Rebased I am quite torn about this patch. I feel it's doing way too many things in one single change. It's really hard to test that way and I am not super confident about doing it, but going to give it a try. Just ask, and I will split it into several patches. Mathieu I will have some comments soon. I am having problems testing pubdate without QueryParser. For the QueryParser pubdate(xxxx) seems to work alright. But pubdate= or pubdate: doesn't seem to work. Any ideas about that? (In reply to Katrin Fischer from comment #50) > I am having problems testing pubdate without QueryParser. For the > QueryParser pubdate(xxxx) seems to work alright. But pubdate= or pubdate: > doesn't seem to work. Any ideas about that? Yes, dates are searched with parentheses not = or : with QP. Ah sorry, I was not clear. When the QueryParser is turned off, none of the syntaxes seem to work. Neither for DOM nor GRS1. What I did after applying the patch: cp etc/searchengine/queryparser.yaml ../koha-devU/etc/searchengine/queryparser.yaml cp etc/zebradb/ccl.properties ../koha-devU/etc/zebradb/ccl.properties cp etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml cp etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl cp etc/zebradb/marc_defs/unimarc/biblios/record.abs ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/record.abs Completely reindexed the bibliographic data using -b -v -r -x Full reindex and searching for Dissertation-information worked nicely in DOM and GRS1. But... 1) In general I am having problems with pubdate and acqdate searches in UNIMARC compared to their MARC21 equivalents Mathieu, could you please take a look? Syntaxes I tried: pubdate=yyyy, pubdate:yyyy With QueryParser: pubdate(yyyy), acqdate(yyyy) 2) It would be nice if Dissertation-information could also be added for MARC21 on a seaparate bug later on. 3) In general I am a bit worried about removing indexes. I can see, why some subfields are not making much sense to index, I also have read the comments on bug 11119, but some of the changes might be debatable. For example the following change could break functionality some libraries rely on. Maybe it would not hurt to keep those around? -#FIXME Fields 603, 630, 631, 632, 633, 634, 635, 636, 646 are not defined in Unimarc, but may be used by some libraries. -melm 603 Subject,Subject:p -melm 630 Subject,Subject:p -melm 631 Subject,Subject:p -melm 632 Subject,Subject:p -melm 633 Subject,Subject:p -melm 634 Subject,Subject:p -melm 635 Subject,Subject:p -melm 636 Subject,Subject:p -melm 646 Subject,Subject:p Failing for now until 1) can be resolved. > > 1) In general I am having problems with pubdate and acqdate searches in > UNIMARC compared to their MARC21 equivalents Mathieu, could you please take > a look? Syntaxes I tried: > pubdate=yyyy, pubdate:yyyy > With QueryParser: pubdate(yyyy), acqdate(yyyy) I will take a look > 3) In general I am a bit worried about removing indexes. I can see, why > some subfields are not making much sense to index, I also have read > the comments on bug 11119, but some of the changes might be debatable. > For example the following change could break functionality some > libraries rely on. Maybe it would not hurt to keep those around? > > -#FIXME Fields 603, 630, 631, 632, 633, 634, 635, 636, 646 are not defined > in Unimarc, but may be used by some libraries. I don't know who added these indexes and why, but they were never used in Unimarc, so it was clearly a mistake to put them in standard configuration files. Maybe some french library made one day a very specific developpment using these fields (I'm thinking of Frantiq network, specialised in Archeology). Or maybe it is just a mistake... I will send a message to french list to warn everybody that if a library want to index those fields in 3.14 they need to edit the config files before reindexing. Mathieu Mathieu I made some tests. Without my patch: 1. with QP enable pubdate(1925) is working pubdate:1925 is working 2. with QP disable pubdate(1925) is not working pubdate:1925 is not working With my patch: Exactly the same behavior So, for me, the patch is correct, regarding pubdate index. It is not possible to build a query with pubdate without QP, but it was still the case in master. Mathieu Oh, that explains. It seems it's possible with MARC21 - what are we missing to make it work? I was always said not to use pubdate index like that, so the issue is old, but maybe unimarc specific. It is working only when you use a date in the advanced search filter see comment 30 By the way: If you or Galen ask it, I can try to split the patch, with non functional changes (comments...) in a separate patch. But I prefer not to, if it is not required, because - I could do a mistake and break the patch, - I won't have time to work on Koha this week. I think the pubdate index does not block this patch then - I will take another look at it later (if noone beats me to it :) ). Maybe you could file a bug for the pubdate index in UNIMARC and link to this patch? Bug 11585 created All looks good to me. I had a discussion with Paul and the only think we found is that the Dissertation-information is not defined for MARC21 (but the index is defined in the ccl.properties). See bug 11620. Marked as Passed QA. Created attachment 24751 [details] [review] Bug 11202 - Improve UNIMARC biblio indexing This patch makes the following changes to UNIMARC biblio indexing : A. Changes to UNIMARC conf files 1. add comments to biblio-koha-indexdefs.xml 2. make biblio-koha-indexdefs.xml more compact by grouping some declarations Ex : 200$f and 200$g => one declaration for 200$fg 3. suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in unimarc format) 4. unindex some (sub)fields unneeded by most users (318, 207,230,210a,215, 4XXd) 5. change the way 308 field is indexed (no visible changes) 6. replace Title-host with Host-item -- see bug 11119 7. index 208 in Material-Type -- see bug 11119 8. index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n 9. index 100 pos 8-9 in pubdate:s instead of 210$d 10. Index all subfields of note 334 and 327 in note index 11. Index 304 and 327 in title index as well as note index 327 can contain a list of titles included in a work 304 can contain the title of the original work in case of a translation 12. Index 314 in author index as well as note index 314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author) 13. Index 328 note in Dissertation-information as well as note 14. Index 328$t in Title B. Changes to ccl.properties : 1. add a new index Dissertation-information (1056) 2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes) C. Changes to Search.pm 1. add Dissertation-information and suppress Title-host and UPC D. Changes to QP config file queryparser.yaml 1. add Dissertation-information 2 fix EAN, pubdate and acqdate Test plan : If you cannot test in GRS1, test only in DOM, as GRS will be deprecated. 1. Apply the patch in a UNIMARC Koha running with DOM and ICU 2. copy src/etc/searchengine/queryparser.yaml into the main config directory of QP 3. copy src/etc/zebradb/ccl.properties into the main config directory of Zebra 4. copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config directory of Zebra 5. reindex biblios (rebuild_zebra.pl -r -b -x -v) 6. test note index : make some searches on 334$b or 327$b 7. test author index : make some searches on 314 field 8. test title index : make some searches on 304 and 327 field, make a search on 328$t subfield 9. test dissertation-information index : make some searches on 328 field 10. In a record, put in the dates of 100 fields the values "1000" (1st date) and "1001" (2d date) ; try to search a book written in year 1000, you should find the record ; idem for year 1001 11. make some searches and sort by date. It should work better as before, especially if you have values like "c2009" or "impr. 2010" in 210 field 12. Regression test : make some searches on several indexes, like EAN, etc. It should work as before Test 10-12 with and without Queryparser activated. Be careful: with Queryparser activated, the index names (title, dissertation-information...) must be entered in lowercase only. Of course, to test search and sort by dates, you need to have full records, with dates in 100 field as well as 210 field. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Hi Jonathan, thx for taking a look - my tests were also looking good so far, but I was hesitating to pass as I don't know UNIMARC really well. I think our libraries would like an index for dissertation-information - this could be another bug. From my review of this patch, five tests in t/QueryParser.t fail. At first glance, I think the problem is with the tests, though. Created attachment 25005 [details] [review] Bug 11202: FIX UT for QueryParser.t Test plan: prove t/QueryParser.t Pushed to master. Thanks, Mathieu! Pushed to 3.14.x, will be in 3.14.07 |