Bug 11202 - Improve UNIMARC biblio indexing
Summary: Improve UNIMARC biblio indexing
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P4 enhancement (vote)
Assignee: Mathieu Saby
QA Contact: Testopia
URL:
Keywords:
: 11026 11119 (view as bug list)
Depends on:
Blocks: 8962
  Show dependency treegraph
 
Reported: 2013-11-05 15:36 UTC by Mathieu Saby
Modified: 2014-12-07 20:05 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Improve UNIMARC biblio indexing (96.93 KB, patch)
2013-11-05 15:37 UTC, Mathieu Saby
Details | Diff | Splinter Review
Improve UNIMARC biblio indexing (142.34 KB, patch)
2013-11-05 16:53 UTC, Mathieu Saby
Details | Diff | Splinter Review
Improve UNIMARC biblio indexing (142.34 KB, patch)
2013-11-06 14:00 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 2] Bug 11202 : Fix biblio-zebra-indexdefs.xsl (1.80 KB, patch)
2013-11-15 18:04 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH 3] Fix indexing of 411, 412, 511 and index new fields 506,507,576,577 (41.17 KB, patch)
2013-11-19 17:11 UTC, Mathieu Saby
Details | Diff | Splinter Review
[PATCH] Bug 11202 - Improve UNIMARC biblio indexing (142.35 KB, patch)
2013-11-22 21:08 UTC, Mathieu Saby
Details | Diff | Splinter Review
Bug 11202 - Improve UNIMARC biblio indexing (142.30 KB, patch)
2013-12-02 15:13 UTC, Mathieu Saby
Details | Diff | Splinter Review
Bug 11202 - Improve UNIMARC biblio indexing (142.42 KB, patch)
2013-12-07 14:04 UTC, Mathieu Saby
Details | Diff | Splinter Review
Improve UNIMARC biblio indexing (142.44 KB, patch)
2013-12-16 17:10 UTC, Paola Rossi
Details | Diff | Splinter Review
[PATCH] Bug 11202 - Improve UNIMARC biblio indexing (142.79 KB, patch)
2013-12-27 14:58 UTC, Mathieu Saby
Details | Diff | Splinter Review
Bug 11202 - Improve UNIMARC biblio indexing (142.81 KB, patch)
2014-01-27 14:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11202: FIX UT for QueryParser.t (3.85 KB, patch)
2014-02-03 12:18 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Saby 2013-11-05 15:36:35 UTC

    
Comment 1 Mathieu Saby 2013-11-05 15:37:36 UTC Comment hidden (obsolete)
Comment 2 Mathieu Saby 2013-11-05 15:45:07 UTC
something went wrong...
Comment 3 Zeno Tajoli 2013-11-05 16:07:24 UTC
So, you prefer to refine the code ?
Comment 4 Mathieu Saby 2013-11-05 16:22:52 UTC
(In reply to Zeno Tajoli from comment #3)
> So, you prefer to refine the code ?

What do you mean by that?

I made a mistake, and forgot to join the zebra xsl file.
I'm testing and correcting the patch in a few minutes.

And I am considering making a date sort on 100 coded field instead of 210$d.

Mathieu
Comment 5 Zeno Tajoli 2013-11-05 16:34:41 UTC
Well, in fact it is strange to see 'In discussion' after only two comment with a patch attached.
So I guess that you want to refine the patch for same motivation.
Only this.
We will try to sign it when you have finish.
Comment 6 Mathieu Saby 2013-11-05 16:39:05 UTC
(In reply to Zeno Tajoli from comment #5)
> Well, in fact it is strange to see 'In discussion' after only two comment
> with a patch attached.
> So I guess that you want to refine the patch for same motivation.
> Only this.
> We will try to sign it when you have finish.

Oh, ok. So, yes, refining and fixing ;-)
Comment 7 Mathieu Saby 2013-11-05 16:42:04 UTC
By the way, could you tell me if 359 field is used in Italy?
Comment 8 Zeno Tajoli 2013-11-05 16:45:15 UTC
No, tag 359 is not wide used in Italy.
Bye
Comment 9 Mathieu Saby 2013-11-05 16:53:13 UTC Comment hidden (obsolete)
Comment 10 Mathieu Saby 2013-11-05 16:54:00 UTC
I forgot to copy the number of the bug in the last attachment ;-) But it can be tested.

Mathieu
Comment 11 Mathieu Saby 2013-11-05 18:08:22 UTC
*** Bug 11119 has been marked as a duplicate of this bug. ***
Comment 12 Mathieu Saby 2013-11-05 18:09:08 UTC
*** Bug 11026 has been marked as a duplicate of this bug. ***
Comment 13 Zeno Tajoli 2013-11-05 18:58:33 UTC
Hi,

how could I test Dissertation-information index ?
It is not clear for me which option of 'Advanced Search' I need to use
to search on Dissertation-information index
Comment 14 Mathieu Saby 2013-11-05 20:09:22 UTC
Not in advanced search, but in normal search, try
"Dissertation-information:xxxxx"

Mathieu
Comment 15 Mathieu Saby 2013-11-06 14:00:11 UTC Comment hidden (obsolete)
Comment 16 Paola Rossi 2013-11-07 15:53:55 UTC
Kind Mathieu Saby, I install master 3.13.00.042 choosing ICU, DOM, unimarc, as required, and the Italian version of koha. A problem appears at 5-th step of the test plan, with this warning messagge:

./rebuild_zebra.pl -r -b -x -v
Zebra configuration information
================================
Zebra biblio directory      = /home/koha/bug-11202/var/lib/zebradb/biblios
Zebra authorities directory = /home/koha/bug-11202/var/lib/zebradb/authorities
Koha directory              = /var/root-koha/bug-11202
BIBLIONUMBER in :     001$@
BIBLIOITEMNUMBER in : 090$a
================================
skipping authorities
====================
exporting biblio
====================
1
Records exported: 1
====================
REINDEXING zebra
====================
16:35:52-07/11 zebraidx(9494) [warn] /home/koha/bug-11202/etc/zebradb/biblios/etc/dom-config.xml: could not parse xslt stylesheet /home/koha/bug-11202/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
16:35:52-07/11 zebraidx(9494) [warn] /home/koha/bug-11202/etc/zebradb/biblios/etc/dom-config.xml: could not parse xslt stylesheet /home/koha/bug-11202/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
====================
CLEANING
====================

Then the search answers erroneously zero findings.
Can you help me?
Comment 17 Paola Rossi 2013-11-15 13:27:49 UTC
I pass the patch to "Failed QA" status.
Comment 18 Mathieu Saby 2013-11-15 13:34:41 UTC
thank you for testing.
I suppose you don't have only 1 record in your catalog?
Could you tell me if italian unimarc has something different from standard ifla unimarc?

Mathieu
Comment 19 Mathieu Saby 2013-11-15 13:35:21 UTC
Oups, and I forgot : if you run rebuild_zebra without applying my patch, do you have some errors too ?

Mathieu
Comment 20 Paola Rossi 2013-11-15 13:43:42 UTC
(In reply to mathieu saby from comment #19)
> [...] if you run rebuild_zebra without applying my patch, do
> you have some errors too ?
> 
> Mathieu

Kind Mathieu, I tested on stable and on master on 2013-11-07, and I had no error in both.
Thanks, Paola
Comment 21 Mathieu Saby 2013-11-15 15:03:31 UTC
(In reply to Paola Rossi from comment #20)
> (In reply to mathieu saby from comment #19)
> > [...] if you run rebuild_zebra without applying my patch, do
> > you have some errors too ?
> > 
> > Mathieu
> 
> Kind Mathieu, I tested on stable and on master on 2013-11-07, and I had no
> error in both.
> Thanks, Paola

Grazie mille. So, I will take a look at my patch.

Mathieu
Comment 22 Paola Rossi 2013-11-15 16:41:11 UTC
(In reply to mathieu saby from comment #18)
> thank you for testing.
> I suppose you don't have only 1 record in your catalog?
> Could you tell me if italian unimarc has something different from standard
> ifla unimarc?
> 
> Mathieu
 
The error occurs indexing only one of these bibliographic records:
Nilo, antico Nilo / Folco Quilici 1989
Il Nilo / Pier Domenico Boccardi 1903
Verde Nilo / Cesare Brandi 2001

in target Italian Bibliographic Service (SBN): opac.sbn.it:2100, database nopac 

These are three standard records.
Comment 23 Mathieu Saby 2013-11-15 16:45:23 UTC
(In reply to Paola Rossi from comment #22)
> (In reply to mathieu saby from comment #18)
> > thank you for testing.
> > I suppose you don't have only 1 record in your catalog?
> > Could you tell me if italian unimarc has something different from standard
> > ifla unimarc?
> > 
> > Mathieu
>  
> The error occurs indexing only one of these bibliographic records:
> Nilo, antico Nilo / Folco Quilici 1989
> Il Nilo / Pier Domenico Boccardi 1903
> Verde Nilo / Cesare Brandi 2001
> 
> in target Italian Bibliographic Service (SBN): opac.sbn.it:2100, database
> nopac 
> 
> These are three standard records.

Funny. Maybe it is the "nilo" word which breaks indexing, but I don't understand how...
Did you use rebuild_zebra_sliced to know that?
Could you run rebuild_zebra on one of this record, with "-v -v" option instead of just "-v", and paste the results here please?

I'm trying to reapply my patch, and to import your record with z3950

Mathieu
Comment 24 Paola Rossi 2013-11-15 17:04:07 UTC
> Could you run rebuild_zebra on one of this record, with "-v -v" option
> instead of just "-v", and paste the results here please?
> 

I've just sent it to your personal email.
Comment 25 Paola Rossi 2013-11-15 17:09:03 UTC
(In reply to mathieu saby from comment #23)
[...]
> Could you run rebuild_zebra on one of this record, with "-v -v" option
> instead of just "-v", and paste the results here please?
> 

output of the command rebuild_zebra :
http://paste.koha-community.org/260
Comment 26 Mathieu Saby 2013-11-15 17:37:53 UTC
Thanks. indeed, it's looong ;-)
I crashed my etc/ dir with a wrong manipulation, so, still a little work to do have an answer...
Comment 27 Mathieu Saby 2013-11-15 18:01:44 UTC
I rebuilt biblio-zebra-indexdefs.xml.
2 lines were missing in previous patch, I don't know why.
Patch coming
Comment 28 Mathieu Saby 2013-11-15 18:04:05 UTC Comment hidden (obsolete)
Comment 29 Paola Rossi 2013-11-19 12:06:22 UTC
Thanks, Mathieu. Now Zebra rebuilds the index.

Step 10:
1) In a record, I put in the dates of 100 fields the values "1000" (1st date, 9-12) and "1001" (2d date, 13-16), and nothing in 210$d. 

Searching (in advanced search) year 1000, I don't find the record; neither searching 1001. The answer is zero findings for 'yr,st-numeric: 1000'.
Searching "yr:1000" in simple search, I don't find the record. The answer is zero findings for 'yr:1000'.

2) In a record, I put in the dates of 100 fields the values "1000" (1st date, 9-12) and "1001" (2d date, 13-16), and the value "1000" in 210$d. 
Searching (in advanced search) year 1000, I find the record.
Searching "yr:1000" in simple search, I don't find the record. The answer is zero findings for 'yr:1000'.

3) In a record, I put in the dates of 100 fields the values "1903" (1st date, 9-12) and "1904" (2d date, 13-16), and nothing in 210$d.
Searching (in advanced search or in simple search) year 1903, I don't find the record (like 1).

So I pass the patch to "Failed QA" status.
Comment 30 Mathieu Saby 2013-11-19 13:18:49 UTC
(In reply to Paola Rossi from comment #29)
> Thanks, Mathieu. Now Zebra rebuilds the index.
> 
> Step 10:
> 1) In a record, I put in the dates of 100 fields the values "1000" (1st
> date, 9-12) and "1001" (2d date, 13-16), and nothing in 210$d. 
> 
> Searching (in advanced search) year 1000, I don't find the record; neither
> searching 1001. The answer is zero findings for 'yr,st-numeric: 1000'.
> Searching "yr:1000" in simple search, I don't find the record. The answer is
> zero findings for 'yr:1000'.
> 
> 2) In a record, I put in the dates of 100 fields the values "1000" (1st
> date, 9-12) and "1001" (2d date, 13-16), and the value "1000" in 210$d. 
> Searching (in advanced search) year 1000, I find the record.
> Searching "yr:1000" in simple search, I don't find the record. The answer is
> zero findings for 'yr:1000'.
> 
> 3) In a record, I put in the dates of 100 fields the values "1903" (1st
> date, 9-12) and "1904" (2d date, 13-16), and nothing in 210$d.
> Searching (in advanced search or in simple search) year 1903, I don't find
> the record (like 1).
> 
> So I pass the patch to "Failed QA" status.


Thank you for testing again.
In fact searching dates with yr index in simple search does not work in master.
What is strange is that you don't find the record in advanced search if the date is only in 100 and not in 210.
I'm taking a look.


Mathieu
Comment 31 Mathieu Saby 2013-11-19 14:03:09 UTC
The year index in dropdown list does not function well. The request sent to zebra is broken (in master)
I've made a ticket to our vendor for that.

For searching by date, the only way is to use the "filters" that appears in advanced search.
These filters can be used even if they are no other search criteria (so they are not really filters ;-)

I've proposed to our vendor to suppress the year index from dropdown list. It's unusable on UX point of view, and it is broken...
We should use the filters.

Well, now, I'm getting back to the bug itself ;-)

Mathieu
Comment 32 Mathieu Saby 2013-11-19 16:05:42 UTC
I modified a record with :

100    _a20001109d10251026k||y0frey50      ba
=> 1st date : 1025
2d date : 1026

210    _aParis
       _cCahiers du cinéma
       _d2000
=> date in 210 : 2000

I applied the patch, copy the conf files, and rebuild zebra.

Search with limit=1025:
......./cgi-bin/koha/catalogue/search.pl?idx=kw&q=&op=and&idx=kw&q=&op=and&idx=kw&q=&limit-yr=1025&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&sort_by=relevance

=> 1 result

Search with limit=1026:
......./cgi-bin/koha/catalogue/search.pl?idx=kw&q=&op=and&idx=kw&q=&op=and&idx=kw&q=&limit-yr=1026&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&sort_by=relevance

=> 1 result

Search with limit=2000:
http://pro.koha-msaby/cgi-bin/koha/catalogue/search.pl?idx=kw&q=&op=and&idx=kw&q=&op=and&idx=kw&q=&limit-yr=2000&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&limit=&sort_by=relevance

=> several results, and my record belongs to them. So the value of 210$d was searched.


==> For me, it works.
Could you try to make a test with the same steps ?

Mathieu
Comment 33 Mathieu Saby 2013-11-19 16:22:46 UTC
oups, I made some mistakes. Fields 411 and others fields 4xx are missing.
please do not test while it is fixed!
Comment 34 Mathieu Saby 2013-11-19 17:11:52 UTC Comment hidden (obsolete)
Comment 35 Mathieu Saby 2013-11-19 17:13:38 UTC
To test this bug: apply the 3 patches.
Look at test plan of 1st patch and 3rd patch (whitch only adds a few indexes), and comments 30 to 32 if you have trouble for testing the new way of indexing dates.

Mathieu
Comment 36 Mathieu Saby 2013-11-22 21:08:12 UTC Comment hidden (obsolete)
Comment 37 I'm just a bot 2013-11-22 22:49:31 UTC
Patch applied cleanly, go forth and signoff
Comment 38 Paola Rossi 2013-12-02 14:28:32 UTC
Hello, Mathieu.

I apply the patch against master 3.15.00.002.

Rebuilding zebra indexes, an error appears:
<http://paste.koha-community.org/10>

Rebuilding zebra indexes against master without patching, the error doesn't appear.

The two notices, coming from opac.sbn.it:2100, DB nopac, are:
Nilo, antico Nilo / Folco Quilici
Il Nilo / Pier Domenico Boccardi

So I pass the patch to "Failed QA" status.
Comment 39 Mathieu Saby 2013-12-02 15:13:27 UTC Comment hidden (obsolete)
Comment 40 Paola Rossi 2013-12-04 10:13:03 UTC
Thanks, Mathieu. Now rebuilding zebra indexes is OK.

I turn "UseQueryParser" preference to the second option "Try". 

As required at step 5, I submit:

Dissertation-information: XXXX

in homepage, selecting tab "Search the catalog".

Koha answers:

 Home › Catalogo › Ricerca di 'pqf=@and @and @or @or @attr 1=1016 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=20 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=34 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc"'
Nessun risultato trovato

Nessun risultato rispondente alla tua ricerca per 'pqf=@and @and @or @or @attr 1=1016 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=20 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @attr 9=34 @attr 2=102 @attr 5=1 @attr 4=6 "Dissertation-information:" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 4=6 "XXXX" @or @or @attr 1=1016 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=20 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc" @attr 9=34 @attr 2=102 @attr 5=1 @attr 5=1 @attr 5=1 @attr 4=6 "#relevance_asc"'.

So I pass the patch to "Failed QA" status.
Comment 41 Mathieu Saby 2013-12-04 13:40:41 UTC
Oh!

I did not found documentation on how to update Queryparser conf, so maybe I did the things wrongly.
I should maybe ask Jared...

Did you try to use this new index without the QueryParser?

And can you say me if searches on other indexes with the Queryparser activated are working well on your instance without my patch?


Mathieu
Comment 42 Mathieu Saby 2013-12-07 14:00:09 UTC
Hello Paola
It's a bug of the queryparser. I've created a new bug: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11353

So, you must use only lowercases in indexes : "dissertation-information" and NOT "Dissertation-information".

I update test plan.

Mathieu
Comment 43 Mathieu Saby 2013-12-07 14:04:10 UTC Comment hidden (obsolete)
Comment 44 Paola Rossi 2013-12-16 17:10:28 UTC Comment hidden (obsolete)
Comment 45 Mathieu Saby 2013-12-16 23:15:53 UTC
Thank you Paola!
Comment 46 Mathieu Saby 2013-12-27 14:58:51 UTC Comment hidden (obsolete)
Comment 47 Katrin Fischer 2014-01-18 08:58:34 UTC
I am quite torn about this patch. I feel it's doing way too many things in one single change. It's really hard to test that way and I am not super confident about doing it, but going to give it a try.
Comment 48 Mathieu Saby 2014-01-18 10:26:25 UTC
Just ask, and I will split it into several patches.

Mathieu
Comment 49 Katrin Fischer 2014-01-18 11:13:20 UTC
I will have some comments soon.
Comment 50 Katrin Fischer 2014-01-18 11:35:08 UTC
I am having problems testing pubdate without QueryParser. For the QueryParser pubdate(xxxx) seems to work alright. But pubdate= or pubdate: doesn't seem to work. Any ideas about that?
Comment 51 Jared Camins-Esakov 2014-01-18 12:50:54 UTC
(In reply to Katrin Fischer from comment #50)
> I am having problems testing pubdate without QueryParser. For the
> QueryParser pubdate(xxxx) seems to work alright. But pubdate= or pubdate:
> doesn't seem to work. Any ideas about that?

Yes, dates are searched with parentheses not = or : with QP.
Comment 52 Katrin Fischer 2014-01-18 13:00:32 UTC
Ah sorry, I was not clear. When the QueryParser is turned off, none of the syntaxes seem to work. Neither for DOM nor GRS1.
Comment 53 Katrin Fischer 2014-01-18 13:18:06 UTC
What I did after applying the patch:
cp etc/searchengine/queryparser.yaml ../koha-devU/etc/searchengine/queryparser.yaml
cp etc/zebradb/ccl.properties ../koha-devU/etc/zebradb/ccl.properties
cp etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml
cp etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
cp etc/zebradb/marc_defs/unimarc/biblios/record.abs  ../koha-devU/etc/zebradb/marc_defs/unimarc/biblios/record.abs

Completely reindexed the bibliographic data using -b -v -r -x

Full reindex and searching for Dissertation-information worked nicely in DOM and GRS1. But...

1) In general I am having problems with pubdate and acqdate searches in UNIMARC compared to their MARC21 equivalents Mathieu, could you please take a look? Syntaxes I tried:
  pubdate=yyyy, pubdate:yyyy
  With QueryParser: pubdate(yyyy), acqdate(yyyy)

2) It would be nice if Dissertation-information could also be added for MARC21
   on a seaparate bug later on.

3) In general I am a bit worried about removing indexes. I can see, why
   some subfields are not making much sense to index, I also have read
   the comments on bug 11119, but some of the  changes might be debatable. 
   For example the following change could break functionality some 
   libraries rely on. Maybe it would not hurt to keep those around?
    
-#FIXME Fields 603, 630, 631, 632, 633, 634, 635, 636, 646 are not defined in Unimarc, but may be used by some libraries.
-melm 603      Subject,Subject:p
-melm 630      Subject,Subject:p
-melm 631      Subject,Subject:p
-melm 632      Subject,Subject:p
-melm 633      Subject,Subject:p
-melm 634      Subject,Subject:p
-melm 635      Subject,Subject:p
-melm 636      Subject,Subject:p
-melm 646      Subject,Subject:p


Failing for now until 1) can be resolved.
Comment 54 Mathieu Saby 2014-01-18 15:01:07 UTC
> 
> 1) In general I am having problems with pubdate and acqdate searches in
> UNIMARC compared to their MARC21 equivalents Mathieu, could you please take
> a look? Syntaxes I tried:
>   pubdate=yyyy, pubdate:yyyy
>   With QueryParser: pubdate(yyyy), acqdate(yyyy)


I will take a look


> 3) In general I am a bit worried about removing indexes. I can see, why
>    some subfields are not making much sense to index, I also have read
>    the comments on bug 11119, but some of the  changes might be debatable. 
>    For example the following change could break functionality some 
>    libraries rely on. Maybe it would not hurt to keep those around?
>     
> -#FIXME Fields 603, 630, 631, 632, 633, 634, 635, 636, 646 are not defined
> in Unimarc, but may be used by some libraries.

I don't know who added these indexes and why, but they were never used in Unimarc, so it was clearly a mistake to put them in standard configuration files.
Maybe some french library made one day a very specific developpment using these fields (I'm thinking of Frantiq network, specialised in Archeology). Or maybe it is just a mistake... I will send a message to french list to warn everybody that if a library want to index those fields in 3.14 they need to edit the config files before reindexing.

Mathieu

Mathieu
Comment 55 Mathieu Saby 2014-01-20 08:44:33 UTC
I made some tests.

Without my patch:

1. with QP enable
pubdate(1925) is working
pubdate:1925 is working

2. with QP disable
pubdate(1925) is not working
pubdate:1925 is not working

With my patch:

Exactly the same behavior

So, for me, the patch is correct, regarding pubdate index. It is not possible to build a query with pubdate without QP, but it was still the case in master.

Mathieu
Comment 56 Katrin Fischer 2014-01-20 08:45:36 UTC
Oh, that explains. It seems it's possible with MARC21 - what are we missing to make it work?
Comment 57 Mathieu Saby 2014-01-20 08:49:51 UTC
I was always said not to use pubdate index like that, so the issue is old, but maybe unimarc specific.
It is working only when you use a date in the advanced search filter
see comment 30


By the way: If you or Galen ask it, I can try to split the patch, with non functional changes (comments...) in a separate patch. But I prefer not to, if it is not required, because 
- I could do a mistake and break the patch, 
- I won't have time to work on Koha this week.
Comment 58 Katrin Fischer 2014-01-20 09:14:22 UTC
I think the pubdate index does not block this patch then - I will take another look at it later (if noone beats me to it :) ). Maybe you could file a bug for the pubdate index in UNIMARC and link to this patch?
Comment 59 Mathieu Saby 2014-01-20 12:23:51 UTC
Bug 11585 created
Comment 60 Jonathan Druart 2014-01-27 14:16:11 UTC
All looks good to me.

I had a discussion with Paul and the only think we found is that the Dissertation-information is not defined for MARC21 (but the index is defined in the ccl.properties). See bug 11620.

Marked as Passed QA.
Comment 61 Jonathan Druart 2014-01-27 14:18:13 UTC
Created attachment 24751 [details] [review]
Bug 11202 - Improve UNIMARC biblio indexing

This patch makes the following changes to UNIMARC biblio indexing :
A. Changes to UNIMARC conf files
1. add comments to biblio-koha-indexdefs.xml
2. make biblio-koha-indexdefs.xml more compact by grouping some declarations
Ex : 200$f and 200$g => one declaration for 200$fg
3. suppress unneeded declarations (indexing of some 4XX fields and 6XX fields not in unimarc format)
4. unindex some (sub)fields unneeded by most users (318, 207,230,210a,215, 4XXd)
5. change the way 308 field is indexed (no visible changes)
6. replace Title-host with Host-item -- see bug 11119
7. index 208 in Material-Type -- see bug 11119
8. index 100 pos 8-9 and 9-12 in pubdate:y and pubdate:n
9. index 100 pos 8-9 in pubdate:s instead of 210$d
10. Index all subfields of note 334 and 327 in note index
11. Index 304 and 327 in title index as well as note index
327 can contain a list of titles included in a work
304 can contain the title of the original work in case of a translation
12. Index 314 in author index as well as note index
314 can contain authors not mentionned in 200$f/g (the 4th, 5th etc author)
13. Index 328 note in Dissertation-information as well as note
14. Index 328$t in Title

B. Changes to ccl.properties :
1. add a new index Dissertation-information (1056)
2. fix EAN, pubdate and acqdate (they were not linked with bib1 attributes)

C. Changes to Search.pm
1. add Dissertation-information and suppress Title-host and UPC

D. Changes to QP config file queryparser.yaml
1. add Dissertation-information
2 fix EAN, pubdate and acqdate

Test plan :
If you cannot test in GRS1, test only in DOM, as GRS will be deprecated.

1. Apply the patch in a UNIMARC Koha running with DOM and ICU
2. copy src/etc/searchengine/queryparser.yaml into the main config directory of QP
3. copy src/etc/zebradb/ccl.properties into the main config directory of Zebra
4. copy src/etc/zebradb/marc_defs/unimarc/biblio/* into the main config directory of Zebra
5. reindex biblios (rebuild_zebra.pl -r -b -x -v)
6. test note index : make some searches on 334$b or 327$b
7. test author index : make some searches on 314 field
8. test title index : make some searches on 304 and 327 field, make a search on 328$t subfield
9. test dissertation-information index : make some searches on 328 field
10. In a record, put in the dates of 100 fields the values "1000" (1st date) and "1001" (2d date) ; try to search a book written in year 1000, you should find the record ; idem for year 1001
11. make some searches and sort by date. It should work better as before, especially if you have values like "c2009" or "impr. 2010" in 210 field
12. Regression test : make some searches on several indexes, like EAN, etc. It should work as before
Test 10-12 with and without Queryparser activated.
Be careful: with Queryparser activated, the index names (title, dissertation-information...) must be entered in lowercase only.
Of course, to test search and sort by dates, you need to have full records, with dates in 100 field as well as 210 field.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Comment 62 Katrin Fischer 2014-01-27 14:22:23 UTC
Hi Jonathan, thx for taking a look - my tests were also looking good so far, but I was hesitating to pass as I don't know UNIMARC really well. I think our libraries would like an index for dissertation-information - this could be another bug.
Comment 63 Galen Charlton 2014-01-27 22:02:26 UTC
From my review of this patch, five tests in t/QueryParser.t fail.  At first glance, I think the problem is with the tests, though.
Comment 64 Jonathan Druart 2014-02-03 12:18:46 UTC
Created attachment 25005 [details] [review]
Bug 11202: FIX UT for QueryParser.t

Test plan:
prove t/QueryParser.t
Comment 65 Galen Charlton 2014-02-19 21:10:29 UTC
Pushed to master.  Thanks, Mathieu!
Comment 66 Fridolin Somers 2014-05-19 15:28:45 UTC
Pushed to 3.14.x, will be in 3.14.07