Bug 7421 - UNIMARC authorities DOM indexing mode
Summary: UNIMARC authorities DOM indexing mode
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P4 enhancement (vote)
Assignee: mathieu saby
QA Contact: Jonathan Druart
URL:
Keywords: rel_3_14_candidate
Depends on:
Blocks: 7284 8958
  Show dependency treegraph
 
Reported: 2012-01-09 07:54 UTC by Frédéric Demians
Modified: 2014-12-07 20:02 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Proposed patch (118.16 KB, patch)
2012-01-09 10:42 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 7421 Add UNIMARC DOM Filter (118.20 KB, patch)
2012-06-29 08:50 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 7421 Add UNIMARC DOM Filter (118.30 KB, patch)
2012-07-17 03:14 UTC, Mason James
Details | Diff | Splinter Review
Add UNIMARC DOM Filter (140.87 KB, patch)
2012-10-16 15:28 UTC, Frédéric Demians
Details | Diff | Splinter Review
UNIMARC authorities file to test the indexing (143.65 KB, application/octet-stream)
2012-10-25 14:07 UTC, laurenthdl
Details
Extract of auth_header for test. (1.35 MB, text/x-sql)
2012-10-25 15:09 UTC, laurenthdl
Details
[PATCH][alternative patch] Bug 7421 Add UNIMARC DOM Filter (78.45 KB, patch)
2013-04-07 16:44 UTC, mathieu saby
Details | Diff | Splinter Review
[SIGNED OFF] Bug 7421 Add UNIMARC DOM Filter (78.72 KB, patch)
2013-09-08 08:30 UTC, Katrin Fischer
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 7421 Add UNIMARC DOM Filter (78.76 KB, patch)
2013-10-06 22:16 UTC, Katrin Fischer
Details | Diff | Splinter Review
MSAby - 11 auth records (11.36 KB, text/plain)
2013-10-11 09:23 UTC, mathieu saby
Details
MSAby - 11 auth records (plain text) (10.37 KB, text/plain)
2013-10-11 09:23 UTC, mathieu saby
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Frédéric Demians 2012-01-09 07:54:01 UTC
MARC21 authorities are indexed by Zebra using DOM or GRS1 filter. DOM is better. For example, it offer a full support of @ATTR 6=3 (field content completeness). Without this attribute, accurate authority headings matching is impossible. 

MARC21 DOM filter must be ported to UNIMARC.
Comment 1 Frédéric Demians 2012-01-09 10:42:02 UTC Comment hidden (obsolete)
Comment 2 Chris Cormack 2012-06-29 08:50:45 UTC Comment hidden (obsolete)
Comment 3 Chris Cormack 2012-06-29 08:52:10 UTC
Applies cleanly, didn't break zebra, I can't vouch it works right for UNIMARC but it doesn't break anything else.
Comment 4 Mason James 2012-07-17 03:14:19 UTC Comment hidden (obsolete)
Comment 5 Mason James 2012-07-17 03:18:15 UTC
(In reply to comment #2)
> Created attachment 10559 [details] [review]
> Bug 7421 Add UNIMARC DOM Filter
> 
> Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

QA comments

 - trivial and safe patch, 
 - doesn't break existing functionality
 - both files pass ./t/00-valid-xml.t xml tests

passing QA on this...
Comment 6 Paul Poulain 2012-07-18 13:54:26 UTC
I started an install from scratch, with UNIMARC.

Trying to reindex authorities, I get:
15:53 ~/koha.dev/koha-community/misc/migration_tools (new/bug_7421 $)$ ./rebuild_zebra.pl -a -v -r
Zebra configuration information
================================
Zebra biblio directory      = /home/paul/koha.dev/installs/unimarc-dom/var/lib/zebradb/biblios
Zebra authorities directory = /home/paul/koha.dev/installs/unimarc-dom/var/lib/zebradb/authorities
Koha directory              = /home/paul/koha.dev/koha-community
BIBLIONUMBER in :     001$@
BIBLIOITEMNUMBER in : 090$a
================================
====================
exporting authority
====================
1
Records exported: 1
====================
REINDEXING zebra
====================
15:53:24-18/07 zebraidx(14752) [warn] /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc/dom-config-marc.xml: stylesheet authority-zebra-indexdefs.xsl not found in path /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/etc:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/marc_defs/unimarc/authorities:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/lang_defs/fr:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/xsl

investigating...
Comment 7 Paul Poulain 2012-07-18 14:21:30 UTC
(In reply to comment #6)
> 15:53:24-18/07 zebraidx(14752) [warn]
> /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc/dom-
> config-marc.xml: stylesheet authority-zebra-indexdefs.xsl not found in path
> /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc:/home/
> paul/koha.dev/installs/unimarc-dom/etc/zebradb/etc:/home/paul/koha.dev/
> installs/unimarc-dom/etc/zebradb/marc_defs/unimarc/authorities:/home/paul/
> koha.dev/installs/unimarc-dom/etc/zebradb/lang_defs/fr:/home/paul/koha.dev/
> installs/unimarc-dom/etc/zebradb/xsl

Strange... the authority-zebra-indexdefs.xsl is in my koha repo:

ls ../../etc/zebradb/marc_defs/unimarc/authorities/
authority-koha-indexdefs.xml  authority-zebra-indexdefs.xsl  record.abs


But looking in my unimarc-dom (the installation directory), they are not:
15:57 ~/koha.dev/koha-community/misc/migration_tools (new/bug_7421 $)$ ls ~/koha.dev/installs/unimarc-dom/etc/zebradb/marc_defs/unimarc/authorities/
record.abs


Is it possible that they have not be copied by the make install ?

reinstalling ...

this time, I get my files at the proper place.

It probably means that I was not on the bug 7421 git branch when I made the previous install, so the file did not exist. My bad probably.

After reindexing, I get no results when searching. zebra logs say:
16:11:18-18/07 zebrasrv(1) [request] Search authorities ERROR 114 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @and @attr 5=100 @attr 1=authtype NP @attr 4=6 @attr 5=1 @attr 1=Heading-Main jean


error 114 is unsupported attribute

Same thing with yaz-client:
Z> base authorities
Z> f jean
Sent searchRequest.
Received SearchResponse.
Search was a bloomin' failure.
Number of hits: 0, setno 1
Result Set Status: none
records returned: 0
Diagnostic message(s) from database:
    [114] Unsupported Use attribute -- v2 addinfo 'any'


Frédéric, hint/explanation welcomed !!!


{{{
Side comment:
at the end of make install, I get
chmod: impossible d'accéder à «/home/paul/koha.dev/installs/unimarcdom2/var/spool»: Aucun fichier ou dossier de ce type

It's related to bug 8268, i'll create another entry for this problem
}}}
Comment 8 Paul Poulain 2012-09-05 13:02:30 UTC
bump ! Frédéric, see comment 7, I can't make it work !
Comment 9 Paul Poulain 2012-09-12 12:34:26 UTC
(In reply to comment #8)
> bump ! Frédéric, see comment 7, I can't make it work !

Frédéric, bump again !
Comment 10 Frédéric Demians 2012-09-19 13:44:52 UTC
It works for me, but I'm not sure to understand what's wrong with your installation. Is it an update or a new installation?
Comment 11 Paul Poulain 2012-09-19 15:14:56 UTC
(In reply to comment #10)
> It works for me, but I'm not sure to understand what's wrong with your
> installation. Is it an update or a new installation?

it was a fresh install
Comment 12 Frédéric Demians 2012-09-19 17:38:04 UTC
> it was a fresh install

I've redone a new installation, choosing UNIMARC, and DOM. And it works
out of the box. I've chosen a complete UNIMARC framework? Have you
chosen the alternative framework?
Comment 13 Jared Camins-Esakov 2012-10-06 04:13:40 UTC
Changing target version so this bug report will show up in the list of 3.12-targeted features and doesn't get lost in the shuffle during feature freeze.
Comment 14 Paul Poulain 2012-10-08 08:51:06 UTC
This ENH has no side effect risk. I plan to try again, and would push it before the string freeze (oct 22th). So switching back to master.
Comment 15 Paul Poulain 2012-10-08 17:26:53 UTC
Frédéric, I tried again, same problem.

I made:
 * a dev mode install
 * from git source
 * All other Makefile.PL to default

Any difference with what you made ?
Comment 16 laurenthdl 2012-10-16 10:07:35 UTC
In my opinion, that patch needs deep work.
In fact, there are no Any, authtype or Heading-Main index defined and those indexes are required at the moment to complete all the requests.

Do you have any fix for marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl file which adds those indexes ?
Comment 17 Frédéric Demians 2012-10-16 15:28:08 UTC Comment hidden (obsolete)
Comment 18 Paul Poulain 2012-10-24 15:17:01 UTC
Good news: i've tested this patch, and it's much better, it worked like a charm on a new setup !!! I'll ask Henri-Damien to do other tests. If he is successful, I think I could push this patch for 3.10 !
Comment 19 laurenthdl 2012-10-25 14:04:57 UTC
Some things could be fixed : 
   - MARC21 should be changed into UNIMARC (in the comments)
   - indexing 008 as it is in MARC21 when there is no UNIMARC specification of 008 field is not really helpfull


Some others HAVE TO be fixed :

Search Main Heading with Default authtype contains wil :
11 results

Some of which contains wil in $b which is not the current behaviour

zebra search launched :
   Search authorities OK 11 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Heading wil


Search Heading with Default authtype contains wil :
1 result

which contains wil in $a only

zebra search launched :
   OK 1 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Match wil


Search Heading with Default authtype contains wil :
1 result

which contains wil in $a only

zebra search launched :
   OK 1 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Match wil


Search Keyword Search with Default authtype contains wil :
7 results

which contains wil in $a or $b

zebra search launched :
 OK 7 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Any wil

I find it much strange that Any is not expanding in all the indexed subfields and that keyword search would not return more results than the heading one.
Comment 20 laurenthdl 2012-10-25 14:07:45 UTC
Created attachment 13037 [details]
UNIMARC authorities file to test the indexing

This file contains 1000 authorities to test the indexing.
Comment 21 laurenthdl 2012-10-25 15:09:36 UTC
Created attachment 13051 [details]
Extract of auth_header for test.
Comment 22 mathieu saby 2013-02-07 09:35:48 UTC
Hello
Is someone still working on that bug?

M. Saby
Comment 23 Paul Poulain 2013-02-11 17:40:51 UTC
(In reply to comment #22)
> Hello
> Is someone still working on that bug?
> 
> M. Saby

I think / it seems no

(reminder: laurenthdl, aka Henri-Damien, that made the last comment, was co-owner & founder of BibLibre with me, and he left BibLibre last dec 31 -see http://www.biblibre.com/en/blog/entry/biblibre-takes-another-next-step -)
Comment 24 mathieu saby 2013-04-07 16:42:49 UTC
I joint an alternative patch. I took as a base the patch of F. Demians, but made a lot of changes, so I think it is more logical to create a new patch as the behavior is not the same as previous patch.
I was not able to test, I don't have currently a Koha with DOM.

M. Saby
Comment 25 mathieu saby 2013-04-07 16:44:08 UTC Comment hidden (obsolete)
Comment 26 mathieu saby 2013-04-13 13:13:01 UTC
I tested my patch (attachment 17261 [details] [review]) on a DOM environment. I think it works. 
But I forgot one step in testing : in order to force Zebra to take into account the new config files, I had to copy them from your sources to your main koha directory, before launching rebuild_zebra -a -x -r -v.

For example, from
home/msaby/koha-dev/src/etc/zebra_db/marc_defs/unimarc/authorities
to 
home/msaby/koha-dev/etc/zebra_db/marc_defs/unimarc/authorities

So, you can test.

M. Saby
Comment 27 Katrin Fischer 2013-09-08 08:30:13 UTC Comment hidden (obsolete)
Comment 28 Katrin Fischer 2013-09-08 08:32:09 UTC
I have gone ahead and signed off on this, as with this patch applied I was able to get authority searching working with DOM indexing on my test installation and it seems to work well. As I am by no means a UNIMARC expert this is a bit of a weak sign off - Jonathan, could you take a closer look in QA and/or request another sign off?
Comment 29 mathieu saby 2013-09-08 09:10:04 UTC
A french librarian wanted to do some tests on "unimarc patches" next week, but I don't know if he had planned to look at this one. I will ask him.

Mathieu
Comment 30 mathieu saby 2013-09-08 09:11:04 UTC
And a great thank you for testing and signing off! I hope french libraries will be able to use DOM in 3.14...

Mathieu
Comment 31 Vitor Fernandes 2013-10-04 13:37:58 UTC
THis patch works fine for me!
Comment 32 Katrin Fischer 2013-10-06 22:16:55 UTC
Created attachment 21840 [details] [review]
[SIGNED-OFF] Bug 7421 Add UNIMARC DOM Filter

I took as a base the patch of F. Demians, but made a lot of changes, so I think it is more logical to create a new patch as the behavior is not the same as previous patch.
I tried to define DOM config files as a "miror" of record.abs, so the behavior be the same.
If it is OK, we will be able to improve indexing later, for example suppressing warns, managing indicators or subdivisions, etc.
I made some little changes to record.abs :
- comments
- 216 was indexed in Conference-name as well as Trademark. I supposes "Conference-name" is an error, so I indexed only in Trademark
- index 2 new notes : 340 / 356

The only difference between record.abs and DOM is that DOM config files does not index complete fields, but subfields.
Ex :
melm 200 ===> <kohaidx:index_subfields tag="200" subfields="abcdfgjxyz">
I took all the subfields from the UNIMARC Authorities manual. The only subfields not indexed are numeric subfields : $7, $8 for language of record, and $0,2,3,5,6 for 4XX/5XX/7XX

To test :
- index a set of bib and auth records with GRS-1
- make some searches on different kind of authorities
- index the same records with DOM
- make the same searches
- You are not supposed to see differences

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
As I am not a UNIMARC user it's hard for me to test this, but
while testing other authority related patches I noticed that I couldn't
index the UNIMARC authorities of the sample base. The files are obviously
missing and reindex_zebra.pl notes this. With this patch applied,
indexing works and authorities are searchable in my installation.

[2013-10-07] Added second sign off from Vitor.

Signed-off-by: Vitor Fernandes <fvernandes@keep.pt>
Comment 33 Jonathan Druart 2013-10-10 14:24:47 UTC
QA comment:

I retry same tests as Henri Damien:

- go on authorities/authorities-home.pl
- Select "default" "contains" "wil" "main heading"
  * GSR-1: 27 results
  * Dom  : 27 results
- Select "default" "contains" "wil" "all headings"
  * GSR-1: 0 result
  * Dom  : 27 results
- Select "default" "contains" "wil" "Keyword search"
  * GSR-1: 27 results
  * Dom  : 29 results

Marked as Failed QA.
Comment 34 Jonathan Druart 2013-10-10 14:25:47 UTC
(Maybe I did an error somewhere, I start to be tired to parse xml with my eyes...)
Comment 35 mathieu saby 2013-10-10 15:16:51 UTC
(In reply to Jonathan Druart from comment #33)
> QA comment:
> 
> I retry same tests as Henri Damien:
> 
> - go on authorities/authorities-home.pl
> - Select "default" "contains" "wil" "main heading"
>   * GSR-1: 27 results
>   * Dom  : 27 results
> - Select "default" "contains" "wil" "all headings"
>   * GSR-1: 0 result
>   * Dom  : 27 results
> - Select "default" "contains" "wil" "Keyword search"
>   * GSR-1: 27 results
>   * Dom  : 29 results
> 
> Marked as Failed QA.

I will look at it.
In what precise field/subfield did you put the "wil" string?

Maybe the small discrepancy for Keyword search exists also between MARC21 DOM and GRS1? 

Mathieu
Comment 36 mathieu saby 2013-10-10 20:15:18 UTC
I am not sure I understood how Koha is searching authorities, so I explored the code.

In authorities-search.inc : 
- you have 3 choices (main heading, match heading, all headings.
- each choice is linked with a value of "marclist" field, passed in the form in an <input>.
- main heading => marclist=mainentry
- match heading => marclist=match
- all heading => marclist=all
These data are passed to authorities-home.pl

In authorities-home.pl : 
my ( $results, $total ) = SearchAuthorities(
        [$marclist],  [$and_or],
        [$excluding], [$operator],
        [$value], ( $startfrom - 1 ) * $resultsperpage,
        $resultsperpage, $authtypecode,
        $orderby
    );


In C4/AuthoritiesMarc.pm :
sub SearchAuthorities {
    my ($tags, $and_or, $excluding, $operator, $value, $offset,$length,$authtypecode,$sortby,$skipmetadata) = @_;
...
elsif ( @$tags[$i] eq "mainentry" ) {
                $attr = " \@attr 1=Heading ";
...
elsif ( @$tags[$i] eq "match" ) {
                $attr = " \@attr 1=Match ";
...
else { # Assume any if no index was specified
                $attr = " \@attr 1=Any ";


There is no specific choice for the "all" value, so the "any" index is used. I wonder if it is normal.

But your failing case
- Select "default" "contains" "wil" "all headings"
>   * GSR-1: 0 result
>   * Dom  : 27 results

probably means that "Any" index is broken.


Mathieu
Comment 37 mathieu saby 2013-10-10 20:20:27 UTC
In record.abs for GRS1 we have :
" all any"

It is "translated" in DOM zebra-xsl file as : 
+  <xslo:template mode="index_all" match="text()">
+    <z:index name="Any:w Any:p">
+      <xslo:value-of select="."/>
+    </z:index>
+  </xslo:template>

And I don't find it in DOM koha-xml file.

I don't remember how I did that. It is probably the xslt transformation stylesheet that added those lines, and maybe it is not good.
More investigation later...

Mathieu
Comment 38 Galen Charlton 2013-10-10 21:19:04 UTC
I'm taking a chance on this and have pushed it to master (and the reason why I'm taking a chance is that I've also written and pushed a regression test that shows, with the dataset provided by Henri-Damien, that GRS-1 and DOM indexing are behaving the same).

I'm not sure where Jonathan's 27 hits on "wil" are coming from -- only 11 records in the sample file have that sequence of letters at all.  Are you using a different dataset?  If so, could you attach the records or a download link to the bug?

I'm setting to in discussion rather than pushed to master in case it turns out that I was too precipitous and a revert is needed.
Comment 39 mathieu saby 2013-10-11 07:20:26 UTC
Hi
(In reply to Galen Charlton from comment #38)
> I'm taking a chance on this and have pushed it to master (and the reason why
> I'm taking a chance is that I've also written and pushed a regression test

Is it attached to an other bug ?

> that shows, with the dataset provided by Henri-Damien,

I forgot there were a sample of records attached to the bug ;-)

 that GRS-1 and DOM
> indexing are behaving the same).
> 

Mathieu
Comment 40 Jonathan Druart 2013-10-11 08:17:59 UTC
Galen,
I tested with the dataset provide by Mathieu (mail "pending bugs about UNIMARC marc flavor": https://www.dropbox.com/s/idw98d7jiy5e5xd/dump_unimarc.zip).
Comment 41 mathieu saby 2013-10-11 08:43:00 UTC
For info, here is the content of HDL file

=LDR  00166    a2200085   4500
=001  1
=100  \\$a20121025              frey50       
=999  \\$a1
=152  \\$bNP
=200  \\$aMeusburger$bMatthias

=LDR  00143    a2200073   4500
=001  2
=100  \\$a20121025              frey50       
=200  \\$aBelaïdi$bAlice
=152  \\$bNP

=LDR  00147    a2200073   4500
=001  3
=100  \\$a20121025              frey50       
=210  \\$aMartires del compas
=152  \\$bCO

=LDR  00138    a2200073   4500
=001  4
=100  \\$a20121025              frey50       
=200  \\$aMrak$bMira
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  5
=100  \\$a20121025              frey50       
=200  \\$aWaffler$bEmmanuel
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  6
=100  \\$a20121025              frey50       
=200  \\$aManjarres$bBruno
=152  \\$bNP

=LDR  00142    a2200073   4500
=001  7
=100  \\$a20121025              frey50       
=200  \\$aMartinez$bPepe
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  8
=100  \\$a20121025              frey50       
=200  \\$aNeveu$bPhilippe
=152  \\$bNP

=LDR  00136    a2200073   4500
=001  9
=100  \\$a20121025              frey50       
=210  \\$aZaragraf
=152  \\$bCO

=LDR  00142    a2200073   4500
=001  12
=100  \\$a20121025              frey50       
=200  \\$aKali Live Dub
=152  \\$bNP

=LDR  00133    a2200073   4500
=001  13
=100  \\$a20121025              frey50       
=200  \\$aAvia
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  14
=100  \\$a20121025              frey50       
=200  \\$aFugitive (The)
=152  \\$bNP

=LDR  00137    a2200073   4500
=001  15
=100  \\$a20121025              frey50       
=200  \\$aThésée
=152  \\$bNP

=LDR  00136    a2200073   4500
=001  16
=100  \\$a20121025              frey50       
=210  \\$aZenzile
=152  \\$bCO

=LDR  00150    a2200073   4500
=001  17
=100  \\$a20121025              frey50       
=210  \\$aAir$cDuo instrumental
=152  \\$bCO

=LDR  00138    a2200073   4500
=001  18
=100  \\$a20121025              frey50       
=210  \\$aRoudoudou
=152  \\$bCO

=LDR  00138    a2200073   4500
=001  19
=100  \\$a20121025              frey50       
=210  \\$aSeven Dub
=152  \\$bCO

=LDR  00145    a2200073   4500
=001  20
=100  \\$a20121025              frey50       
=210  \\$aWilliams Traffic
=152  \\$bCO

=LDR  00152    a2200073   4500
=001  21
=100  \\$a20121025              frey50       
=210  \\$a Le  Peuple de l'Herbe 
=152  \\$bCO

=LDR  00142    a2200073   4500
=001  22
=100  \\$a20121025              frey50       
=210  \\$aSporto Kantes
=152  \\$bCO

=LDR  00141    a2200073   4500
=001  23
=100  \\$a20121025              frey50       
=210  \\$aOto Electrik
=152  \\$bCO

=LDR  00142    a2200073   4500
=001  24
=100  \\$a20121025              frey50       
=210  \\$aGotan Project
=152  \\$bCO

=LDR  00136    a2200073   4500
=001  25
=100  \\$a20121025              frey50       
=210  \\$aEz3kiel
=152  \\$bCO

=LDR  00141    a2200073   4500
=001  26
=100  \\$a20121025              frey50       
=210  \\$aBrain Damage
=152  \\$bCO

=LDR  00132    a2200073   4500
=001  27
=100  \\$a20121025              frey50       
=210  \\$aArk
=152  \\$bCO

=LDR  00134    a2200073   4500
=001  28
=100  \\$a20121025              frey50       
=210  \\$aOctet
=152  \\$bCO

=LDR  00137    a2200073   4500
=001  29
=100  \\$a20121025              frey50       
=200  \\$aBeyoncé
=152  \\$bNP

=LDR  00149    a2200073   4500
=001  30
=100  \\$a20121025              frey50       
=200  \\$aWashington$bBrittani
=152  \\$bNP

=LDR  00142    a2200073   4500
=001  31
=100  \\$a20121025              frey50       
=200  \\$aChapa$bMarcie
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  32
=100  \\$a20121025              frey50       
=200  \\$aGlaspie$bNikki
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  33
=100  \\$a20121025              frey50       
=200  \\$aIfrah$bLaurence
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  34
=100  \\$a20121025              frey50       
=200  \\$aCassourra$bAlain
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  35
=100  \\$a20121025              frey50       
=200  \\$aGrimm$bWilhelm
=152  \\$bNP

=LDR  00149    a2200073   4500
=001  36
=100  \\$a20121025              frey50       
=200  \\$aSpirine$bGuennadi K.
=152  \\$bNP

=LDR  00155    a2200073   4500
=001  37
=100  \\$a20121025              frey50       
=200  \\$aTchekhov$bAnton Pavlovitch
=152  \\$bNP

=LDR  00142    a2200073   4500
=001  38
=100  \\$a20121025              frey50       
=200  \\$aLissitsky$bEl
=152  \\$bNP

=LDR  00148    a2200073   4500
=001  39
=100  \\$a20121025              frey50       
=200  \\$aOusatschev$bAndreï
=152  \\$bNP

=LDR  00140    a2200073   4500
=001  40
=100  \\$a20121025              frey50       
=200  \\$aLô$bCheikh
=152  \\$bNP

=LDR  00142    a2200073   4500
=001  41
=100  \\$a20121025              frey50       
=200  \\$aSarr$bThierno
=152  \\$bNP

=LDR  00138    a2200073   4500
=001  42
=100  \\$a20121025              frey50       
=200  \\$aSow$bOmar
=152  \\$bNP

=LDR  00140    a2200073   4500
=001  43
=100  \\$a20121025              frey50       
=200  \\$aAllen$bTony
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  44
=100  \\$a20121025              frey50       
=200  \\$aEllis$bPee Wee
=152  \\$bNP

=LDR  00149    a2200073   4500
=001  45
=100  \\$a20121025              frey50       
=200  \\$aTall$bCheikh Tidiane
=152  \\$bNP

=LDR  00142    a2200073   4500
=001  46
=100  \\$a20121025              frey50       
=200  \\$aShin$bSe-jung
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  47
=100  \\$a20121025              frey50       
=200  \\$aZanasi$bGianni
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  48
=100  \\$a20121025              frey50       
=200  \\$aMurino$bCaterina
=152  \\$bNP

=LDR  00148    a2200073   4500
=001  49
=100  \\$a20121025              frey50       
=200  \\$aMastandrea$bValerio
=152  \\$bNP

=LDR  00148    a2200073   4500
=001  50
=100  \\$a20121025              frey50       
=200  \\$aBattiston$bGiuseppe
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  51
=100  \\$a20121025              frey50       
=200  \\$aRobillot$bHenri
=152  \\$bNP

=LDR  00150    a2200073   4500
=001  52
=100  \\$a20121025              frey50       
=200  \\$aSan Millan$bFrançois
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  53
=100  \\$a20121025              frey50       
=200  \\$aLecaye$bEmmanuel
=152  \\$bNP

=LDR  00148    a2200073   4500
=001  54
=152  \\$bSNC
=100  \\$a20121025              frey50       
=250  \\$aMusique$ySénégal

=LDR  00143    a2200073   4500
=001  55
=100  \\$a20121025              frey50       
=200  \\$aScherfig$bLone
=152  \\$bNP

=LDR  00141    a2200073   4500
=001  56
=100  \\$a20121025              frey50       
=200  \\$aHornby$bNick
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  57
=100  \\$a20121025              frey50       
=200  \\$aEnglishby$bPaul
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  58
=100  \\$a20121025              frey50       
=200  \\$aSarsgaard$bPeter
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  59
=100  \\$a20121025              frey50       
=200  \\$aMulligan$bCarey
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  60
=100  \\$a20121025              frey50       
=200  \\$aMolina$bAlfred
=152  \\$bNP

=LDR  00174    a2200073   4500
=001  61
=100  \\$a20121025              frey50       
=250  \\$aExtraterrestres$xCédéroms pour la jeunesse
=152  \\$bSNC

=LDR  00173    a2200073   4500
=001  62
=100  \\$a20121025              frey50       
=250  \\$aChauves-souris$xCédéroms pour la jeunesse
=152  \\$bSNC

=LDR  00165    a2200073   4500
=001  63
=100  \\$a20121025              frey50       
=250  \\$aBourse$xCédéroms pour la jeunesse
=152  \\$bSNC

=LDR  00165    a2200073   4500
=001  64
=100  \\$a20121025              frey50       
=250  \\$aFêtes$xCédéroms pour la jeunesse
=152  \\$bSNC

=LDR  00166    a2200073   4500
=001  65
=100  \\$a20121025              frey50       
=250  \\$aAutomne$xCédéroms pour la jeunesse
=152  \\$bSNC

=LDR  00151    a2200073   4500
=001  66
=152  \\$bSNC
=100  \\$a20121025              frey50       
=250  \\$aMusique$yBurkina Faso

=LDR  00143    a2200073   4500
=001  67
=100  \\$a20121025              frey50       
=200  \\$aDémé$bVictor
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  68
=100  \\$a20121025              frey50       
=200  \\$aDiabaté$bIssouf
=152  \\$bNP

=LDR  00142    a2200073   4500
=001  69
=100  \\$a20121025              frey50       
=200  \\$aDiarra$bSalif
=152  \\$bNP

=LDR  00140    a2200073   4500
=001  70
=100  \\$a20121025              frey50       
=200  \\$aDiarra$bAli
=152  \\$bNP

=LDR  00138    a2200073   4500
=001  71
=100  \\$a20121025              frey50       
=200  \\$aKoudédé
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  72
=100  \\$a20121025              frey50       
=200  \\$aAnderson$bLaurie
=152  \\$bNP

=LDR  00138    a2200073   4500
=001  73
=100  \\$a20121025              frey50       
=200  \\$aReed$bLou
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  74
=100  \\$a20121025              frey50       
=200  \\$aScherer$bPeter
=152  \\$bNP

=LDR  00140    a2200073   4500
=001  75
=100  \\$a20121025              frey50       
=200  \\$aBurger$bRob
=152  \\$bNP

=LDR  00140    a2200073   4500
=001  76
=100  \\$a20121025              frey50       
=200  \\$aHakim$bOmar
=152  \\$bNP

=LDR  00139    a2200073   4500
=001  77
=100  \\$a20121025              frey50       
=200  \\$aZorn$bJohn
=152  \\$bNP

=LDR  00135    a2200073   4500
=001  78
=100  \\$a20121025              frey50       
=200  \\$aAntony
=152  \\$bNP

=LDR  00160    a2200073   4500
=001  79
=100  \\$a20121025              frey50       
=250  \\$aJeux de simulation$xCédéroms
=152  \\$bSNC

=LDR  00168    a2200073   4500
=001  80
=100  \\$a20121025              frey50       
=250  \\$aGuerre mondiale (1939-1945)$xLudiciels
=152  \\$bSNC

=LDR  00142    a2200073   4500
=001  81
=100  \\$a20121025              frey50       
=200  \\$aAimée$bAnouk
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  82
=100  \\$a20121025              frey50       
=200  \\$aPiccoli$bMichel
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  83
=100  \\$a20121025              frey50       
=200  \\$aPlacido$bMichele
=152  \\$bNP

=LDR  00142    a2200073   4500
=001  84
=100  \\$a20121025              frey50       
=200  \\$aMiller$bSteve
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  85
=100  \\$a20121025              frey50       
=200  \\$aBuffalo$bNorton
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  86
=100  \\$a20121025              frey50       
=200  \\$aCharles$bSonny
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  87
=100  \\$a20121025              frey50       
=200  \\$aKnudtson$bGordy
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  88
=100  \\$a20121025              frey50       
=200  \\$aPeterson$bBilly
=152  \\$bNP

=LDR  00144    a2200073   4500
=001  89
=100  \\$a20121025              frey50       
=200  \\$aClaerr$bThierry
=152  \\$bNP

=LDR  00146    a2200073   4500
=001  90
=100  \\$a20121025              frey50       
=200  \\$aWesteel$bIsabelle
=152  \\$bNP

=LDR  00148    a2200073   4500
=001  91
=100  \\$a20121025              frey50       
=200  \\$aSauvage$bChristiane
=152  \\$bNP

=LDR  00171    a2200073   4500
=001  92
=100  \\$a20121025              frey50       
=210  \\$aAssociation des bibliothécaires français
=152  \\$bCO

=LDR  00143    a2200073   4500
=001  93
=100  \\$a20121025              frey50       
=200  \\$aMundy$bAnthony
=152  \\$bNP

=LDR  00150    a2200073   4500
=001  94
=100  \\$a20121025              frey50       
=200  \\$aMuldrow$bGeorgia Anne
=152  \\$bNP

=LDR  00174    a2200073   4500
=001  95
=152  \\$bSNC
=100  \\$a20121025              frey50       
=250  \\$aMusique$yCongo (République Démocratique)  

=LDR  00141    a2200073   4500
=001  96
=100  \\$a20121025              frey50       
=200  \\$aNsangue$bGuy
=152  \\$bNP

=LDR  00139    a2200073   4500
=001  97
=100  \\$a20121025              frey50       
=200  \\$aTran$bAlex
=152  \\$bNP

=LDR  00141    a2200073   4500
=001  98
=100  \\$a20121025              frey50       
=200  \\$aIpupa$bFally
=152  \\$bNP

=LDR  00145    a2200073   4500
=001  99
=100  \\$a20121025              frey50       
=200  \\$aGalison$bWilliam
=152  \\$bNP

=LDR  00143    a2200073   4500
=001  100
=100  \\$a20121025              frey50       
=200  \\$aClerc$bArnaud
=152  \\$bNP

=LDR  00138    a2200073   4500
=001  101
=100  \\$a20121025              frey50       
=200  \\$aWiretree
=152  \\$bNP

=LDR  00147    a2200073   4500
=001  102
=100  \\$a20121025              frey50       
=200  \\$aBlanchard$bDaniel
=152  \\$bNP



Mathieu
Comment 42 mathieu saby 2013-10-11 08:44:18 UTC
oups, maybe it is was not a good idea to copy all the records. too long...

Mathieu
Comment 43 mathieu saby 2013-10-11 08:47:13 UTC
I think if we want a set of records for testing, we should take only a small file with 10 records from the "real word".
With a small set, if there is a oddity in results it would be easier to find why.

Mathieu
Comment 44 mathieu saby 2013-10-11 09:22:39 UTC
Here is a set of 11 auth record from my catalog. unicode encoding.
Diffent types of auths, rejected forms etc.

I join 2 file : marc raw file, and marc file "translated" to plain text, to make verifications easier.
Comment 45 mathieu saby 2013-10-11 09:23:15 UTC
Created attachment 21956 [details]
MSAby - 11 auth records
Comment 46 mathieu saby 2013-10-11 09:23:50 UTC
Created attachment 21957 [details]
MSAby - 11 auth records (plain text)
Comment 47 Galen Charlton 2013-10-11 14:53:20 UTC
(In reply to mathieu saby from comment #39)
> Hi
> (In reply to Galen Charlton from comment #38)
> > I'm taking a chance on this and have pushed it to master (and the reason why
> > I'm taking a chance is that I've also written and pushed a regression test
> 
> Is it attached to an other bug ?

No -- see the following commit:

http://git.koha-community.org/gitweb/?p=koha.git;a=commitdiff;h=521e599dd82061fd10321d681d06d1409134d39c
Comment 48 Galen Charlton 2013-10-11 15:16:38 UTC
(In reply to Jonathan Druart from comment #40)
> Galen,
> I tested with the dataset provide by Mathieu (mail "pending bugs about
> UNIMARC marc flavor":
> https://www.dropbox.com/s/idw98d7jiy5e5xd/dump_unimarc.zip).

Thanks.  I took this file and adapted the test cases I wrote yesterday.  Here are the results I got:

[1] main heading contains "wil"

GRS-1 = 27 hits
DOM   = 27 hits

[2] all headings contains "wil"

GRS-1 = 27 hits
DOM   = 27 hits

[3] keyword contains "wil"

GRS-1 = 27 hits
DOM   = 29 hits

The following indicates that the DOM filter is not necessarily wrong:

$ perl -ne 'BEGIN { $/ = "\035"; } END { print "$ct\n" } { $ct++ if /wil/i; }' < unimarc_auth_3.mrc 
29

IOW, there really are 29 records that contain the substring "wil".

I'll investigate further to see why the GRS-1 any index is behaving slightly differently.
Comment 49 Jared Camins-Esakov 2013-10-11 15:19:30 UTC
(In reply to Galen Charlton from comment #48)
> I'll investigate further to see why the GRS-1 any index is behaving slightly
> differently.

Because GRS-1's "any" directive does not include all fields, whereas the DOM "equivalent" does.
Comment 50 mathieu saby 2013-10-11 15:28:57 UTC
(In reply to Jared Camins-Esakov from comment #49)
> (In reply to Galen Charlton from comment #48)
> > I'll investigate further to see why the GRS-1 any index is behaving slightly
> > differently.
> 
> Because GRS-1's "any" directive does not include all fields, whereas the DOM
> "equivalent" does.

all fields? you mean really ALL fields even if they are not used by an other index? That must very annoying for users!
I made a patch some time ago for repacing "any" with "anywhere" and make a reasonable "any" index, like we can found in other ILS. I made it for GRS1, but it may be even more usefull for DOM.
Bug 8962 (need some rewrite I suppose)

M. Saby
Comment 51 Katrin Fischer 2013-10-11 15:31:05 UTC
I am not sure if it's annoying - I always tell people that everything that is indexed can be searched in keyword search and the reactions so far are quite positive.
Comment 52 Jared Camins-Esakov 2013-10-11 15:37:48 UTC
(In reply to mathieu saby from comment #50)
> (In reply to Jared Camins-Esakov from comment #49)
> > (In reply to Galen Charlton from comment #48)
> > > I'll investigate further to see why the GRS-1 any index is behaving slightly
> > > differently.
> > 
> > Because GRS-1's "any" directive does not include all fields, whereas the DOM
> > "equivalent" does.
> 
> all fields? you mean really ALL fields even if they are not used by an other
> index? That must very annoying for users!
> I made a patch some time ago for repacing "any" with "anywhere" and make a
> reasonable "any" index, like we can found in other ILS. I made it for GRS1,
> but it may be even more usefull for DOM.
> Bug 8962 (need some rewrite I suppose)

Yeah, ideally there would be a special keyword index for DOM that included all the normal keyword-ish fields and note fields, but for me a higher priority was getting keyword working at all.
Comment 53 Galen Charlton 2013-10-11 15:44:16 UTC
(In reply to Jared Camins-Esakov from comment #49)
> (In reply to Galen Charlton from comment #48)
> > I'll investigate further to see why the GRS-1 any index is behaving slightly
> > differently.
> 
> Because GRS-1's "any" directive does not include all fields, whereas the DOM
> "equivalent" does.

The "all" directive, in particular, just specifies one or more attributes that are tacked on to the attribute list for each element being indexed.

The two "wil" records that DOM includes in the "any" index have the "William" string in the 810 field.  Since the UNIMARC GRS-1 record.abs does not index the 810 at all, it is likewise not included in the "any" index.  By contrast, the DOM XSL index_all option indexes all text strings present in the record.
Comment 54 Galen Charlton 2013-10-11 15:48:30 UTC
(In reply to Katrin Fischer from comment #51)
> I am not sure if it's annoying - I always tell people that everything that
> is indexed can be searched in keyword search and the reactions so far are
> quite positive.

I agree -- it's easier to explain that an "any" index includes everything (though whether or not fixed fields should be included is a matter of opinion), so I for one don't really see a problem with the change in behavior of the "any" index.

Note, though, that that's a separate question from whether the "any" index should be included in, say, auth_finder.pl, as somebody looking for authority records via that interface has little reason to be using anything other than the headings searches to begin with.
Comment 55 mathieu saby 2013-10-11 21:14:45 UTC
(In reply to Galen Charlton from comment #54)
> (In reply to Katrin Fischer from comment #51)
> > I am not sure if it's annoying - I always tell people that everything that
> > is indexed can be searched in keyword search and the reactions so far are
> > quite positive.
> 
> I agree -- it's easier to explain that an "any" index includes everything
> (though whether or not fixed fields should be included is a matter of
> opinion), so I for one don't really see a problem with the change in
> behavior of the "any" index.
> 
> Note, though, that that's a separate question from whether the "any" index
> should be included in, say, auth_finder.pl, as somebody looking for
> authority records via that interface has little reason to be using anything
> other than the headings searches to begin with.

Sorry, but I do not agree with both of you.
I'm speaking here in general, not for authorities (we don't give access to authorities on opac). The OPAC search with GRS1 is very "noisy", with a lot of unwanted and non understandable results.
Ex : I have a book published in "Roma". It has "1 vol. (215 p.)", it is written in french ("fre" in 105a) with an item having an INTERNAL note (not shown on opac) in 995 field, like "book given by a teacher".
This book will currently show on opac when some user search "vol", "fre", "book", "given", "teacher" or "Roma". For "Roma" I could maybe understand, but I don't like that. For the other points I don't like that AT ALL.

We have a rather big catalog (400 000 records), to Koha can easily find a lot of unrelated records for any search. With DOM I understand it will be worse.

Of course this issue could been less problematic, if Koha had a good pertinence sorting (it is mad), a good date sorting (it is broken, for unimarc at least), and a good facetting system (all we have are pseudofacets, non parametrable for the moment, based only on the X first results, not cumulable, and whose general presentation is not user friendly). We can work on these points, of course, but it would be easier to define a more reasonable "any" index.

Mathieu
Comment 56 Galen Charlton 2013-10-12 19:14:23 UTC
(In reply to mathieu saby from comment #55)
> (In reply to Galen Charlton from comment #54)
> > (In reply to Katrin Fischer from comment #51)
> > > I am not sure if it's annoying - I always tell people that everything that
> > > is indexed can be searched in keyword search and the reactions so far are
> > > quite positive.
> > 
> > I agree -- it's easier to explain that an "any" index includes everything
> > (though whether or not fixed fields should be included is a matter of
> > opinion), so I for one don't really see a problem with the change in
> > behavior of the "any" index.
> > 
> > Note, though, that that's a separate question from whether the "any" index
> > should be included in, say, auth_finder.pl, as somebody looking for
> > authority records via that interface has little reason to be using anything
> > other than the headings searches to begin with.
> 
> Sorry, but I do not agree with both of you.
> I'm speaking here in general, not for authorities (we don't give access to
> authorities on opac). The OPAC search with GRS1 is very "noisy", with a lot
> of unwanted and non understandable results.
> Ex : I have a book published in "Roma". It has "1 vol. (215 p.)", it is
> written in french ("fre" in 105a) with an item having an INTERNAL note (not
> shown on opac) in 995 field, like "book given by a teacher".
> This book will currently show on opac when some user search "vol", "fre",
> "book", "given", "teacher" or "Roma". For "Roma" I could maybe understand,
> but I don't like that. For the other points I don't like that AT ALL.
> 
> We have a rather big catalog (400 000 records), to Koha can easily find a
> lot of unrelated records for any search. With DOM I understand it will be
> worse.
> 
> Of course this issue could been less problematic, if Koha had a good
> pertinence sorting (it is mad), a good date sorting (it is broken, for
> unimarc at least), and a good facetting system (all we have are
> pseudofacets, non parametrable for the moment, based only on the X first
> results, not cumulable, and whose general presentation is not user
> friendly). We can work on these points, of course, but it would be easier to
> define a more reasonable "any" index.

These are reasonable concerns and coments, but IMO they are more for bug 8962 than this one.  Since the immediate concern of this bug -- adding an initial DOM configuration for UNIMARC authorities -- is now addressed in master and we have an explanation of the differences in "Any" authority searches in DOM vs. GRS-1, I'm setting this one to pushed to master.
Comment 57 mathieu saby 2013-10-13 08:21:43 UTC
> 
> These are reasonable concerns and coments, but IMO they are more for bug
> 8962 than this one.  

Right ;-)
Now that this one and the one for 1XX are pushed, I will rewrite 8962.

Mathieu