MARC21 authorities are indexed by Zebra using DOM or GRS1 filter. DOM is better. For example, it offer a full support of @ATTR 6=3 (field content completeness). Without this attribute, accurate authority headings matching is impossible. MARC21 DOM filter must be ported to UNIMARC.
Created attachment 7089 [details] [review] Proposed patch
Created attachment 10559 [details] [review] Bug 7421 Add UNIMARC DOM Filter Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Applies cleanly, didn't break zebra, I can't vouch it works right for UNIMARC but it doesn't break anything else.
Created attachment 10881 [details] [review] Bug 7421 Add UNIMARC DOM Filter Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
(In reply to comment #2) > Created attachment 10559 [details] [review] > Bug 7421 Add UNIMARC DOM Filter > > Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> QA comments - trivial and safe patch, - doesn't break existing functionality - both files pass ./t/00-valid-xml.t xml tests passing QA on this...
I started an install from scratch, with UNIMARC. Trying to reindex authorities, I get: 15:53 ~/koha.dev/koha-community/misc/migration_tools (new/bug_7421 $)$ ./rebuild_zebra.pl -a -v -r Zebra configuration information ================================ Zebra biblio directory = /home/paul/koha.dev/installs/unimarc-dom/var/lib/zebradb/biblios Zebra authorities directory = /home/paul/koha.dev/installs/unimarc-dom/var/lib/zebradb/authorities Koha directory = /home/paul/koha.dev/koha-community BIBLIONUMBER in : 001$@ BIBLIOITEMNUMBER in : 090$a ================================ ==================== exporting authority ==================== 1 Records exported: 1 ==================== REINDEXING zebra ==================== 15:53:24-18/07 zebraidx(14752) [warn] /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc/dom-config-marc.xml: stylesheet authority-zebra-indexdefs.xsl not found in path /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/etc:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/marc_defs/unimarc/authorities:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/lang_defs/fr:/home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/xsl investigating...
(In reply to comment #6) > 15:53:24-18/07 zebraidx(14752) [warn] > /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc/dom- > config-marc.xml: stylesheet authority-zebra-indexdefs.xsl not found in path > /home/paul/koha.dev/installs/unimarc-dom/etc/zebradb/authorities/etc:/home/ > paul/koha.dev/installs/unimarc-dom/etc/zebradb/etc:/home/paul/koha.dev/ > installs/unimarc-dom/etc/zebradb/marc_defs/unimarc/authorities:/home/paul/ > koha.dev/installs/unimarc-dom/etc/zebradb/lang_defs/fr:/home/paul/koha.dev/ > installs/unimarc-dom/etc/zebradb/xsl Strange... the authority-zebra-indexdefs.xsl is in my koha repo: ls ../../etc/zebradb/marc_defs/unimarc/authorities/ authority-koha-indexdefs.xml authority-zebra-indexdefs.xsl record.abs But looking in my unimarc-dom (the installation directory), they are not: 15:57 ~/koha.dev/koha-community/misc/migration_tools (new/bug_7421 $)$ ls ~/koha.dev/installs/unimarc-dom/etc/zebradb/marc_defs/unimarc/authorities/ record.abs Is it possible that they have not be copied by the make install ? reinstalling ... this time, I get my files at the proper place. It probably means that I was not on the bug 7421 git branch when I made the previous install, so the file did not exist. My bad probably. After reindexing, I get no results when searching. zebra logs say: 16:11:18-18/07 zebrasrv(1) [request] Search authorities ERROR 114 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @and @attr 5=100 @attr 1=authtype NP @attr 4=6 @attr 5=1 @attr 1=Heading-Main jean error 114 is unsupported attribute Same thing with yaz-client: Z> base authorities Z> f jean Sent searchRequest. Received SearchResponse. Search was a bloomin' failure. Number of hits: 0, setno 1 Result Set Status: none records returned: 0 Diagnostic message(s) from database: [114] Unsupported Use attribute -- v2 addinfo 'any' Frédéric, hint/explanation welcomed !!! {{{ Side comment: at the end of make install, I get chmod: impossible d'accéder à «/home/paul/koha.dev/installs/unimarcdom2/var/spool»: Aucun fichier ou dossier de ce type It's related to bug 8268, i'll create another entry for this problem }}}
bump ! Frédéric, see comment 7, I can't make it work !
(In reply to comment #8) > bump ! Frédéric, see comment 7, I can't make it work ! Frédéric, bump again !
It works for me, but I'm not sure to understand what's wrong with your installation. Is it an update or a new installation?
(In reply to comment #10) > It works for me, but I'm not sure to understand what's wrong with your > installation. Is it an update or a new installation? it was a fresh install
> it was a fresh install I've redone a new installation, choosing UNIMARC, and DOM. And it works out of the box. I've chosen a complete UNIMARC framework? Have you chosen the alternative framework?
Changing target version so this bug report will show up in the list of 3.12-targeted features and doesn't get lost in the shuffle during feature freeze.
This ENH has no side effect risk. I plan to try again, and would push it before the string freeze (oct 22th). So switching back to master.
Frédéric, I tried again, same problem. I made: * a dev mode install * from git source * All other Makefile.PL to default Any difference with what you made ?
In my opinion, that patch needs deep work. In fact, there are no Any, authtype or Heading-Main index defined and those indexes are required at the moment to complete all the requests. Do you have any fix for marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl file which adds those indexes ?
Created attachment 12852 [details] [review] Add UNIMARC DOM Filter Rebasing this patch on new installed server, I get your error! I fix it. Do we have UNIMARC 'communities' authorities records to test it more deeply?
Good news: i've tested this patch, and it's much better, it worked like a charm on a new setup !!! I'll ask Henri-Damien to do other tests. If he is successful, I think I could push this patch for 3.10 !
Some things could be fixed : - MARC21 should be changed into UNIMARC (in the comments) - indexing 008 as it is in MARC21 when there is no UNIMARC specification of 008 field is not really helpfull Some others HAVE TO be fixed : Search Main Heading with Default authtype contains wil : 11 results Some of which contains wil in $b which is not the current behaviour zebra search launched : Search authorities OK 11 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Heading wil Search Heading with Default authtype contains wil : 1 result which contains wil in $a only zebra search launched : OK 1 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Match wil Search Heading with Default authtype contains wil : 1 result which contains wil in $a only zebra search launched : OK 1 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Match wil Search Keyword Search with Default authtype contains wil : 7 results which contains wil in $a or $b zebra search launched : OK 7 1 1+0 RPN @attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 4=6 @attr 5=1 @attr 1=Any wil I find it much strange that Any is not expanding in all the indexed subfields and that keyword search would not return more results than the heading one.
Created attachment 13037 [details] UNIMARC authorities file to test the indexing This file contains 1000 authorities to test the indexing.
Created attachment 13051 [details] Extract of auth_header for test.
Hello Is someone still working on that bug? M. Saby
(In reply to comment #22) > Hello > Is someone still working on that bug? > > M. Saby I think / it seems no (reminder: laurenthdl, aka Henri-Damien, that made the last comment, was co-owner & founder of BibLibre with me, and he left BibLibre last dec 31 -see http://www.biblibre.com/en/blog/entry/biblibre-takes-another-next-step -)
I joint an alternative patch. I took as a base the patch of F. Demians, but made a lot of changes, so I think it is more logical to create a new patch as the behavior is not the same as previous patch. I was not able to test, I don't have currently a Koha with DOM. M. Saby
Created attachment 17261 [details] [review] [PATCH][alternative patch] Bug 7421 Add UNIMARC DOM Filter I tried to define DOM config files as a "miror" of record.abs, so the behavior be the same. If it is OK, we will be able to improve indexing later, for example suppressing warns, managing indicators or subdivisions, etc. I made some little changes to record.abs : - comments - 216 was indexed in Conference-name as well as Trademark. I supposes "Conference-name" is an error, so I indexed only in Trademark - index 2 new notes : 340 / 356 The only difference between record.abs and DOM is that DOM config files does not index complete fields, but subfields. Ex : melm 200 ===> <kohaidx:index_subfields tag="200" subfields="abcdfgjxyz"> I took all the subfields from the UNIMARC Authorities manual. The only subfields not indexed are numeric subfields : $7, $8 for language of record, and $0,2,3,5,6 for 4XX/5XX/7XX To test : - index a set of bib and auth records with GRS-1 - make some searches on different kind of authorities - index the same records with DOM - make the same searches - You are not supposed to see differences
I tested my patch (attachment 17261 [details] [review]) on a DOM environment. I think it works. But I forgot one step in testing : in order to force Zebra to take into account the new config files, I had to copy them from your sources to your main koha directory, before launching rebuild_zebra -a -x -r -v. For example, from home/msaby/koha-dev/src/etc/zebra_db/marc_defs/unimarc/authorities to home/msaby/koha-dev/etc/zebra_db/marc_defs/unimarc/authorities So, you can test. M. Saby
Created attachment 20876 [details] [review] [SIGNED OFF] Bug 7421 Add UNIMARC DOM Filter I took as a base the patch of F. Demians, but made a lot of changes, so I think it is more logical to create a new patch as the behavior is not the same as previous patch. I tried to define DOM config files as a "miror" of record.abs, so the behavior be the same. If it is OK, we will be able to improve indexing later, for example suppressing warns, managing indicators or subdivisions, etc. I made some little changes to record.abs : - comments - 216 was indexed in Conference-name as well as Trademark. I supposes "Conference-name" is an error, so I indexed only in Trademark - index 2 new notes : 340 / 356 The only difference between record.abs and DOM is that DOM config files does not index complete fields, but subfields. Ex : melm 200 ===> <kohaidx:index_subfields tag="200" subfields="abcdfgjxyz"> I took all the subfields from the UNIMARC Authorities manual. The only subfields not indexed are numeric subfields : $7, $8 for language of record, and $0,2,3,5,6 for 4XX/5XX/7XX To test : - index a set of bib and auth records with GRS-1 - make some searches on different kind of authorities - index the same records with DOM - make the same searches - You are not supposed to see differences Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> As I am not a UNIMARC user it's hard for me to test this, but while testing other authority related patches I noticed that I couldn't index the UNIMARC authorities of the sample base. The files are obviously missing and reindex_zebra.pl notes this. With this patch applied, indexing works and authorities are searchable in my installation.
I have gone ahead and signed off on this, as with this patch applied I was able to get authority searching working with DOM indexing on my test installation and it seems to work well. As I am by no means a UNIMARC expert this is a bit of a weak sign off - Jonathan, could you take a closer look in QA and/or request another sign off?
A french librarian wanted to do some tests on "unimarc patches" next week, but I don't know if he had planned to look at this one. I will ask him. Mathieu
And a great thank you for testing and signing off! I hope french libraries will be able to use DOM in 3.14... Mathieu
THis patch works fine for me!
Created attachment 21840 [details] [review] [SIGNED-OFF] Bug 7421 Add UNIMARC DOM Filter I took as a base the patch of F. Demians, but made a lot of changes, so I think it is more logical to create a new patch as the behavior is not the same as previous patch. I tried to define DOM config files as a "miror" of record.abs, so the behavior be the same. If it is OK, we will be able to improve indexing later, for example suppressing warns, managing indicators or subdivisions, etc. I made some little changes to record.abs : - comments - 216 was indexed in Conference-name as well as Trademark. I supposes "Conference-name" is an error, so I indexed only in Trademark - index 2 new notes : 340 / 356 The only difference between record.abs and DOM is that DOM config files does not index complete fields, but subfields. Ex : melm 200 ===> <kohaidx:index_subfields tag="200" subfields="abcdfgjxyz"> I took all the subfields from the UNIMARC Authorities manual. The only subfields not indexed are numeric subfields : $7, $8 for language of record, and $0,2,3,5,6 for 4XX/5XX/7XX To test : - index a set of bib and auth records with GRS-1 - make some searches on different kind of authorities - index the same records with DOM - make the same searches - You are not supposed to see differences Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> As I am not a UNIMARC user it's hard for me to test this, but while testing other authority related patches I noticed that I couldn't index the UNIMARC authorities of the sample base. The files are obviously missing and reindex_zebra.pl notes this. With this patch applied, indexing works and authorities are searchable in my installation. [2013-10-07] Added second sign off from Vitor. Signed-off-by: Vitor Fernandes <fvernandes@keep.pt>
QA comment: I retry same tests as Henri Damien: - go on authorities/authorities-home.pl - Select "default" "contains" "wil" "main heading" * GSR-1: 27 results * Dom : 27 results - Select "default" "contains" "wil" "all headings" * GSR-1: 0 result * Dom : 27 results - Select "default" "contains" "wil" "Keyword search" * GSR-1: 27 results * Dom : 29 results Marked as Failed QA.
(Maybe I did an error somewhere, I start to be tired to parse xml with my eyes...)
(In reply to Jonathan Druart from comment #33) > QA comment: > > I retry same tests as Henri Damien: > > - go on authorities/authorities-home.pl > - Select "default" "contains" "wil" "main heading" > * GSR-1: 27 results > * Dom : 27 results > - Select "default" "contains" "wil" "all headings" > * GSR-1: 0 result > * Dom : 27 results > - Select "default" "contains" "wil" "Keyword search" > * GSR-1: 27 results > * Dom : 29 results > > Marked as Failed QA. I will look at it. In what precise field/subfield did you put the "wil" string? Maybe the small discrepancy for Keyword search exists also between MARC21 DOM and GRS1? Mathieu
I am not sure I understood how Koha is searching authorities, so I explored the code. In authorities-search.inc : - you have 3 choices (main heading, match heading, all headings. - each choice is linked with a value of "marclist" field, passed in the form in an <input>. - main heading => marclist=mainentry - match heading => marclist=match - all heading => marclist=all These data are passed to authorities-home.pl In authorities-home.pl : my ( $results, $total ) = SearchAuthorities( [$marclist], [$and_or], [$excluding], [$operator], [$value], ( $startfrom - 1 ) * $resultsperpage, $resultsperpage, $authtypecode, $orderby ); In C4/AuthoritiesMarc.pm : sub SearchAuthorities { my ($tags, $and_or, $excluding, $operator, $value, $offset,$length,$authtypecode,$sortby,$skipmetadata) = @_; ... elsif ( @$tags[$i] eq "mainentry" ) { $attr = " \@attr 1=Heading "; ... elsif ( @$tags[$i] eq "match" ) { $attr = " \@attr 1=Match "; ... else { # Assume any if no index was specified $attr = " \@attr 1=Any "; There is no specific choice for the "all" value, so the "any" index is used. I wonder if it is normal. But your failing case - Select "default" "contains" "wil" "all headings" > * GSR-1: 0 result > * Dom : 27 results probably means that "Any" index is broken. Mathieu
In record.abs for GRS1 we have : " all any" It is "translated" in DOM zebra-xsl file as : + <xslo:template mode="index_all" match="text()"> + <z:index name="Any:w Any:p"> + <xslo:value-of select="."/> + </z:index> + </xslo:template> And I don't find it in DOM koha-xml file. I don't remember how I did that. It is probably the xslt transformation stylesheet that added those lines, and maybe it is not good. More investigation later... Mathieu
I'm taking a chance on this and have pushed it to master (and the reason why I'm taking a chance is that I've also written and pushed a regression test that shows, with the dataset provided by Henri-Damien, that GRS-1 and DOM indexing are behaving the same). I'm not sure where Jonathan's 27 hits on "wil" are coming from -- only 11 records in the sample file have that sequence of letters at all. Are you using a different dataset? If so, could you attach the records or a download link to the bug? I'm setting to in discussion rather than pushed to master in case it turns out that I was too precipitous and a revert is needed.
Hi (In reply to Galen Charlton from comment #38) > I'm taking a chance on this and have pushed it to master (and the reason why > I'm taking a chance is that I've also written and pushed a regression test Is it attached to an other bug ? > that shows, with the dataset provided by Henri-Damien, I forgot there were a sample of records attached to the bug ;-) that GRS-1 and DOM > indexing are behaving the same). > Mathieu
Galen, I tested with the dataset provide by Mathieu (mail "pending bugs about UNIMARC marc flavor": https://www.dropbox.com/s/idw98d7jiy5e5xd/dump_unimarc.zip).
For info, here is the content of HDL file =LDR 00166 a2200085 4500 =001 1 =100 \\$a20121025 frey50 =999 \\$a1 =152 \\$bNP =200 \\$aMeusburger$bMatthias =LDR 00143 a2200073 4500 =001 2 =100 \\$a20121025 frey50 =200 \\$aBelaïdi$bAlice =152 \\$bNP =LDR 00147 a2200073 4500 =001 3 =100 \\$a20121025 frey50 =210 \\$aMartires del compas =152 \\$bCO =LDR 00138 a2200073 4500 =001 4 =100 \\$a20121025 frey50 =200 \\$aMrak$bMira =152 \\$bNP =LDR 00145 a2200073 4500 =001 5 =100 \\$a20121025 frey50 =200 \\$aWaffler$bEmmanuel =152 \\$bNP =LDR 00144 a2200073 4500 =001 6 =100 \\$a20121025 frey50 =200 \\$aManjarres$bBruno =152 \\$bNP =LDR 00142 a2200073 4500 =001 7 =100 \\$a20121025 frey50 =200 \\$aMartinez$bPepe =152 \\$bNP =LDR 00143 a2200073 4500 =001 8 =100 \\$a20121025 frey50 =200 \\$aNeveu$bPhilippe =152 \\$bNP =LDR 00136 a2200073 4500 =001 9 =100 \\$a20121025 frey50 =210 \\$aZaragraf =152 \\$bCO =LDR 00142 a2200073 4500 =001 12 =100 \\$a20121025 frey50 =200 \\$aKali Live Dub =152 \\$bNP =LDR 00133 a2200073 4500 =001 13 =100 \\$a20121025 frey50 =200 \\$aAvia =152 \\$bNP =LDR 00143 a2200073 4500 =001 14 =100 \\$a20121025 frey50 =200 \\$aFugitive (The) =152 \\$bNP =LDR 00137 a2200073 4500 =001 15 =100 \\$a20121025 frey50 =200 \\$aThésée =152 \\$bNP =LDR 00136 a2200073 4500 =001 16 =100 \\$a20121025 frey50 =210 \\$aZenzile =152 \\$bCO =LDR 00150 a2200073 4500 =001 17 =100 \\$a20121025 frey50 =210 \\$aAir$cDuo instrumental =152 \\$bCO =LDR 00138 a2200073 4500 =001 18 =100 \\$a20121025 frey50 =210 \\$aRoudoudou =152 \\$bCO =LDR 00138 a2200073 4500 =001 19 =100 \\$a20121025 frey50 =210 \\$aSeven Dub =152 \\$bCO =LDR 00145 a2200073 4500 =001 20 =100 \\$a20121025 frey50 =210 \\$aWilliams Traffic =152 \\$bCO =LDR 00152 a2200073 4500 =001 21 =100 \\$a20121025 frey50 =210 \\$a Le Peuple de l'Herbe =152 \\$bCO =LDR 00142 a2200073 4500 =001 22 =100 \\$a20121025 frey50 =210 \\$aSporto Kantes =152 \\$bCO =LDR 00141 a2200073 4500 =001 23 =100 \\$a20121025 frey50 =210 \\$aOto Electrik =152 \\$bCO =LDR 00142 a2200073 4500 =001 24 =100 \\$a20121025 frey50 =210 \\$aGotan Project =152 \\$bCO =LDR 00136 a2200073 4500 =001 25 =100 \\$a20121025 frey50 =210 \\$aEz3kiel =152 \\$bCO =LDR 00141 a2200073 4500 =001 26 =100 \\$a20121025 frey50 =210 \\$aBrain Damage =152 \\$bCO =LDR 00132 a2200073 4500 =001 27 =100 \\$a20121025 frey50 =210 \\$aArk =152 \\$bCO =LDR 00134 a2200073 4500 =001 28 =100 \\$a20121025 frey50 =210 \\$aOctet =152 \\$bCO =LDR 00137 a2200073 4500 =001 29 =100 \\$a20121025 frey50 =200 \\$aBeyoncé =152 \\$bNP =LDR 00149 a2200073 4500 =001 30 =100 \\$a20121025 frey50 =200 \\$aWashington$bBrittani =152 \\$bNP =LDR 00142 a2200073 4500 =001 31 =100 \\$a20121025 frey50 =200 \\$aChapa$bMarcie =152 \\$bNP =LDR 00143 a2200073 4500 =001 32 =100 \\$a20121025 frey50 =200 \\$aGlaspie$bNikki =152 \\$bNP =LDR 00144 a2200073 4500 =001 33 =100 \\$a20121025 frey50 =200 \\$aIfrah$bLaurence =152 \\$bNP =LDR 00145 a2200073 4500 =001 34 =100 \\$a20121025 frey50 =200 \\$aCassourra$bAlain =152 \\$bNP =LDR 00143 a2200073 4500 =001 35 =100 \\$a20121025 frey50 =200 \\$aGrimm$bWilhelm =152 \\$bNP =LDR 00149 a2200073 4500 =001 36 =100 \\$a20121025 frey50 =200 \\$aSpirine$bGuennadi K. =152 \\$bNP =LDR 00155 a2200073 4500 =001 37 =100 \\$a20121025 frey50 =200 \\$aTchekhov$bAnton Pavlovitch =152 \\$bNP =LDR 00142 a2200073 4500 =001 38 =100 \\$a20121025 frey50 =200 \\$aLissitsky$bEl =152 \\$bNP =LDR 00148 a2200073 4500 =001 39 =100 \\$a20121025 frey50 =200 \\$aOusatschev$bAndreï =152 \\$bNP =LDR 00140 a2200073 4500 =001 40 =100 \\$a20121025 frey50 =200 \\$aLô$bCheikh =152 \\$bNP =LDR 00142 a2200073 4500 =001 41 =100 \\$a20121025 frey50 =200 \\$aSarr$bThierno =152 \\$bNP =LDR 00138 a2200073 4500 =001 42 =100 \\$a20121025 frey50 =200 \\$aSow$bOmar =152 \\$bNP =LDR 00140 a2200073 4500 =001 43 =100 \\$a20121025 frey50 =200 \\$aAllen$bTony =152 \\$bNP =LDR 00143 a2200073 4500 =001 44 =100 \\$a20121025 frey50 =200 \\$aEllis$bPee Wee =152 \\$bNP =LDR 00149 a2200073 4500 =001 45 =100 \\$a20121025 frey50 =200 \\$aTall$bCheikh Tidiane =152 \\$bNP =LDR 00142 a2200073 4500 =001 46 =100 \\$a20121025 frey50 =200 \\$aShin$bSe-jung =152 \\$bNP =LDR 00143 a2200073 4500 =001 47 =100 \\$a20121025 frey50 =200 \\$aZanasi$bGianni =152 \\$bNP =LDR 00145 a2200073 4500 =001 48 =100 \\$a20121025 frey50 =200 \\$aMurino$bCaterina =152 \\$bNP =LDR 00148 a2200073 4500 =001 49 =100 \\$a20121025 frey50 =200 \\$aMastandrea$bValerio =152 \\$bNP =LDR 00148 a2200073 4500 =001 50 =100 \\$a20121025 frey50 =200 \\$aBattiston$bGiuseppe =152 \\$bNP =LDR 00144 a2200073 4500 =001 51 =100 \\$a20121025 frey50 =200 \\$aRobillot$bHenri =152 \\$bNP =LDR 00150 a2200073 4500 =001 52 =100 \\$a20121025 frey50 =200 \\$aSan Millan$bFrançois =152 \\$bNP =LDR 00145 a2200073 4500 =001 53 =100 \\$a20121025 frey50 =200 \\$aLecaye$bEmmanuel =152 \\$bNP =LDR 00148 a2200073 4500 =001 54 =152 \\$bSNC =100 \\$a20121025 frey50 =250 \\$aMusique$ySénégal =LDR 00143 a2200073 4500 =001 55 =100 \\$a20121025 frey50 =200 \\$aScherfig$bLone =152 \\$bNP =LDR 00141 a2200073 4500 =001 56 =100 \\$a20121025 frey50 =200 \\$aHornby$bNick =152 \\$bNP =LDR 00144 a2200073 4500 =001 57 =100 \\$a20121025 frey50 =200 \\$aEnglishby$bPaul =152 \\$bNP =LDR 00145 a2200073 4500 =001 58 =100 \\$a20121025 frey50 =200 \\$aSarsgaard$bPeter =152 \\$bNP =LDR 00144 a2200073 4500 =001 59 =100 \\$a20121025 frey50 =200 \\$aMulligan$bCarey =152 \\$bNP =LDR 00143 a2200073 4500 =001 60 =100 \\$a20121025 frey50 =200 \\$aMolina$bAlfred =152 \\$bNP =LDR 00174 a2200073 4500 =001 61 =100 \\$a20121025 frey50 =250 \\$aExtraterrestres$xCédéroms pour la jeunesse =152 \\$bSNC =LDR 00173 a2200073 4500 =001 62 =100 \\$a20121025 frey50 =250 \\$aChauves-souris$xCédéroms pour la jeunesse =152 \\$bSNC =LDR 00165 a2200073 4500 =001 63 =100 \\$a20121025 frey50 =250 \\$aBourse$xCédéroms pour la jeunesse =152 \\$bSNC =LDR 00165 a2200073 4500 =001 64 =100 \\$a20121025 frey50 =250 \\$aFêtes$xCédéroms pour la jeunesse =152 \\$bSNC =LDR 00166 a2200073 4500 =001 65 =100 \\$a20121025 frey50 =250 \\$aAutomne$xCédéroms pour la jeunesse =152 \\$bSNC =LDR 00151 a2200073 4500 =001 66 =152 \\$bSNC =100 \\$a20121025 frey50 =250 \\$aMusique$yBurkina Faso =LDR 00143 a2200073 4500 =001 67 =100 \\$a20121025 frey50 =200 \\$aDémé$bVictor =152 \\$bNP =LDR 00145 a2200073 4500 =001 68 =100 \\$a20121025 frey50 =200 \\$aDiabaté$bIssouf =152 \\$bNP =LDR 00142 a2200073 4500 =001 69 =100 \\$a20121025 frey50 =200 \\$aDiarra$bSalif =152 \\$bNP =LDR 00140 a2200073 4500 =001 70 =100 \\$a20121025 frey50 =200 \\$aDiarra$bAli =152 \\$bNP =LDR 00138 a2200073 4500 =001 71 =100 \\$a20121025 frey50 =200 \\$aKoudédé =152 \\$bNP =LDR 00145 a2200073 4500 =001 72 =100 \\$a20121025 frey50 =200 \\$aAnderson$bLaurie =152 \\$bNP =LDR 00138 a2200073 4500 =001 73 =100 \\$a20121025 frey50 =200 \\$aReed$bLou =152 \\$bNP =LDR 00143 a2200073 4500 =001 74 =100 \\$a20121025 frey50 =200 \\$aScherer$bPeter =152 \\$bNP =LDR 00140 a2200073 4500 =001 75 =100 \\$a20121025 frey50 =200 \\$aBurger$bRob =152 \\$bNP =LDR 00140 a2200073 4500 =001 76 =100 \\$a20121025 frey50 =200 \\$aHakim$bOmar =152 \\$bNP =LDR 00139 a2200073 4500 =001 77 =100 \\$a20121025 frey50 =200 \\$aZorn$bJohn =152 \\$bNP =LDR 00135 a2200073 4500 =001 78 =100 \\$a20121025 frey50 =200 \\$aAntony =152 \\$bNP =LDR 00160 a2200073 4500 =001 79 =100 \\$a20121025 frey50 =250 \\$aJeux de simulation$xCédéroms =152 \\$bSNC =LDR 00168 a2200073 4500 =001 80 =100 \\$a20121025 frey50 =250 \\$aGuerre mondiale (1939-1945)$xLudiciels =152 \\$bSNC =LDR 00142 a2200073 4500 =001 81 =100 \\$a20121025 frey50 =200 \\$aAimée$bAnouk =152 \\$bNP =LDR 00144 a2200073 4500 =001 82 =100 \\$a20121025 frey50 =200 \\$aPiccoli$bMichel =152 \\$bNP =LDR 00145 a2200073 4500 =001 83 =100 \\$a20121025 frey50 =200 \\$aPlacido$bMichele =152 \\$bNP =LDR 00142 a2200073 4500 =001 84 =100 \\$a20121025 frey50 =200 \\$aMiller$bSteve =152 \\$bNP =LDR 00144 a2200073 4500 =001 85 =100 \\$a20121025 frey50 =200 \\$aBuffalo$bNorton =152 \\$bNP =LDR 00143 a2200073 4500 =001 86 =100 \\$a20121025 frey50 =200 \\$aCharles$bSonny =152 \\$bNP =LDR 00144 a2200073 4500 =001 87 =100 \\$a20121025 frey50 =200 \\$aKnudtson$bGordy =152 \\$bNP =LDR 00144 a2200073 4500 =001 88 =100 \\$a20121025 frey50 =200 \\$aPeterson$bBilly =152 \\$bNP =LDR 00144 a2200073 4500 =001 89 =100 \\$a20121025 frey50 =200 \\$aClaerr$bThierry =152 \\$bNP =LDR 00146 a2200073 4500 =001 90 =100 \\$a20121025 frey50 =200 \\$aWesteel$bIsabelle =152 \\$bNP =LDR 00148 a2200073 4500 =001 91 =100 \\$a20121025 frey50 =200 \\$aSauvage$bChristiane =152 \\$bNP =LDR 00171 a2200073 4500 =001 92 =100 \\$a20121025 frey50 =210 \\$aAssociation des bibliothécaires français =152 \\$bCO =LDR 00143 a2200073 4500 =001 93 =100 \\$a20121025 frey50 =200 \\$aMundy$bAnthony =152 \\$bNP =LDR 00150 a2200073 4500 =001 94 =100 \\$a20121025 frey50 =200 \\$aMuldrow$bGeorgia Anne =152 \\$bNP =LDR 00174 a2200073 4500 =001 95 =152 \\$bSNC =100 \\$a20121025 frey50 =250 \\$aMusique$yCongo (République Démocratique) =LDR 00141 a2200073 4500 =001 96 =100 \\$a20121025 frey50 =200 \\$aNsangue$bGuy =152 \\$bNP =LDR 00139 a2200073 4500 =001 97 =100 \\$a20121025 frey50 =200 \\$aTran$bAlex =152 \\$bNP =LDR 00141 a2200073 4500 =001 98 =100 \\$a20121025 frey50 =200 \\$aIpupa$bFally =152 \\$bNP =LDR 00145 a2200073 4500 =001 99 =100 \\$a20121025 frey50 =200 \\$aGalison$bWilliam =152 \\$bNP =LDR 00143 a2200073 4500 =001 100 =100 \\$a20121025 frey50 =200 \\$aClerc$bArnaud =152 \\$bNP =LDR 00138 a2200073 4500 =001 101 =100 \\$a20121025 frey50 =200 \\$aWiretree =152 \\$bNP =LDR 00147 a2200073 4500 =001 102 =100 \\$a20121025 frey50 =200 \\$aBlanchard$bDaniel =152 \\$bNP Mathieu
oups, maybe it is was not a good idea to copy all the records. too long... Mathieu
I think if we want a set of records for testing, we should take only a small file with 10 records from the "real word". With a small set, if there is a oddity in results it would be easier to find why. Mathieu
Here is a set of 11 auth record from my catalog. unicode encoding. Diffent types of auths, rejected forms etc. I join 2 file : marc raw file, and marc file "translated" to plain text, to make verifications easier.
Created attachment 21956 [details] MSAby - 11 auth records
Created attachment 21957 [details] MSAby - 11 auth records (plain text)
(In reply to mathieu saby from comment #39) > Hi > (In reply to Galen Charlton from comment #38) > > I'm taking a chance on this and have pushed it to master (and the reason why > > I'm taking a chance is that I've also written and pushed a regression test > > Is it attached to an other bug ? No -- see the following commit: http://git.koha-community.org/gitweb/?p=koha.git;a=commitdiff;h=521e599dd82061fd10321d681d06d1409134d39c
(In reply to Jonathan Druart from comment #40) > Galen, > I tested with the dataset provide by Mathieu (mail "pending bugs about > UNIMARC marc flavor": > https://www.dropbox.com/s/idw98d7jiy5e5xd/dump_unimarc.zip). Thanks. I took this file and adapted the test cases I wrote yesterday. Here are the results I got: [1] main heading contains "wil" GRS-1 = 27 hits DOM = 27 hits [2] all headings contains "wil" GRS-1 = 27 hits DOM = 27 hits [3] keyword contains "wil" GRS-1 = 27 hits DOM = 29 hits The following indicates that the DOM filter is not necessarily wrong: $ perl -ne 'BEGIN { $/ = "\035"; } END { print "$ct\n" } { $ct++ if /wil/i; }' < unimarc_auth_3.mrc 29 IOW, there really are 29 records that contain the substring "wil". I'll investigate further to see why the GRS-1 any index is behaving slightly differently.
(In reply to Galen Charlton from comment #48) > I'll investigate further to see why the GRS-1 any index is behaving slightly > differently. Because GRS-1's "any" directive does not include all fields, whereas the DOM "equivalent" does.
(In reply to Jared Camins-Esakov from comment #49) > (In reply to Galen Charlton from comment #48) > > I'll investigate further to see why the GRS-1 any index is behaving slightly > > differently. > > Because GRS-1's "any" directive does not include all fields, whereas the DOM > "equivalent" does. all fields? you mean really ALL fields even if they are not used by an other index? That must very annoying for users! I made a patch some time ago for repacing "any" with "anywhere" and make a reasonable "any" index, like we can found in other ILS. I made it for GRS1, but it may be even more usefull for DOM. Bug 8962 (need some rewrite I suppose) M. Saby
I am not sure if it's annoying - I always tell people that everything that is indexed can be searched in keyword search and the reactions so far are quite positive.
(In reply to mathieu saby from comment #50) > (In reply to Jared Camins-Esakov from comment #49) > > (In reply to Galen Charlton from comment #48) > > > I'll investigate further to see why the GRS-1 any index is behaving slightly > > > differently. > > > > Because GRS-1's "any" directive does not include all fields, whereas the DOM > > "equivalent" does. > > all fields? you mean really ALL fields even if they are not used by an other > index? That must very annoying for users! > I made a patch some time ago for repacing "any" with "anywhere" and make a > reasonable "any" index, like we can found in other ILS. I made it for GRS1, > but it may be even more usefull for DOM. > Bug 8962 (need some rewrite I suppose) Yeah, ideally there would be a special keyword index for DOM that included all the normal keyword-ish fields and note fields, but for me a higher priority was getting keyword working at all.
(In reply to Jared Camins-Esakov from comment #49) > (In reply to Galen Charlton from comment #48) > > I'll investigate further to see why the GRS-1 any index is behaving slightly > > differently. > > Because GRS-1's "any" directive does not include all fields, whereas the DOM > "equivalent" does. The "all" directive, in particular, just specifies one or more attributes that are tacked on to the attribute list for each element being indexed. The two "wil" records that DOM includes in the "any" index have the "William" string in the 810 field. Since the UNIMARC GRS-1 record.abs does not index the 810 at all, it is likewise not included in the "any" index. By contrast, the DOM XSL index_all option indexes all text strings present in the record.
(In reply to Katrin Fischer from comment #51) > I am not sure if it's annoying - I always tell people that everything that > is indexed can be searched in keyword search and the reactions so far are > quite positive. I agree -- it's easier to explain that an "any" index includes everything (though whether or not fixed fields should be included is a matter of opinion), so I for one don't really see a problem with the change in behavior of the "any" index. Note, though, that that's a separate question from whether the "any" index should be included in, say, auth_finder.pl, as somebody looking for authority records via that interface has little reason to be using anything other than the headings searches to begin with.
(In reply to Galen Charlton from comment #54) > (In reply to Katrin Fischer from comment #51) > > I am not sure if it's annoying - I always tell people that everything that > > is indexed can be searched in keyword search and the reactions so far are > > quite positive. > > I agree -- it's easier to explain that an "any" index includes everything > (though whether or not fixed fields should be included is a matter of > opinion), so I for one don't really see a problem with the change in > behavior of the "any" index. > > Note, though, that that's a separate question from whether the "any" index > should be included in, say, auth_finder.pl, as somebody looking for > authority records via that interface has little reason to be using anything > other than the headings searches to begin with. Sorry, but I do not agree with both of you. I'm speaking here in general, not for authorities (we don't give access to authorities on opac). The OPAC search with GRS1 is very "noisy", with a lot of unwanted and non understandable results. Ex : I have a book published in "Roma". It has "1 vol. (215 p.)", it is written in french ("fre" in 105a) with an item having an INTERNAL note (not shown on opac) in 995 field, like "book given by a teacher". This book will currently show on opac when some user search "vol", "fre", "book", "given", "teacher" or "Roma". For "Roma" I could maybe understand, but I don't like that. For the other points I don't like that AT ALL. We have a rather big catalog (400 000 records), to Koha can easily find a lot of unrelated records for any search. With DOM I understand it will be worse. Of course this issue could been less problematic, if Koha had a good pertinence sorting (it is mad), a good date sorting (it is broken, for unimarc at least), and a good facetting system (all we have are pseudofacets, non parametrable for the moment, based only on the X first results, not cumulable, and whose general presentation is not user friendly). We can work on these points, of course, but it would be easier to define a more reasonable "any" index. Mathieu
(In reply to mathieu saby from comment #55) > (In reply to Galen Charlton from comment #54) > > (In reply to Katrin Fischer from comment #51) > > > I am not sure if it's annoying - I always tell people that everything that > > > is indexed can be searched in keyword search and the reactions so far are > > > quite positive. > > > > I agree -- it's easier to explain that an "any" index includes everything > > (though whether or not fixed fields should be included is a matter of > > opinion), so I for one don't really see a problem with the change in > > behavior of the "any" index. > > > > Note, though, that that's a separate question from whether the "any" index > > should be included in, say, auth_finder.pl, as somebody looking for > > authority records via that interface has little reason to be using anything > > other than the headings searches to begin with. > > Sorry, but I do not agree with both of you. > I'm speaking here in general, not for authorities (we don't give access to > authorities on opac). The OPAC search with GRS1 is very "noisy", with a lot > of unwanted and non understandable results. > Ex : I have a book published in "Roma". It has "1 vol. (215 p.)", it is > written in french ("fre" in 105a) with an item having an INTERNAL note (not > shown on opac) in 995 field, like "book given by a teacher". > This book will currently show on opac when some user search "vol", "fre", > "book", "given", "teacher" or "Roma". For "Roma" I could maybe understand, > but I don't like that. For the other points I don't like that AT ALL. > > We have a rather big catalog (400 000 records), to Koha can easily find a > lot of unrelated records for any search. With DOM I understand it will be > worse. > > Of course this issue could been less problematic, if Koha had a good > pertinence sorting (it is mad), a good date sorting (it is broken, for > unimarc at least), and a good facetting system (all we have are > pseudofacets, non parametrable for the moment, based only on the X first > results, not cumulable, and whose general presentation is not user > friendly). We can work on these points, of course, but it would be easier to > define a more reasonable "any" index. These are reasonable concerns and coments, but IMO they are more for bug 8962 than this one. Since the immediate concern of this bug -- adding an initial DOM configuration for UNIMARC authorities -- is now addressed in master and we have an explanation of the differences in "Any" authority searches in DOM vs. GRS-1, I'm setting this one to pushed to master.
> > These are reasonable concerns and coments, but IMO they are more for bug > 8962 than this one. Right ;-) Now that this one and the one for 1XX are pushed, I will rewrite 8962. Mathieu