Currently, 024$a is indexed into Identifier-other:w, even when it is a URI (e.g. http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14217) This causes problems because the "w" index type replaces punctuation with spaces, and tokenizes on spaces, so that the URI is decomposed into a series of values which are indexed separately. This is definitely not what you want when indexing a 024$a when it is a URI. For example: The url "http://libris.kb.se/resource/bib/219553" becomes the following: <index name="Identifier-other" type="w" seq="28">@^</index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="29">http</index> <index name="Identifier-other" type="w" seq="30">libris</index> <index name="Identifier-other" type="w" seq="31">kb</index> <index name="Identifier-other" type="w" seq="32">se</index> <index name="Identifier-other" type="w" seq="33">resource</index> <index name="Identifier-other" type="w" seq="34">bib</index> <index name="Identifier-other" type="w" seq="35">219553</index> Fortunately, the 024$2 subfield value tells us the source of the identifier, and "uri" is one of the valid options. So, when we have a 024$2=uri, we can index the 024$a using the "url" index type. (I'm also planning to index into the "phrase" index type for all 024$a as it performs the normalization but it doesn't tokenize based on the spaces, so this normal form may still be of use for urls and other identifiers that rely on punctuation for meaning.)
Created attachment 46512 [details] [review] Bug 15555 - Index 024$a into Identifier-other:u url register when source $2 is uri This patch indexes 024$a into the "phrase" index type, and the "url" index type, if the 024$2 equals "uri". TEST PLAN 1) Apply the patch. 1b) If you're using a gitified Koha or a git install, you'll need to upgrade your instance or copy your zebradb files over to /etc/koha/zebradb or your "kohadev" directory. 2) Add a 024$a with a URL like http://libris.kb.se/resource/bib/219553 to a bibliographic record 3) Re-index Zebra 4) Type "id-other,st-urx,fuzzy=http://libris.kb.se/resource/bib/219553" into the "Search the catalog" box in the Staff Client and search 5) Note that you retrieve your record NOTE: The fuzzy is required because Koha's query "parsing" functions change http:// to http=// which won't correctly match the value in the "Identifier-other:u" index. NOTE: Alternatively, you could do the following search instead: "id-other,phr=http libris kb se resource bib 219553". It would work as well by using the "Identifier-other:p" index. Advanced tester version: 4) In a terminal window, find the "koha-conf.xml" file in your "etc" directory. 5) Open "koha-conf.xml" and find <listen id="biblioserver">. Copy the URI you find there. (e.g. unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket). 6) Type "yaz-client unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket" 7) After it connects, type "base biblios" and press enter 8) Type "format xml" and press enter 9) Type "elements zebra::index" and press enter 10) Type "f id-other,st-urx=http://libris.kb.se/resource/bib/219553" and press enter 11) Note that you should have at least one result 12) Type "show 1" 13) If you scroll through the results, you should find something like the following: <index name="Identifier-other" type="w" seq="28">@^</index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="29">http</index> <index name="Identifier-other" type="w" seq="30">libris</index> <index name="Identifier-other" type="w" seq="31">kb</index> <index name="Identifier-other" type="w" seq="32">se</index> <index name="Identifier-other" type="w" seq="33">resource</index> <index name="Identifier-other" type="w" seq="34">bib</index> <index name="Identifier-other" type="w" seq="35">219553</index> <index name="Identifier-other" type="p" seq="28">http libris kb se resource bib 219553</index> <index name="Identifier-other" type="u" seq="36">http://libris.kb.se/resource/bib/219553</index>
Advanced testers: Here's the steps you need to do to upgrade your Koha's Zebra config, if you're using a git dev install: cd /home/dcook/git INSTALL_LOG="/home/dcook/koha-dev/misc/koha-install-log" cp $INSTALL_LOG backup-koha-install-log perl Makefile.PL --prev-install-log "$INSTALL_LOG" make make upgrade
NOTE: The output I've provided in the above comments has come from a Koha using CHR indexing. ICU indexing has different output, but the same behaviour. Here's what I see with "format xml" and "elements zebra::index": <index name="Identifier-other" type="w" seq="21"></index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="22"></index> <index name="Identifier-other" type="w" seq="23"></index> <index name="Identifier-other" type="w" seq="24"></index> <index name="Identifier-other" type="w" seq="25"></index> <index name="Identifier-other" type="p" seq="21"></index> <index name="Identifier-other" type="p" seq="22"></index> <index name="Identifier-other" type="p" seq="23"></index> <index name="Identifier-other" type="p" seq="24"></index> <index name="Identifier-other" type="p" seq="25"></index> <index name="Identifier-other" type="u" seq="26">http://libris.kb.se/resource/bib/219553</index> Here's what I see with "format xml" and "elements index": <z:index name="Identifier-other:w Identifier-other:p">http://libris.kb.se/resource/bib/219553</z:index> <z:index name="Identifier-other:u">http://libris.kb.se/resource/bib/219553</z:index> However, this output is misleading. That's basically just the output of "xsltproc biblio-zebra-indexdefs.xsl <record>". It's pre-normalization and thus essentially meaningless. ----------- Only advanced users will look at yaz-client though. That being said, there are functional differences between ICU and CHR. For instance, the following query will work in CHR but NOT in ICU: id-other,phr=http libris kb se resource bib 219553 Conversely, the following query will work in ICU but not in CHR: id-other,phr=http libriskbse resource bib 219553 That's because we've configured tokenization and normalization to work differently between the two schemes. Fun, right? ICU uses the "l" tokenize rule from http://www.indexdata.com/yaz/doc/yaz-icu.html. That means it tokenizes based on slashes, spaces, and maybe some other characters I haven't discovered yet. You can verify that with the following commands: echo "THIS IS A TEST" | yaz-icu -x -c ./etc/zebradb/etc/phrases-icu.xml echo "THIS/IS/A/TEST" | yaz-icu -x -c ./etc/zebradb/etc/phrases-icu.xml Indeed, check out the following yaz-client output when using ICU: Z> f id-other,phr=http://libris.kb.se/resource/bib/219553 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 12, setno 17 SearchResult-1: term=http cnt=12, term=libriskbse cnt=12, term=resource cnt=12, term=bib cnt=12, term=219553 cnt=12 records returned: 0 Elapsed: 0.001458 You can see the URL has been broken into 5 terms/tokens with ICU, while CHR does the following: Z> f id-other,phr=http://libris.kb.se/resource/bib/219553 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 12, setno 13 SearchResult-1: term=http cnt=12, term=libris cnt=12, term=kb cnt=12, term=se cnt=12, term=resource cnt=12, term=bib cnt=12, term=219553 cnt=12 records returned: 0 Elapsed: 0.001119 We actually have 7 terms/tokens in the case of CHR! And that means that we don't want to try to outsmart Zebra by pre-normalizing our queries. We want to query Zebra with the exact same data that it indexed, because that way the normalization will be the same! Science!
Note that the above pertains to "phrases-icu.xml" specifically.
Created attachment 46567 [details] [review] [SIGNED-OFF]Bug 15555: Index 024$a into Identifier-other:u url register when source $2 is uri This patch indexes 024$a into the "phrase" index type, and the "url" index type, if the 024$2 equals "uri". TEST PLAN 1) Apply the patch. 1b) If you're using a gitified Koha or a git install, you'll need to upgrade your instance or copy your zebradb files over to /etc/koha/zebradb or your "kohadev" directory. 2) Add a 024$a with a URL like http://libris.kb.se/resource/bib/219553 to a bibliographic record 3) Re-index Zebra 4) Type "id-other,st-urx,fuzzy=http://libris.kb.se/resource/bib/219553" into the "Search the catalog" box in the Staff Client and search 5) Note that you retrieve your record NOTE: The fuzzy is required because Koha's query "parsing" functions change http:// to http=// which won't correctly match the value in the "Identifier-other:u" index. NOTE: Alternatively, you could do the following search instead: "id-other,phr=http libris kb se resource bib 219553". It would work as well by using the "Identifier-other:p" index. Advanced tester version: 4) In a terminal window, find the "koha-conf.xml" file in your "etc" directory. 5) Open "koha-conf.xml" and find <listen id="biblioserver">. Copy the URI you find there. (e.g. unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket). 6) Type "yaz-client unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket" 7) After it connects, type "base biblios" and press enter 8) Type "format xml" and press enter 9) Type "elements zebra::index" and press enter 10) Type "f id-other,st-urx=http://libris.kb.se/resource/bib/219553" and press enter 11) Note that you should have at least one result 12) Type "show 1" 13) If you scroll through the results, you should find something like the following: <index name="Identifier-other" type="w" seq="28">@^</index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="29">http</index> <index name="Identifier-other" type="w" seq="30">libris</index> <index name="Identifier-other" type="w" seq="31">kb</index> <index name="Identifier-other" type="w" seq="32">se</index> <index name="Identifier-other" type="w" seq="33">resource</index> <index name="Identifier-other" type="w" seq="34">bib</index> <index name="Identifier-other" type="w" seq="35">219553</index> <index name="Identifier-other" type="p" seq="28">http libris kb se resource bib 219553</index> <index name="Identifier-other" type="u" seq="36">http://libris.kb.se/resource/bib/219553</index> Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com> Works as advertised the record is retrieved
Because of your NOTE I think IMHO this needs to go in Manual. Add here Documentation Manager
Where do we think this should go in the manual? We don't really get this nitty gritting in the searching section. Nicole
Yeah, that might be tough to add to the manual. Prior to this patch, only the 952$u had an index with a "u" (ie url) register, and that's only accessible using "st-urx" as a CCL qualifier. Actually, in the future, I'd like to add the 856$u to the "url:u" index as well. I suppose that might be added to the manual... saying that "url,st-urx" will let you look up URLs in the 952$u and 856$u. I've had some people ask in the past about easy ways to find records with URLs and that would do it... But yeah. Not sure about the manual.
Actually, regarding the manual, I'm tempted to personally write up an appendix regarding searching in Koha using CCL... but I'm not sure when that's going to happen.
It would also be a bit advanced, as I would need to get into the stuff about the "p", "n", "w", and "u" registers for it to make complete sense.
Created attachment 50261 [details] [review] Bug 15555: Index 024$a into Identifier-other:u url register when source $2 is uri This patch indexes 024$a into the "phrase" index type, and the "url" index type, if the 024$2 equals "uri". TEST PLAN 1) Apply the patch. 1b) If you're using a gitified Koha or a git install, you'll need to upgrade your instance or copy your zebradb files over to /etc/koha/zebradb or your "kohadev" directory. 2) Add a 024$a with a URL like http://libris.kb.se/resource/bib/219553 to a bibliographic record 3) Re-index Zebra 4) Type "id-other,st-urx,fuzzy=http://libris.kb.se/resource/bib/219553" into the "Search the catalog" box in the Staff Client and search 5) Note that you retrieve your record NOTE: The fuzzy is required because Koha's query "parsing" functions change http:// to http=// which won't correctly match the value in the "Identifier-other:u" index. NOTE: Alternatively, you could do the following search instead: "id-other,phr=http libris kb se resource bib 219553". It would work as well by using the "Identifier-other:p" index. Advanced tester version: 4) In a terminal window, find the "koha-conf.xml" file in your "etc" directory. 5) Open "koha-conf.xml" and find <listen id="biblioserver">. Copy the URI you find there. (e.g. unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket). 6) Type "yaz-client unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket" 7) After it connects, type "base biblios" and press enter 8) Type "format xml" and press enter 9) Type "elements zebra::index" and press enter 10) Type "f id-other,st-urx=http://libris.kb.se/resource/bib/219553" and press enter 11) Note that you should have at least one result 12) Type "show 1" 13) If you scroll through the results, you should find something like the following: <index name="Identifier-other" type="w" seq="28">@^</index> <index name="Identifier-other" type="w" seq="1"></index> <index name="Identifier-other" type="w" seq="29">http</index> <index name="Identifier-other" type="w" seq="30">libris</index> <index name="Identifier-other" type="w" seq="31">kb</index> <index name="Identifier-other" type="w" seq="32">se</index> <index name="Identifier-other" type="w" seq="33">resource</index> <index name="Identifier-other" type="w" seq="34">bib</index> <index name="Identifier-other" type="w" seq="35">219553</index> <index name="Identifier-other" type="p" seq="28">http libris kb se resource bib 219553</index> <index name="Identifier-other" type="u" seq="36">http://libris.kb.se/resource/bib/219553</index> Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com> Works as advertised the record is retrieved Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Some of the statements in the commit message do not work for me. A search like "id-other,phr=http libris kb se resource bib 219553" does not have results. Searching for "id-other,phr=libris.kb.se resource" does. The steps in the advanced tester version do not work for me too. I verified the following in yaz-client: [1] Z> f @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 1, setno 16 [2] First removed $2 and reindexed. Then searched again: Z> f @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553 Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 0, setno 1
Pushed to master for the May 2016 release. Thanks David!