Bug 15555 - Index 024$a into Identifier-other:u url register when source $2 is uri
Summary: Index 024$a into Identifier-other:u url register when source $2 is uri
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Cook
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on: 14217
Blocks: 15541
  Show dependency treegraph
 
Reported: 2016-01-12 04:17 UTC by David Cook
Modified: 2018-06-04 20:10 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 15555 - Index 024$a into Identifier-other:u url register when source $2 is uri (4.91 KB, patch)
2016-01-12 04:19 UTC, David Cook
Details | Diff | Splinter Review
[SIGNED-OFF]Bug 15555: Index 024$a into Identifier-other:u url register when source $2 is uri (5.02 KB, patch)
2016-01-13 05:01 UTC, Héctor Eduardo Castro Avalos
Details | Diff | Splinter Review
Bug 15555: Index 024$a into Identifier-other:u url register when source $2 is uri (5.80 KB, patch)
2016-04-15 08:40 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2016-01-12 04:17:21 UTC
Currently, 024$a is indexed into Identifier-other:w, even when it is a URI (e.g. http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=14217)

This causes problems because the "w" index type replaces punctuation with spaces, and tokenizes on spaces, so that the URI is decomposed into a series of values which are indexed separately. This is definitely not what you want when indexing a 024$a when it is a URI.

For example:

The url "http://libris.kb.se/resource/bib/219553" becomes the following:

<index name="Identifier-other" type="w" seq="28">@^</index>
<index name="Identifier-other" type="w" seq="1"></index>
<index name="Identifier-other" type="w" seq="29">http</index>
<index name="Identifier-other" type="w" seq="30">libris</index>
<index name="Identifier-other" type="w" seq="31">kb</index>
<index name="Identifier-other" type="w" seq="32">se</index>
<index name="Identifier-other" type="w" seq="33">resource</index>
<index name="Identifier-other" type="w" seq="34">bib</index>
<index name="Identifier-other" type="w" seq="35">219553</index>

Fortunately, the 024$2 subfield value tells us the source of the identifier, and "uri" is one of the valid options. So, when we have a 024$2=uri, we can index the 024$a using the "url" index type. 

(I'm also planning to index into the "phrase" index type for all 024$a as it performs the normalization but it doesn't tokenize based on the spaces, so this normal form may still be of use for urls and other identifiers that rely on punctuation for meaning.)
Comment 1 David Cook 2016-01-12 04:19:35 UTC Comment hidden (obsolete)
Comment 2 David Cook 2016-01-12 04:39:21 UTC
Advanced testers:

Here's the steps you need to do to upgrade your Koha's Zebra config, if you're using a git dev install:

cd /home/dcook/git
INSTALL_LOG="/home/dcook/koha-dev/misc/koha-install-log"
cp $INSTALL_LOG backup-koha-install-log
perl Makefile.PL --prev-install-log "$INSTALL_LOG"
make
make upgrade
Comment 3 David Cook 2016-01-12 05:43:48 UTC
NOTE: The output I've provided in the above comments has come from a Koha using CHR indexing.

ICU indexing has different output, but the same behaviour.

Here's what I see with "format xml" and "elements zebra::index":

<index name="Identifier-other" type="w" seq="21"></index>
<index name="Identifier-other" type="w" seq="1"></index>
<index name="Identifier-other" type="w" seq="22"></index>
<index name="Identifier-other" type="w" seq="23"></index>
<index name="Identifier-other" type="w" seq="24"></index>
<index name="Identifier-other" type="w" seq="25"></index>
<index name="Identifier-other" type="p" seq="21"></index>
<index name="Identifier-other" type="p" seq="22"></index>
<index name="Identifier-other" type="p" seq="23"></index>
<index name="Identifier-other" type="p" seq="24"></index>
<index name="Identifier-other" type="p" seq="25"></index>
<index name="Identifier-other" type="u" seq="26">http://libris.kb.se/resource/bib/219553</index>

Here's what I see with "format xml" and "elements index":

<z:index name="Identifier-other:w Identifier-other:p">http://libris.kb.se/resource/bib/219553</z:index>
<z:index name="Identifier-other:u">http://libris.kb.se/resource/bib/219553</z:index>

However, this output is misleading. That's basically just the output of "xsltproc biblio-zebra-indexdefs.xsl <record>". It's pre-normalization and thus essentially meaningless.

-----------

Only advanced users will look at yaz-client though.

That being said, there are functional differences between ICU and CHR.


For instance, the following query will work in CHR but NOT in ICU:
id-other,phr=http libris kb se resource bib 219553

Conversely, the following query will work in ICU but not in CHR:
id-other,phr=http libriskbse resource bib 219553

That's because we've configured tokenization and normalization to work differently between the two schemes. Fun, right?

ICU uses the "l" tokenize rule from http://www.indexdata.com/yaz/doc/yaz-icu.html. That means it tokenizes based on slashes, spaces, and maybe some other characters I haven't discovered yet. You can verify that with the following commands:

echo "THIS IS A TEST" | yaz-icu -x -c ./etc/zebradb/etc/phrases-icu.xml
echo "THIS/IS/A/TEST" | yaz-icu -x -c ./etc/zebradb/etc/phrases-icu.xml

Indeed, check out the following yaz-client output when using ICU:

Z> f id-other,phr=http://libris.kb.se/resource/bib/219553
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 12, setno 17
SearchResult-1: term=http cnt=12, term=libriskbse cnt=12, term=resource cnt=12, term=bib cnt=12, term=219553 cnt=12
records returned: 0
Elapsed: 0.001458

You can see the URL has been broken into 5 terms/tokens with ICU, while CHR does the following:

Z> f id-other,phr=http://libris.kb.se/resource/bib/219553
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 12, setno 13
SearchResult-1: term=http cnt=12, term=libris cnt=12, term=kb cnt=12, term=se cnt=12, term=resource cnt=12, term=bib cnt=12, term=219553 cnt=12
records returned: 0
Elapsed: 0.001119

We actually have 7 terms/tokens in the case of CHR!

And that means that we don't want to try to outsmart Zebra by pre-normalizing our queries. We want to query Zebra with the exact same data that it indexed, because that way the normalization will be the same! Science!
Comment 4 David Cook 2016-01-12 05:44:44 UTC
Note that the above pertains to "phrases-icu.xml" specifically.
Comment 5 Héctor Eduardo Castro Avalos 2016-01-13 05:01:52 UTC Comment hidden (obsolete)
Comment 6 Héctor Eduardo Castro Avalos 2016-01-13 05:06:53 UTC
Because of your NOTE I think IMHO this needs to go in Manual.

Add here Documentation Manager
Comment 7 Nicole C. Engard 2016-01-14 20:33:32 UTC
Where do we think this should go in the manual? We don't really get this nitty gritting in the searching section.

Nicole
Comment 8 David Cook 2016-01-17 22:19:28 UTC
Yeah, that might be tough to add to the manual. 

Prior to this patch, only the 952$u had an index with a "u" (ie url) register, and that's only accessible using "st-urx" as a CCL qualifier. 

Actually, in the future, I'd like to add the 856$u to the "url:u" index as well. I suppose that might be added to the manual... saying that "url,st-urx" will let you look up URLs in the 952$u and 856$u. I've had some people ask in the past about easy ways to find records with URLs and that would do it...

But yeah. Not sure about the manual.
Comment 9 David Cook 2016-01-18 04:43:52 UTC
Actually, regarding the manual, I'm tempted to personally write up an appendix regarding searching in Koha using CCL... but I'm not sure when that's going to happen.
Comment 10 David Cook 2016-01-18 04:44:40 UTC
It would also be a bit advanced, as I would need to get into the stuff about the "p", "n", "w", and "u" registers for it to make complete sense.
Comment 11 Marcel de Rooy 2016-04-15 08:40:07 UTC
Created attachment 50261 [details] [review]
Bug 15555: Index 024$a into Identifier-other:u url register when source $2 is uri

This patch indexes 024$a into the "phrase" index type, and the "url" index type,
if the 024$2 equals "uri".

TEST PLAN

1) Apply the patch.
1b) If you're using a gitified Koha or a git install,
you'll need to upgrade your instance or copy your zebradb files
over to /etc/koha/zebradb or your "kohadev" directory.
2) Add a 024$a with a URL like http://libris.kb.se/resource/bib/219553
 to a bibliographic record
3) Re-index Zebra
4) Type "id-other,st-urx,fuzzy=http://libris.kb.se/resource/bib/219553"
 into the "Search the catalog" box in the Staff Client and search
5) Note that you retrieve your record

NOTE: The fuzzy is required because Koha's query "parsing" functions change
http:// to http=// which won't correctly match the value in the "Identifier-other:u" index.
NOTE: Alternatively, you could do the following search instead:
"id-other,phr=http libris kb se resource bib 219553".
 It would work as well by using the "Identifier-other:p" index.

Advanced tester version:
4) In a terminal window, find the "koha-conf.xml" file in your "etc" directory.
5) Open "koha-conf.xml" and find <listen id="biblioserver">.
Copy the URI you find there. (e.g. unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket).
6) Type "yaz-client unix:/home/dcook/koha-dev/var/run/zebradb/bibliosocket"
7) After it connects, type "base biblios" and press enter
8) Type "format xml" and press enter
9) Type "elements zebra::index" and press enter
10) Type "f id-other,st-urx=http://libris.kb.se/resource/bib/219553" and press enter
11) Note that you should have at least one result
12) Type "show 1"
13) If you scroll through the results, you should find something like the following:

<index name="Identifier-other" type="w" seq="28">@^</index>
<index name="Identifier-other" type="w" seq="1"></index>
<index name="Identifier-other" type="w" seq="29">http</index>
<index name="Identifier-other" type="w" seq="30">libris</index>
<index name="Identifier-other" type="w" seq="31">kb</index>
<index name="Identifier-other" type="w" seq="32">se</index>
<index name="Identifier-other" type="w" seq="33">resource</index>
<index name="Identifier-other" type="w" seq="34">bib</index>
<index name="Identifier-other" type="w" seq="35">219553</index>
<index name="Identifier-other" type="p" seq="28">http libris kb se resource bib 219553</index>
<index name="Identifier-other" type="u" seq="36">http://libris.kb.se/resource/bib/219553</index>

Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Works as advertised the record is retrieved

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Some of the statements in the commit message do not work for me.
A search like "id-other,phr=http libris kb se resource bib 219553" does not
have results. Searching for "id-other,phr=libris.kb.se resource" does.
The steps in the advanced tester version do not work for me too.
I verified the following in yaz-client:
[1] Z> f @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 1, setno 16
[2] First removed $2 and reindexed. Then searched again:
Z> f @attr 1=9012 @attr 4=104 http://libris.kb.se/resource/bib/219553
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 0, setno 1
Comment 12 Kyle M Hall 2016-04-29 13:26:01 UTC
Pushed to master for the May 2016 release. Thanks David!