Bug 11232

Summary: Retrieve facets from Zebra
Product: Koha Reporter: David Cook <dcook>
Component: Architecture, internals, and plumbingAssignee: Tomás Cohen Arazi <tomascohen>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: new feature    
Priority: P5 - low CC: f.demians, fridolin.somers, gwilliams, jonathan.druart, katrin.fischer, mathsabypro, matted-34813, mtompset, nick, paul.poulain, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17123
Change sponsored?: --- Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on:    
Bug Blocks: 13665, 13087, 13088    
Attachments: 0001-Bug-11232-support-zebra-facet-queries-in-search.patch
Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl
Bug 11232: MARC21 facet definition and updated XSL file for DOM
Bug 11232: UNIMARC facet definition and updated XSL file for DOM
Bug 11232: NORMARC facet definition and updated XSL file for DOM
Bug 11232: retrieve facets from Zebra's indexes in DOM
Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl
Bug 11232: MARC21 facet definition and updated XSL file for DOM
Bug 11232: UNIMARC facet definition and updated XSL file for DOM
Bug 11232: NORMARC facet definition and updated XSL file for DOM
Bug 11232: retrieve facets from Zebra's indexes in DOM
Sample record, items not faceted by holding library in search
Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl
Bug 11232: MARC21 facet definition and updated XSL file for DOM
Bug 11232: UNIMARC facet definition and updated XSL file for DOM
Bug 11232: NORMARC facet definition and updated XSL file for DOM
Bug 11232: retrieve facets from Zebra's indexes in DOM
Bug 11232: (followup) remove unnecesary namespace definition from all XML elements
Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl
Bug 11232: MARC21 facet definition and updated XSL file for DOM
Bug 11232: UNIMARC facet definition and updated XSL file for DOM
Bug 11232: NORMARC facet definition and updated XSL file for DOM
Bug 11232: retrieve facets from Zebra's indexes in DOM
Bug 11232: (followup) remove unnecesary namespace definition from all XML elements
Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl
Bug 11232: MARC21 facet definition and updated XSL file for DOM
Bug 11232: UNIMARC facet definition and updated XSL file for DOM
Bug 11232: NORMARC facet definition and updated XSL file for DOM
Bug 11232: retrieve facets from Zebra's indexes in DOM
Bug 11232: (followup) remove unnecesary namespace definition from all XML elements
Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl
Bug 11232: MARC21 facet definition and updated XSL file for DOM
Bug 11232: UNIMARC facet definition and updated XSL file for DOM
Bug 11232: NORMARC facet definition and updated XSL file for DOM
Bug 11232: retrieve facets from Zebra's indexes in DOM
Bug 11232: (followup) remove unnecesary namespace definition from all XML elements
Bug 11232: (followup) Configuration variable for enabling Zebra facets
Bug 11232: (qa followup) Add missing fields/subfields to the item types faceta
Bug 11232: (qa followup) empty ID due to namespace mistake
[PASSED QA] Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl
[PASSED QA] Bug 11232: MARC21 facet definition and updated XSL file for DOM
[PASSED QA] Bug 11232: UNIMARC facet definition and updated XSL file for DOM
[PASSED QA] Bug 11232: NORMARC facet definition and updated XSL file for DOM
[PASSED QA] Bug 11232: retrieve facets from Zebra's indexes in DOM
[PASSED QA] Bug 11232: (followup) remove unnecesary namespace definition from all XML elements
[PASSED QA] Bug 11232: (followup) Configuration variable for enabling Zebra facets
[PASSED QA] Bug 11232: (qa followup) Add missing fields/subfields to the item types faceta
[PASSED QA] Bug 11232: (qa followup) empty ID due to namespace mistake

Description David Cook 2013-11-11 00:13:13 UTC
At Kohacon, a few of us (Jared, Paul P, Tomas, myself, and possibly others) looked at using Zebra's native facetting. 

After a bit of trial and error, in the final 15 minutes of the hackfest, we were able to get it working using queries in YAZ(high fives all around), but I'm not sure if anyone has followed up on that work (as I've been on holiday). I don't see any existing bug reports for adding this functionality to Koha, but I think it would be preferable to how we build facets at the moment (hence this bug). 

This method should be:

1) faster (since it is pulling the facets from the search results using indexes rather than iterating through every record);
2) more comprehensive (it is based on the full result set and not just the first 50 or so results [depending on your sysprefs]);
3) able to build facets for control fields and the leader (since it pulls data using indexes, we should be able to easily build facets for publication year, record type, etc.).

In terms of customizability (bug 10891), it might be more difficult to change on the fly.

This is because we need to explicitly define each facet index in our configuration files for Zebra.

I'll need to look over my notes and do a few more experiments, but I'm quite certain this is the case.

While we could add a mechanism for allowing users to specify which facets to retrieve and display (probably a good idea), we couldn't do the same for the static XML config file. So we would need to either include all the indexes in the XML config file or mark it as a place for local changes.

I'm not sure about the cost/benefit of telling Zebra that every index could be used as a facet. I don't imagine it would be too much of a cost though, since the list of facets would be fed to Zebra on start up and not runtime.

--

As I said, I'm  not sure if anyone is working on the native facetting at the moment. I would like to but I'll have to look at my workload and chat to my boss.

I'll be happy to post my notes in a comment tomorrow on how to get it to work. The config is slightly different for DOM and GRS1 but it works well for both.
Comment 1 Mathieu Saby 2013-11-11 11:59:53 UTC
Hi
Well, I will wait to see some patch attached here to know if bug 10891 should be closed, or maybe rethink on a different base (maybe just to hide unwanted facets, and not providing a real customization for example).

Mathieu
Comment 2 Mathieu Saby 2013-11-11 12:09:01 UTC
> 
> I'm not sure about the cost/benefit of telling Zebra that every index could
> be used as a facet. I don't imagine it would be too much of a cost though,
> since the list of facets would be fed to Zebra on start up and not runtime.


Of course ;-)
Just take a look at some library or bookseller's websites.
Most fields must NOT be used as facets.

For me facets should made on :
- availability (make a distinction between items notforloan, items on loan, items available for loan)
- date (on coded fields)
- language
- language of original material (for translations)
- subject (with facets based on subfields of 6XX, not on complete fields : a facet for topic subjects, a facet for chronological, a facet for geographical etc)
- type of document and carriage (various fields and coded fields in MARC21 and UNIMARC)
- coded informations in 0XX MARC21 and 1XX UNIMARC fields
- coded informations in label
- authors

and maybe titles, and series

Mathieu
Comment 3 Mathieu Saby 2013-11-11 12:14:43 UTC
Note that there is currently a big issue for UNIMARC facets:
the facet build on type of document (equivalent of 942 in MARC21) is not visible.

That's because some libraries use 099$t for that, and others use other fields like 210$b that are not made for that.
That was the main goal of my not-yet-written-patch for make facets customisable...

But maybe it could be fixed by telling UNIMARC ppl that from 3.16 version onwards the type of document is stored in 099t, and provide a script for copying types of documents from other fields to this one for non standard configuration.

Mathieu
Comment 4 David Cook 2013-12-02 23:16:11 UTC
(In reply to mathieu saby from comment #3)
> Note that there is currently a big issue for UNIMARC facets:
> the facet build on type of document (equivalent of 942 in MARC21) is not
> visible.
> 
> That's because some libraries use 099$t for that, and others use other
> fields like 210$b that are not made for that.
> That was the main goal of my not-yet-written-patch for make facets
> customisable...
> 
> But maybe it could be fixed by telling UNIMARC ppl that from 3.16 version
> onwards the type of document is stored in 099t, and provide a script for
> copying types of documents from other fields to this one for non standard
> configuration.
> 
> Mathieu

With zebra facets, it uses the index. So if both those marc tags are filling the same index, it should be fine.
Comment 5 Mathieu Saby 2013-12-02 23:47:51 UTC
(In reply to David Cook from comment #4)
> (In reply to mathieu saby from comment #3)
> > Note that there is currently a big issue for UNIMARC facets:
> > the facet build on type of document (equivalent of 942 in MARC21) is not
> > visible.
> > 
> > That's because some libraries use 099$t for that, and others use other
> > fields like 210$b that are not made for that.
> > That was the main goal of my not-yet-written-patch for make facets
> > customisable...
> > 
> > But maybe it could be fixed by telling UNIMARC ppl that from 3.16 version
> > onwards the type of document is stored in 099t, and provide a script for
> > copying types of documents from other fields to this one for non standard
> > configuration.
> > 
> > Mathieu
> 
> With zebra facets, it uses the index. So if both those marc tags are filling
> the same index, it should be fine.

I did not think of it... Of course ;-)
So, it will really be a nice improvement for 3.16 if Zebra facets could be used.
Comment 6 David Cook 2013-12-02 23:49:42 UTC
I'm outlining below a rough guide to how to enable Zebra facetting. Note that this has nothing to do with Koha per se. This is just about setting up Zebra. The actual Koha code to make use of Zebra's facets will have to come after initial setup, and at the moment it's just not at the top of my list of priorities.

First, take stock of which version of Koha you're running and note which type of indexing you're using (DOM or GRS1).

Look for these two files in your /etc/zebradb/ folder:
"retrieval-info-bib-dom.xml"
"retrieval-info-bib-grs1.xml"

If you can't find them, you're probably running an older version of Koha and you'll want to modify your "koha-conf.xml". If you do find them, the config in "koha-conf.xml" is only a fallback, so not necessary for testing.


Step One:
a) Edit the file that suits your version of Koha and method of indexing.
(i.e. if you're running the latest Koha with dom, edit "retrieval-info-bib-dom.xml". If you're running the latest Koha with grs1, edit "retrieval-info-bib-grs1.xml". If you're running an older version of Koha, you're only using grs1 and you'll need to edit "koha-conf.xml"

Step Two:
a) Add lines like the following as children to the "<retrievalinfo>" node (N.B. do not add them to the top as they will displace the default retrieval syntax setting. Add them at the bottom.):

<retrieval syntax="xml" name="zebra::facet::any:w"/>
<retrieval syntax="xml" name="zebra::facet::title:w"/>
<retrieval syntax="xml" name="zebra::facet::title:0"/>

(N.B. Paul Poulain mentioned that the "0" register is a "raw"(?) register and that this should be used if you have issues with diacritics or non-Latin characters. I imagine that this needs to be set up in your records.abs or DOM XSLT.)

(N.B. You can substitute any index in for "any" or "title". You can also use any register instead of "w" or "0", such as "p".)

Step Three: 
a) Restart your Zebra server

Step Four:
a) Connect to Zebra using yaz-client (e.g. yaz-client open "unix:/zebra/test/var/run/zebradb/bibliosocket")(You can find your specific path by looking at the top of koha-conf.xml for a "listen" node with an ID of "biblioserver")

b) Tell Zebra to use facets (e.g. elements zebra::facet::any:w,title:w,title:0)
c) Do a search (e.g. find test)
d) Show your results (e.g. show 1+1)
e) You should now see a list of facets for your search results

--

If you have any issues with these instructions, comment here, find me in the IRC channel or visit the sites that I'm linking below.

--

For more information, visit the following links:

http://www.indexdata.com/zebra/doc/zebra.pdf (Search in the document for "facet")

http://lists.indexdata.dk/pipermail/zebralist/2007-July/001682.html (This is a discussion of setting up Zebra facets)
Comment 7 David Cook 2013-12-02 23:58:07 UTC
Also note that you will have to individually set up each index that you want to use for facet queries.

As per this email from an IndexData person: http://lists.indexdata.dk/pipermail/zebralist/2007-July/001694.html

In general, this shouldn't case a problem, since we only use a short list of facets. However, it would create a barrier to allowing users to customize their results.

In theory, we could list all or most of the available indexes in the Zebra config files. Then, we could use a system preference or a DB table to choose which indexes we want to use for facets. This would generate the "elements" string that we pass to Zebra everytime we want our facets.

Actually, now that I think about it...that might not be a bad idea. We already have that Search.pm sub where we're manually keeping track of indexes, so we could keep that or use something similar to fill an HTML list and have that create our Zebra facet string. 

So long as we generate the string once and then use it multiple times...that should be all right. We could also have a hardcoded fallback string in case someone deletes the facet config in Koha or if their facet config is bad. 

Hmm. Ideas. Ideas. Ideas...
Comment 8 Jared Camins-Esakov 2013-12-03 00:09:52 UTC
(In reply to David Cook from comment #7)
> Also note that you will have to individually set up each index that you want
> to use for facet queries.
> 
> As per this email from an IndexData person:
> http://lists.indexdata.dk/pipermail/zebralist/2007-July/001694.html
> 
> In general, this shouldn't case a problem, since we only use a short list of
> facets. However, it would create a barrier to allowing users to customize
> their results.

This has to be changed in two places. In the server elements configuration, as described at that link, *and* an additional index has to be configured for each facet.
Comment 9 Mark Tompsett 2013-12-03 00:12:45 UTC
Could not this bug just get the zebra facets working, and then whatever needs to be done to filter be worked on in bug 10891?
Comment 10 Mathieu Saby 2013-12-03 08:16:00 UTC
> Actually, now that I think about it...that might not be a bad idea. We
> already have that Search.pm sub where we're manually keeping track of
> indexes, so we could keep that or use something similar to fill an HTML list
> and have that create our Zebra facet string. 
> 


I thought this list of index in search.pm was going to be deprecated with the adoption of QueryParser ?

Mathieu
Comment 11 Mathieu Saby 2013-12-03 08:18:23 UTC
(In reply to David Cook from comment #6)
> I'm outlining below a rough guide to how to enable Zebra facetting. Note
> that this has nothing to do with Koha per se. This is just about setting up
> Zebra. The actual Koha code to make use of Zebra's facets will have to come
> after initial setup, and at the moment it's just not at the top of my list
> of priorities.
> 
> First, take stock of which version of Koha you're running and note which
> type of indexing you're using (DOM or GRS1).
> 
> Look for these two files in your /etc/zebradb/ folder:
> "retrieval-info-bib-dom.xml"
> "retrieval-info-bib-grs1.xml"
> 
> If you can't find them, you're probably running an older version of Koha and
> you'll want to modify your "koha-conf.xml". If you do find them, the config
> in "koha-conf.xml" is only a fallback, so not necessary for testing.
> 
> 
> Step One:
> a) Edit the file that suits your version of Koha and method of indexing.
> (i.e. if you're running the latest Koha with dom, edit
> "retrieval-info-bib-dom.xml". If you're running the latest Koha with grs1,
> edit "retrieval-info-bib-grs1.xml". If you're running an older version of
> Koha, you're only using grs1 and you'll need to edit "koha-conf.xml"
> 
> Step Two:
> a) Add lines like the following as children to the "<retrievalinfo>" node
> (N.B. do not add them to the top as they will displace the default retrieval
> syntax setting. Add them at the bottom.):
> 
> <retrieval syntax="xml" name="zebra::facet::any:w"/>
> <retrieval syntax="xml" name="zebra::facet::title:w"/>
> <retrieval syntax="xml" name="zebra::facet::title:0"/>
> 
> (N.B. Paul Poulain mentioned that the "0" register is a "raw"(?) register
> and that this should be used if you have issues with diacritics or non-Latin
> characters. I imagine that this needs to be set up in your records.abs or
> DOM XSLT.)
> 
> (N.B. You can substitute any index in for "any" or "title". You can also use
> any register instead of "w" or "0", such as "p".)
> 
> Step Three: 
> a) Restart your Zebra server
> 
> Step Four:
> a) Connect to Zebra using yaz-client (e.g. yaz-client open
> "unix:/zebra/test/var/run/zebradb/bibliosocket")(You can find your specific
> path by looking at the top of koha-conf.xml for a "listen" node with an ID
> of "biblioserver")
> 
> b) Tell Zebra to use facets (e.g. elements
> zebra::facet::any:w,title:w,title:0)
> c) Do a search (e.g. find test)
> d) Show your results (e.g. show 1+1)
> e) You should now see a list of facets for your search results
> 
> --
> 
> If you have any issues with these instructions, comment here, find me in the
> IRC channel or visit the sites that I'm linking below.
> 
> --
> 
> For more information, visit the following links:
> 
> http://www.indexdata.com/zebra/doc/zebra.pdf (Search in the document for
> "facet")
> 
> http://lists.indexdata.dk/pipermail/zebralist/2007-July/001682.html (This is
> a discussion of setting up Zebra facets)

As DOM is now the new standard way of indexing, and GRS1 is going to be deprecated, maybe the work could be made only for DOM files?
Comment 12 Katrin Fischer 2013-12-03 11:32:15 UTC
I think it would be possible to agree that this is only done for DOM, as we are trying to phase out/deprecate GRS-1.
There are some bugs in QueryParser that I would like to see addressed before we make it the standard (for example bug 10542).
Comment 13 David Cook 2013-12-04 05:29:05 UTC
(In reply to Jared Camins-Esakov from comment #8)
> (In reply to David Cook from comment #7)
> > Also note that you will have to individually set up each index that you want
> > to use for facet queries.
> > 
> > As per this email from an IndexData person:
> > http://lists.indexdata.dk/pipermail/zebralist/2007-July/001694.html
> > 
> > In general, this shouldn't case a problem, since we only use a short list of
> > facets. However, it would create a barrier to allowing users to customize
> > their results.
> 
> This has to be changed in two places. In the server elements configuration,
> as described at that link, *and* an additional index has to be configured
> for each facet.

Jared, I'm not sure I understand what you mean by "an additional index has to be configured for each facet". 

Could you elaborate? 

Once the server element is set up for a facet, it should be all good to go. 

--

That said, I was thinking about this a bit last night, and I can't recall exactly what is retrieved with a facet query.

I have yet to set up a VM for Koha on my new desktop so I'll have to wait a bit before testing it out again.
Comment 14 Jared Camins-Esakov 2013-12-04 12:44:23 UTC
(In reply to David Cook from comment #13)
> Jared, I'm not sure I understand what you mean by "an additional index has
> to be configured for each facet". 
> 
> Could you elaborate? 

You have to add a element:0 index for each facet you want to use.
Comment 15 David Cook 2013-12-06 05:34:56 UTC
(In reply to Jared Camins-Esakov from comment #14)
> (In reply to David Cook from comment #13)
> > Jared, I'm not sure I understand what you mean by "an additional index has
> > to be configured for each facet". 
> > 
> > Could you elaborate? 
> 
> You have to add a element:0 index for each facet you want to use.

Hmm, I'm not sure I understand. I didn't do that and it seemed to work all right.
Comment 16 Paul Poulain 2013-12-06 07:48:46 UTC
(In reply to David Cook from comment #15)
> (In reply to Jared Camins-Esakov from comment #14)
> > (In reply to David Cook from comment #13)
> > > Jared, I'm not sure I understand what you mean by "an additional index has
> > > to be configured for each facet". 
> > > 
> > > Could you elaborate? 
> > 
> > You have to add a element:0 index for each facet you want to use.
> 
> Hmm, I'm not sure I understand. I didn't do that and it seemed to work all
> right.

Do you have tested a facet with diacritics or non latin characters. iirc, the :0 means "raw storing", and fixes diacritics/non-latin display problems
Comment 17 David Cook 2013-12-09 00:50:51 UTC
(In reply to Paul Poulain from comment #16)
> (In reply to David Cook from comment #15)
> > (In reply to Jared Camins-Esakov from comment #14)
> > > (In reply to David Cook from comment #13)
> > > > Jared, I'm not sure I understand what you mean by "an additional index has
> > > > to be configured for each facet". 
> > > > 
> > > > Could you elaborate? 
> > > 
> > > You have to add a element:0 index for each facet you want to use.
> > 
> > Hmm, I'm not sure I understand. I didn't do that and it seemed to work all
> > right.
> 
> Do you have tested a facet with diacritics or non latin characters. iirc,
> the :0 means "raw storing", and fixes diacritics/non-latin display problems

I haven't tried with non-latin characters, but I remember at Kohacon when you used the :0 to fix the issue you were having with Arabic characters.
Comment 18 David Cook 2014-03-20 03:04:29 UTC
For package install users, double-check your koha-conf.xml. Look for a block like the following:

<!-- BIBLIOGRAPHIC RECORDS -->
<server id="biblioserver"  listenref="biblioserver">
    <directory>/var/lib/koha/devone/biblios</directory>
    <config>/etc/koha/sites/devone/zebra-biblios-dom.cfg</config>
    <cql2rpn>/etc/koha/zebradb/pqf.properties</cql2rpn>

    <xi:include href="/etc/koha/marc21-retrieval-info-bib-dom.xml"
 xmlns:xi="http://www.w3.org/2001/XInclude">

The file that you need to update would be "marc21-retrieval-info-bib-dom.xml".

(My bad for saying "retrieval-info-bib-dom.xml". On a tarball dev install, it will probably be "retrieval-info-bib-dom.xml", but it's worth double-checking in any case.)

--

Another note for restarting Zebra...

I don't know the best way to restart Zebra on a non-package install.

However, sudo koha-restart-zebra [instancename] works a treat on a package install.

--

I just tried this again, and facets worked quite neatly.
Comment 19 David Cook 2014-03-20 04:25:09 UTC
IMPORTANT CONFIGURATION UPDATE:

So I was wrong before about the need to define "zebra::facet::field:type" style retrieval elements within retrievalinfo.

In fact, we just need to add one "retrieval" element to the bottom of the config before the closing </retrievalinfo> tag, and it is this:

<retrieval syntax="xml"/>

YAY!!!

This means that we don't have to individually define every single facet index. This means that facets could be highly flexible!

--

The solution became apparent after consulting the following documents:

http://lists.indexdata.dk/pipermail/zebralist/2009-June/002251.html

http://www.indexdata.com/yaz/doc/tools.retrieval.html

http://lists.indexdata.dk/pipermail/zebralist/2012-May/002536.html

Hdl stated "if I enable multiple retrieval facilities (MODS, XML, DC etc...), then zebra::meta and zebra::facet seems to be
disabled ([25]". 

So, I looked into the Zebralist a bit and noticed someone who "wanted" to block the Zebra Special Retrieval Elements (such as zebra::meta and zebra::facet). 

Adam from Indexdata said, "There is no to disable it as such. But you can use the retrieval facility of YAZ to specify valid element set names + formats."

So I looked into the retrieval facility a bit more...and noticed that the "name" attribute of the "retrieval" node was optional. If you used it, you could specify a single element. 

However, what happens if you remove the name completely? What if you make the "retrieval" element less restrictive?

It turns out that you re-gain access to those Zebra Special Retrieval Elements, which we were excluding because our configuration was too restrictive!

I'm placing my updated guide to the bottom of this bug. 

I have a few more notes before that though:

First, I should mention that using something like "zebra::facet::Title:0" won't work, unless you're indexing strings with a type of "0" in record.abs or the DOM equivalent (which I can't recall at the moment). At least, that's my best guess. I've seen Paul Poulain use something like "zebra::facet::Title:0" and it worked with Arabic, so I assume they were indexing with the type 0 in their indexing config.

Second, you have to use actual indexes for facets. You can't use the CCL short-hand like "ti". Also, capitals matter. In my previous examples, I said you could use "title", but I should've checked first as it's actually "Title". 

Third, chaining indexes and types (e.g. "zebra::facet::f1:t1,f2:t2,.." as noted in the Indexdata config) will now work, while it wouldn't when defining facets individually.

--

Updated Guide

1) Check your koha-conf.xml and check for where your <retrievalinfo> is defined. It should be based on an <xi:include>. If you're using packages, it'll probably be something like "marc21-retrieval-info-bib-dom.xml" (depending on your marc flavour and indexing filter).

2) Add the line "<retrieval syntax="xml"/>" at the bottom of the document before the closing </retrievalinfo> tag. 

3) Restart Zebra (using koha-reset-zebra with the packages or manually otherwise)

4) yaz-client [unix socket defined in koha-conf.xml]

base biblios
format xml
elements zebra::facet::Title:w,Author:w,itype:w
find test
show

--

You should be presented with a facet list for "Title", "Author", and "Itype". 

I'm currently not sure about the meaning of "coccur" (co-occurrence?) but "occur" is the number of times the term shows up in the search results.

--

Cool, eh?
Comment 20 David Cook 2014-03-20 04:36:39 UTC
Latest trial...

Facets seem to work great with my package install but not with my tarball dev install.

PACKAGE INSTALL:
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="w" index="Title">
    <term coccur="12" occur="7">of</term>
    <term coccur="10" occur="6">australia</term>
    <term coccur="8" occur="6">tax</term>
    <term coccur="9" occur="6">test</term>
    <term coccur="5" occur="5">institute</term>
    <term coccur="8" occur="4">and</term>
    <term coccur="8" occur="4">the</term>
    <term coccur="5" occur="3">convention</term>
    <term coccur="4" occur="3">taxation</term>
    <term coccur="4" occur="2">an</term>
    <term coccur="4" occur="2">fiscal</term>
    <term coccur="4" occur="2">illusion</term>
    <term coccur="4" occur="2">in</term>
    <term coccur="3" occur="2">melbourne</term>
    <term coccur="5" occur="2">papers</term>
    <term coccur="3" occur="2">seminar</term>
    <term coccur="4" occur="2">to</term>
    <term coccur="2" occur="1">1</term>
    <term coccur="2" occur="1">a</term>
    <term coccur="4" occur="1">dominion</term>
  </facet>
  <facet type="w" index="Author">
    <term coccur="5" occur="5">110</term>
    <term coccur="7" occur="4">institute</term>
    <term coccur="9" occur="4">of</term>
    <term coccur="6" occur="3">andrew</term>
    <term coccur="5" occur="3">australia</term>
    <term coccur="6" occur="3">e</term>
    <term coccur="6" occur="3">research</term>
    <term coccur="5" occur="3">taxation</term>
    <term coccur="6" occur="3">university</term>
    <term coccur="4" occur="2">australian</term>
    <term coccur="4" occur="2">brian</term>
    <term coccur="4" occur="2">c</term>
    <term coccur="4" occur="2">centre</term>
    <term coccur="4" occur="2">dollery</term>
    <term coccur="4" occur="2">federalism</term>
    <term coccur="4" occur="2">national</term>
    <term coccur="4" occur="2">r</term>
    <term coccur="4" occur="2">worthington</term>
    <term coccur="2" occur="1">g</term>
    <term coccur="2" occur="1">j</term>
  </facet>
  <facet type="w" index="itype">
    <term coccur="115" occur="9">book</term>
    <term coccur="5" occur="4">text</term>
  </facet>
</record>

TARBALL DEV INSTALL:
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="w" index="Title">
    <term coccur="14" occur="7">+I7</term>
    <term coccur="12" occur="6">+I7</term>
    <term coccur="7" occur="5">+I7</term>
    <term coccur="10" occur="5">+I7</term>
    <term coccur="8" occur="5">+I7</term>
    <term coccur="8" occur="4">+I7</term>
    <term coccur="8" occur="4">+I7</term>
    <term coccur="5" occur="3">+I7</term>
    <term coccur="6" occur="3">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="3" occur="2">+I7</term>
    <term coccur="5" occur="2">+I7</term>
    <term coccur="3" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="1">+I7</term>
  </facet>
</record>

--

Will have to look into possible causes for this...
Comment 21 David Cook 2014-03-20 04:38:57 UTC
Also worthwhile on the Zebra front...

http://www.indexdata.com/yaz/doc/server.vhosts.html
Comment 22 David Cook 2014-03-20 05:51:05 UTC
Alas, the problem appears to be with ICU indexing. 

Wajasu provided some links which makes it seem like there might not be a way to get Zebra facets to work when using ICU indexing...

http://lists.indexdata.dk/pipermail/zebralist/2008-January/001859.html
http://lists.indexdata.dk/pipermail/zebralist/2009-April/002190.html

Yet, I swear that Paul Poulain was using ICU indexing and was able to make use of the :0 index type to get around this very problem.

I've tried adding Title:0 to record.abs and biblio-koha-indexdefs.xml, but to no avail. 

Happy to hear from Paul or Jared if they have any ideas on this one...
Comment 23 Tomás Cohen Arazi 2014-03-21 18:47:51 UTC
I'd like to add that the default behaviour for facets is to limit them to 20...
The indexdata folks explained to me that a higher count can be hardcoded in the configuration file. For instance:

 <retrieval syntax="xml" name="zebra::facet::subject:p:100"/>

will set the limit to 100.

It is worth mentioning that Zebra tokenizes data for indexing purposes, and also lower cases it. Thus, to have facets work as we use them now (they show the exact data that is on the record) then :0 should be used.

With :p :

<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="p" index="subject">
    <term coccur="1" occur="49">sang</term>
    <term coccur="1" occur="45">tegneserier amerikanske 25458100</term>
    <term coccur="1" occur="42">02</term>
    <term coccur="1" occur="42">klaver</term>
    <term coccur="1" occur="28">spillefilmer amerikanske 28231100</term>
    <term coccur="1" occur="17">filmer komedie 2040551600</term>
    <term coccur="1" occur="16">sang klaver 02</term>
    <term coccur="1" occur="12">filmer romantisk 2040551900</term>
    <term coccur="1" occur="5">tidsskrifter norske 13171900</term>
    <term coccur="1" occur="3">barnehager tidsskrifter 372 2105 15283900</term>
    <term coccur="1" occur="3">basketball 796 323 d5 10329200</term>
    <term coccur="1" occur="3">litteratur historie og kritikk 809 d5 10045100</term>
    <term coccur="1" occur="3">malerkunst historie generelt 759 d5 750 z 24632900</term>
    <term coccur="1" occur="3">sporreboker 793 73 d5 12683600</term>
    <term coccur="1" occur="3">sprak 400 d5 2041875500</term>
    <term coccur="1" occur="3">tegneserier science fiction bs 26382800</term>
  </facet>
</record>

So we should define facets like this:

 <retrieval syntax="xml" name="zebra::facet::subject:0:100"/>

The element set in the call should be the same and will get this results:

Z> elem zebra::facet::subject:0:100
Z> s
Sent presentRequest (1+1).
Records: 1
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="0" index="subject">
    <term coccur="1" occur="49">Sang</term>
    <term coccur="1" occur="45">Tegneserier, Amerikanske 25458100</term>
    <term coccur="1" occur="42">02</term>
    <term coccur="1" occur="42">Klaver</term>
    <term coccur="1" occur="28">Spillefilmer, Amerikanske 28231100</term>
    <term coccur="1" occur="17">Filmer Komedie 2040551600</term>
    <term coccur="1" occur="16">Sang Klaver 02</term>
    <term coccur="1" occur="12">Filmer Romantisk 2040551900</term>
    <term coccur="1" occur="5">Tidsskrifter, Norske 13171900</term>
    <term coccur="1" occur="3">Barnehager Tidsskrifter 372.2105 15283900</term>
    <term coccur="1" occur="3">Basketball 796.323 d5 10329200</term>
    <term coccur="1" occur="3">Litteratur Historie og kritikk 809 d5 10045100</term>
    <term coccur="1" occur="3">Malerkunst Historie Generelt 759 d5 750 z 24632900</term>
    <term coccur="1" occur="3">Språk 400 d5 2041875500</term>
    <term coccur="1" occur="3">Spørrebøker 793.73 d5 12683600</term>
    <term coccur="1" occur="3">Tegneserier Science fiction BS 26382800</term>
  </facet>
</record>


---
And I'm sure we'll be getting better results in the encoding front, because Zebra should be returning the exact data we sent for indexing.
Comment 24 Tomás Cohen Arazi 2014-03-21 19:25:22 UTC
(In reply to David Cook from comment #20)
> Latest trial...
> 
> Facets seem to work great with my package install but not with my tarball
> dev install.
...
>     <term coccur="4" occur="2">+I7</term>
>     <term coccur="4" occur="1">+I7</term>
>   </facet>
> </record>
> 
> --
> 
> Will have to look into possible causes for this...

David, the problem with your setup is that you inadvertedly set ICU for the :0 index which you shouldn't.

Just set:

icuchain words-icu.xml

for 'index w' and 'index p'.

We are using the :0 index for display purposes, so no comparisson or order definition is in place.

I think it is time to start coding :-D
Comment 25 David Cook 2014-03-23 22:52:26 UTC
(In reply to Tomás Cohen Arazi from comment #24)
> (In reply to David Cook from comment #20)
> > Latest trial...
> > 
> > Facets seem to work great with my package install but not with my tarball
> > dev install.
> ...
> >     <term coccur="4" occur="2">+I7</term>
> >     <term coccur="4" occur="1">+I7</term>
> >   </facet>
> > </record>
> > 
> > --
> > 
> > Will have to look into possible causes for this...
> 
> David, the problem with your setup is that you inadvertedly set ICU for the
> :0 index which you shouldn't.
> 
> Just set:
> 
> icuchain words-icu.xml
> 
> for 'index w' and 'index p'.
> 
> We are using the :0 index for display purposes, so no comparisson or order
> definition is in place.
> 
> I think it is time to start coding :-D

Thanks for all that info, Tomas! You're the best!

However, I don't believe that I have ICU set for the :0 index. I only have icuchain words-icu.xml set up for 'index w' and 'index p'. Here's what I have in my default.idx for "index 0":

# Null map index (no mapping at all)
# Used if structure=key (@attr 4=3)
index 0
completeness 0
position 1
charmap @

Here's the diagnostic message I get when I try to use "index 0":

Records: 1
Diagnostic message(s) from database:
    [25] Specified element set name not valid for specified database -- v2 addinfo ''

I get the same message when I try with my non-ICU system :/.
Comment 26 Tomás Cohen Arazi 2014-03-23 23:39:33 UTC
(In reply to David Cook from comment #25)
> (In reply to Tomás Cohen Arazi from comment #24)
> > (In reply to David Cook from comment #20)
> > > Latest trial...
> > > 
> > > Facets seem to work great with my package install but not with my tarball
> > > dev install.
> > ...
> > >     <term coccur="4" occur="2">+I7</term>
> > >     <term coccur="4" occur="1">+I7</term>
> > >   </facet>
> > > </record>
> > > 
> > > --
> > > 
> > > Will have to look into possible causes for this...
> > 
> > David, the problem with your setup is that you inadvertedly set ICU for the
> > :0 index which you shouldn't.
> > 
> > Just set:
> > 
> > icuchain words-icu.xml
> > 
> > for 'index w' and 'index p'.
> > 
> > We are using the :0 index for display purposes, so no comparisson or order
> > definition is in place.
> > 
> > I think it is time to start coding :-D
> 
> Thanks for all that info, Tomas! You're the best!
> 
> However, I don't believe that I have ICU set for the :0 index. I only have
> icuchain words-icu.xml set up for 'index w' and 'index p'. Here's what I
> have in my default.idx for "index 0":
> 
> # Null map index (no mapping at all)
> # Used if structure=key (@attr 4=3)
> index 0
> completeness 0
> position 1
> charmap @
> 
> Here's the diagnostic message I get when I try to use "index 0":
> 
> Records: 1
> Diagnostic message(s) from database:
>     [25] Specified element set name not valid for specified database -- v2
> addinfo ''
> 
> I get the same message when I try with my non-ICU system :/.

What I did was: 

-added some :0 use in zebradb/retrieval-info-bib-dom.xml like:

<retrieval syntax="xml" name="zebra::facet::subject:0:100"/>

- Edited marc_defs/marc21/biblios/biblio-koha-indexdefs.xml and added an occurence of

<target_index>Subject:0</target_index>
after every
<target_index>Subject:p</target_index>

this could be narrowed down later (we might not need facets for all thsoe fields), but I wanted to make sure :0 contained the same as :p for subject, for testing purposes.

- Run 
xsltproc ~/koha-dev/etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl biblio-koha-indexdefs.xml > biblio-zebra-indexdefs.xsl

to regenerate the xsl file,

- Then reindex :-(

And then search using:
 Z> format xml
 Z> elem zebra::facet::subject:0:100

I also had zebra::facet::subject:p defined to compare the results over the same search.

Regards
To+
Comment 27 David Cook 2014-03-24 00:56:47 UTC
(In reply to Tomás Cohen Arazi from comment #26)
> (In reply to David Cook from comment #25)
> > (In reply to Tomás Cohen Arazi from comment #24)
> > > (In reply to David Cook from comment #20)
> > > > Latest trial...
> > > > 
> > > > Facets seem to work great with my package install but not with my tarball
> > > > dev install.
> > > ...
> > > >     <term coccur="4" occur="2">+I7</term>
> > > >     <term coccur="4" occur="1">+I7</term>
> > > >   </facet>
> > > > </record>
> > > > 
> > > > --
> > > > 
> > > > Will have to look into possible causes for this...
> > > 
> > > David, the problem with your setup is that you inadvertedly set ICU for the
> > > :0 index which you shouldn't.
> > > 
> > > Just set:
> > > 
> > > icuchain words-icu.xml
> > > 
> > > for 'index w' and 'index p'.
> > > 
> > > We are using the :0 index for display purposes, so no comparisson or order
> > > definition is in place.
> > > 
> > > I think it is time to start coding :-D
> > 
> > Thanks for all that info, Tomas! You're the best!
> > 
> > However, I don't believe that I have ICU set for the :0 index. I only have
> > icuchain words-icu.xml set up for 'index w' and 'index p'. Here's what I
> > have in my default.idx for "index 0":
> > 
> > # Null map index (no mapping at all)
> > # Used if structure=key (@attr 4=3)
> > index 0
> > completeness 0
> > position 1
> > charmap @
> > 
> > Here's the diagnostic message I get when I try to use "index 0":
> > 
> > Records: 1
> > Diagnostic message(s) from database:
> >     [25] Specified element set name not valid for specified database -- v2
> > addinfo ''
> > 
> > I get the same message when I try with my non-ICU system :/.
> 
> What I did was: 
> 
> -added some :0 use in zebradb/retrieval-info-bib-dom.xml like:
> 
> <retrieval syntax="xml" name="zebra::facet::subject:0:100"/>
> 
> - Edited marc_defs/marc21/biblios/biblio-koha-indexdefs.xml and added an
> occurence of
> 
> <target_index>Subject:0</target_index>
> after every
> <target_index>Subject:p</target_index>
> 
> this could be narrowed down later (we might not need facets for all thsoe
> fields), but I wanted to make sure :0 contained the same as :p for subject,
> for testing purposes.
> 
> - Run 
> xsltproc ~/koha-dev/etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl
> biblio-koha-indexdefs.xml > biblio-zebra-indexdefs.xsl
> 
> to regenerate the xsl file,
> 
> - Then reindex :-(
> 
> And then search using:
>  Z> format xml
>  Z> elem zebra::facet::subject:0:100
> 
> I also had zebra::facet::subject:p defined to compare the results over the
> same search.
> 
> Regards
> To+

I believe that I've done the same thing and am getting that error instead :(.

I'll try again though just to make sure :).
Comment 28 David Cook 2014-03-24 01:32:25 UTC
Thanks again, Tomas!

I'm accustomed to using the GRS-1 filter, so I forgot about transforming the XML into XSL for DOM indexing. Problem solved! Yay!

That's great information about appending "100" to the element to get a longer facet list as well! I don't know if we would necessarily want more than 20 results per facet, but it's nice to have the configurability to change it.

However, I would recommend "not" hardcoding "zebra::facet::subject:0:100" into zebradb/retrieval-info-bib-dom.xml.

Try just using "<retrieval syntax="xml"/>" in "zebradb/retrieval-info-bib-dom.xml". It's a much more flexible configuration.

It lets you change "100" to "50" or whatever just by using the "elements" command in YAZ. You can also chain facets in elements so that you can run 1 query but get the facets for 5 different elements. (e.g.  elements zebra::facet::title:0:100,subject:0:25,itype:0:30)

[Sidenote: I see my earlier comments about index case sensitivity aren't correct, at least when using the <retrieval syntax="xml"/> config.]

Yes, I think we've found a solution! Time to start writing code!

The only obstacle I see now is determining which indexes need the "index 0". I suppose all the fields currently used for facets...plus pubdate.

This will also be an obstacle in terms of having configurable facets. Using "<retrieval syntax="xml"/>" in the Zebra config would let you spin a query with any facets we want, but relying on that "index 0" will determine which results return usable data...

Still...very exciting!

I imagine it would be useful to write something a bit abstract for getting facets, so that we can make it easier to plugin other indexers like Solr and ElasticSearch...
Comment 29 David Cook 2014-03-24 02:33:01 UTC
PS Is that your way of saying that you're working on this project, Tomas? ;)
Comment 30 Paul Poulain 2014-03-24 07:48:30 UTC
(In reply to David Cook from comment #29)
> PS Is that your way of saying that you're working on this project, Tomas? ;)

For now, he's honeymooning ;-) (should be back around april, 15th)

> I imagine it would be useful to write something a bit abstract for getting 
> facets, so that we can make it easier to plugin other indexers like Solr and 
> ElasticSearch...
Of course !
Comment 31 wajasu 2014-03-27 22:42:48 UTC
Idea: To determine which indexes get the additional index:0, we could:
provide a UI(similiar to msaby's suggestion that allows us to select an index by grabbing the <target_index> nodes from biblios-koha-indexdefs.xml, and search replacing
<target_index>index:p</target_index> with
<target_index>index:p</target_index><target_index>index:0</target_index>
(control_field, data_field, sub_field savy UI is possible)

That would at least let the selection be flexible according to the current configuration, though that file must be located for "koha packages"(TBD).

We could roll out with things initially matching today's facets, and let them select predefined profiles at some point(future feature).

The facet selection UI should highlight configured(non-disabled) facets that are may not exist anymore in the biblios-koha-indexdefs.xml, to aid debugging when that file is subsequently modified (admin or future koha update).

When querying the chosen facets, it might be worth querying the whole list at once and if it fails, querying each sequentially such that if one fails, we just don't supply that label to the UI and log the failing facet, refering them to the facet config UI to reconcile.  But things continue to run.

It might be worth looking into generating retreival-info-bib-grs1-faceted.xml and biblios-koha-indexdefs-faceted.xml and launching xsltproc to generate biblio-zebra-indexdefs-faceted.xsl, without touching the original configs. Facets could be disabled to enable non-faceted searching until the issue is resolved.

We might also employ xi:include and fallback in koha-conf.xml to use the faceted file(s) which could be generated/removed by the facet config UI. 
<xi:include href="/home/koha/koha-dev/etc/zebradb/retrieval-info-bib-dom.xml"
      xmlns:xi="http://www.w3.org/2001/XInclude">
<xi:fallback>
Thus leaving koha-conf.xml flexible.

Mapping tags to new indexes that go in biblios-koha-indexdefs.xml can possibly be a separate effort.

Of course this is all very zebra focused.  But the idea is to restrict free form editing (for security) and allow newly introduced indexes to roll out without concern of breaking updates.
Comment 32 David Cook 2014-03-28 01:52:42 UTC
(In reply to wajasu from comment #31)
> Idea: To determine which indexes get the additional index:0, we could:
> provide a UI(similiar to msaby's suggestion that allows us to select an
> index by grabbing the <target_index> nodes from biblios-koha-indexdefs.xml,
> and search replacing
> <target_index>index:p</target_index> with
> <target_index>index:p</target_index><target_index>index:0</target_index>
> (control_field, data_field, sub_field savy UI is possible)
> 

I think that this is a bad idea.

1) I'm not 100% sure, but I think that many installs of Koha run multiple sites using one set of Zebra configs. At least, that's how it appears to work on package installs.

I suppose you could have this be an additive process that checks if "index:0" exists and if not, you can add it, when you want to use that index. That wouldn't screw with other people's indexing too much.  

But I'm not sure allowing the web server to write files in /etc/koha/ is the best idea.

2) Increasingly, I'm becoming less sure that librarians themselves actually need web access to this type of configuration. I think it is a good idea to have configurable facets, but perhaps having a separate file for this would be best. A sysadmin could then edit it as needed.

> When querying the chosen facets, it might be worth querying the whole list
> at once and if it fails, querying each sequentially such that if one fails,
> we just don't supply that label to the UI and log the failing facet,
> refering them to the facet config UI to reconcile.  But things continue to
> run.
> 

I don't follow you 100% here. I originally thought that you meant we could query zebra for our chosen facets, and if that failed, that we could choose do each individually. That would probably make sense so that facets didn't entirely bust for one badly configured facet.

> It might be worth looking into generating
> retreival-info-bib-grs1-faceted.xml and biblios-koha-indexdefs-faceted.xml
> and launching xsltproc to generate biblio-zebra-indexdefs-faceted.xsl,
> without touching the original configs. Facets could be disabled to enable
> non-faceted searching until the issue is resolved.
> 
> We might also employ xi:include and fallback in koha-conf.xml to use the
> faceted file(s) which could be generated/removed by the facet config UI. 
> <xi:include href="/home/koha/koha-dev/etc/zebradb/retrieval-info-bib-dom.xml"
>       xmlns:xi="http://www.w3.org/2001/XInclude">
> <xi:fallback>
> Thus leaving koha-conf.xml flexible.
> 
> Mapping tags to new indexes that go in biblios-koha-indexdefs.xml can
> possibly be a separate effort.
> 

I really think this is getting overly complex. This would likely take quite a bit of code just to add a web UI that most librarians will probably (and should probably) never touch.

--

I think the best idea is probably to add "index:0" for our current range of facets, and add it to a few other desired facets (such as pubdate). 

We put the list of indexes in a file, make sure that those indexes are sent off to Zebra when doing a facet query, and that's that.

If a web UI is really necessary, perhaps reading the list of indexes out of that file would make sense.

Perhaps the file would be unnecessary and we could just read out the "index:0" facets from the Zebra config, but I would be hesitant about adding them in...

...especially since adding "index:0" and sending a facet query referencing that new index:0 before doing a full re-index will break the facets (if sent as one combined query and not lots of separate ones).

--

I'm not saying I have a perfect idea in mind, but I think... we shouldn't try to be too fancy or convoluted.

Simple and elegant. We provide a set of default facets, and if folks want to add their own, we leave them with good enough instructions to do so.

I really don't want to be dynamically messing with files that should be static.
Comment 33 Paul Poulain 2014-03-28 08:07:23 UTC
(In reply to David Cook from comment #32)
> (In reply to wajasu from comment #31)
> > Idea: To determine which indexes get the additional index:0, we could:
> > provide a UI(similiar to msaby's suggestion that allows us to select an
> > index by grabbing the <target_index> nodes from biblios-koha-indexdefs.xml,
> > and search replacing
> > <target_index>index:p</target_index> with
> > <target_index>index:p</target_index><target_index>index:0</target_index>
> > (control_field, data_field, sub_field savy UI is possible)
> > 
> 
> I think that this is a bad idea.
Agreed with most of your points. One of the reason being that it's impossible to have everything made by the librarian (any change require rebuilding zebra DB, plus some/many config file change. and yes, it's not a good idea to let the webserver write in /etc ...). So I prefer to have everything made by a sysadmin.

When we worked on Solr, we made indexes accessible by the web interface, but it's something shipped with Solr. That's not the case for Zebra.
> We put the list of indexes in a file, make sure that those indexes are sent
> off to Zebra when doing a facet query, and that's that.
Zebra is highly sensitive on how the config files are written. I'm not sure you can have :0 indexes in a separate file.
AFAIK, when you double declare a line, only the last one is taken into account.
So:
melm 245$a Title:w,Title:p
melm 245$a Title:0
would result in Title:0 being the only index. Which is not what we want.

> Simple and elegant. We provide a set of default facets, and if folks want to
> add their own, we leave them with good enough instructions to do so.
+1
Comment 34 wajasu 2014-03-30 05:46:15 UTC
OK, David and Paul.

I'll code a facet query in our current search for a set of expected facets, as we use today, plus pubdate, or others added with followup patches.

I'll query for all in one go.  If that fails, then might try individual queries as a fallback.  Maybe even keep the current code as a fallback.

If sysadmin updates the configs, then we get the enhanced behavior.

We can NYTprof and see impact.  If a discernable impact, we could add syspref "ZebraNativeFaceting" on/off.

Let me attempt to throw something simple together before i forget how all this zebra search stuff works.
Comment 35 Paul Poulain 2014-03-31 07:03:39 UTC
(In reply to wajasu from comment #34)
> We can NYTprof and see impact.  If a discernable impact, we could add
> syspref "ZebraNativeFaceting" on/off.
mmm... Why would we want to have this syspref OFF ? Currently, the facets are wrong (at least incomplete), slow to calculate.
With this patch they will become more reliable & faster. So ...

If we implement this, that could be a requirement for upgrade (to check/update zebra configuration). For libraries using package, it's automatic. For others, they are supposed to know what they're doing, they're able to update the config.
This already happened (for example the bug 3087)

> Let me attempt to throw something simple together before i forget how all
> this zebra search stuff works.
+1 !
Comment 36 wajasu 2014-04-09 04:08:46 UTC Comment hidden (obsolete)
Comment 37 Mark Tompsett 2014-08-05 19:15:58 UTC
Is this still being worked on? Does this need sign off?
Comment 38 David Cook 2014-08-06 00:14:22 UTC
(In reply to M. Tompsett from comment #37)
> Is this still being worked on? Does this need sign off?

Looks like there is still lots of debugging code (even if it's commented out).

Personally, I would prefer to have the first step be to decompose the existing code into functions (ie reduce the spaghetti) so that it's easier to read and switch between different facet creation methods.

I'm interested in Zebra native facets, but sadly it's not on my "to do" list at the moment, so I won't be contributing anything (besides constructive criticism :P) in the foreseeable future.
Comment 39 Tomás Cohen Arazi 2014-09-06 15:21:04 UTC Comment hidden (obsolete)
Comment 40 Tomás Cohen Arazi 2014-09-06 15:21:14 UTC Comment hidden (obsolete)
Comment 41 Tomás Cohen Arazi 2014-09-06 15:21:20 UTC Comment hidden (obsolete)
Comment 42 Tomás Cohen Arazi 2014-09-06 15:21:27 UTC Comment hidden (obsolete)
Comment 43 Tomás Cohen Arazi 2014-09-06 15:21:38 UTC Comment hidden (obsolete)
Comment 44 Tomás Cohen Arazi 2014-09-06 15:36:05 UTC Comment hidden (obsolete)
Comment 45 Tomás Cohen Arazi 2014-09-06 15:36:19 UTC Comment hidden (obsolete)
Comment 46 Tomás Cohen Arazi 2014-09-06 15:36:28 UTC Comment hidden (obsolete)
Comment 47 Tomás Cohen Arazi 2014-09-06 15:36:36 UTC Comment hidden (obsolete)
Comment 48 Tomás Cohen Arazi 2014-09-06 15:36:43 UTC Comment hidden (obsolete)
Comment 49 Nick Clemens 2014-09-06 16:32:35 UTC
Just a very quick look, I seem to have lost 'Holding libraries' as a facet.  We definitely need that for our consortium
Comment 50 Tomás Cohen Arazi 2014-09-06 16:40:47 UTC
(In reply to Nick Clemens from comment #49)
> Just a very quick look, I seem to have lost 'Holding libraries' as a facet. 
> We definitely need that for our consortium

Thanks for the feedback, Nick. Please send me a sample record. And also tell me this marc dialect the record is written in, and the one you chose on install.
Comment 51 Nick Clemens 2014-09-06 16:49:25 UTC
Created attachment 31436 [details]
Sample record, items not faceted by holding library in search

Marc21 for record and for install
Comment 52 Tomás Cohen Arazi 2014-09-08 02:07:39 UTC
(In reply to Nick Clemens from comment #51)
> Created attachment 31436 [details]
> Sample record, items not faceted by holding library in search
> 
> Marc21 for record and for install

Nick, I've spotted the problem. Thanks for testing, will provide a new set of patches tomorrow.
Comment 53 Tomás Cohen Arazi 2014-09-08 13:34:30 UTC Comment hidden (obsolete)
Comment 54 Tomás Cohen Arazi 2014-09-08 13:34:37 UTC Comment hidden (obsolete)
Comment 55 Tomás Cohen Arazi 2014-09-08 13:34:45 UTC Comment hidden (obsolete)
Comment 56 Tomás Cohen Arazi 2014-09-08 13:34:52 UTC Comment hidden (obsolete)
Comment 57 Tomás Cohen Arazi 2014-09-08 13:35:01 UTC Comment hidden (obsolete)
Comment 58 Tomás Cohen Arazi 2014-09-08 13:35:12 UTC Comment hidden (obsolete)
Comment 59 Tomás Cohen Arazi 2014-09-08 13:40:39 UTC
(In reply to Nick Clemens from comment #51)
> Created attachment 31436 [details]
> Sample record, items not faceted by holding library in search
> 
> Marc21 for record and for install

Nick, I've fixed the problem on the latest patch set.
Comment 60 Jonathan Druart 2014-09-17 08:36:01 UTC
Unimarc testing:

For author and topic facets, spaces are replaced by "::*::".
The facet list looks like:
  Ganz::*::Axel
  Gardel::*::Louis
  Hall::*::John D 

The links are catalogue/search.pl?idx=kw&q=d&limit=au:Ganz%3A%3A*%3A%3AAxel

And if I follow it:
   No results match your search for 'kw,wrdl: d' with limit(s): 'au:Hall::*::John D'
Comment 61 Jonathan Druart 2014-09-17 08:39:04 UTC
And in the Koha logs:
[Wed Sep 17 10:34:04 2014] search.pl: WARNING: query problem with kw,wrdl=d and au=Hall=*=John D ZOOM error 10014 "CCL parsing error" (addinfo: "Embedded truncation not supported") from diag-set 'ZOOM' at /home/koha/src/C4/Search.pm line 380.
Comment 62 Tomás Cohen Arazi 2014-09-19 22:48:12 UTC Comment hidden (obsolete)
Comment 63 Tomás Cohen Arazi 2014-09-19 22:48:22 UTC Comment hidden (obsolete)
Comment 64 Tomás Cohen Arazi 2014-09-19 22:48:31 UTC Comment hidden (obsolete)
Comment 65 Tomás Cohen Arazi 2014-09-19 22:48:43 UTC Comment hidden (obsolete)
Comment 66 Tomás Cohen Arazi 2014-09-19 22:48:51 UTC Comment hidden (obsolete)
Comment 67 Tomás Cohen Arazi 2014-09-19 22:49:02 UTC Comment hidden (obsolete)
Comment 68 David Cook 2014-09-22 01:39:12 UTC Comment hidden (obsolete)
Comment 69 David Cook 2014-09-22 01:39:21 UTC Comment hidden (obsolete)
Comment 70 David Cook 2014-09-22 01:39:28 UTC Comment hidden (obsolete)
Comment 71 David Cook 2014-09-22 01:39:36 UTC Comment hidden (obsolete)
Comment 72 David Cook 2014-09-22 01:39:43 UTC Comment hidden (obsolete)
Comment 73 David Cook 2014-09-22 01:39:52 UTC Comment hidden (obsolete)
Comment 74 David Cook 2014-09-22 01:40:29 UTC
Putting this back to "Needs signoff" so that other people can test NORMARC and UNIMARC...
Comment 75 Jonathan Druart 2014-10-10 14:34:16 UTC Comment hidden (obsolete)
Comment 76 Jonathan Druart 2014-10-10 14:34:34 UTC Comment hidden (obsolete)
Comment 77 Jonathan Druart 2014-10-10 14:34:44 UTC Comment hidden (obsolete)
Comment 78 Jonathan Druart 2014-10-10 14:34:51 UTC Comment hidden (obsolete)
Comment 79 Jonathan Druart 2014-10-10 14:34:59 UTC Comment hidden (obsolete)
Comment 80 Jonathan Druart 2014-10-10 14:35:09 UTC Comment hidden (obsolete)
Comment 81 Jonathan Druart 2014-10-10 14:35:56 UTC
Tested with Tomas yesterday on a UNIMARC installation. All works fine :)
Comment 82 Katrin Fischer 2014-10-11 20:32:44 UTC
Hi Tomas,

I am seeing problem with this comparing the facets before and after appyling the patch. Testing on a fairly small database with 120 records there appear to be less facets appearing after applying the patch then before. Can you check? I will send you the records by email, search term was "de".
Comment 83 Tomás Cohen Arazi 2014-10-15 13:34:29 UTC Comment hidden (obsolete)
Comment 84 Tomás Cohen Arazi 2014-10-15 13:35:50 UTC Comment hidden (obsolete)
Comment 85 Tomás Cohen Arazi 2014-10-15 15:17:22 UTC Comment hidden (obsolete)
Comment 86 Katrin Fischer 2014-10-15 15:29:49 UTC
Created attachment 32352 [details] [review]
[PASSED QA] Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl

This patch changes koha-indexdefs-to-zebra.xsl to correctly process a new syntax
for defining facet indexes on the XML files.

It also changes the retrieval file to allow access to Zebra's internal data from
Zoom (i.e. access to zebra::facet:*).

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 87 Katrin Fischer 2014-10-15 15:29:57 UTC
Created attachment 32353 [details] [review]
[PASSED QA] Bug 11232: MARC21 facet definition and updated XSL file for DOM

This patch adds the facets definitions to the biblio-koha-indexdefs.xml, based
on what is hardcoded on C4::Koha::getFacets().

The biblio-zebra-indexdefs.xsl file for MARC21 is generated using the usual:

xsltproc ...koha-indexdefs-to-zebra.xsl ...marc21/biblios/biblio-koha-indexdefs.xml > \
    ...marc21/biblios/biblio-zebra-indexdefs.xsl

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 88 Katrin Fischer 2014-10-15 15:30:24 UTC
Created attachment 32354 [details] [review]
[PASSED QA] Bug 11232: UNIMARC facet definition and updated XSL file for DOM

This patch adds the facets definitions to the biblio-koha-indexdefs.xml, based
on what is hardcoded on C4::Koha::getFacets().

The biblio-zebra-indexdefs.xsl file for UNIMARC is generated using the usual:

xsltproc ...koha-indexdefs-to-zebra.xsl ...unimarc/biblios/biblio-koha-indexdefs.xml > \
    ...unimarc/biblios/biblio-zebra-indexdefs.xsl

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 89 Katrin Fischer 2014-10-15 15:30:53 UTC
Created attachment 32355 [details] [review]
[PASSED QA] Bug 11232: NORMARC facet definition and updated XSL file for DOM

This patch adds the facets definitions to the biblio-koha-indexdefs.xml, based
on what is hardcoded on C4::Koha::getFacets().

The biblio-zebra-indexdefs.xsl file for NORMARC is generated using the usual:

xsltproc ...koha-indexdefs-to-zebra.xsl ...normarc/biblios/biblio-koha-indexdefs.xml > \
    ...normarc/biblios/biblio-zebra-indexdefs.xsl

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 90 Katrin Fischer 2014-10-15 15:31:29 UTC
Created attachment 32356 [details] [review]
[PASSED QA] Bug 11232: retrieve facets from Zebra's indexes in DOM

This patch adds the following routines to C4::Search

- GetFacets
   This is a wrapper routine, that given a ZOOM::ResultSet, extracts
   the relevant facets. To do the job, it uses the internal functions:
   _get_facets_from_records and _get_facets_from_zebra. The choice is done
   on querying the indexing mode: grs1 will use records, and dom zebra's facets.
- _get_facets_from_records
   Just refactoring the already existent main loop from getRecords into a function.
- _get_facets_from_zebra
   Given a result set, loop through all defined facets in C4::Koha::getFacets
   and call _get_facet_from_result_set for each, to build the facets
   information. It retrieves the facets from Zebra's facets.
- _get_facet_from_result_set
   Given a result set and a facet index name, retrieve the facets
   for the given index, and build the result for rendering.

To test this preliminay work:
- Apply the patches, install on DOM, using MARC21, NORMARC and UNIMARC.
- Reindex some DB with lots of records.
- Check facets work.

Note: UNIMARC is the only dialect that has more than one subfield (concatenated)
for facets values, so it is better to test on uNIMARC. The approach leaves room
for re-thinking facets in MARC21/NORMARC, but it is outside of the scope of the bug
(e.g. we could have more author facets)

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 91 Katrin Fischer 2014-10-15 15:31:49 UTC
Created attachment 32357 [details] [review]
[PASSED QA] Bug 11232: (followup) remove unnecesary namespace definition from all XML elements

The previous patches for facet extraction from Zebra indexes set a default
namespace on the following files:

etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
etc/zebradb/marc_defs/normarc/biblios/biblio-koha-indexdefs.xml
etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml

and hence the XML file index_subfields can be cleaned by removing the namespace.

To test:
- Apply this patch
- Run

$ for i in marc21 normarc unimarc
  do xsltproc etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl \
              etc/zebradb/marc_defs/$i/biblios/biblio-koha-indexdefs.xml \
              > etc/zebradb/marc_defs/$i/biblios/biblio-zebra-indexdefs.xsl
  done

=> SUCCESS: no errors reported

- Run
$ git diff
=> SUCCESS: no differences on the xsl files

- Sign off :-D

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 92 Katrin Fischer 2014-10-15 15:32:06 UTC
Created attachment 32358 [details] [review]
[PASSED QA] Bug 11232: (followup) Configuration variable for enabling Zebra facets

This patch adds a variable to koha-conf.xml controlling the use of Zebra facets.

Usage:
 - use_zebra_facets = 1 | 0

Zebra facets work only on DOM.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 93 Katrin Fischer 2014-10-15 15:32:15 UTC
Created attachment 32359 [details] [review]
[PASSED QA] Bug 11232: (qa followup) Add missing fields/subfields to the item types faceta

The itype facet was missing 952$y for both MARC21 and NORMARC.
This patch adds that. And also modifies the zebra-biblios-dom.cfg file
(also the debian/ version) so facetNumRecs is set to 1000 for zebra.

It is the amount of records that are taken into account. The more record,
the more exact the facets for the result set. 1000 was chosen as it changed
the time to reindex 1000 records from 18s to 19s.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 94 Katrin Fischer 2014-10-15 15:32:23 UTC
Created attachment 32360 [details] [review]
[PASSED QA] Bug 11232: (qa followup) empty ID due to namespace mistake

Note: NORMARC is missing the id field.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
This patch makes t/db_dependent/Search.t pass again.
NORMARC is currently not tested.

I checked the results before and after applying the patch
and the facets are now looking the same as before.
Passes all tests and QA script.
Comment 95 Tomás Cohen Arazi 2014-10-15 16:03:29 UTC
Patch pushed to master.
Comment 96 David Cook 2014-10-27 23:29:20 UTC
Quick question. Is there a reason why we needed to add "facetNumRecs"? 

The speed already seemed pretty good. Even if it were a bit slower, I think I'd rather use all retrieved records than an arbitrary number of them :/. I suppose 1000 is pretty good, but if I click a facet, I want to know that I'm getting all the relevant records from a result set and not just the records that are from the first 1000 records.

Of course, perhaps I'm being a bit ridiculous. If a person is getting over 1000 results, their search isn't very exact. But... sometimes you want to do a broad search then narrow it down using multiple facets :/.

I suppose this is the kind of configuration that individual libraries could change on their own though.
Comment 97 Mark Tompsett 2014-10-27 23:41:39 UTC
Actually, I haven't looked at the code, but David's comment does resonate with me. If I can retrieve all the facets, why would I limit them? Is there a significant speed difference at a particular number of facets? If I have to wait 20 seconds for all the facets, is that really so bad for a large search result?
Comment 98 Tomás Cohen Arazi 2014-10-27 23:59:46 UTC
(In reply to David Cook from comment #96)
> Quick question. Is there a reason why we needed to add "facetNumRecs"? 

Zebra defaults to 10 [1] (which is way too low) and I found no way to define a wildcard.

Any ideas for improving this are welcome.

[1] Line 902 of http://www.indexdata.com/zebra/dox/html/retrieve_8c_source.html