Hi
Well, I will wait to see some patch attached here to know if bug 10891 should be closed, or maybe rethink on a different base (maybe just to hide unwanted facets, and not providing a real customization for example).

Mathieu

> 
> I'm not sure about the cost/benefit of telling Zebra that every index could
> be used as a facet. I don't imagine it would be too much of a cost though,
> since the list of facets would be fed to Zebra on start up and not runtime.


Of course ;-)
Just take a look at some library or bookseller's websites.
Most fields must NOT be used as facets.

For me facets should made on :
- availability (make a distinction between items notforloan, items on loan, items available for loan)
- date (on coded fields)
- language
- language of original material (for translations)
- subject (with facets based on subfields of 6XX, not on complete fields : a facet for topic subjects, a facet for chronological, a facet for geographical etc)
- type of document and carriage (various fields and coded fields in MARC21 and UNIMARC)
- coded informations in 0XX MARC21 and 1XX UNIMARC fields
- coded informations in label
- authors

and maybe titles, and series

Mathieu

Note that there is currently a big issue for UNIMARC facets:
the facet build on type of document (equivalent of 942 in MARC21) is not visible.

That's because some libraries use 099$t for that, and others use other fields like 210$b that are not made for that.
That was the main goal of my not-yet-written-patch for make facets customisable...

But maybe it could be fixed by telling UNIMARC ppl that from 3.16 version onwards the type of document is stored in 099t, and provide a script for copying types of documents from other fields to this one for non standard configuration.

Mathieu

(In reply to mathieu saby from comment #3)
> Note that there is currently a big issue for UNIMARC facets:
> the facet build on type of document (equivalent of 942 in MARC21) is not
> visible.
> 
> That's because some libraries use 099$t for that, and others use other
> fields like 210$b that are not made for that.
> That was the main goal of my not-yet-written-patch for make facets
> customisable...
> 
> But maybe it could be fixed by telling UNIMARC ppl that from 3.16 version
> onwards the type of document is stored in 099t, and provide a script for
> copying types of documents from other fields to this one for non standard
> configuration.
> 
> Mathieu

With zebra facets, it uses the index. So if both those marc tags are filling the same index, it should be fine.

(In reply to David Cook from comment #4)
> (In reply to mathieu saby from comment #3)
> > Note that there is currently a big issue for UNIMARC facets:
> > the facet build on type of document (equivalent of 942 in MARC21) is not
> > visible.
> > 
> > That's because some libraries use 099$t for that, and others use other
> > fields like 210$b that are not made for that.
> > That was the main goal of my not-yet-written-patch for make facets
> > customisable...
> > 
> > But maybe it could be fixed by telling UNIMARC ppl that from 3.16 version
> > onwards the type of document is stored in 099t, and provide a script for
> > copying types of documents from other fields to this one for non standard
> > configuration.
> > 
> > Mathieu
> 
> With zebra facets, it uses the index. So if both those marc tags are filling
> the same index, it should be fine.

I did not think of it... Of course ;-)
So, it will really be a nice improvement for 3.16 if Zebra facets could be used.

I'm outlining below a rough guide to how to enable Zebra facetting. Note that this has nothing to do with Koha per se. This is just about setting up Zebra. The actual Koha code to make use of Zebra's facets will have to come after initial setup, and at the moment it's just not at the top of my list of priorities.

First, take stock of which version of Koha you're running and note which type of indexing you're using (DOM or GRS1).

Look for these two files in your /etc/zebradb/ folder:
"retrieval-info-bib-dom.xml"
"retrieval-info-bib-grs1.xml"

If you can't find them, you're probably running an older version of Koha and you'll want to modify your "koha-conf.xml". If you do find them, the config in "koha-conf.xml" is only a fallback, so not necessary for testing.


Step One:
a) Edit the file that suits your version of Koha and method of indexing.
(i.e. if you're running the latest Koha with dom, edit "retrieval-info-bib-dom.xml". If you're running the latest Koha with grs1, edit "retrieval-info-bib-grs1.xml". If you're running an older version of Koha, you're only using grs1 and you'll need to edit "koha-conf.xml"

Step Two:
a) Add lines like the following as children to the "<retrievalinfo>" node (N.B. do not add them to the top as they will displace the default retrieval syntax setting. Add them at the bottom.):

<retrieval syntax="xml" name="zebra::facet::any:w"/>
<retrieval syntax="xml" name="zebra::facet::title:w"/>
<retrieval syntax="xml" name="zebra::facet::title:0"/>

(N.B. Paul Poulain mentioned that the "0" register is a "raw"(?) register and that this should be used if you have issues with diacritics or non-Latin characters. I imagine that this needs to be set up in your records.abs or DOM XSLT.)

(N.B. You can substitute any index in for "any" or "title". You can also use any register instead of "w" or "0", such as "p".)

Step Three: 
a) Restart your Zebra server

Step Four:
a) Connect to Zebra using yaz-client (e.g. yaz-client open "unix:/zebra/test/var/run/zebradb/bibliosocket")(You can find your specific path by looking at the top of koha-conf.xml for a "listen" node with an ID of "biblioserver")

b) Tell Zebra to use facets (e.g. elements zebra::facet::any:w,title:w,title:0)
c) Do a search (e.g. find test)
d) Show your results (e.g. show 1+1)
e) You should now see a list of facets for your search results

--

If you have any issues with these instructions, comment here, find me in the IRC channel or visit the sites that I'm linking below.

--

For more information, visit the following links:

http://www.indexdata.com/zebra/doc/zebra.pdf (Search in the document for "facet")

http://lists.indexdata.dk/pipermail/zebralist/2007-July/001682.html (This is a discussion of setting up Zebra facets)

Also note that you will have to individually set up each index that you want to use for facet queries.

As per this email from an IndexData person: http://lists.indexdata.dk/pipermail/zebralist/2007-July/001694.html

In general, this shouldn't case a problem, since we only use a short list of facets. However, it would create a barrier to allowing users to customize their results.

In theory, we could list all or most of the available indexes in the Zebra config files. Then, we could use a system preference or a DB table to choose which indexes we want to use for facets. This would generate the "elements" string that we pass to Zebra everytime we want our facets.

Actually, now that I think about it...that might not be a bad idea. We already have that Search.pm sub where we're manually keeping track of indexes, so we could keep that or use something similar to fill an HTML list and have that create our Zebra facet string. 

So long as we generate the string once and then use it multiple times...that should be all right. We could also have a hardcoded fallback string in case someone deletes the facet config in Koha or if their facet config is bad. 

Hmm. Ideas. Ideas. Ideas...

(In reply to David Cook from comment #7)
> Also note that you will have to individually set up each index that you want
> to use for facet queries.
> 
> As per this email from an IndexData person:
> http://lists.indexdata.dk/pipermail/zebralist/2007-July/001694.html
> 
> In general, this shouldn't case a problem, since we only use a short list of
> facets. However, it would create a barrier to allowing users to customize
> their results.

This has to be changed in two places. In the server elements configuration, as described at that link, *and* an additional index has to be configured for each facet.

Could not this bug just get the zebra facets working, and then whatever needs to be done to filter be worked on in bug 10891?

> Actually, now that I think about it...that might not be a bad idea. We
> already have that Search.pm sub where we're manually keeping track of
> indexes, so we could keep that or use something similar to fill an HTML list
> and have that create our Zebra facet string. 
> 


I thought this list of index in search.pm was going to be deprecated with the adoption of QueryParser ?

Mathieu

(In reply to David Cook from comment #6)
> I'm outlining below a rough guide to how to enable Zebra facetting. Note
> that this has nothing to do with Koha per se. This is just about setting up
> Zebra. The actual Koha code to make use of Zebra's facets will have to come
> after initial setup, and at the moment it's just not at the top of my list
> of priorities.
> 
> First, take stock of which version of Koha you're running and note which
> type of indexing you're using (DOM or GRS1).
> 
> Look for these two files in your /etc/zebradb/ folder:
> "retrieval-info-bib-dom.xml"
> "retrieval-info-bib-grs1.xml"
> 
> If you can't find them, you're probably running an older version of Koha and
> you'll want to modify your "koha-conf.xml". If you do find them, the config
> in "koha-conf.xml" is only a fallback, so not necessary for testing.
> 
> 
> Step One:
> a) Edit the file that suits your version of Koha and method of indexing.
> (i.e. if you're running the latest Koha with dom, edit
> "retrieval-info-bib-dom.xml". If you're running the latest Koha with grs1,
> edit "retrieval-info-bib-grs1.xml". If you're running an older version of
> Koha, you're only using grs1 and you'll need to edit "koha-conf.xml"
> 
> Step Two:
> a) Add lines like the following as children to the "<retrievalinfo>" node
> (N.B. do not add them to the top as they will displace the default retrieval
> syntax setting. Add them at the bottom.):
> 
> <retrieval syntax="xml" name="zebra::facet::any:w"/>
> <retrieval syntax="xml" name="zebra::facet::title:w"/>
> <retrieval syntax="xml" name="zebra::facet::title:0"/>
> 
> (N.B. Paul Poulain mentioned that the "0" register is a "raw"(?) register
> and that this should be used if you have issues with diacritics or non-Latin
> characters. I imagine that this needs to be set up in your records.abs or
> DOM XSLT.)
> 
> (N.B. You can substitute any index in for "any" or "title". You can also use
> any register instead of "w" or "0", such as "p".)
> 
> Step Three: 
> a) Restart your Zebra server
> 
> Step Four:
> a) Connect to Zebra using yaz-client (e.g. yaz-client open
> "unix:/zebra/test/var/run/zebradb/bibliosocket")(You can find your specific
> path by looking at the top of koha-conf.xml for a "listen" node with an ID
> of "biblioserver")
> 
> b) Tell Zebra to use facets (e.g. elements
> zebra::facet::any:w,title:w,title:0)
> c) Do a search (e.g. find test)
> d) Show your results (e.g. show 1+1)
> e) You should now see a list of facets for your search results
> 
> --
> 
> If you have any issues with these instructions, comment here, find me in the
> IRC channel or visit the sites that I'm linking below.
> 
> --
> 
> For more information, visit the following links:
> 
> http://www.indexdata.com/zebra/doc/zebra.pdf (Search in the document for
> "facet")
> 
> http://lists.indexdata.dk/pipermail/zebralist/2007-July/001682.html (This is
> a discussion of setting up Zebra facets)

As DOM is now the new standard way of indexing, and GRS1 is going to be deprecated, maybe the work could be made only for DOM files?

I think it would be possible to agree that this is only done for DOM, as we are trying to phase out/deprecate GRS-1.
There are some bugs in QueryParser that I would like to see addressed before we make it the standard (for example bug 10542).

(In reply to Jared Camins-Esakov from comment #8)
> (In reply to David Cook from comment #7)
> > Also note that you will have to individually set up each index that you want
> > to use for facet queries.
> > 
> > As per this email from an IndexData person:
> > http://lists.indexdata.dk/pipermail/zebralist/2007-July/001694.html
> > 
> > In general, this shouldn't case a problem, since we only use a short list of
> > facets. However, it would create a barrier to allowing users to customize
> > their results.
> 
> This has to be changed in two places. In the server elements configuration,
> as described at that link, *and* an additional index has to be configured
> for each facet.

Jared, I'm not sure I understand what you mean by "an additional index has to be configured for each facet". 

Could you elaborate? 

Once the server element is set up for a facet, it should be all good to go. 

--

That said, I was thinking about this a bit last night, and I can't recall exactly what is retrieved with a facet query.

I have yet to set up a VM for Koha on my new desktop so I'll have to wait a bit before testing it out again.

(In reply to David Cook from comment #13)
> Jared, I'm not sure I understand what you mean by "an additional index has
> to be configured for each facet". 
> 
> Could you elaborate? 

You have to add a element:0 index for each facet you want to use.

(In reply to Jared Camins-Esakov from comment #14)
> (In reply to David Cook from comment #13)
> > Jared, I'm not sure I understand what you mean by "an additional index has
> > to be configured for each facet". 
> > 
> > Could you elaborate? 
> 
> You have to add a element:0 index for each facet you want to use.

Hmm, I'm not sure I understand. I didn't do that and it seemed to work all right.

(In reply to David Cook from comment #15)
> (In reply to Jared Camins-Esakov from comment #14)
> > (In reply to David Cook from comment #13)
> > > Jared, I'm not sure I understand what you mean by "an additional index has
> > > to be configured for each facet". 
> > > 
> > > Could you elaborate? 
> > 
> > You have to add a element:0 index for each facet you want to use.
> 
> Hmm, I'm not sure I understand. I didn't do that and it seemed to work all
> right.

Do you have tested a facet with diacritics or non latin characters. iirc, the :0 means "raw storing", and fixes diacritics/non-latin display problems

(In reply to Paul Poulain from comment #16)
> (In reply to David Cook from comment #15)
> > (In reply to Jared Camins-Esakov from comment #14)
> > > (In reply to David Cook from comment #13)
> > > > Jared, I'm not sure I understand what you mean by "an additional index has
> > > > to be configured for each facet". 
> > > > 
> > > > Could you elaborate? 
> > > 
> > > You have to add a element:0 index for each facet you want to use.
> > 
> > Hmm, I'm not sure I understand. I didn't do that and it seemed to work all
> > right.
> 
> Do you have tested a facet with diacritics or non latin characters. iirc,
> the :0 means "raw storing", and fixes diacritics/non-latin display problems

I haven't tried with non-latin characters, but I remember at Kohacon when you used the :0 to fix the issue you were having with Arabic characters.

For package install users, double-check your koha-conf.xml. Look for a block like the following:

<!-- BIBLIOGRAPHIC RECORDS -->
<server id="biblioserver"  listenref="biblioserver">
    <directory>/var/lib/koha/devone/biblios</directory>
    <config>/etc/koha/sites/devone/zebra-biblios-dom.cfg</config>
    <cql2rpn>/etc/koha/zebradb/pqf.properties</cql2rpn>

    <xi:include href="/etc/koha/marc21-retrieval-info-bib-dom.xml"
 xmlns:xi="http://www.w3.org/2001/XInclude">

The file that you need to update would be "marc21-retrieval-info-bib-dom.xml".

(My bad for saying "retrieval-info-bib-dom.xml". On a tarball dev install, it will probably be "retrieval-info-bib-dom.xml", but it's worth double-checking in any case.)

--

Another note for restarting Zebra...

I don't know the best way to restart Zebra on a non-package install.

However, sudo koha-restart-zebra [instancename] works a treat on a package install.

--

I just tried this again, and facets worked quite neatly.

IMPORTANT CONFIGURATION UPDATE:

So I was wrong before about the need to define "zebra::facet::field:type" style retrieval elements within retrievalinfo.

In fact, we just need to add one "retrieval" element to the bottom of the config before the closing </retrievalinfo> tag, and it is this:

<retrieval syntax="xml"/>

YAY!!!

This means that we don't have to individually define every single facet index. This means that facets could be highly flexible!

--

The solution became apparent after consulting the following documents:

http://lists.indexdata.dk/pipermail/zebralist/2009-June/002251.html

http://www.indexdata.com/yaz/doc/tools.retrieval.html

http://lists.indexdata.dk/pipermail/zebralist/2012-May/002536.html

Hdl stated "if I enable multiple retrieval facilities (MODS, XML, DC etc...), then zebra::meta and zebra::facet seems to be
disabled ([25]". 

So, I looked into the Zebralist a bit and noticed someone who "wanted" to block the Zebra Special Retrieval Elements (such as zebra::meta and zebra::facet). 

Adam from Indexdata said, "There is no to disable it as such. But you can use the retrieval facility of YAZ to specify valid element set names + formats."

So I looked into the retrieval facility a bit more...and noticed that the "name" attribute of the "retrieval" node was optional. If you used it, you could specify a single element. 

However, what happens if you remove the name completely? What if you make the "retrieval" element less restrictive?

It turns out that you re-gain access to those Zebra Special Retrieval Elements, which we were excluding because our configuration was too restrictive!

I'm placing my updated guide to the bottom of this bug. 

I have a few more notes before that though:

First, I should mention that using something like "zebra::facet::Title:0" won't work, unless you're indexing strings with a type of "0" in record.abs or the DOM equivalent (which I can't recall at the moment). At least, that's my best guess. I've seen Paul Poulain use something like "zebra::facet::Title:0" and it worked with Arabic, so I assume they were indexing with the type 0 in their indexing config.

Second, you have to use actual indexes for facets. You can't use the CCL short-hand like "ti". Also, capitals matter. In my previous examples, I said you could use "title", but I should've checked first as it's actually "Title". 

Third, chaining indexes and types (e.g. "zebra::facet::f1:t1,f2:t2,.." as noted in the Indexdata config) will now work, while it wouldn't when defining facets individually.

--

Updated Guide

1) Check your koha-conf.xml and check for where your <retrievalinfo> is defined. It should be based on an <xi:include>. If you're using packages, it'll probably be something like "marc21-retrieval-info-bib-dom.xml" (depending on your marc flavour and indexing filter).

2) Add the line "<retrieval syntax="xml"/>" at the bottom of the document before the closing </retrievalinfo> tag. 

3) Restart Zebra (using koha-reset-zebra with the packages or manually otherwise)

4) yaz-client [unix socket defined in koha-conf.xml]

base biblios
format xml
elements zebra::facet::Title:w,Author:w,itype:w
find test
show

--

You should be presented with a facet list for "Title", "Author", and "Itype". 

I'm currently not sure about the meaning of "coccur" (co-occurrence?) but "occur" is the number of times the term shows up in the search results.

--

Cool, eh?

Latest trial...

Facets seem to work great with my package install but not with my tarball dev install.

PACKAGE INSTALL:
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="w" index="Title">
    <term coccur="12" occur="7">of</term>
    <term coccur="10" occur="6">australia</term>
    <term coccur="8" occur="6">tax</term>
    <term coccur="9" occur="6">test</term>
    <term coccur="5" occur="5">institute</term>
    <term coccur="8" occur="4">and</term>
    <term coccur="8" occur="4">the</term>
    <term coccur="5" occur="3">convention</term>
    <term coccur="4" occur="3">taxation</term>
    <term coccur="4" occur="2">an</term>
    <term coccur="4" occur="2">fiscal</term>
    <term coccur="4" occur="2">illusion</term>
    <term coccur="4" occur="2">in</term>
    <term coccur="3" occur="2">melbourne</term>
    <term coccur="5" occur="2">papers</term>
    <term coccur="3" occur="2">seminar</term>
    <term coccur="4" occur="2">to</term>
    <term coccur="2" occur="1">1</term>
    <term coccur="2" occur="1">a</term>
    <term coccur="4" occur="1">dominion</term>
  </facet>
  <facet type="w" index="Author">
    <term coccur="5" occur="5">110</term>
    <term coccur="7" occur="4">institute</term>
    <term coccur="9" occur="4">of</term>
    <term coccur="6" occur="3">andrew</term>
    <term coccur="5" occur="3">australia</term>
    <term coccur="6" occur="3">e</term>
    <term coccur="6" occur="3">research</term>
    <term coccur="5" occur="3">taxation</term>
    <term coccur="6" occur="3">university</term>
    <term coccur="4" occur="2">australian</term>
    <term coccur="4" occur="2">brian</term>
    <term coccur="4" occur="2">c</term>
    <term coccur="4" occur="2">centre</term>
    <term coccur="4" occur="2">dollery</term>
    <term coccur="4" occur="2">federalism</term>
    <term coccur="4" occur="2">national</term>
    <term coccur="4" occur="2">r</term>
    <term coccur="4" occur="2">worthington</term>
    <term coccur="2" occur="1">g</term>
    <term coccur="2" occur="1">j</term>
  </facet>
  <facet type="w" index="itype">
    <term coccur="115" occur="9">book</term>
    <term coccur="5" occur="4">text</term>
  </facet>
</record>

TARBALL DEV INSTALL:
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="w" index="Title">
    <term coccur="14" occur="7">+I7</term>
    <term coccur="12" occur="6">+I7</term>
    <term coccur="7" occur="5">+I7</term>
    <term coccur="10" occur="5">+I7</term>
    <term coccur="8" occur="5">+I7</term>
    <term coccur="8" occur="4">+I7</term>
    <term coccur="8" occur="4">+I7</term>
    <term coccur="5" occur="3">+I7</term>
    <term coccur="6" occur="3">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="3" occur="2">+I7</term>
    <term coccur="5" occur="2">+I7</term>
    <term coccur="3" occur="2">+I7</term>
    <term coccur="4" occur="2">+I7</term>
    <term coccur="4" occur="1">+I7</term>
  </facet>
</record>

--

Will have to look into possible causes for this...

Also worthwhile on the Zebra front...

http://www.indexdata.com/yaz/doc/server.vhosts.html

Alas, the problem appears to be with ICU indexing. 

Wajasu provided some links which makes it seem like there might not be a way to get Zebra facets to work when using ICU indexing...

http://lists.indexdata.dk/pipermail/zebralist/2008-January/001859.html
http://lists.indexdata.dk/pipermail/zebralist/2009-April/002190.html

Yet, I swear that Paul Poulain was using ICU indexing and was able to make use of the :0 index type to get around this very problem.

I've tried adding Title:0 to record.abs and biblio-koha-indexdefs.xml, but to no avail. 

Happy to hear from Paul or Jared if they have any ideas on this one...

I'd like to add that the default behaviour for facets is to limit them to 20...
The indexdata folks explained to me that a higher count can be hardcoded in the configuration file. For instance:

 <retrieval syntax="xml" name="zebra::facet::subject:p:100"/>

will set the limit to 100.

It is worth mentioning that Zebra tokenizes data for indexing purposes, and also lower cases it. Thus, to have facets work as we use them now (they show the exact data that is on the record) then :0 should be used.

With :p :

<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="p" index="subject">
    <term coccur="1" occur="49">sang</term>
    <term coccur="1" occur="45">tegneserier amerikanske 25458100</term>
    <term coccur="1" occur="42">02</term>
    <term coccur="1" occur="42">klaver</term>
    <term coccur="1" occur="28">spillefilmer amerikanske 28231100</term>
    <term coccur="1" occur="17">filmer komedie 2040551600</term>
    <term coccur="1" occur="16">sang klaver 02</term>
    <term coccur="1" occur="12">filmer romantisk 2040551900</term>
    <term coccur="1" occur="5">tidsskrifter norske 13171900</term>
    <term coccur="1" occur="3">barnehager tidsskrifter 372 2105 15283900</term>
    <term coccur="1" occur="3">basketball 796 323 d5 10329200</term>
    <term coccur="1" occur="3">litteratur historie og kritikk 809 d5 10045100</term>
    <term coccur="1" occur="3">malerkunst historie generelt 759 d5 750 z 24632900</term>
    <term coccur="1" occur="3">sporreboker 793 73 d5 12683600</term>
    <term coccur="1" occur="3">sprak 400 d5 2041875500</term>
    <term coccur="1" occur="3">tegneserier science fiction bs 26382800</term>
  </facet>
</record>

So we should define facets like this:

 <retrieval syntax="xml" name="zebra::facet::subject:0:100"/>

The element set in the call should be the same and will get this results:

Z> elem zebra::facet::subject:0:100
Z> s
Sent presentRequest (1+1).
Records: 1
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="0" index="subject">
    <term coccur="1" occur="49">Sang</term>
    <term coccur="1" occur="45">Tegneserier, Amerikanske 25458100</term>
    <term coccur="1" occur="42">02</term>
    <term coccur="1" occur="42">Klaver</term>
    <term coccur="1" occur="28">Spillefilmer, Amerikanske 28231100</term>
    <term coccur="1" occur="17">Filmer Komedie 2040551600</term>
    <term coccur="1" occur="16">Sang Klaver 02</term>
    <term coccur="1" occur="12">Filmer Romantisk 2040551900</term>
    <term coccur="1" occur="5">Tidsskrifter, Norske 13171900</term>
    <term coccur="1" occur="3">Barnehager Tidsskrifter 372.2105 15283900</term>
    <term coccur="1" occur="3">Basketball 796.323 d5 10329200</term>
    <term coccur="1" occur="3">Litteratur Historie og kritikk 809 d5 10045100</term>
    <term coccur="1" occur="3">Malerkunst Historie Generelt 759 d5 750 z 24632900</term>
    <term coccur="1" occur="3">Språk 400 d5 2041875500</term>
    <term coccur="1" occur="3">Spørrebøker 793.73 d5 12683600</term>
    <term coccur="1" occur="3">Tegneserier Science fiction BS 26382800</term>
  </facet>
</record>


---
And I'm sure we'll be getting better results in the encoding front, because Zebra should be returning the exact data we sent for indexing.

(In reply to David Cook from comment #20)
> Latest trial...
> 
> Facets seem to work great with my package install but not with my tarball
> dev install.
...
>     <term coccur="4" occur="2">+I7</term>
>     <term coccur="4" occur="1">+I7</term>
>   </facet>
> </record>
> 
> --
> 
> Will have to look into possible causes for this...

David, the problem with your setup is that you inadvertedly set ICU for the :0 index which you shouldn't.

Just set:

icuchain words-icu.xml

for 'index w' and 'index p'.

We are using the :0 index for display purposes, so no comparisson or order definition is in place.

I think it is time to start coding :-D

(In reply to Tomás Cohen Arazi from comment #24)
> (In reply to David Cook from comment #20)
> > Latest trial...
> > 
> > Facets seem to work great with my package install but not with my tarball
> > dev install.
> ...
> >     <term coccur="4" occur="2">+I7</term>
> >     <term coccur="4" occur="1">+I7</term>
> >   </facet>
> > </record>
> > 
> > --
> > 
> > Will have to look into possible causes for this...
> 
> David, the problem with your setup is that you inadvertedly set ICU for the
> :0 index which you shouldn't.
> 
> Just set:
> 
> icuchain words-icu.xml
> 
> for 'index w' and 'index p'.
> 
> We are using the :0 index for display purposes, so no comparisson or order
> definition is in place.
> 
> I think it is time to start coding :-D

Thanks for all that info, Tomas! You're the best!

However, I don't believe that I have ICU set for the :0 index. I only have icuchain words-icu.xml set up for 'index w' and 'index p'. Here's what I have in my default.idx for "index 0":

# Null map index (no mapping at all)
# Used if structure=key (@attr 4=3)
index 0
completeness 0
position 1
charmap @

Here's the diagnostic message I get when I try to use "index 0":

Records: 1
Diagnostic message(s) from database:
    [25] Specified element set name not valid for specified database -- v2 addinfo ''

I get the same message when I try with my non-ICU system :/.

(In reply to David Cook from comment #25)
> (In reply to Tomás Cohen Arazi from comment #24)
> > (In reply to David Cook from comment #20)
> > > Latest trial...
> > > 
> > > Facets seem to work great with my package install but not with my tarball
> > > dev install.
> > ...
> > >     <term coccur="4" occur="2">+I7</term>
> > >     <term coccur="4" occur="1">+I7</term>
> > >   </facet>
> > > </record>
> > > 
> > > --
> > > 
> > > Will have to look into possible causes for this...
> > 
> > David, the problem with your setup is that you inadvertedly set ICU for the
> > :0 index which you shouldn't.
> > 
> > Just set:
> > 
> > icuchain words-icu.xml
> > 
> > for 'index w' and 'index p'.
> > 
> > We are using the :0 index for display purposes, so no comparisson or order
> > definition is in place.
> > 
> > I think it is time to start coding :-D
> 
> Thanks for all that info, Tomas! You're the best!
> 
> However, I don't believe that I have ICU set for the :0 index. I only have
> icuchain words-icu.xml set up for 'index w' and 'index p'. Here's what I
> have in my default.idx for "index 0":
> 
> # Null map index (no mapping at all)
> # Used if structure=key (@attr 4=3)
> index 0
> completeness 0
> position 1
> charmap @
> 
> Here's the diagnostic message I get when I try to use "index 0":
> 
> Records: 1
> Diagnostic message(s) from database:
>     [25] Specified element set name not valid for specified database -- v2
> addinfo ''
> 
> I get the same message when I try with my non-ICU system :/.

What I did was: 

-added some :0 use in zebradb/retrieval-info-bib-dom.xml like:

<retrieval syntax="xml" name="zebra::facet::subject:0:100"/>

- Edited marc_defs/marc21/biblios/biblio-koha-indexdefs.xml and added an occurence of

<target_index>Subject:0</target_index>
after every
<target_index>Subject:p</target_index>

this could be narrowed down later (we might not need facets for all thsoe fields), but I wanted to make sure :0 contained the same as :p for subject, for testing purposes.

- Run 
xsltproc ~/koha-dev/etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl biblio-koha-indexdefs.xml > biblio-zebra-indexdefs.xsl

to regenerate the xsl file,

- Then reindex :-(

And then search using:
 Z> format xml
 Z> elem zebra::facet::subject:0:100

I also had zebra::facet::subject:p defined to compare the results over the same search.

Regards
To+

(In reply to Tomás Cohen Arazi from comment #26)
> (In reply to David Cook from comment #25)
> > (In reply to Tomás Cohen Arazi from comment #24)
> > > (In reply to David Cook from comment #20)
> > > > Latest trial...
> > > > 
> > > > Facets seem to work great with my package install but not with my tarball
> > > > dev install.
> > > ...
> > > >     <term coccur="4" occur="2">+I7</term>
> > > >     <term coccur="4" occur="1">+I7</term>
> > > >   </facet>
> > > > </record>
> > > > 
> > > > --
> > > > 
> > > > Will have to look into possible causes for this...
> > > 
> > > David, the problem with your setup is that you inadvertedly set ICU for the
> > > :0 index which you shouldn't.
> > > 
> > > Just set:
> > > 
> > > icuchain words-icu.xml
> > > 
> > > for 'index w' and 'index p'.
> > > 
> > > We are using the :0 index for display purposes, so no comparisson or order
> > > definition is in place.
> > > 
> > > I think it is time to start coding :-D
> > 
> > Thanks for all that info, Tomas! You're the best!
> > 
> > However, I don't believe that I have ICU set for the :0 index. I only have
> > icuchain words-icu.xml set up for 'index w' and 'index p'. Here's what I
> > have in my default.idx for "index 0":
> > 
> > # Null map index (no mapping at all)
> > # Used if structure=key (@attr 4=3)
> > index 0
> > completeness 0
> > position 1
> > charmap @
> > 
> > Here's the diagnostic message I get when I try to use "index 0":
> > 
> > Records: 1
> > Diagnostic message(s) from database:
> >     [25] Specified element set name not valid for specified database -- v2
> > addinfo ''
> > 
> > I get the same message when I try with my non-ICU system :/.
> 
> What I did was: 
> 
> -added some :0 use in zebradb/retrieval-info-bib-dom.xml like:
> 
> <retrieval syntax="xml" name="zebra::facet::subject:0:100"/>
> 
> - Edited marc_defs/marc21/biblios/biblio-koha-indexdefs.xml and added an
> occurence of
> 
> <target_index>Subject:0</target_index>
> after every
> <target_index>Subject:p</target_index>
> 
> this could be narrowed down later (we might not need facets for all thsoe
> fields), but I wanted to make sure :0 contained the same as :p for subject,
> for testing purposes.
> 
> - Run 
> xsltproc ~/koha-dev/etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl
> biblio-koha-indexdefs.xml > biblio-zebra-indexdefs.xsl
> 
> to regenerate the xsl file,
> 
> - Then reindex :-(
> 
> And then search using:
>  Z> format xml
>  Z> elem zebra::facet::subject:0:100
> 
> I also had zebra::facet::subject:p defined to compare the results over the
> same search.
> 
> Regards
> To+

I believe that I've done the same thing and am getting that error instead :(.

I'll try again though just to make sure :).

Thanks again, Tomas!

I'm accustomed to using the GRS-1 filter, so I forgot about transforming the XML into XSL for DOM indexing. Problem solved! Yay!

That's great information about appending "100" to the element to get a longer facet list as well! I don't know if we would necessarily want more than 20 results per facet, but it's nice to have the configurability to change it.

However, I would recommend "not" hardcoding "zebra::facet::subject:0:100" into zebradb/retrieval-info-bib-dom.xml.

Try just using "<retrieval syntax="xml"/>" in "zebradb/retrieval-info-bib-dom.xml". It's a much more flexible configuration.

It lets you change "100" to "50" or whatever just by using the "elements" command in YAZ. You can also chain facets in elements so that you can run 1 query but get the facets for 5 different elements. (e.g.  elements zebra::facet::title:0:100,subject:0:25,itype:0:30)

[Sidenote: I see my earlier comments about index case sensitivity aren't correct, at least when using the <retrieval syntax="xml"/> config.]

Yes, I think we've found a solution! Time to start writing code!

The only obstacle I see now is determining which indexes need the "index 0". I suppose all the fields currently used for facets...plus pubdate.

This will also be an obstacle in terms of having configurable facets. Using "<retrieval syntax="xml"/>" in the Zebra config would let you spin a query with any facets we want, but relying on that "index 0" will determine which results return usable data...

Still...very exciting!

I imagine it would be useful to write something a bit abstract for getting facets, so that we can make it easier to plugin other indexers like Solr and ElasticSearch...

PS Is that your way of saying that you're working on this project, Tomas? ;)

(In reply to David Cook from comment #29)
> PS Is that your way of saying that you're working on this project, Tomas? ;)

For now, he's honeymooning ;-) (should be back around april, 15th)

> I imagine it would be useful to write something a bit abstract for getting 
> facets, so that we can make it easier to plugin other indexers like Solr and 
> ElasticSearch...
Of course !

Idea: To determine which indexes get the additional index:0, we could:
provide a UI(similiar to msaby's suggestion that allows us to select an index by grabbing the <target_index> nodes from biblios-koha-indexdefs.xml, and search replacing
<target_index>index:p</target_index> with
<target_index>index:p</target_index><target_index>index:0</target_index>
(control_field, data_field, sub_field savy UI is possible)

That would at least let the selection be flexible according to the current configuration, though that file must be located for "koha packages"(TBD).

We could roll out with things initially matching today's facets, and let them select predefined profiles at some point(future feature).

The facet selection UI should highlight configured(non-disabled) facets that are may not exist anymore in the biblios-koha-indexdefs.xml, to aid debugging when that file is subsequently modified (admin or future koha update).

When querying the chosen facets, it might be worth querying the whole list at once and if it fails, querying each sequentially such that if one fails, we just don't supply that label to the UI and log the failing facet, refering them to the facet config UI to reconcile.  But things continue to run.

It might be worth looking into generating retreival-info-bib-grs1-faceted.xml and biblios-koha-indexdefs-faceted.xml and launching xsltproc to generate biblio-zebra-indexdefs-faceted.xsl, without touching the original configs. Facets could be disabled to enable non-faceted searching until the issue is resolved.

We might also employ xi:include and fallback in koha-conf.xml to use the faceted file(s) which could be generated/removed by the facet config UI. 
<xi:include href="/home/koha/koha-dev/etc/zebradb/retrieval-info-bib-dom.xml"
      xmlns:xi="http://www.w3.org/2001/XInclude">
<xi:fallback>
Thus leaving koha-conf.xml flexible.

Mapping tags to new indexes that go in biblios-koha-indexdefs.xml can possibly be a separate effort.

Of course this is all very zebra focused.  But the idea is to restrict free form editing (for security) and allow newly introduced indexes to roll out without concern of breaking updates.

(In reply to wajasu from comment #31)
> Idea: To determine which indexes get the additional index:0, we could:
> provide a UI(similiar to msaby's suggestion that allows us to select an
> index by grabbing the <target_index> nodes from biblios-koha-indexdefs.xml,
> and search replacing
> <target_index>index:p</target_index> with
> <target_index>index:p</target_index><target_index>index:0</target_index>
> (control_field, data_field, sub_field savy UI is possible)
> 

I think that this is a bad idea.

1) I'm not 100% sure, but I think that many installs of Koha run multiple sites using one set of Zebra configs. At least, that's how it appears to work on package installs.

I suppose you could have this be an additive process that checks if "index:0" exists and if not, you can add it, when you want to use that index. That wouldn't screw with other people's indexing too much.  

But I'm not sure allowing the web server to write files in /etc/koha/ is the best idea.

2) Increasingly, I'm becoming less sure that librarians themselves actually need web access to this type of configuration. I think it is a good idea to have configurable facets, but perhaps having a separate file for this would be best. A sysadmin could then edit it as needed.

> When querying the chosen facets, it might be worth querying the whole list
> at once and if it fails, querying each sequentially such that if one fails,
> we just don't supply that label to the UI and log the failing facet,
> refering them to the facet config UI to reconcile.  But things continue to
> run.
> 

I don't follow you 100% here. I originally thought that you meant we could query zebra for our chosen facets, and if that failed, that we could choose do each individually. That would probably make sense so that facets didn't entirely bust for one badly configured facet.

> It might be worth looking into generating
> retreival-info-bib-grs1-faceted.xml and biblios-koha-indexdefs-faceted.xml
> and launching xsltproc to generate biblio-zebra-indexdefs-faceted.xsl,
> without touching the original configs. Facets could be disabled to enable
> non-faceted searching until the issue is resolved.
> 
> We might also employ xi:include and fallback in koha-conf.xml to use the
> faceted file(s) which could be generated/removed by the facet config UI. 
> <xi:include href="/home/koha/koha-dev/etc/zebradb/retrieval-info-bib-dom.xml"
>       xmlns:xi="http://www.w3.org/2001/XInclude">
> <xi:fallback>
> Thus leaving koha-conf.xml flexible.
> 
> Mapping tags to new indexes that go in biblios-koha-indexdefs.xml can
> possibly be a separate effort.
> 

I really think this is getting overly complex. This would likely take quite a bit of code just to add a web UI that most librarians will probably (and should probably) never touch.

--

I think the best idea is probably to add "index:0" for our current range of facets, and add it to a few other desired facets (such as pubdate). 

We put the list of indexes in a file, make sure that those indexes are sent off to Zebra when doing a facet query, and that's that.

If a web UI is really necessary, perhaps reading the list of indexes out of that file would make sense.

Perhaps the file would be unnecessary and we could just read out the "index:0" facets from the Zebra config, but I would be hesitant about adding them in...

...especially since adding "index:0" and sending a facet query referencing that new index:0 before doing a full re-index will break the facets (if sent as one combined query and not lots of separate ones).

--

I'm not saying I have a perfect idea in mind, but I think... we shouldn't try to be too fancy or convoluted.

Simple and elegant. We provide a set of default facets, and if folks want to add their own, we leave them with good enough instructions to do so.

I really don't want to be dynamically messing with files that should be static.

(In reply to David Cook from comment #32)
> (In reply to wajasu from comment #31)
> > Idea: To determine which indexes get the additional index:0, we could:
> > provide a UI(similiar to msaby's suggestion that allows us to select an
> > index by grabbing the <target_index> nodes from biblios-koha-indexdefs.xml,
> > and search replacing
> > <target_index>index:p</target_index> with
> > <target_index>index:p</target_index><target_index>index:0</target_index>
> > (control_field, data_field, sub_field savy UI is possible)
> > 
> 
> I think that this is a bad idea.
Agreed with most of your points. One of the reason being that it's impossible to have everything made by the librarian (any change require rebuilding zebra DB, plus some/many config file change. and yes, it's not a good idea to let the webserver write in /etc ...). So I prefer to have everything made by a sysadmin.

When we worked on Solr, we made indexes accessible by the web interface, but it's something shipped with Solr. That's not the case for Zebra.
> We put the list of indexes in a file, make sure that those indexes are sent
> off to Zebra when doing a facet query, and that's that.
Zebra is highly sensitive on how the config files are written. I'm not sure you can have :0 indexes in a separate file.
AFAIK, when you double declare a line, only the last one is taken into account.
So:
melm 245$a Title:w,Title:p
melm 245$a Title:0
would result in Title:0 being the only index. Which is not what we want.

> Simple and elegant. We provide a set of default facets, and if folks want to
> add their own, we leave them with good enough instructions to do so.
+1

OK, David and Paul.

I'll code a facet query in our current search for a set of expected facets, as we use today, plus pubdate, or others added with followup patches.

I'll query for all in one go.  If that fails, then might try individual queries as a fallback.  Maybe even keep the current code as a fallback.

If sysadmin updates the configs, then we get the enhanced behavior.

We can NYTprof and see impact.  If a discernable impact, we could add syspref "ZebraNativeFaceting" on/off.

Let me attempt to throw something simple together before i forget how all this zebra search stuff works.

(In reply to wajasu from comment #34)
> We can NYTprof and see impact.  If a discernable impact, we could add
> syspref "ZebraNativeFaceting" on/off.
mmm... Why would we want to have this syspref OFF ? Currently, the facets are wrong (at least incomplete), slow to calculate.
With this patch they will become more reliable & faster. So ...

If we implement this, that could be a requirement for upgrade (to check/update zebra configuration). For libraries using package, it's automatic. For others, they are supposed to know what they're doing, they're able to update the config.
This already happened (for example the bug 3087)

> Let me attempt to throw something simple together before i forget how all
> this zebra search stuff works.
+1 !

Is this still being worked on? Does this need sign off?

(In reply to M. Tompsett from comment #37)
> Is this still being worked on? Does this need sign off?

Looks like there is still lots of debugging code (even if it's commented out).

Personally, I would prefer to have the first step be to decompose the existing code into functions (ie reduce the spaghetti) so that it's easier to read and switch between different facet creation methods.

I'm interested in Zebra native facets, but sadly it's not on my "to do" list at the moment, so I won't be contributing anything (besides constructive criticism :P) in the foreseeable future.

Just a very quick look, I seem to have lost 'Holding libraries' as a facet.  We definitely need that for our consortium

(In reply to Nick Clemens from comment #49)
> Just a very quick look, I seem to have lost 'Holding libraries' as a facet. 
> We definitely need that for our consortium

Thanks for the feedback, Nick. Please send me a sample record. And also tell me this marc dialect the record is written in, and the one you chose on install.

Created attachment 31436 [details]
Sample record, items not faceted by holding library in search

Marc21 for record and for install

(In reply to Nick Clemens from comment #51)
> Created attachment 31436 [details]
> Sample record, items not faceted by holding library in search
> 
> Marc21 for record and for install

Nick, I've spotted the problem. Thanks for testing, will provide a new set of patches tomorrow.

(In reply to Nick Clemens from comment #51)
> Created attachment 31436 [details]
> Sample record, items not faceted by holding library in search
> 
> Marc21 for record and for install

Nick, I've fixed the problem on the latest patch set.

Unimarc testing:

For author and topic facets, spaces are replaced by "::*::".
The facet list looks like:
  Ganz::*::Axel
  Gardel::*::Louis
  Hall::*::John D 

The links are catalogue/search.pl?idx=kw&q=d&limit=au:Ganz%3A%3A*%3A%3AAxel

And if I follow it:
   No results match your search for 'kw,wrdl: d' with limit(s): 'au:Hall::*::John D'

And in the Koha logs:
[Wed Sep 17 10:34:04 2014] search.pl: WARNING: query problem with kw,wrdl=d and au=Hall=*=John D ZOOM error 10014 "CCL parsing error" (addinfo: "Embedded truncation not supported") from diag-set 'ZOOM' at /home/koha/src/C4/Search.pm line 380.

Putting this back to "Needs signoff" so that other people can test NORMARC and UNIMARC...

Tested with Tomas yesterday on a UNIMARC installation. All works fine :)

Hi Tomas,

I am seeing problem with this comparing the facets before and after appyling the patch. Testing on a fairly small database with 120 records there appear to be less facets appearing after applying the patch then before. Can you check? I will send you the records by email, search term was "de".

Created attachment 32352 [details] [review]
[PASSED QA] Bug 11232: Add new syntax for facets definition on koha-indexdefs-to-zebra.xsl

This patch changes koha-indexdefs-to-zebra.xsl to correctly process a new syntax
for defining facet indexes on the XML files.

It also changes the retrieval file to allow access to Zebra's internal data from
Zoom (i.e. access to zebra::facet:*).

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32353 [details] [review]
[PASSED QA] Bug 11232: MARC21 facet definition and updated XSL file for DOM

This patch adds the facets definitions to the biblio-koha-indexdefs.xml, based
on what is hardcoded on C4::Koha::getFacets().

The biblio-zebra-indexdefs.xsl file for MARC21 is generated using the usual:

xsltproc ...koha-indexdefs-to-zebra.xsl ...marc21/biblios/biblio-koha-indexdefs.xml > \
    ...marc21/biblios/biblio-zebra-indexdefs.xsl

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32354 [details] [review]
[PASSED QA] Bug 11232: UNIMARC facet definition and updated XSL file for DOM

This patch adds the facets definitions to the biblio-koha-indexdefs.xml, based
on what is hardcoded on C4::Koha::getFacets().

The biblio-zebra-indexdefs.xsl file for UNIMARC is generated using the usual:

xsltproc ...koha-indexdefs-to-zebra.xsl ...unimarc/biblios/biblio-koha-indexdefs.xml > \
    ...unimarc/biblios/biblio-zebra-indexdefs.xsl

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32355 [details] [review]
[PASSED QA] Bug 11232: NORMARC facet definition and updated XSL file for DOM

This patch adds the facets definitions to the biblio-koha-indexdefs.xml, based
on what is hardcoded on C4::Koha::getFacets().

The biblio-zebra-indexdefs.xsl file for NORMARC is generated using the usual:

xsltproc ...koha-indexdefs-to-zebra.xsl ...normarc/biblios/biblio-koha-indexdefs.xml > \
    ...normarc/biblios/biblio-zebra-indexdefs.xsl

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32356 [details] [review]
[PASSED QA] Bug 11232: retrieve facets from Zebra's indexes in DOM

This patch adds the following routines to C4::Search

- GetFacets
   This is a wrapper routine, that given a ZOOM::ResultSet, extracts
   the relevant facets. To do the job, it uses the internal functions:
   _get_facets_from_records and _get_facets_from_zebra. The choice is done
   on querying the indexing mode: grs1 will use records, and dom zebra's facets.
- _get_facets_from_records
   Just refactoring the already existent main loop from getRecords into a function.
- _get_facets_from_zebra
   Given a result set, loop through all defined facets in C4::Koha::getFacets
   and call _get_facet_from_result_set for each, to build the facets
   information. It retrieves the facets from Zebra's facets.
- _get_facet_from_result_set
   Given a result set and a facet index name, retrieve the facets
   for the given index, and build the result for rendering.

To test this preliminay work:
- Apply the patches, install on DOM, using MARC21, NORMARC and UNIMARC.
- Reindex some DB with lots of records.
- Check facets work.

Note: UNIMARC is the only dialect that has more than one subfield (concatenated)
for facets values, so it is better to test on uNIMARC. The approach leaves room
for re-thinking facets in MARC21/NORMARC, but it is outside of the scope of the bug
(e.g. we could have more author facets)

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32357 [details] [review]
[PASSED QA] Bug 11232: (followup) remove unnecesary namespace definition from all XML elements

The previous patches for facet extraction from Zebra indexes set a default
namespace on the following files:

etc/zebradb/marc_defs/marc21/biblios/biblio-koha-indexdefs.xml
etc/zebradb/marc_defs/normarc/biblios/biblio-koha-indexdefs.xml
etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml

and hence the XML file index_subfields can be cleaned by removing the namespace.

To test:
- Apply this patch
- Run

$ for i in marc21 normarc unimarc
  do xsltproc etc/zebradb/xsl/koha-indexdefs-to-zebra.xsl \
              etc/zebradb/marc_defs/$i/biblios/biblio-koha-indexdefs.xml \
              > etc/zebradb/marc_defs/$i/biblios/biblio-zebra-indexdefs.xsl
  done

=> SUCCESS: no errors reported

- Run
$ git diff
=> SUCCESS: no differences on the xsl files

- Sign off :-D

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: David Cook <dcook@prosentient.com.au>

Seems to work with DOM and MARC21.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32358 [details] [review]
[PASSED QA] Bug 11232: (followup) Configuration variable for enabling Zebra facets

This patch adds a variable to koha-conf.xml controlling the use of Zebra facets.

Usage:
 - use_zebra_facets = 1 | 0

Zebra facets work only on DOM.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32359 [details] [review]
[PASSED QA] Bug 11232: (qa followup) Add missing fields/subfields to the item types faceta

The itype facet was missing 952$y for both MARC21 and NORMARC.
This patch adds that. And also modifies the zebra-biblios-dom.cfg file
(also the debian/ version) so facetNumRecs is set to 1000 for zebra.

It is the amount of records that are taken into account. The more record,
the more exact the facets for the result set. 1000 was chosen as it changed
the time to reindex 1000 records from 18s to 19s.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 32360 [details] [review]
[PASSED QA] Bug 11232: (qa followup) empty ID due to namespace mistake

Note: NORMARC is missing the id field.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
This patch makes t/db_dependent/Search.t pass again.
NORMARC is currently not tested.

I checked the results before and after applying the patch
and the facets are now looking the same as before.
Passes all tests and QA script.

Patch pushed to master.

Quick question. Is there a reason why we needed to add "facetNumRecs"? 

The speed already seemed pretty good. Even if it were a bit slower, I think I'd rather use all retrieved records than an arbitrary number of them :/. I suppose 1000 is pretty good, but if I click a facet, I want to know that I'm getting all the relevant records from a result set and not just the records that are from the first 1000 records.

Of course, perhaps I'm being a bit ridiculous. If a person is getting over 1000 results, their search isn't very exact. But... sometimes you want to do a broad search then narrow it down using multiple facets :/.

I suppose this is the kind of configuration that individual libraries could change on their own though.

Actually, I haven't looked at the code, but David's comment does resonate with me. If I can retrieve all the facets, why would I limit them? Is there a significant speed difference at a particular number of facets? If I have to wait 20 seconds for all the facets, is that really so bad for a large search result?

(In reply to David Cook from comment #96)
> Quick question. Is there a reason why we needed to add "facetNumRecs"? 

Zebra defaults to 10 [1] (which is way too low) and I found no way to define a wildcard.

Any ideas for improving this are welcome.

[1] Line 902 of http://www.indexdata.com/zebra/dox/html/retrieve_8c_source.html