Bug 13665 - Retrieve facets from zebra is slow
Summary: Retrieve facets from zebra is slow
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on: 11232
Blocks:
  Show dependency treegraph
 
Reported: 2015-02-04 11:14 UTC by Jonathan Druart
Modified: 2022-05-25 03:03 UTC (History)
11 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2015-02-04 11:14:22 UTC
With a 1 million biblio records installation (MARC21 + DOM), a search is very slow if facets are retrieved from zebra.

Debugging, I found where the processing time is spent: 
In C4::Search::_get_facet_from_result_set, the line
    my $facet = $rs->record( 0 )->raw;
can spent up to 3 seconds!
Actually it's the ->record call, not the ->raw.
In ZOOM->record, the time is spent in
  my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which);

I stopped the track game at this point.

So for instance, with a facet holdingbranch, (FacetMaxCount set to 20) the element zebra::facet::su-to:0:20" is set, $rs->size returns 962076, and the total execution time for _get_facet_from_result_set (only for this facet!) is 3.2sec.

If I set FacetMaxCount to 1, I got: 0.3 sec, for 10: 1.67. So quite linear.

Lets compare with yaz-client:
Z> open unix:/home/koha/var/run/zebradb/bibliosocket
Connecting...OK.
Sent initrequest.
Connection accepted by v3 target.
ID     : 81
Name   : Zebra Information Server/GFS/YAZ
Version: 4.2.30 98864b44c654645bc16b2c54f822dc2e45a93031
Options: search present delSet triggerResourceCtrl scan sort extendedServices namedResultSets
Elapsed: 0.009150
Z> base biblios
Z> format xml
Z> elem zebra::facet::holdingbranch:0:20
Z> f d
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 809470, setno 1
SearchResult-1: term=d cnt=809470
records returned: 0
Elapsed: 0.078905
Z> s 1+1
Sent presentRequest (1+1).
Records: 1
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="0" index="holdingbranch">
    <term coccur="941" occur="103144">br1</term>
    [...]
  </facet>
</record>
nextResultSetPosition = 2
Elapsed: 1.393694

To compare with the old facet method, I calculated the time spend in 
C4::Search::GetFacets (for 20 facets):
From zebra (new): 9.3sec
From Records (old): 0.16sec (with maxRecordsForFacets = 20)
From Records (old): 1.85sec (with maxRecordsForFacets = 100)
From Records (old): 15.1sec (with maxRecordsForFacets = 1000)

Note that the machine is a test machine (VE) and is quite slow.
Firebug tells me that with the new method (20 facets), the total load page is ~30sec, with the old method (20 facets calculated from 20 records) is 8-9sec
Comment 1 Katrin Fischer 2015-05-15 06:12:19 UTC
I can confirm that the new facets are too slow to be used for a big library currently. We tried with a database of rougly 1.000.000 as well and saw even slower response times than in Jonathan's test below. Searches took up to 40 seconds. With the old facets the time came down dramatically to about 4-6 seconds for the same search and is generally much faster.
Comment 2 Martin Renvoize 2015-10-26 10:29:48 UTC
Have we reported this upstream at all?
Comment 3 Tomás Cohen Arazi 2015-10-26 11:47:05 UTC
(In reply to Martin Renvoize from comment #2)
> Have we reported this upstream at all?

I don't think so. The main problem is that we are retrieving each facet at a time. This gets really better when we retrieve all facets at once. The problem is that if you ask for a setSpec containing all facets, and one is empty, then the result is empty. I've tried to patch zebra for that, but didn't finish it. The IndexData guys didn't seem to accept such a solution anyway.

The best approach IMO is to make rebuild_zebra.pl create phantom entries for all facets. But didn't have the time and memory to do it.

PS: When we first tried zebra facets in Reno, they were really better speed-wise. I think something changed between 2.0.47 and 2.0.60 that made them worse.
Comment 4 Hugo Agud 2016-11-07 14:19:42 UTC
We have faced a similar issue about zebra facet performance, but we have identified another variables that affect the zebra performance



We were working with koha 16.05.05 debian 8 plack enabled, 4 cpu , 8 RAM + 100 GB on SSD.

From fresh install, load a database of 225.000 bib records + 300.000 authority + 260.000 item record.

When searching we found that stopwords increase the time response up to 12 times...

Koha Search: Felipe Rey España (15 secons)
Koha Search: Felipe Rey de España (110 seconds)

If we turn on queryfuzz and query stemming the speed increase sharply

Koha Search: Felipe Rey España (5 secons)
Koha Search: Felipe Rey de España (20 seconds)

If we turn on zebra facets, it becomes very fast
Koha Search: Felipe Rey España (1,5 secons)
Koha Search: Felipe Rey de España (2,5 seconds)


If we turn off zebra facet and keep query steemming and qeryfuzzy on it has the same performance behaviour that turning off facets

Koha Search: Felipe Rey España (5 secons)
Koha Search: Felipe Rey de España (20 seconds)

then here we have a mix of issues with stopwords, querystemming, queryfuzzy and zebra facets.

Hope it helps
Comment 5 David Cook 2016-11-24 23:02:15 UTC
(In reply to Hugo Agud from comment #4)
> Hope it helps

That's really interesting!
Comment 6 David Cook 2020-09-10 02:42:39 UTC
Jonathan and Katrin:

Is the issue the large database or the large resultset? Looking at Jonathan's example, it seems like the issue is the large resultset. 

It makes sense to me that a large result set would take a long time to extract facets from (depending on the efficiency of that facet extraction code of course).

Without Zebra facets, it looks like we're only naively extracting facets from 20 records (using "maxRecordsForFacets" which defaults to 20), so it's going to be faster for a large result set but much worse quality.
Comment 7 David Cook 2020-09-10 02:48:44 UTC
I think the most robust solution would be to asynchronously load the facets into the Koha Web UI after Koha has returned the search page to the user's browser. (The search results should be asynchronously loaded too but that's for another day.)

If I recall correctly, Net::Z3950::ZOOM has the ability to do a certain amount of non-blocking I/O, so in theory you should also be able to do the facet fetching in parallel. I'd have to look into that further...

My TODO list is already overfilled, but it would be interesting work to do. Actually, that could make for an interesting plugin. If we turned off facet fetching in the Perl search script and let a plugin take over. That might be a good way to do a proof-of-concept at the very least. I'll add it to my list in any case... lol.
Comment 8 David Cook 2022-05-23 23:20:04 UTC
(In reply to Martin Renvoize from comment #2)
> Have we reported this upstream at all?

I reported something in this area yesterday: https://github.com/indexdata/idzebra/issues/35

As Hugo notes, stop words seem to cause big problems, although it doesn't seem tied to the result set but rather the initial records fetched and filtered. 

I had 2 nearly identical queries on a 1,000,000+ record database. The first search returned instantly and its 1st facet returned in 2 seconds. The second search (including "the") returned after 30 seconds and its 1st facet returned after 60 seconds. Both had result sets of 3323-3325 records. Since "the" had matched over 1,000,000 records, it seemed clear that the facet calculation involved more than just the result set (although my initial skimming through the Zebra C code couldn't confirm that).

I'm going to see if I can compile Zebra and figure out what's going on there, so that I can either send a pull request or at least put Indexdata onto the root cause and rough solution. 

But I'm also not going to commit heaps of time to this. It's fresh in my mind, so now is a good time to do it, but we'll see...
Comment 9 Tomás Cohen Arazi 2022-05-24 11:02:39 UTC
David, whlie you're digging that hole... One of the problems for the facet retrieval step in Koha is that we look through the defined facets instead of fetching them all at once.

The reason behind that is that if some facet is not populated at all (i.e. no record has field X which is mapped to the facet) then the resultset is always empty in Zebra. I filed an issue about that (a lot of time ago) but the answer was negative: that's the expected behavior.

So, would it be possible to add some ghost record (e.g. ID = -1) with all the fields (one space or special character?) so we convert it into one query instead of many?
Comment 10 David Cook 2022-05-25 03:03:08 UTC
(In reply to Tomás Cohen Arazi from comment #9)
> David, whlie you're digging that hole... One of the problems for the facet
> retrieval step in Koha is that we look through the defined facets instead of
> fetching them all at once.
> 
> The reason behind that is that if some facet is not populated at all (i.e.
> no record has field X which is mapped to the facet) then the resultset is
> always empty in Zebra. I filed an issue about that (a lot of time ago) but
> the answer was negative: that's the expected behavior.

Yeah, I've been thinking a little bit about that as well, although I didn't look at the code for that. I think that's probably handled within the facet_fetch() function. 

I do wonder if they'd be interested in adding the "14" attribute to the faceting as well.

(https://software.indexdata.com/zebra/doc/querymodel-zebra.html#querymodel-zebra-attr-search)

Adam is out of the office this week, but I'm hoping that he'll help me to understand the faceting code. 

> So, would it be possible to add some ghost record (e.g. ID = -1) with all
> the fields (one space or special character?) so we convert it into one query
> instead of many?

I think that might be an option. 

When I run zebrasrv in this directory https://github.com/indexdata/idzebra/tree/master/examples/marcxml, I noticed that there were records that weren't indexed and records that seemed to have empty indexes. 

In theory, we could add a record to Zebra like you describe, but I haven't tested it in practice. 

--

I wouldn't mind looking at trying to fetch the facets asynchronously although that's less pressing for me atm...