Bug 13665 - Retrieve facets from zebra is slow
Summary: Retrieve facets from zebra is slow
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact: Testopia
Depends on: 11232
  Show dependency treegraph
Reported: 2015-02-04 11:14 UTC by Jonathan Druart
Modified: 2020-09-10 02:48 UTC (History)
11 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:


Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2015-02-04 11:14:22 UTC
With a 1 million biblio records installation (MARC21 + DOM), a search is very slow if facets are retrieved from zebra.

Debugging, I found where the processing time is spent: 
In C4::Search::_get_facet_from_result_set, the line
    my $facet = $rs->record( 0 )->raw;
can spent up to 3 seconds!
Actually it's the ->record call, not the ->raw.
In ZOOM->record, the time is spent in
  my $_rec = Net::Z3950::ZOOM::resultset_record($this->_rs(), $which);

I stopped the track game at this point.

So for instance, with a facet holdingbranch, (FacetMaxCount set to 20) the element zebra::facet::su-to:0:20" is set, $rs->size returns 962076, and the total execution time for _get_facet_from_result_set (only for this facet!) is 3.2sec.

If I set FacetMaxCount to 1, I got: 0.3 sec, for 10: 1.67. So quite linear.

Lets compare with yaz-client:
Z> open unix:/home/koha/var/run/zebradb/bibliosocket
Sent initrequest.
Connection accepted by v3 target.
ID     : 81
Name   : Zebra Information Server/GFS/YAZ
Version: 4.2.30 98864b44c654645bc16b2c54f822dc2e45a93031
Options: search present delSet triggerResourceCtrl scan sort extendedServices namedResultSets
Elapsed: 0.009150
Z> base biblios
Z> format xml
Z> elem zebra::facet::holdingbranch:0:20
Z> f d
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 809470, setno 1
SearchResult-1: term=d cnt=809470
records returned: 0
Elapsed: 0.078905
Z> s 1+1
Sent presentRequest (1+1).
Records: 1
Record type: XML
<record xmlns="http://www.indexdata.com/zebra/">
  <facet type="0" index="holdingbranch">
    <term coccur="941" occur="103144">br1</term>
nextResultSetPosition = 2
Elapsed: 1.393694

To compare with the old facet method, I calculated the time spend in 
C4::Search::GetFacets (for 20 facets):
From zebra (new): 9.3sec
From Records (old): 0.16sec (with maxRecordsForFacets = 20)
From Records (old): 1.85sec (with maxRecordsForFacets = 100)
From Records (old): 15.1sec (with maxRecordsForFacets = 1000)

Note that the machine is a test machine (VE) and is quite slow.
Firebug tells me that with the new method (20 facets), the total load page is ~30sec, with the old method (20 facets calculated from 20 records) is 8-9sec
Comment 1 Katrin Fischer 2015-05-15 06:12:19 UTC
I can confirm that the new facets are too slow to be used for a big library currently. We tried with a database of rougly 1.000.000 as well and saw even slower response times than in Jonathan's test below. Searches took up to 40 seconds. With the old facets the time came down dramatically to about 4-6 seconds for the same search and is generally much faster.
Comment 2 Martin Renvoize 2015-10-26 10:29:48 UTC
Have we reported this upstream at all?
Comment 3 Tomás Cohen Arazi 2015-10-26 11:47:05 UTC
(In reply to Martin Renvoize from comment #2)
> Have we reported this upstream at all?

I don't think so. The main problem is that we are retrieving each facet at a time. This gets really better when we retrieve all facets at once. The problem is that if you ask for a setSpec containing all facets, and one is empty, then the result is empty. I've tried to patch zebra for that, but didn't finish it. The IndexData guys didn't seem to accept such a solution anyway.

The best approach IMO is to make rebuild_zebra.pl create phantom entries for all facets. But didn't have the time and memory to do it.

PS: When we first tried zebra facets in Reno, they were really better speed-wise. I think something changed between 2.0.47 and 2.0.60 that made them worse.
Comment 4 Hugo Agud 2016-11-07 14:19:42 UTC
We have faced a similar issue about zebra facet performance, but we have identified another variables that affect the zebra performance

We were working with koha 16.05.05 debian 8 plack enabled, 4 cpu , 8 RAM + 100 GB on SSD.

From fresh install, load a database of 225.000 bib records + 300.000 authority + 260.000 item record.

When searching we found that stopwords increase the time response up to 12 times...

Koha Search: Felipe Rey España (15 secons)
Koha Search: Felipe Rey de España (110 seconds)

If we turn on queryfuzz and query stemming the speed increase sharply

Koha Search: Felipe Rey España (5 secons)
Koha Search: Felipe Rey de España (20 seconds)

If we turn on zebra facets, it becomes very fast
Koha Search: Felipe Rey España (1,5 secons)
Koha Search: Felipe Rey de España (2,5 seconds)

If we turn off zebra facet and keep query steemming and qeryfuzzy on it has the same performance behaviour that turning off facets

Koha Search: Felipe Rey España (5 secons)
Koha Search: Felipe Rey de España (20 seconds)

then here we have a mix of issues with stopwords, querystemming, queryfuzzy and zebra facets.

Hope it helps
Comment 5 David Cook 2016-11-24 23:02:15 UTC
(In reply to Hugo Agud from comment #4)
> Hope it helps

That's really interesting!
Comment 6 David Cook 2020-09-10 02:42:39 UTC
Jonathan and Katrin:

Is the issue the large database or the large resultset? Looking at Jonathan's example, it seems like the issue is the large resultset. 

It makes sense to me that a large result set would take a long time to extract facets from (depending on the efficiency of that facet extraction code of course).

Without Zebra facets, it looks like we're only naively extracting facets from 20 records (using "maxRecordsForFacets" which defaults to 20), so it's going to be faster for a large result set but much worse quality.
Comment 7 David Cook 2020-09-10 02:48:44 UTC
I think the most robust solution would be to asynchronously load the facets into the Koha Web UI after Koha has returned the search page to the user's browser. (The search results should be asynchronously loaded too but that's for another day.)

If I recall correctly, Net::Z3950::ZOOM has the ability to do a certain amount of non-blocking I/O, so in theory you should also be able to do the facet fetching in parallel. I'd have to look into that further...

My TODO list is already overfilled, but it would be interesting work to do. Actually, that could make for an interesting plugin. If we turned off facet fetching in the Perl search script and let a plugin take over. That might be a good way to do a proof-of-concept at the very least. I'll add it to my list in any case... lol.