Summary: | Retrieve facets from zebra is slow | ||
---|---|---|---|
Product: | Koha | Reporter: | Jonathan Druart <jonathan.druart> |
Component: | Searching | Assignee: | Galen Charlton <gmcharlt> |
Status: | NEW --- | QA Contact: | Testopia <testopia> |
Severity: | normal | ||
Priority: | P5 - low | CC: | dcook, eric.begin, fridolin.somers, hagud, jean-manuel.broust, magnus, marjorie.barry-vila, martin.renvoize, nick, tomascohen |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
Change sponsored?: | --- | Patch complexity: | --- |
Documentation contact: | Documentation submission: | ||
Text to go in the release notes: | Version(s) released in: | ||
Circulation function: | |||
Bug Depends on: | 11232 | ||
Bug Blocks: |
Description
Jonathan Druart
2015-02-04 11:14:22 UTC
I can confirm that the new facets are too slow to be used for a big library currently. We tried with a database of rougly 1.000.000 as well and saw even slower response times than in Jonathan's test below. Searches took up to 40 seconds. With the old facets the time came down dramatically to about 4-6 seconds for the same search and is generally much faster. Have we reported this upstream at all? (In reply to Martin Renvoize from comment #2) > Have we reported this upstream at all? I don't think so. The main problem is that we are retrieving each facet at a time. This gets really better when we retrieve all facets at once. The problem is that if you ask for a setSpec containing all facets, and one is empty, then the result is empty. I've tried to patch zebra for that, but didn't finish it. The IndexData guys didn't seem to accept such a solution anyway. The best approach IMO is to make rebuild_zebra.pl create phantom entries for all facets. But didn't have the time and memory to do it. PS: When we first tried zebra facets in Reno, they were really better speed-wise. I think something changed between 2.0.47 and 2.0.60 that made them worse. We have faced a similar issue about zebra facet performance, but we have identified another variables that affect the zebra performance We were working with koha 16.05.05 debian 8 plack enabled, 4 cpu , 8 RAM + 100 GB on SSD. From fresh install, load a database of 225.000 bib records + 300.000 authority + 260.000 item record. When searching we found that stopwords increase the time response up to 12 times... Koha Search: Felipe Rey España (15 secons) Koha Search: Felipe Rey de España (110 seconds) If we turn on queryfuzz and query stemming the speed increase sharply Koha Search: Felipe Rey España (5 secons) Koha Search: Felipe Rey de España (20 seconds) If we turn on zebra facets, it becomes very fast Koha Search: Felipe Rey España (1,5 secons) Koha Search: Felipe Rey de España (2,5 seconds) If we turn off zebra facet and keep query steemming and qeryfuzzy on it has the same performance behaviour that turning off facets Koha Search: Felipe Rey España (5 secons) Koha Search: Felipe Rey de España (20 seconds) then here we have a mix of issues with stopwords, querystemming, queryfuzzy and zebra facets. Hope it helps (In reply to Hugo Agud from comment #4) > Hope it helps That's really interesting! Jonathan and Katrin: Is the issue the large database or the large resultset? Looking at Jonathan's example, it seems like the issue is the large resultset. It makes sense to me that a large result set would take a long time to extract facets from (depending on the efficiency of that facet extraction code of course). Without Zebra facets, it looks like we're only naively extracting facets from 20 records (using "maxRecordsForFacets" which defaults to 20), so it's going to be faster for a large result set but much worse quality. I think the most robust solution would be to asynchronously load the facets into the Koha Web UI after Koha has returned the search page to the user's browser. (The search results should be asynchronously loaded too but that's for another day.) If I recall correctly, Net::Z3950::ZOOM has the ability to do a certain amount of non-blocking I/O, so in theory you should also be able to do the facet fetching in parallel. I'd have to look into that further... My TODO list is already overfilled, but it would be interesting work to do. Actually, that could make for an interesting plugin. If we turned off facet fetching in the Perl search script and let a plugin take over. That might be a good way to do a proof-of-concept at the very least. I'll add it to my list in any case... lol. (In reply to Martin Renvoize from comment #2) > Have we reported this upstream at all? I reported something in this area yesterday: https://github.com/indexdata/idzebra/issues/35 As Hugo notes, stop words seem to cause big problems, although it doesn't seem tied to the result set but rather the initial records fetched and filtered. I had 2 nearly identical queries on a 1,000,000+ record database. The first search returned instantly and its 1st facet returned in 2 seconds. The second search (including "the") returned after 30 seconds and its 1st facet returned after 60 seconds. Both had result sets of 3323-3325 records. Since "the" had matched over 1,000,000 records, it seemed clear that the facet calculation involved more than just the result set (although my initial skimming through the Zebra C code couldn't confirm that). I'm going to see if I can compile Zebra and figure out what's going on there, so that I can either send a pull request or at least put Indexdata onto the root cause and rough solution. But I'm also not going to commit heaps of time to this. It's fresh in my mind, so now is a good time to do it, but we'll see... David, whlie you're digging that hole... One of the problems for the facet retrieval step in Koha is that we look through the defined facets instead of fetching them all at once. The reason behind that is that if some facet is not populated at all (i.e. no record has field X which is mapped to the facet) then the resultset is always empty in Zebra. I filed an issue about that (a lot of time ago) but the answer was negative: that's the expected behavior. So, would it be possible to add some ghost record (e.g. ID = -1) with all the fields (one space or special character?) so we convert it into one query instead of many? (In reply to Tomás Cohen Arazi from comment #9) > David, whlie you're digging that hole... One of the problems for the facet > retrieval step in Koha is that we look through the defined facets instead of > fetching them all at once. > > The reason behind that is that if some facet is not populated at all (i.e. > no record has field X which is mapped to the facet) then the resultset is > always empty in Zebra. I filed an issue about that (a lot of time ago) but > the answer was negative: that's the expected behavior. Yeah, I've been thinking a little bit about that as well, although I didn't look at the code for that. I think that's probably handled within the facet_fetch() function. I do wonder if they'd be interested in adding the "14" attribute to the faceting as well. (https://software.indexdata.com/zebra/doc/querymodel-zebra.html#querymodel-zebra-attr-search) Adam is out of the office this week, but I'm hoping that he'll help me to understand the faceting code. > So, would it be possible to add some ghost record (e.g. ID = -1) with all > the fields (one space or special character?) so we convert it into one query > instead of many? I think that might be an option. When I run zebrasrv in this directory https://github.com/indexdata/idzebra/tree/master/examples/marcxml, I noticed that there were records that weren't indexed and records that seemed to have empty indexes. In theory, we could add a record to Zebra like you describe, but I haven't tested it in practice. -- I wouldn't mind looking at trying to fetch the facets asynchronously although that's less pressing for me atm... |