Koha search got considerably slower between 3.16 and 3.18.
With 3.22 we got Plack support out of the box for packages. Time to look into the speed issues and get the fastest Koha ever.
This bug will be used as an umbrella bug to gather all related information.
Bug 11842 introduced problematic cache flushing behaviour.
The cache is deleted all the time, eg. the framework structure is
fetched again and again for every single result in a search (related
to 2. below)
Reverting the patch leads to a significant increase in search speed (building the results page really), supposedly because the MARC structure is not fetched for every single result anymore.
The MARC structure needs to be deleted from cache only after a framework is changed, not while searching the OPAC.
Search without XSLT is much faster than without.
Search with a single Result is faster now than search with multiple results.
A lot of time is lost when building the results page with XSLT
For every single result in a search, a lot of things are repeated
all over again. A huge bunch of sysprefs is fetched every time, even
though it is not likely they will change while fetching all results
of one search query. GetMarcStructure seems to be a problem (see 1.
above) but others may be too.
Bug 11051 seems related.
"Benchmarks for 3.22" in the Koha wiki: http://wiki.koha-community.org/wiki/Benchmark_for_3.22
Allow me to go on a limb here, and dare to raise the importance of this generic "feature".
At this moment, hardware requirements to make a search feel "instantaneous" are unrealistic to most libraries. It shoudn't be and I think it would be a worthy goal to improve on the current benchmarks by x10.
With the patch on bug 15263 and reverting bug 11842, I can reduce search time on a Raspberry Pi from 30 to 18 seconds for 1000 hits.
In the long run, we probably should change things more deeply. Related part of a dicussion from IRC:
(13:51:58) tcohen: drojf: my plan with https://github.com/tomascohen/koha/tree/xslt_filtering was to avoid useless filtering of the retrieved records
(13:52:14) tcohen: so, put search results in Koha::MetadataRecord object
(13:52:42) tcohen: and pass them through a pipe of Koha::Filter-s using the Koha::RecordProcessor thing
(13:53:09) tcohen: that's from a perspective in which we re-organize the code
(13:53:29) tcohen: and start abstracting things a bit for a future multi-metadata schema support
(13:54:10) tcohen: regarding zebra, in 3.16 we made DOM setups exchange XML between Perl and Zebra
(13:54:15) tcohen: both for indexing (which is OK)
(13:54:34) tcohen: and for retrieval (which is OK, but seems really under performant in the current state of the way we do things)
(13:55:20) tcohen: so my opinion regarding the syspref retrieval for each record is that we should optimize that in a short term
(13:55:35) tcohen: but move to Koha::Fitler ofr the mid/long-term
(13:55:58) tcohen: if you think about it, we are serializing/deserializing things more than 3 times
(13:56:17) tcohen: and this might get worse with JSON data from ES
(13:56:36) tcohen: get JSON data => have filters for JSON data => period
(13:56:39) drojf: that sounds very good to me
(13:56:51) tcohen: get MARC(XML) data => have filters for MARC(XML) data => period
(13:57:26) tcohen: also, with the current indexer, we shouldn't re-calculate loan status, etc for rendering
(13:57:33) tcohen: that information is already in zebra
(13:57:35) tcohen: or it should
(13:58:13) tcohen: it is cheaper to prepare the data so zebra has it, than calculate it on rendering
(13:58:31) tcohen: because de indexing process is asynchronous
(13:58:40) tcohen: and is queued, etc
(In reply to Blou from comment #4)
> At this moment, hardware requirements to make a search feel "instantaneous"
> are unrealistic to most libraries.
I heartily disagree. Zebra is actually extremely fast. Often instantly fast. It's Koha that makes search slow, and we have the power to change that.
(In reply to Mirko Tietgen from comment #6)
> (13:52:14) tcohen: so, put search results in Koha::MetadataRecord object
> (13:52:42) tcohen: and pass them through a pipe of Koha::Filter-s using the
> Koha::RecordProcessor thing
> (13:53:09) tcohen: that's from a perspective in which we re-organize the code
> (13:53:29) tcohen: and start abstracting things a bit for a future
> multi-metadata schema support
> (13:55:58) tcohen: if you think about it, we are serializing/deserializing things more than 3 times
We're doing too much work processing search results. We need to streamline this process to avoid repeated handling, unnecessary lookups, etc.
Besides getting performance improvements, we'll also gain some future-proofing by abstracting away from MARC based search result processing.
I'd argue that we should be retrieving the bare minimum from Zebra and then doing the minimal amount of transforming possible. Failing that, if we have to fetch whole MARC records from Zebra, let's set up an optimized pipeline.
I reckon that there's no *technical* reason that search shouldn't be fast in Koha.