Bug 15262 - Run Koha Run. Koha search should be fast again.
Summary: Run Koha Run. Koha search should be fast again.
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: unspecified
Hardware: All All
: P2 enhancement (vote)
Assignee: Mirko Tietgen
QA Contact: Testopia
URL: http://wiki.koha-community.org/wiki/B...
Keywords:
Depends on: 15264 15263 16041 16734
Blocks: 15342
  Show dependency treegraph
 
Reported: 2015-11-30 10:53 UTC by Mirko Tietgen
Modified: 2019-09-19 02:01 UTC (History)
12 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mirko Tietgen 2015-11-30 10:53:36 UTC
Koha search got considerably slower between 3.16 and 3.18.

With 3.22 we got Plack support out of the box for packages. Time to look into the speed issues and get the fastest Koha ever.

This bug will be used as an umbrella bug to gather all related information.
Comment 1 Mirko Tietgen 2015-11-30 10:59:53 UTC
Bug 11842 introduced problematic cache flushing behaviour.

The cache is deleted all the time, eg. the framework structure is
fetched again and again for every single result in a search (related
to 2. below)

Reverting the patch leads to a significant increase in search speed (building the results page really), supposedly because the MARC structure is not fetched for every single result anymore.

The MARC structure needs to be deleted from cache only after a framework is changed, not while searching the OPAC.
Comment 2 Mirko Tietgen 2015-11-30 11:02:10 UTC
Search without XSLT is much faster than without.
Search with a single Result is faster now than search with multiple results.

A lot of time is lost when building the results page with XSLT

For every single result in a search, a lot of things are repeated
all over again. A huge bunch of sysprefs is fetched every time, even
though it is not likely they will change while fetching all results
of one search query. GetMarcStructure seems to be a problem (see 1.
above) but others may be too.
Bug 11051 seems related.
Comment 3 Mirko Tietgen 2015-11-30 11:03:12 UTC
"Benchmarks for 3.22" in the Koha wiki: http://wiki.koha-community.org/wiki/Benchmark_for_3.22
Comment 4 Blou 2015-11-30 13:20:30 UTC
Allow me to go on a limb here, and dare to raise the importance of this generic "feature".

At this moment, hardware requirements to make a search feel "instantaneous" are unrealistic to most libraries.  It shoudn't be and I think it would be a worthy goal to improve on the current benchmarks by x10.
Comment 5 Mirko Tietgen 2015-11-30 14:17:07 UTC
With the patch on bug 15263 and reverting bug 11842, I can reduce search time on a Raspberry Pi from 30 to 18 seconds for 1000 hits.
Comment 6 Mirko Tietgen 2015-11-30 14:46:02 UTC
In the long run, we probably should change things more deeply. Related part of a dicussion from IRC:

(13:51:58) tcohen: drojf: my plan with https://github.com/tomascohen/koha/tree/xslt_filtering was to avoid useless filtering of the retrieved records
(13:52:14) tcohen: so, put search results in Koha::MetadataRecord object
(13:52:42) tcohen: and pass them through a pipe of Koha::Filter-s using the Koha::RecordProcessor thing
(13:53:09) tcohen: that's from a perspective in which we re-organize the code
(13:53:29) tcohen: and start abstracting things a bit for a future multi-metadata schema support
(13:54:10) tcohen: regarding zebra, in 3.16 we made DOM setups exchange XML between Perl and Zebra
(13:54:15) tcohen: both for indexing (which is OK)
(13:54:34) tcohen: and for retrieval (which is OK, but seems really under performant in the current state of the way we do things)
(13:55:20) tcohen: so my opinion regarding the syspref retrieval for each record is that we should optimize that in a short term
(13:55:35) tcohen: but move to Koha::Fitler ofr the mid/long-term
(13:55:58) tcohen: if you think about it, we are serializing/deserializing things more than 3 times
(13:56:17) tcohen: and this might get worse with JSON data from ES
(13:56:36) tcohen: get JSON data => have filters for JSON data => period
(13:56:39) drojf: that sounds very good to me
(13:56:51) tcohen: get MARC(XML) data => have filters for MARC(XML) data => period
(13:57:26) tcohen: also, with the current indexer, we shouldn't re-calculate loan status, etc for rendering
(13:57:33) tcohen: that information is already in zebra
(13:57:35) tcohen: or it should
(13:58:13) tcohen: it is cheaper to prepare the data so zebra has it, than calculate it on rendering
(13:58:31) tcohen: because de indexing process is asynchronous 
(13:58:40) tcohen: and is queued, etc
Comment 7 David Cook 2019-09-19 01:53:27 UTC
(In reply to Blou from comment #4)
> At this moment, hardware requirements to make a search feel "instantaneous"
> are unrealistic to most libraries.

I heartily disagree. Zebra is actually extremely fast. Often instantly fast. It's Koha that makes search slow, and we have the power to change that.
Comment 8 David Cook 2019-09-19 02:01:23 UTC
(In reply to Mirko Tietgen from comment #6)
> (13:52:14) tcohen: so, put search results in Koha::MetadataRecord object
> (13:52:42) tcohen: and pass them through a pipe of Koha::Filter-s using the
> Koha::RecordProcessor thing
> (13:53:09) tcohen: that's from a perspective in which we re-organize the code
> (13:53:29) tcohen: and start abstracting things a bit for a future
> multi-metadata schema support
...
> (13:55:58) tcohen: if you think about it, we are serializing/deserializing things more than 3 times

Exactly.

We're doing too much work processing search results. We need to streamline this process to avoid repeated handling, unnecessary lookups, etc. 

Besides getting performance improvements, we'll also gain some future-proofing by abstracting away from MARC based search result processing.

I'd argue that we should be retrieving the bare minimum from Zebra and then doing the minimal amount of transforming possible. Failing that, if we have to fetch whole MARC records from Zebra, let's set up an optimized pipeline.

I reckon that there's no *technical* reason that search shouldn't be fast in Koha.