Bug 14476

Summary: Improving opac-search performances
Product: Koha Reporter: Julian FIOL <julian.fiol>
Component: Architecture, internals, and plumbingAssignee: Julian Maurice <julian.maurice>
Status: RESOLVED WONTFIX QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: dcook, fridolin.somers, gaetan.boisson, julian.maurice, katrin.fischer, tomascohen
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Who signed the patch off: Text to go in the release notes:
Version(s) released in:
Bug Depends on:    
Bug Blocks: 7923    
Attachments: bug 14476 : Caching C4::XSLT::transformMARCXML4XSLT
[SIGNED-OFF] bug 14476 : Caching C4::XSLT::transformMARCXML4XSLT

Description Julian FIOL 2015-06-30 14:07:35 UTC
Improving opac-search performances by caching the results of C4::XSLT::transformMARCXML4XSLT.

This also impact the performances of C4::XSLT::XSLTParse4Display.

My tests (with Devel::NYTProf) showed a gain of about 
65% on C4::XSLT::transformMARCXML4XSLT and 
35% on C4::XSLT::XSLTParse4Display.

The gain of performances is increasing with the numbers of results.

Examples:
--------
1/
searching : "sinatra", results : 2
gain :
C4::XSLT::transformMARCXML4XSLT = 40%
C4::XSLT::XSLTParse4Display     = 19%

2/
searching : "Victor Hugo", results : 16
gain :
C4::XSLT::transformMARCXML4XSLT = 71%
C4::XSLT::XSLTParse4Display     = 36%

3/
searching : "a", results : 2060
gain :
C4::XSLT::transformMARCXML4XSLT = 82%
C4::XSLT::XSLTParse4Display     = 51%
Comment 1 Julian FIOL 2015-06-30 14:12:39 UTC Comment hidden (obsolete)
Comment 2 Gaetan Boisson 2015-09-01 18:17:22 UTC
Created attachment 42179 [details] [review]
[SIGNED-OFF] bug 14476 : Caching C4::XSLT::transformMARCXML4XSLT

This patch is improving the opac-search performances by
caching the results of C4::XSLT::transformMARCXML4XSLT

My tests (with Devel::NYTProf) showed a gain of about
65% on C4::XSLT::transformMARCXML4XSLT and
35% on C4::XSLT::XSLTParse4Display.

The gain of performances is increasing with the numbers
of results.

Signed-off-by: Gaetan Boisson <gaetan.boisson@biblibre.com>
Comment 3 Gaetan Boisson 2015-09-01 18:18:44 UTC
Works for me, although the performance gain was not as big as expected. I will test it again on a larger data set.
Comment 4 Tomás Cohen Arazi 2015-09-01 18:21:35 UTC
I have a concern: if we cache the record to be displayed, then the item statuses won't be up-to-date. If the tradeoff is not enough, maybe this is just not worth. Because people expect to see updated information on the records.
Comment 5 Fridolin SOMERS 2019-01-08 11:57:22 UTC
(In reply to Tomás Cohen Arazi from comment #4)
> I have a concern: if we cache the record to be displayed, then the item
> statuses won't be up-to-date. If the tradeoff is not enough, maybe this is
> just not worth. Because people expect to see updated information on the
> records.

Indeed.

I think we need to change this by moving items out of this XSLT file.
I think we should use Ajax to load in realtime items infos, with and API for that.
This will also avoid indexing all record on item circulation.
Dreams ...
Comment 6 David Cook 2019-01-08 23:15:47 UTC
(In reply to Fridolin SOMERS from comment #5)
> (In reply to Tomás Cohen Arazi from comment #4)
> > I have a concern: if we cache the record to be displayed, then the item
> > statuses won't be up-to-date. If the tradeoff is not enough, maybe this is
> > just not worth. Because people expect to see updated information on the
> > records.
> 
> Indeed.
> 
> I think we need to change this by moving items out of this XSLT file.
> I think we should use Ajax to load in realtime items infos, with and API for
> that.
> This will also avoid indexing all record on item circulation.
> Dreams ...

I often think using Ajax to load real time item info is a good idea too, although I think we'd still have the indexing issue because of the "limit to currently available items" filter on the search. 

Often, I think we should think about our OPAC as if it were a totally separate system, and just link it up to the Koha instance using APIs.
Comment 7 Tomás Cohen Arazi 2019-01-09 10:02:49 UTC
(In reply to Fridolin SOMERS from comment #5)
> (In reply to Tomás Cohen Arazi from comment #4)
> > I have a concern: if we cache the record to be displayed, then the item
> > statuses won't be up-to-date. If the tradeoff is not enough, maybe this is
> > just not worth. Because people expect to see updated information on the
> > records.
> 
> Indeed.
> 
> I think we need to change this by moving items out of this XSLT file.
> I think we should use Ajax to load in realtime items infos, with and API for
> that.

I agree! Let's have that endpoint!

> This will also avoid indexing all record on item circulation.

Not sure about it, but we could certainly index items and make JOIN operations in our queries ;-) (for limits, as David also suggests).
Comment 8 David Cook 2019-01-10 02:50:13 UTC
(In reply to Tomás Cohen Arazi from comment #7)
> (In reply to Fridolin SOMERS from comment #5)
> > This will also avoid indexing all record on item circulation.
> 
> Not sure about it, but we could certainly index items and make JOIN
> operations in our queries ;-) (for limits, as David also suggests).

I was totally thinking about that, as I was thinking about Solr query joins. I imagine ElasticSearch must be able to do the same thing, although I don't think Zebra could. Or did you have something in mind for JOIN operations?
Comment 9 Tomás Cohen Arazi 2019-01-10 11:22:46 UTC
(In reply to David Cook from comment #8)
> (In reply to Tomás Cohen Arazi from comment #7)
> > (In reply to Fridolin SOMERS from comment #5)
> > > This will also avoid indexing all record on item circulation.
> > 
> > Not sure about it, but we could certainly index items and make JOIN
> > operations in our queries ;-) (for limits, as David also suggests).
> 
> I was totally thinking about that, as I was thinking about Solr query joins.
> I imagine ElasticSearch must be able to do the same thing, although I don't
> think Zebra could. Or did you have something in mind for JOIN operations?

I was thinking ES. Zebra support is in freeze mode. And I'm sure you cannot mix different databases on a query.
Comment 10 David Cook 2019-01-11 02:55:46 UTC
(In reply to Tomás Cohen Arazi from comment #9)
> Zebra support is in freeze mode. 

That's good to hear. I've been wondering how long we'd try to support both. I look forward to a time where we're just focusing on one.
Comment 11 Katrin Fischer 2019-01-11 07:11:48 UTC
As far as I know there has been no official decision on Zebra. There are still Zebra bugs being worked on.
Comment 12 Julian Maurice 2020-01-17 15:44:26 UTC
Since caching the XSLT result is not desirable, should we close this bug ?
Comment 13 Julian Maurice 2020-04-10 08:53:57 UTC
(In reply to Julian Maurice from comment #12)
> Since caching the XSLT result is not desirable, should we close this bug ?

No answers in 3 months, so I'm changing status to WONTFIX. Feel free to reopen if you are willing to work on this.