Bug 7511 - Caching Templates
Summary: Caching Templates
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: 3.8
Hardware: All All
: P1 - high enhancement (vote)
Assignee: Paul Poulain
QA Contact: Ian Walls
URL:
Keywords:
Depends on:
Blocks: 7533
  Show dependency treegraph
 
Reported: 2012-02-08 15:48 UTC by Paul Poulain
Modified: 2016-08-30 21:59 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 7511 [ENH] Template caching (T::T feature) (1.37 KB, patch)
2012-02-10 08:46 UTC, Paul Poulain
Details | Diff | Splinter Review
Bug 7511 [ENH] Template caching (T::T feature) (1.66 KB, patch)
2012-02-12 19:31 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Poulain 2012-02-08 15:48:07 UTC
(cc of a mail sent to koha-devel on 2011-09-12)


The loading of the template seems to cost 150ms, and I feel that those 2
options could be usefull to speed up the process. Did anyone already saw
this option before ? Is it something worth investigating more ?

Here is the documentation of the 1st option:

COMPILE_EXT

>From version 2 onwards, the Template Toolkit has the ability to compile
templates to Perl code and save them to disk for subsequent use (i.e.
cache persistence). The COMPILE_EXT option may be provided to specify a
filename extension for compiled template files. It is undefined by
default and no attempt will be made to read or write any compiled
template files.

    my $template = Template->new({
        COMPILE_EXT => '.ttc',
    });

If COMPILE_EXT is defined (and COMPILE_DIR isn't, see below) then
compiled template files with the COMPILE_EXT extension will be written
to the same directory from which the source template files were loaded.


I made some tests and the results are great:

NYTprof comparison:
WITHOUT caching: Profile of mainpage.pl for 2.18s (of 2.79s), executing 199503 statements and 60033 subroutine calls in 279 source files and 81 string evals.
WITH caching: Profile of mainpage.pl for 1.86s (of 2.27s), executing 131771 statements and 47501 subroutine calls in 281 source files and 73 string evals.

The same kind of improvements can be seen with misc/load_testing/benchmark_staff.pl
WITHOUT caching:
Benchmarking with 20 occurences of each operation and 30 concurrent sessions 
Step 1: staff client main page     	23465ms	0.852 pages/sec
Step 2: catalog detail page        	34507ms	0.579 biblios/sec
Step 3: catalogue search               	26909ms	0.743 biblios/sec
Step 5: patron detail page         	27872ms	0.717 borrowers/sec
Step 5: patron search page             	52577ms	0.76 borrowers/sec
Step 6a circulation (checkouts)        	32309ms	0.619 checkouts/sec
Step 6b circulation (checkins)         	32959ms	0.606 checkins/sec
all transactions at once               	103897ms	1.732 operations/sec

WITH caching:
Benchmarking with 20 occurences of each operation and 30 concurrent sessions 
Step 1: staff client main page     	21286ms	0.939 pages/sec
Step 2: catalog detail page        	27213ms	0.734 biblios/sec
Step 3: catalogue search               	22091ms	0.905 biblios/sec
Step 5: patron detail page         	22015ms	0.908 borrowers/sec
Step 5: patron search page             	43871ms	0.911 borrowers/sec
Step 6a circulation (checkouts)        	27601ms	0.724 checkouts/sec
Step 6b circulation (checkins)         	28860ms	0.693 checkins/sec
all transactions at once               	87597ms	2.054 operations/sec
Comment 1 Paul Poulain 2012-02-10 08:43:02 UTC
Yesterday, I had a long trip in a train, so I investigated more, and wrote a patch.

Just a performance thing: I tried to checkout a book, with and without caching, on a copy of a real database. The result is absolutly astonishing:

The command run:
perl -d:NYTProf circ/circulation.pl userid=test password=test borrowernumber=7852 barcode=00519000000272

Without T::T cache =
Profile of circ/circulation.pl for 3.28s (of 4.74s), executing 486760 statements and 88239 subroutine calls in 284 source files and 87 string evals.

With T::T cache = 
Profile of circ/circulation.pl for 2.37s (of 2.83s), executing 144410 statements and 53427 subroutine calls in 293 source files and 74 string evals.

The circulation.tt is one of the largest template we have, the result is ... well, use the word you want.
I was so surprised that I tried to find a mistake in my test, but could not.
without cache = run the command, run nytprofhtml to generate the result page
with cache =
* run the command once
* check that /tmp now has a cached version
* run the command again and run nytprofhtml to generate the result page

Patch to follow, it's a 2 lines one !
Comment 2 Paul Poulain 2012-02-10 08:46:16 UTC Comment hidden (obsolete)
Comment 3 Paul Poulain 2012-02-10 08:54:16 UTC
(In reply to comment #2)
> Just add to your Koha configuration file
> <template_cache_dir>/path/writable/by/apache/user</template_cache_dir>
> and Koha will use template caching, for about 10% CPU time saving.

1- don't forget to reload apache
2- to check that cache now work: (I assume that KOHA_DIR is /home/koha/src and the cache dir is /tmp) there is now a directory /tmp/home/koha/src, with sub directories, owned by apache user

I encountered a problem when trying to run NYTProf and caching at the same time: the NYTProf is run as "paul" while Koha run as "apache2". Once you've started running Koha with caching as one of those 2 users, you can't run a script as the other user. This is because of linux permissions rwxr--r--
A little sudo rm -fr /tmp/home fixes this annoying problem if you need to run a script as another user
a chmod +w+x could also work, I didn't tried
Comment 4 Jared Camins-Esakov 2012-02-12 19:31:02 UTC
Created attachment 7589 [details] [review]
Bug 7511 [ENH] Template caching (T::T feature)

Just add to your Koha configuration file
<template_cache_dir>/path/writable/by/apache/user</template_cache_dir>
in the <config> block, and Koha will use template caching, for about
10% CPU time saving.

on linux servers, /tmp is usually OK

(also fixes an indenting with a TAB)

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
The results of using this are very striking. Based on an insufficient
sample size, it would seem that the time spent in T::T is reduced by a
factor of at least 5.
Comment 5 Ian Walls 2012-02-12 22:12:39 UTC
Safely implements a new <config> option using the features of T:T.  Marking Passed QA.

Next steps are to add this <config> option to the Makefile.PL so it does not need to be manually edited into existence.
Comment 6 Paul Poulain 2012-02-13 10:04:08 UTC
(In reply to comment #5)
> Next steps are to add this <config> option to the Makefile.PL so it does not
> need to be manually edited into existence.

Right. I've added a bug for that, and made it dependant from this one (bug 7533)
Comment 7 Jared Camins-Esakov 2012-12-31 00:36:59 UTC
There have been no further reports of problems so I am marking this bug resolved.