Summary: | Introduce new tied cache system | ||
---|---|---|---|
Product: | Koha | Reporter: | Jared Camins-Esakov <jcamins> |
Component: | Architecture, internals, and plumbing | Assignee: | Galen Charlton <gmcharlt> |
Status: | CLOSED FIXED | QA Contact: | Paul Poulain <paul.poulain> |
Severity: | major | ||
Priority: | P5 - low | CC: | elliott, paul.poulain |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | Large patch | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: | Circulation function: | ||
Bug Depends on: | |||
Bug Blocks: | 8089 | ||
Attachments: |
Bug 9434: Introduce new tied cache system
Bug 9434: Introduce new tied cache system Bug 9434: Introduce new tied cache system [PASSED QA] Bug 9434: Introduce new tied cache system |
Description
Jared Camins-Esakov
2013-01-19 19:32:44 UTC
Created attachment 14764 [details] [review] Bug 9434: Introduce new tied cache system At the moment we cache numerous pieces of information in module-level variables which then do not get updated in other threads/processes when they are changed by the user. This is a serious usability issue. Examples of this include the way we treat sysprefs (there is now a method to disable the syspref cache, but by default it is enabled), notices, frameworks, field mappings, and koha-conf.xml, at least. This patch sets the stage for eliminating this problem by making it possible to convert module-level cache variables into variables that are actually backed by whatever caching system may be configured. This is done through a special Koha::Cache::Object class which can be tied to the variables that are being used for caching and provided with a constructor method/closure to allow the cache to be reloaded when it expires. For example: my $cache = Koha::Cache->new(); my $data = 'whatever'; my $variable = Koha::Cache->create_scalar( { 'key' => 'whatever', 'timeout' => 2, 'constructor' => sub { return $data; }, } ); print "$$variable\n"; # Prints "whatever" The one change this necessitates for accessing the data is that the variable must be dereferenced an additional time before use (i.e. $$variable instead of $variable). There is no difference when the variable tied is a hash (created with Koha::Cache->create_hash). This is a small price to pay for Koha working in a multi-threaded, persistent environment. This change will also make caching easier in general. CHI was incompatible with the variable tying, so this patch also removes the dependency on CHI, using instead Cache::Memcached::Fast, Cache::FastMmap, and Cache::Memory, when they are available. To test: 1) Apply patch. 2) Run unit test t/Cache.t (after setting the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables). As no changes were made to the tests already in that file, this passing demonstrates there are no regressions. 3) With memcached caching enabled (you must set the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables) and DEBUG turned on (i.e. the DEBUG environment variable set to 1), try running a report via the web service ([intranet]/cgi-bin/koha/svc/report?id=1 and check your web server logs to confirm that there are messages like "get_from_cache for intranet:report:id:1" in them. 4) If the reports worked, sign off. NOTE: Technically you could test this without needing memcached by installing libcache-fastmmap-perl and setting CACHING_SYSTEM to 'fastmmap' instead of 'memcached'. You could also install libcache-perl and set CACHING_SYSTEM to 'memory' but there would be little point as the cached variables would go out of scope in between runs. Created attachment 14786 [details] [review] Bug 9434: Introduce new tied cache system At the moment we cache numerous pieces of information in module-level variables which then do not get updated in other threads/processes when they are changed by the user. This is a serious usability issue. Examples of this include the way we treat sysprefs (there is now a method to disable the syspref cache, but by default it is enabled), notices, frameworks, field mappings, and koha-conf.xml, at least. This patch sets the stage for eliminating this problem by making it possible to convert module-level cache variables into variables that are actually backed by whatever caching system may be configured. This is done through a special Koha::Cache::Object class which can be tied to the variables that are being used for caching and provided with a constructor method/closure to allow the cache to be reloaded when it expires. For example: my $cache = Koha::Cache->new(); my $data = 'whatever'; my $variable = Koha::Cache->create_scalar( { 'key' => 'whatever', 'timeout' => 2, 'constructor' => sub { return $data; }, } ); print "$$variable\n"; # Prints "whatever" The one change this necessitates for accessing the data is that the variable must be dereferenced an additional time before use (i.e. $$variable instead of $variable). There is no difference when the variable tied is a hash (created with Koha::Cache->create_hash). This is a small price to pay for Koha working in a multi-threaded, persistent environment. This change will also make caching easier in general. CHI was incompatible with the variable tying, so this patch also removes the dependency on CHI, using instead Cache::Memcached::Fast, Cache::FastMmap, and Cache::Memory, when they are available. To test: 1) Apply patch. 2) Run unit test t/Cache.t (after setting the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables). As no changes were made to the tests already in that file, this passing demonstrates there are no regressions. 3) With memcached caching enabled (you must set the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables) and DEBUG turned on (i.e. the DEBUG environment variable set to 1), try running a report via the web service ([intranet]/cgi-bin/koha/svc/report?id=1 and check your web server logs to confirm that there are messages like "get_from_cache for intranet:report:id:1" in them. 4) If the reports worked, sign off. NOTE: Technically you could test this without needing memcached by installing libcache-fastmmap-perl and setting CACHING_SYSTEM to 'fastmmap' instead of 'memcached'. You could also install libcache-perl and set CACHING_SYSTEM to 'memory' but there would be little point as the cached variables would go out of scope in between runs. Created attachment 14894 [details] [review] Bug 9434: Introduce new tied cache system At the moment we cache numerous pieces of information in module-level variables which then do not get updated in other threads/processes when they are changed by the user. This is a serious usability issue. Examples of this include the way we treat sysprefs (there is now a method to disable the syspref cache, but by default it is enabled), notices, frameworks, field mappings, and koha-conf.xml, at least. This patch sets the stage for eliminating this problem by making it possible to convert module-level cache variables into variables that are actually backed by whatever caching system may be configured. This is done through a special Koha::Cache::Object class which can be tied to the variables that are being used for caching and provided with a constructor method/closure to allow the cache to be reloaded when it expires. For example: my $cache = Koha::Cache->new(); my $data = 'whatever'; my $variable = Koha::Cache->create_scalar( { 'key' => 'whatever', 'timeout' => 2, 'constructor' => sub { return $data; }, } ); print "$$variable\n"; # Prints "whatever" The one change this necessitates for accessing the data is that the variable must be dereferenced an additional time before use (i.e. $$variable instead of $variable). There is no difference when the variable tied is a hash (created with Koha::Cache->create_hash). This is a small price to pay for Koha working in a multi-threaded, persistent environment. This change will also make caching easier in general. CHI was incompatible with the variable tying, so this patch also removes the dependency on CHI, using instead Cache::Memcached::Fast, Cache::FastMmap, and Cache::Memory, when they are available. To test: 1) Apply patch. 2) Run unit test t/Cache.t (after setting the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables). As no changes were made to the tests already in that file, this passing demonstrates there are no regressions. 3) With memcached caching enabled (you must set the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables) and DEBUG turned on (i.e. the DEBUG environment variable set to 1), try running a report via the web service ([intranet]/cgi-bin/koha/svc/report?id=1 and check your web server logs to confirm that there are messages like "get_from_cache for intranet:report:id:1" in them. 4) If the reports worked, sign off. NOTE: Technically you could test this without needing memcached by installing libcache-fastmmap-perl and setting CACHING_SYSTEM to 'fastmmap' instead of 'memcached'. You could also install libcache-perl and set CACHING_SYSTEM to 'memory' but there would be little point as the cached variables would go out of scope in between runs. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> I am getting a failed test with the unit test provided. elliott@ByWater-Test:~/kohaclone$ prove ./t/Cache.t ./t/Cache.t .. 1/29 Use of uninitialized value $ip in hash element at /usr/share/perl5/Cache/Memcached.pm line 253. Use of uninitialized value in subroutine entry at /usr/share/perl5/Cache/Memcached.pm line 288. # Failed test 'use C4::Context;' # at ./t/Cache.t line 15. # Tried to use 'C4::Context'. # Error: Bad arg length for Socket::pack_sockaddr_in, length is 0, should be 4 at /usr/lib/perl/5.10/Socket.pm line 214. # BEGIN failed--compilation aborted at /home/elliott/kohaclone/C4/Context.pm line 99. # Compilation failed in require at (eval 14) line 2. # BEGIN failed--compilation aborted at (eval 14) line 2. ./t/Cache.t .. 5/29 # Failed test 'fetching valid item from cache' # at ./t/Cache.t line 39. # got: undef # expected: 'I AM MORE DATA' # Failed test 'fetching valid item from cache (after clearing another item)' # at ./t/Cache.t line 49. # got: undef # expected: 'I AM MORE DATA22' # Failed test 'Data retrieved from cache' # at ./t/Cache.t line 76. # got: '2' # expected: '1' ./t/Cache.t .. 14/29 # Failed test 'Constructor called again when timeout reached' # at ./t/Cache.t line 78. # got: '3' # expected: '2' # Failed test 'Stored new value to cache' # at ./t/Cache.t line 80. # got: '4' # expected: '5' # Failed test 'Constructor not called after storing value' # at ./t/Cache.t line 81. # got: '4' # expected: '2' # Failed test 'Setting hash member worked' # at ./t/Cache.t line 104. # got: undef # expected: 'surprise' ./t/Cache.t .. 25/29 # Looks like you failed 8 tests of 29. ./t/Cache.t .. Dubious, test returned 8 (wstat 2048, 0x800) Failed 8/29 subtests Test Summary Report ------------------- ./t/Cache.t (Wstat: 2048 Tests: 29 Failed: 8) Failed tests: 3, 6, 8, 13-16, 21 Non-zero exit status: 8 Files=1, Tests=29, 8 wallclock secs ( 0.04 usr 0.01 sys + 0.14 cusr 0.03 csys = 0.22 CPU) Result: FAIL (In reply to comment #4) > I am getting a failed test with the unit test provided. > Failed 8/29 subtests The fail probably came from the fact that the following prerequisite were failing: sudo apt-get install libcache-fastmmap-perl export CACHING_SYSTEM=fastmmap It works OK for me when I have it, and I get errors when I don't. So QAing again: * passes koha-qa.pl * unit test provided * I trieds without caching and couldn't see anything wrong. I tried with cache enabled, and it still work. passed QA (and changed the bug description, to reflect the patch description) Created attachment 16547 [details] [review] [PASSED QA] Bug 9434: Introduce new tied cache system At the moment we cache numerous pieces of information in module-level variables which then do not get updated in other threads/processes when they are changed by the user. This is a serious usability issue. Examples of this include the way we treat sysprefs (there is now a method to disable the syspref cache, but by default it is enabled), notices, frameworks, field mappings, and koha-conf.xml, at least. This patch sets the stage for eliminating this problem by making it possible to convert module-level cache variables into variables that are actually backed by whatever caching system may be configured. This is done through a special Koha::Cache::Object class which can be tied to the variables that are being used for caching and provided with a constructor method/closure to allow the cache to be reloaded when it expires. For example: my $cache = Koha::Cache->new(); my $data = 'whatever'; my $variable = Koha::Cache->create_scalar( { 'key' => 'whatever', 'timeout' => 2, 'constructor' => sub { return $data; }, } ); print "$$variable\n"; # Prints "whatever" The one change this necessitates for accessing the data is that the variable must be dereferenced an additional time before use (i.e. $$variable instead of $variable). There is no difference when the variable tied is a hash (created with Koha::Cache->create_hash). This is a small price to pay for Koha working in a multi-threaded, persistent environment. This change will also make caching easier in general. CHI was incompatible with the variable tying, so this patch also removes the dependency on CHI, using instead Cache::Memcached::Fast, Cache::FastMmap, and Cache::Memory, when they are available. To test: 1) Apply patch. 2) Run unit test t/Cache.t (after setting the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables). As no changes were made to the tests already in that file, this passing demonstrates there are no regressions. 3) With memcached caching enabled (you must set the MEMCACHED_SERVERS and CACHING_SYSTEM environment variables) and DEBUG turned on (i.e. the DEBUG environment variable set to 1), try running a report via the web service ([intranet]/cgi-bin/koha/svc/report?id=1 and check your web server logs to confirm that there are messages like "get_from_cache for intranet:report:id:1" in them. 4) If the reports worked, sign off. NOTE: Technically you could test this without needing memcached by installing libcache-fastmmap-perl and setting CACHING_SYSTEM to 'fastmmap' instead of 'memcached'. You could also install libcache-perl and set CACHING_SYSTEM to 'memory' but there would be little point as the cached variables would go out of scope in between runs. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com> This patch has been pushed to master. |