Bug 35799 - Loading svc/cataloguing/framework bottlenecks advanced cataloging editor
Summary: Loading svc/cataloguing/framework bottlenecks advanced cataloging editor
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-01-12 20:29 UTC by Phil Ringnalda
Modified: 2024-04-05 02:55 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Phil Ringnalda 2024-01-12 20:29:19 UTC
The advanced editor loads /cgi-bin/koha/svc/cataloguing/framework?frameworkcode=&callback=define with RequireJS, with a 30 second timeout and no failure handling. That gives you 30 seconds for request, generation, and download, or the advanced editor simply won't load and will sit with a progress indicator spinning forever (and in versions before bug 34275 landed, with no way to edit bibs without editing your cookie so you can get back to using the basic editor).

There are two ways (plus the combination) to have it fail to load, generation or network.

Network is easy enough to reproduce even in koha-testing-docker, since Firefox's devtools have network throttling that goes back to "Regular 2G" which will certainly time it out.

Generation is a bit harder without a decent sized database: with 300K bibs and 600K authorities in my production database it's easy to either write reports that ExtractValue(biblio_metadata.metadata...) and run several at once, or to notice that authority search results actually support &resultsperpage=1000 despite not having UI to set it, and opening a few pages of a thousand authority records that require searching for the number of bibs using each of those thousands of authorities.

svc/cataloguing/framework is a json representation of the requested framework plus all authorized values.

Frameworks are incredibly compressible, consisting of things like 4273 instances of value_builder of which 4253 are "value_builder":"", but it's sent uncompressed, so 1.8MB for something that gzips to 160KB.

It's sent with Cache-Control: no-cache and Pragma: no-cache headers, so even though it continues to load after the RequireJS timeout, just sitting waiting for it (I've had a painfully large number of times where it would consistently load in 31 or 32 seconds) and then reloading does no good. Not caching doesn't really do much of anything to ensure freshness, since if you stick to just one framework in an open instance of the editor, you won't ever reload svc/cataloguing/framework, but it does at least mean that if you know you've changed a framework and you don't see the change you can just open the editor in a new tab without having to actually clear the browser cache.

The authorized values are too small of a percentage of the file size to make much difference there, but, they don't vary with the framework, so there's no reason to reload them for each change of framework, and they are 3/4 of the database connections (one for AV, one for itemtypes, one for classification sources), so loading them once separately probably would help for some generation-limited cases and certainly would for the combination of generation- and network-limited cases. Getting three db connections and then loading a tiny file in less than 30 seconds, then getting one db connection and loading a large file in less than a separate 30 seconds is easier than getting four connections and a large file in one 30 second period.

Beyond that, there's no clear easy answer. Solving for my sketchy internet at home is easy, just gzip the json after generating it, but that adds server (though not db server) load. Okay, generate an already gzipped file once and serve that, after all, I typically only change my frameworks twice a year, how fresh do they need to be? But, editing frameworks consists of hundreds of tiny saves of each subfield of each tag, so making each one regenerate a static file is going to be painful. Okay, let the browser cache the output so at least a reload after getting it to load in 35 seconds works. Cache it for how long?
Comment 1 Phil Ringnalda 2024-03-05 00:32:39 UTC
Actually, since we die with an uncaught exception with no notification and nothing the user can do but reload or close the tab, there's no reason why we should be using "waitSeconds: 30" when 0 (== infinite) exists.
Comment 2 Phil Ringnalda 2024-03-28 23:07:01 UTC
Moved setting waitSeconds: 0 to bug 36461, so it can be about making it possible to load the editor, and this can be about making it pleasant to load it.
Comment 3 Nick Clemens (kidclamp) 2024-03-29 12:59:32 UTC
Filed bug 36463 which improves size of loading subsequent frameworks, however, the initial load is not compressed for some reason - I think because of the loading by requireJS?
Comment 4 Phil Ringnalda 2024-03-29 15:50:17 UTC
Not sure why, but the initial framework load is served as text/javascript, so it's missing compression from that.
Comment 5 Phil Ringnalda 2024-03-29 18:08:44 UTC
Oh, heh. That's what &callback= does, https://git.koha-community.org/Koha-community/Koha/src/branch/master/C4/Service.pm#L56 serves it as 'js' rather than 'json', and although we deflate application/javascript, we don't serve that, https://git.koha-community.org/Koha-community/Koha/src/branch/master/C4/Output.pm#L250 maps 'js' to text/javascript.
Comment 6 Phil Ringnalda 2024-03-29 18:23:26 UTC
And because the names of mimetypes are names, and thus a Hard Problem, rfc9239 says text/javascript is correct and application/javascript is an obsolete alias, so the correct fix is to deflate text/javascript.
Comment 7 Phil Ringnalda 2024-04-05 02:55:26 UTC
Filed bug 36531 for compressing the initial load. So snappy now!