Bug 32716

Summary: update NGINX config examples to increase proxy_buffer_size
Product: Koha Reporter: Galen Charlton <gmcharlt>
Component: Architecture, internals, and plumbingAssignee: Galen Charlton <gmc>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: minor    
Priority: P5 - low CC: arthur.suzuki, brendan, lucas, m.de.rooy
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: Unsponsored
Patch complexity: Trivial patch Documentation contact:
Documentation submission: Text to go in the release notes:
Set proxy_buffer_size in the example NGINX configuration to reduce chances that REST API responses that use pagination get dropped by NGINX
Version(s) released in:
23.05.00,22.11.05,22.05.13,21.11.21
Circulation function:
Attachments: Bug 32716: larger proxy_buffer_size in NGINX example config
Bug 32716: larger proxy_buffer_size in NGINX example config
Bug 32716: larger proxy_buffer_size in NGINX example config
Bug 32716: larger proxy_buffer_size in NGINX example config
Bug 32716: larger proxy_buffer_size in NGINX example config

Description Galen Charlton 2023-01-24 19:04:50 UTC
We recently encountered a situation with a Starman --> Apache --> NGINX setup where responses from /api/v1/patrons (particularly when doing pagination) could fail with an NGINX error message like this:

  upstream sent too big header while reading response header from upstream

We determined that on the proxy server in question, the default NGINX proxy_buffer_size value of 4096 (the PAGESIZE on that server) was a bit too small for the Link header returned by the API response.

Consequently, example NGINX configs in the code and the wiki should include the following:

  proxy_buffer_size 8k;

This is related to bug 31104, but its fix of splitting up the Link headers won't be sufficient, as NGINX will drop the response if the total size of the headers is greater than proxy_buffer_size.
Comment 3 Galen Charlton 2023-01-24 19:47:08 UTC Comment hidden (obsolete)
Comment 4 Brendan Gallagher 2023-03-29 12:32:12 UTC
Created attachment 148911 [details] [review]
Bug 32716: larger proxy_buffer_size in NGINX example config

This patch updates the example NGINX config to increase the
proxy_buffer_size to 16k. The default value of 4k (on some platforms)
has empirically been shown to be a bit too small for the Link
headers emitted by the REST API when pagination is requested.

To test
-------
[1] Set up a Koha system with NGINX as a reverse proxy in
    front of it (either in front of Apache or in front of
    of Starman).
[2] Perform a patron search that returns at least two pages
    of results and navigate to the second page.
[3] Note that the navigation can fail with a 502 HTTP error
    and an "upstream sent too big header while reading response
    header from upstream" error in the NGINX log.

    The problem is most likely when the pagesize of the server
    running NGINX is 4096 bytes.
[4] Update the NGINX configuration per this patch and restart
    NGINX.
[5] This time, repeating step 2 should work.

Signed-off-by: Galen Charlton <gmc@equinoxOLI.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 5 Marcel de Rooy 2023-04-14 06:37:05 UTC
Created attachment 149641 [details] [review]
Bug 32716: larger proxy_buffer_size in NGINX example config

This patch updates the example NGINX config to increase the
proxy_buffer_size to 16k. The default value of 4k (on some platforms)
has empirically been shown to be a bit too small for the Link
headers emitted by the REST API when pagination is requested.

To test
-------
[1] Set up a Koha system with NGINX as a reverse proxy in
    front of it (either in front of Apache or in front of
    of Starman).
[2] Perform a patron search that returns at least two pages
    of results and navigate to the second page.
[3] Note that the navigation can fail with a 502 HTTP error
    and an "upstream sent too big header while reading response
    header from upstream" error in the NGINX log.

    The problem is most likely when the pagesize of the server
    running NGINX is 4096 bytes.
[4] Update the NGINX configuration per this patch and restart
    NGINX.
[5] This time, repeating step 2 should work.

Signed-off-by: Galen Charlton <gmc@equinoxOLI.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 6 Tomás Cohen Arazi (tcohen) 2023-04-14 14:19:53 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 7 Matt Blenkinsop 2023-04-18 12:06:13 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 8 Lucas Gass (lukeg) 2023-05-05 22:11:19 UTC
Backported to 22.05.x for upcoming 22.05.13.
Comment 9 Arthur Suzuki 2023-05-23 09:57:03 UTC
applied to 21.11 for 21.11.21