Bug 32716 - update NGINX config examples to increase proxy_buffer_size
Summary: update NGINX config examples to increase proxy_buffer_size
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor
Assignee: Galen Charlton
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-01-24 19:04 UTC by Galen Charlton
Modified: 2023-12-28 20:44 UTC (History)
4 users (show)

See Also:
Change sponsored?: Unsponsored
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Set proxy_buffer_size in the example NGINX configuration to reduce chances that REST API responses that use pagination get dropped by NGINX
Version(s) released in:
23.05.00,22.11.05,22.05.13,21.11.21
Circulation function:


Attachments
Bug 32716: larger proxy_buffer_size in NGINX example config (1.82 KB, patch)
2023-01-24 19:16 UTC, Galen Charlton
Details | Diff | Splinter Review
Bug 32716: larger proxy_buffer_size in NGINX example config (1.82 KB, patch)
2023-01-24 19:46 UTC, Galen Charlton
Details | Diff | Splinter Review
Bug 32716: larger proxy_buffer_size in NGINX example config (1.82 KB, patch)
2023-01-24 19:47 UTC, Galen Charlton
Details | Diff | Splinter Review
Bug 32716: larger proxy_buffer_size in NGINX example config (1.89 KB, patch)
2023-03-29 12:32 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 32716: larger proxy_buffer_size in NGINX example config (2.00 KB, patch)
2023-04-14 06:37 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Galen Charlton 2023-01-24 19:04:50 UTC
We recently encountered a situation with a Starman --> Apache --> NGINX setup where responses from /api/v1/patrons (particularly when doing pagination) could fail with an NGINX error message like this:

  upstream sent too big header while reading response header from upstream

We determined that on the proxy server in question, the default NGINX proxy_buffer_size value of 4096 (the PAGESIZE on that server) was a bit too small for the Link header returned by the API response.

Consequently, example NGINX configs in the code and the wiki should include the following:

  proxy_buffer_size 8k;

This is related to bug 31104, but its fix of splitting up the Link headers won't be sufficient, as NGINX will drop the response if the total size of the headers is greater than proxy_buffer_size.
Comment 3 Galen Charlton 2023-01-24 19:47:08 UTC Comment hidden (obsolete)
Comment 4 Brendan Gallagher 2023-03-29 12:32:12 UTC
Created attachment 148911 [details] [review]
Bug 32716: larger proxy_buffer_size in NGINX example config

This patch updates the example NGINX config to increase the
proxy_buffer_size to 16k. The default value of 4k (on some platforms)
has empirically been shown to be a bit too small for the Link
headers emitted by the REST API when pagination is requested.

To test
-------
[1] Set up a Koha system with NGINX as a reverse proxy in
    front of it (either in front of Apache or in front of
    of Starman).
[2] Perform a patron search that returns at least two pages
    of results and navigate to the second page.
[3] Note that the navigation can fail with a 502 HTTP error
    and an "upstream sent too big header while reading response
    header from upstream" error in the NGINX log.

    The problem is most likely when the pagesize of the server
    running NGINX is 4096 bytes.
[4] Update the NGINX configuration per this patch and restart
    NGINX.
[5] This time, repeating step 2 should work.

Signed-off-by: Galen Charlton <gmc@equinoxOLI.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Comment 5 Marcel de Rooy 2023-04-14 06:37:05 UTC
Created attachment 149641 [details] [review]
Bug 32716: larger proxy_buffer_size in NGINX example config

This patch updates the example NGINX config to increase the
proxy_buffer_size to 16k. The default value of 4k (on some platforms)
has empirically been shown to be a bit too small for the Link
headers emitted by the REST API when pagination is requested.

To test
-------
[1] Set up a Koha system with NGINX as a reverse proxy in
    front of it (either in front of Apache or in front of
    of Starman).
[2] Perform a patron search that returns at least two pages
    of results and navigate to the second page.
[3] Note that the navigation can fail with a 502 HTTP error
    and an "upstream sent too big header while reading response
    header from upstream" error in the NGINX log.

    The problem is most likely when the pagesize of the server
    running NGINX is 4096 bytes.
[4] Update the NGINX configuration per this patch and restart
    NGINX.
[5] This time, repeating step 2 should work.

Signed-off-by: Galen Charlton <gmc@equinoxOLI.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 6 Tomás Cohen Arazi (tcohen) 2023-04-14 14:19:53 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 7 Matt Blenkinsop 2023-04-18 12:06:13 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 8 Lucas Gass (lukeg) 2023-05-05 22:11:19 UTC
Backported to 22.05.x for upcoming 22.05.13.
Comment 9 Arthur Suzuki 2023-05-23 09:57:03 UTC
applied to 21.11 for 21.11.21