Bug 38382

Summary: Need a fresh connection when CSRF has expired for connexion daemon
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: Command-line UtilitiesAssignee: Nick Clemens (kidclamp) <nick>
Status: RESOLVED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: normal    
Priority: P5 - low CC: aleisha, alexbuckley, blawlor, caroline.cyr-la-rose, david, dcook, fridolin.somers, m.de.rooy, robin
Version: MainKeywords: rel_24_05_candidate
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: Trivial patch Documentation contact:
Documentation submission: Text to go in the release notes:
This fixes the OCLC Connexion import daemon (misc/bin/connexion_import_daemon.pl) - the connection was failing after the CSRF token expired (after 8 hours). It now generates a new user agent when reauthenticating when the CSRF token for the session has expired. (This is related to the CSRF changes added in Koha 24.05 to improve form security.)
Version(s) released in:
25.05.00,24.11.02,24.05.08
Circulation function:
Bug Depends on: 36192    
Bug Blocks:    
Attachments: Bug 38382: Fresh connection when connexion CSRF token expires
Sample record for import
Bug 38382: Fresh connection when connexion CSRF token expires
Bug 38382: Fresh connection when connexion CSRF token expires
Bug 38382: Fresh connection when connexion CSRF token expires

Description Nick Clemens (kidclamp) 2024-11-06 20:28:45 UTC
The CSRF token generated for Koha has a default 8 hour expiration. The connexion daemon tries to reauthnticate when it expires, however, it uses the same user agent, which has the session in the cookie, and so the CSRF fails the check.
Comment 1 Nick Clemens (kidclamp) 2024-11-06 20:37:52 UTC
Created attachment 174096 [details] [review]
Bug 38382: Fresh connection when connexion CSRF token expires

This patch simply generates a new user agent when reauthenticating when the CSRF token
for the session has expired.

To test:
1 - Edit Koha/Token.pm, line 275
from:    { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS * 3600 ) },
to:      { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS ) },
2 - This will make your CSRF tokens expire after 8 seconds
3 - vim /etc/koha/sites/kohadev/connexion.cnf
4 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
5 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
6 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
7 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
8 - Successful request
9 - Wait 10 seconds (more than 8 anyways)
10 - Repeat 7
11 - Unsuccessful request - Invalid CSRF token
12 - Ctrl+C to stop the connexion daemon
13 - Apply patch
14 - repeat 6
15 - repeat 7, success
16 - Wait more than 8 seconds
17 - repeat 7, success!
18 - repeat again until you are satisfied
19 - Sign off!
Comment 2 Nick Clemens (kidclamp) 2024-11-06 20:47:51 UTC
Created attachment 174097 [details]
Sample record for import
Comment 3 Brendan Lawlor 2024-11-06 21:45:55 UTC
Created attachment 174098 [details] [review]
Bug 38382: Fresh connection when connexion CSRF token expires

This patch simply generates a new user agent when reauthenticating when the CSRF token
for the session has expired.

To test:
1 - Edit Koha/Token.pm, line 275
from:    { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS * 3600 ) },
to:      { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS ) },
2 - This will make your CSRF tokens expire after 8 seconds
3 - vim /etc/koha/sites/kohadev/connexion.cnf
4 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
5 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
6 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
7 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
8 - Successful request
9 - Wait 10 seconds (more than 8 anyways)
10 - Repeat 7
11 - Unsuccessful request - Invalid CSRF token
12 - Ctrl+C to stop the connexion daemon
13 - Apply patch
14 - repeat 6
15 - repeat 7, success
16 - Wait more than 8 seconds
17 - repeat 7, success!
18 - repeat again until you are satisfied
19 - Sign off!

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Comment 4 Brendan Lawlor 2024-11-06 21:51:42 UTC
Test notes:

This patch works as described.
Updated Token.pm MaxAge

Before patch after 8 seconds resopnses would come back like
Wed Nov  6 21:39:09 2024: HTTP/1.1 403 Forbidden
Connection: close
Date: Wed, 06 Nov 2024 21:39:09 GMT
Server: Apache/2.4.62 (Debian)
Vary: User-Agent
Content-Type: text/plain
Client-Date: Wed, 06 Nov 2024 21:39:09 GMT
Client-Peer: 127.0.0.1:8081
Client-Response-Num: 1
Client-Transfer-Encoding: chunked

Wrong CSRF token

Wed Nov  6 21:39:09 2024: Response: Unsuccessful request

After patch response would continue to come back successful:
Wed Nov  6 21:44:16 2024: U8kohauserP8kohapass00575nam  22002055  450000100070000000500170000700800410002402000180006503500180008304000140010110000300011524500250014526000200017030000200019094200120021095201070022295500210032995600080035099900110035896475820240222153924.0981015s1948    enk           000   eng d  a9780679777427  aCAT1-0666609A  aNLScKOHA1 aYerby, Frank Garvin.923514aThe vixens. A novel.  aLondon, c1948.  a326 p. ; c8vo.  2ddccBK  0010406_708REF960aMPLbMPLcGENd2014-09-04p39999000001051r2014-09-04 00:00:00w2014-09-04yBK  aVts.178.i.2bVts  aNLS  c24d24

Wed Nov  6 21:44:16 2024: Response: Success.  Batch number 9 - biblio record number 442 overlaid by import.

http://kohadev-intra.myDNSname.org:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=442
Comment 5 Brendan Lawlor 2024-11-06 21:57:24 UTC
When starting and stopping the daemon sometimes I'd get a port in use error and this command came in handy:
sudo fuser -k 8888/tcp

Also When I followed the test plan for Bug 37543 I saw the success message showing HASH(#) instead of the bib number as described in bug 38230

But when I followed this test plan the response correctly displayed the bib number.

So I guess that issue seems to be replated to config since I was sending the same requests when I was testing.
Comment 6 Marcel de Rooy 2024-11-22 08:07:24 UTC
 WARN   misc/bin/connexion_import_daemon.pl
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 0, now: 3)
Comment 7 Nick Clemens (kidclamp) 2024-12-05 16:52:53 UTC
Created attachment 175250 [details] [review]
Bug 38382: Fresh connection when connexion CSRF token expires

This patch simply generates a new user agent when reauthenticating when the CSRF token
for the session has expired.

To test:
1 - Edit Koha/Token.pm, line 275
from:    { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS * 3600 ) },
to:      { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS ) },
2 - This will make your CSRF tokens expire after 8 seconds
3 - vim /etc/koha/sites/kohadev/connexion.cnf
4 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
5 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
6 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
7 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
8 - Successful request
9 - Wait 10 seconds (more than 8 anyways)
10 - Repeat 7
11 - Unsuccessful request - Invalid CSRF token
12 - Ctrl+C to stop the connexion daemon
13 - Apply patch
14 - repeat 6
15 - repeat 7, success
16 - Wait more than 8 seconds
17 - repeat 7, success!
18 - repeat again until you are satisfied
19 - Sign off!

WNC amended and tidied

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Comment 8 Marcel de Rooy 2024-12-13 10:42:13 UTC
Created attachment 175444 [details] [review]
Bug 38382: Fresh connection when connexion CSRF token expires

This patch simply generates a new user agent when reauthenticating when the CSRF token
for the session has expired.

To test:
1 - Edit Koha/Token.pm, line 275
from:    { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS * 3600 ) },
to:      { MaxAge => $params->{MaxAge} // ( CSRF_EXPIRY_HOURS ) },
2 - This will make your CSRF tokens expire after 8 seconds
3 - vim /etc/koha/sites/kohadev/connexion.cnf
4 - Set content:
    host:
    port: 8888
    koha:http://localhost:8081
    log:/var/log/koha/kohadev/connexion.log
    match:ISBN
    user:kohauser
    password:kohapass
    overlay_action:replace
    nomatch_action:create_new
    item_action:always_add
    import_mode:direct
    framework:BKS
    overlay_framework:
    debug:1
5 - Save the sample file from this bug into your kohaclone (or copy and paste into a file your koha test site can reach)
6 - On the command line:
    perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
7 - In another terminal:
    cat bug_33418.test  | nc -v localhost 8888
8 - Successful request
9 - Wait 10 seconds (more than 8 anyways)
10 - Repeat 7
11 - Unsuccessful request - Invalid CSRF token
12 - Ctrl+C to stop the connexion daemon
13 - Apply patch
14 - repeat 6
15 - repeat 7, success
16 - Wait more than 8 seconds
17 - repeat 7, success!
18 - repeat again until you are satisfied
19 - Sign off!

WNC amended and tidied

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 9 Katrin Fischer 2024-12-23 08:38:08 UTC
Pushed for 25.05!

Well done everyone, thank you!
Comment 10 Paul Derscheid 2025-02-03 19:09:10 UTC
Nice work everyone!

Pushed to 24.11.x for 24.11.01
Comment 11 Alex Buckley 2025-02-19 00:28:15 UTC
Hi, 

1) Can you please give us a more detailed test plan?

2) Are we supposed to make the /etc/koha/sites/kohadev/connexion.cnf file?

3) We noticed that there are some strange characters in the sample record for import file ( https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=174097 ), which is giving us the following output in step 6:

kohadev-koha@kohadevbox:koha((v24.05.06))$ perl misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
no mapping found for [0x90] at position 0 in ��96475 g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0x90] at position 2 in 8␞2024022215392 g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0x90] at position 4 in 4.0␞981015s1948    enk           000    g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.


4) For step 8 we get the following, is this to be expected?

kohadev-koha@kohadevbox:koha((v24.05.06))$     cat bug_33418.test  | nc -v localhost 8888
nc: connect to localhost (::1) port 8888 (tcp) failed: Connection refused
Connection to localhost (127.0.0.1) 8888 port [tcp/*] succeeded!
Unsuccessful requestkohadev-koha@kohadevbox:koha((v24.05.06))$
Comment 12 Nick Clemens (kidclamp) 2025-03-04 14:21:17 UTC
(In reply to Alex Buckley from comment #11)
> Hi, 
> 
> 1) Can you please give us a more detailed test plan?
    Which parts are unclear? I should have mentioned creating a user with kohauser/kohapass as the credentials, or use the credentials for an existing account with the correct permissions

> 2) Are we supposed to make the /etc/koha/sites/kohadev/connexion.cnf file?
    Yes, create this file if it doesn't exist

> 3) We noticed that there are some strange characters in the sample record
> for import file (
> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=174097 ), which
> is giving us the following output in step 6:
> 
> kohadev-koha@kohadevbox:koha((v24.05.06))$ perl
> misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
> no mapping found for [0x90] at position 0 in ��96475 g0=ASCII_DEFAULT
> g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
> no mapping found for [0x90] at position 2 in 8␞2024022215392
> g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line
> 308.
> no mapping found for [0x90] at position 4 in 4.0␞981015s1948    enk         
> 000    g0=ASCII_DEFAULT g1=EXTENDED_LATIN at
> /usr/share/perl5/MARC/Charset.pm line 308.

I think this may be from copying/pasting or the way it was downloaded - you can simply export a single record from your koha in marc format and add:
U8kohauserP8kohapass
at the start - connection expects those two fields then parses the rest as marc - you will need to create a user with those credentials and permissions to edit the catalogue - or substitute your own credentials



> 4) For step 8 we get the following, is this to be expected?
> 
> kohadev-koha@kohadevbox:koha((v24.05.06))$     cat bug_33418.test  | nc -v
> localhost 8888
> nc: connect to localhost (::1) port 8888 (tcp) failed: Connection refused
> Connection to localhost (127.0.0.1) 8888 port [tcp/*] succeeded!
> Unsuccessful requestkohadev-koha@kohadevbox:koha((v24.05.06))$

No, the request shoudl be successful, but perhaps the user wasn't created?
I get:
kohadev-koha@kohadevbox:koha(main)$ cat bug_33418.test  | nc -v localhost 8888
nc: connect to localhost (::1) port 8888 (tcp) failed: Connection refused
Connection to localhost (127.0.0.1) 8888 port [tcp/*] succeeded!
Success.  Batch number 2 - biblio record number 439 overlaid by import.

http://kohadev-intra.myDNSname.org:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439kohadev-koha@kohadevbox:koha(main)$
Comment 13 Alex Buckley 2025-03-20 00:54:09 UTC
(In reply to Nick Clemens (kidclamp) from comment #12)
> (In reply to Alex Buckley from comment #11)
> > Hi, 
> > 
> > 1) Can you please give us a more detailed test plan?
>     Which parts are unclear? I should have mentioned creating a user with
> kohauser/kohapass as the credentials, or use the credentials for an existing
> account with the correct permissions
> 
> > 2) Are we supposed to make the /etc/koha/sites/kohadev/connexion.cnf file?
>     Yes, create this file if it doesn't exist
> 
> > 3) We noticed that there are some strange characters in the sample record
> > for import file (
> > https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=174097 ), which
> > is giving us the following output in step 6:
> > 
> > kohadev-koha@kohadevbox:koha((v24.05.06))$ perl
> > misc/bin/connexion_import_daemon.pl -c /etc/koha/sites/kohadev/connexion.cnf
> > no mapping found for [0x90] at position 0 in ��96475 g0=ASCII_DEFAULT
> > g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
> > no mapping found for [0x90] at position 2 in 8␞2024022215392
> > g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line
> > 308.
> > no mapping found for [0x90] at position 4 in 4.0␞981015s1948    enk         
> > 000    g0=ASCII_DEFAULT g1=EXTENDED_LATIN at
> > /usr/share/perl5/MARC/Charset.pm line 308.
> 
> I think this may be from copying/pasting or the way it was downloaded - you
> can simply export a single record from your koha in marc format and add:
> U8kohauserP8kohapass
> at the start - connection expects those two fields then parses the rest as
> marc - you will need to create a user with those credentials and permissions
> to edit the catalogue - or substitute your own credentials
> 
> 
> 
> > 4) For step 8 we get the following, is this to be expected?
> > 
> > kohadev-koha@kohadevbox:koha((v24.05.06))$     cat bug_33418.test  | nc -v
> > localhost 8888
> > nc: connect to localhost (::1) port 8888 (tcp) failed: Connection refused
> > Connection to localhost (127.0.0.1) 8888 port [tcp/*] succeeded!
> > Unsuccessful requestkohadev-koha@kohadevbox:koha((v24.05.06))$
> 
> No, the request shoudl be successful, but perhaps the user wasn't created?
> I get:
> kohadev-koha@kohadevbox:koha(main)$ cat bug_33418.test  | nc -v localhost
> 8888
> nc: connect to localhost (::1) port 8888 (tcp) failed: Connection refused
> Connection to localhost (127.0.0.1) 8888 port [tcp/*] succeeded!
> Success.  Batch number 2 - biblio record number 439 overlaid by import.
> 
> http://kohadev-intra.myDNSname.org:8081/cgi-bin/koha/catalogue/detail.
> pl?biblionumber=439kohadev-koha@kohadevbox:koha(main)$

Sorry for the delay getting back to you Nick.

Thank you for the following clarifications:
- Creating a user with kohauser/kohapass as the credentials
- Exporting a single record from Koha in marc format and adding 'U8kohauserP8kohapass'

that did the trick, thank you!

I have been able to successfully run through the test plan on 24.05.x and am backporting
Comment 14 Alex Buckley 2025-03-20 00:56:38 UTC
Nice work everyone!

Pushed to 24.05.x for 24.05.08
Comment 15 Fridolin Somers 2025-03-20 15:02:58 UTC
Not for 23.11.x
Comment 16 Caroline Cyr La Rose 2025-04-18 18:47:01 UTC
Bug fix, nothing to add/edit in the manual