Bug 32934 - SIP checkouts using "no block" flag have a calculated due rather than the specified due date
Summary: SIP checkouts using "no block" flag have a calculated due rather than the spe...
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: SIP2 (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Lari Strand
QA Contact: Kyle M Hall (khall)
URL:
Keywords: release-notes-needed
Depends on:
Blocks:
 
Reported: 2023-02-09 19:48 UTC by Kyle M Hall (khall)
Modified: 2025-07-31 23:27 UTC (History)
7 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
25.11.00,25.05.02,24.11.08
Circulation function:


Attachments
Bug 32934: Correctly honor the no block due date for SIP checkout messages (1.37 KB, patch)
2023-02-09 19:58 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 32934: Correctly honor the no block due date for SIP checkout messages (2.33 KB, patch)
2023-02-10 11:42 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 32934: Correctly honor the no block due date for SIP checkout messages (2.17 KB, patch)
2024-03-05 10:40 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 32934: Correctly honor the no block due date for SIP checkout messages (2.26 KB, patch)
2024-10-04 12:20 UTC, Lari Strand
Details | Diff | Splinter Review
Bug 32934: Correctly honor the no block due date for SIP checkout messages (2.31 KB, patch)
2024-10-04 13:40 UTC, Brendan Lawlor
Details | Diff | Splinter Review
Bug 32934: Correctly honor the no block due date for SIP checkout messages (2.38 KB, patch)
2024-10-09 11:00 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 32934: (QA follow-up) Tidy code for qa script (6.07 KB, patch)
2024-10-09 11:00 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 32934: Unit tests (3.73 KB, patch)
2025-07-08 12:15 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 32934: Correctly honor the no block due date for SIP checkout messages (4.73 KB, patch)
2025-07-08 12:15 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 32934: Unit tests (5.07 KB, patch)
2025-07-08 15:28 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 32934: Correctly honor the no block due date for SIP checkout messages (4.73 KB, patch)
2025-07-08 15:28 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Kyle M Hall (khall) 2023-02-09 19:48:06 UTC
SIP checkouts using the "no block" flag will cause the due date to wrongly be used as the transaction date, and the due date will instead be calculated using the circulation rules. In addition, the due date field in the SIP checkout response ( AH ) will be empty.
Comment 1 Kyle M Hall (khall) 2023-02-09 19:58:28 UTC
Created attachment 146450 [details] [review]
Bug 32934: Correctly honor the no block due date for SIP checkout messages
Comment 2 Kyle M Hall (khall) 2023-02-10 11:42:49 UTC
Created attachment 146467 [details] [review]
Bug 32934: Correctly honor the no block due date for SIP checkout messages
Comment 3 Lari Strand 2024-03-05 08:54:19 UTC
Patch doesn't apply to master/23x. Applied it to 22.11 and seems to be working ok.
Comment 4 Kyle M Hall (khall) 2024-03-05 10:40:59 UTC
Created attachment 162737 [details] [review]
Bug 32934: Correctly honor the no block due date for SIP checkout messages
Comment 5 David Nind 2024-03-31 22:08:32 UTC
Could you provide a test plan? I'm not that familiar with SIP and need example commands.
Comment 6 Kyle M Hall (khall) 2024-04-11 15:26:52 UTC
(In reply to David Nind from comment #5)
> Could you provide a test plan? I'm not that familiar with SIP and need
> example commands.

Test Plan ( assumes KTD with default data ):
1) Connect to SIP server via Telnet, send the following messages:
9300CNterm1|COterm1|CPCPL|
11YY20240411    14540120270411    145401AOCPL|AA23529001000463|AB39999000001259|ACterm1|BIN|
2) Check Edna's checkouts, not the checkout is not due in 2027
3) Apply this patch, restart the SIP server
4) Check in the item to remove it from Edna's record
5) Repeat step 1
6) The checkout is now due in 2027 as expected!
Comment 7 Brendan Lawlor 2024-04-18 15:22:18 UTC
Is there something you need to do to start the SIP server in KTD?

I read these docs: https://wiki.koha-community.org/wiki/Koha_SIP2_server_setup

I tried running this command which appears to have done something:
kohadev-koha@kohadevbox:koha(master)$ perl C4/S
IP/SIPServer.pm /etc/koha/SIPconfig.xml &
[1] 25565

But then when I try to connect via telnet it fails:
$ telnet localhost 6001
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

I ran the telnet command from a different terminal because ktd doesn't seem to include telnet.

I checked SIPconfig.xml and see the config for the user term1, and confirmed that user exists when I run KTD. I think I am missing a step or doing something wrong.
Comment 8 Kyle M Hall (khall) 2024-04-18 15:43:04 UTC
(In reply to Brendan Lawlor from comment #7)
> Is there something you need to do to start the SIP server in KTD?
> 
> I read these docs:
> https://wiki.koha-community.org/wiki/Koha_SIP2_server_setup
> 
> I tried running this command which appears to have done something:
> kohadev-koha@kohadevbox:koha(master)$ perl C4/S
> IP/SIPServer.pm /etc/koha/SIPconfig.xml &
> [1] 25565
> 
> But then when I try to connect via telnet it fails:
> $ telnet localhost 6001
> Trying 127.0.0.1...
> telnet: Unable to connect to remote host: Connection refused
> 
> I ran the telnet command from a different terminal because ktd doesn't seem
> to include telnet.
> 
> I checked SIPconfig.xml and see the config for the user term1, and confirmed
> that user exists when I run KTD. I think I am missing a step or doing
> something wrong.

In KTD, you can run the bash alias "restart_all". That should do it for you! KTD does indeed not include telnet but you can install it by running "sudo apt update && sudo apt install -y telnet".
Comment 9 Brendan Lawlor 2024-04-18 16:38:49 UTC
Thanks for the reply Kyle!

Unfortunately I still can't connect after running restart_all

kohadev-koha@kohadevbox:koha(master)$ telnet localhost 6001
Trying 127.0.0.1...
Connection failed: Connection refused
Trying ::1...
telnet: Unable to connect to remote host: Cannot assign requested address

I also tried using this sip emulator but also got denied:

kohadev-koha@kohadevbox:koha(master)$ ./misc/sip_cli_emulator.pl -a 127.0.0.1 -p 6001 -su term1 -sp term1 \
  -l CPL --patron 23529001000463 -m checkout --item 39999000001259
Attempting socket connection to 127.0.0.1:6001...failed! : Connection refused

I am using WSL in case that's relevant.
Comment 10 Brendan Lawlor 2024-04-19 15:25:31 UTC
Test Notes:

Add port mapping to ktd docker-compose-lihgt.yml
services: koha: ports: - "6001:6001"

In ktd --shell install telnet
sudo apt update && sudo apt install -y telnet

Check that SIP server is running:
sudo koha-sip --status kohadev

Connect to SIP server and send messages:
kohadev-koha@kohadevbox:koha(main)$ telnet localhost 6001
Trying ::1...
Connection failed: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
9300CNterm1|COterm1|CPCPL|
941
11YY20240411    14540120270411    145401AOCPL|AA23529001000463|AB39999000001259|ACterm1|BIN|
121NNY20240419    151631AOCPL|AA23529001000463|AB39999000001259|AJDream of fair to middling women /|AH|
^]
96
Connection closed by foreign host.

Check Edna's checkouts, checkout date is 4/11/2027 and due date is 4/16/2027
Check the item in

Apply patch. The patch applies without an error, but then there's an issue when you try to restart_all

kohadev-koha@kohadevbox:koha(main)$ restart_all
Restarting Apache httpd web server: apache2.
Restarting Koha ILS: koha-commonStopping Plack daemon for kohadev:.
Stopping Z39.50/SRU daemon for kohadev:.
Stopping Koha worker daemon for kohadev (default):.
Stopping Koha worker daemon for kohadev (long_tasks):.
Stopping Koha indexing daemon for kohadev:.
Successfully stopped all services for instance koha-common:.
Starting Plack daemon for kohadev:.
Starting Z39.50/SRU daemon for kohadev:Global symbol "$checkout" requires explicit package name (did you forget to declare "my $checkout"?) at /kohadevbox/koha/C4/Circulation.pm line 4307.
Compilation failed in require at /kohadevbox/koha/Koha/Items.pm line 27.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Items.pm line 27.
Compilation failed in require at /kohadevbox/koha/Koha/Statistic.pm line 25.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Statistic.pm line 25.
Compilation failed in require at /kohadevbox/koha/Koha/Statistics.pm line 23.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Statistics.pm line 23.
Compilation failed in require at /kohadevbox/koha/C4/Stats.pm line 22.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Stats.pm line 22.
Compilation failed in require at /kohadevbox/koha/C4/Accounts.pm line 23.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Accounts.pm line 23.
Compilation failed in require at /kohadevbox/koha/C4/Overdues.pm line 30.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Overdues.pm line 30.
Compilation failed in require at /kohadevbox/koha/C4/Members.pm line 27.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Members.pm line 27.
Compilation failed in require at /kohadevbox/koha/C4/Letters.pm line 29.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Letters.pm line 29.
Compilation failed in require at /kohadevbox/koha/Koha/Auth/TwoFactorAuth.pm line 23.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Auth/TwoFactorAuth.pm line 23.
Compilation failed in require at /kohadevbox/koha/C4/Auth.pm line 40.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Auth.pm line 40.
Compilation failed in require at /kohadevbox/koha/C4/Serials.pm line 37.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Serials.pm line 37.
Compilation failed in require at /kohadevbox/koha/Koha/Biblio.pm line 29.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Biblio.pm line 29.
Compilation failed in require at /kohadevbox/koha/Koha/Biblios.pm line 25.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Biblios.pm line 25.
Compilation failed in require at /kohadevbox/koha/Koha/Libraries.pm line 25.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Libraries.pm line 25.
Compilation failed in require at /kohadevbox/koha/Koha/Object/Limit/Library.pm line 22.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Object/Limit/Library.pm line 22.
Compilation failed in require at /usr/lib/x86_64-linux-gnu/perl-base/base.pm line 135.
        ...propagated at /usr/lib/x86_64-linux-gnu/perl-base/base.pm line 157.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/AuthorisedValue.pm line 25.
Compilation failed in require at /kohadevbox/koha/Koha/AuthorisedValues.pm line 25.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/AuthorisedValues.pm line 25.
Compilation failed in require at /kohadevbox/koha/C4/Koha.pm line 27.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Koha.pm line 27.
Compilation failed in require at /kohadevbox/koha/C4/Biblio.pm line 88.
BEGIN failed--compilation aborted at /kohadevbox/koha/C4/Biblio.pm line 88.
Compilation failed in require at /kohadevbox/koha/Koha/Z3950Responder.pm line 22.
BEGIN failed--compilation aborted at /kohadevbox/koha/Koha/Z3950Responder.pm line 22.
Compilation failed in require at /kohadevbox/koha/misc/z3950_responder.pl line 27.
BEGIN failed--compilation aborted at /kohadevbox/koha/misc/z3950_responder.pl line 27.
 failed!
Starting Koha worker daemon for kohadev (default):.
Starting Koha worker daemon for kohadev (long_tasks):.
Starting Koha indexing daemon for kohadev:.
.
Comment 11 Lari Strand 2024-10-04 12:20:12 UTC
Created attachment 172389 [details] [review]
Bug 32934: Correctly honor the no block due date for SIP checkout messages
Comment 12 Brendan Lawlor 2024-10-04 13:40:04 UTC
Created attachment 172399 [details] [review]
Bug 32934: Correctly honor the no block due date for SIP checkout messages

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Comment 13 Brendan Lawlor 2024-10-04 13:40:29 UTC
Test notes:

Before applying the patch I got:
Checked out on	04/11/2027 14:54
Due date        04/16/2027

After applying patch I got:
Checked out on  10/04/2024 13:36
Due date        04/11/2027 14:54
Comment 14 Kyle M Hall (khall) 2024-10-09 11:00:50 UTC
Created attachment 172566 [details] [review]
Bug 32934: Correctly honor the no block due date for SIP checkout messages

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 15 Kyle M Hall (khall) 2024-10-09 11:00:57 UTC
Created attachment 172567 [details] [review]
Bug 32934: (QA follow-up) Tidy code for qa script

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 16 Katrin Fischer 2024-10-11 09:45:14 UTC
There is no explanation of the change in the commit message and there are no unit tests. Can you please improve?

For me it's not clear right now what the correct behavior is: Should the due date if included in the SIP message only be processed if the block flag is set or always? I had assumed it should only use it when the flag is set. 

Please add some explanation!
Comment 17 Ryan Henderson 2025-06-20 08:49:33 UTC
Just to add to this, we've recently encountered this issue with one of our partners.  

For us the issue occurred where an offline transaction file was uploaded to the Koha server once the SIP2 service was brought online again.  However, the SIP2 service would ignore the checkout date, and would instead take the due date as the checkout.  

This obviously caused a few head scratches, but whenever we applied these patches to our test server the SIP2 service started to honour the checkout date properly along with the due date.
Comment 18 Martin Renvoize (ashimema) 2025-07-08 12:15:56 UTC Comment hidden (obsolete)
Comment 19 Martin Renvoize (ashimema) 2025-07-08 12:15:58 UTC Comment hidden (obsolete)
Comment 20 Martin Renvoize (ashimema) 2025-07-08 15:28:07 UTC
Created attachment 183869 [details] [review]
Bug 32934: Unit tests
Comment 21 Martin Renvoize (ashimema) 2025-07-08 15:28:09 UTC
Created attachment 183870 [details] [review]
Bug 32934: Correctly honor the no block due date for SIP checkout messages

**Problem:**
When using SIP checkout messages with the "no block" flag, the specified due date
was being ignored. Instead of honoring the provided due date, the system would:
1. Use the due date as the transaction timestamp (incorrect behavior)
2. Calculate a new due date based on circulation rules
3. Return an empty due date field (AH) in the SIP response

This affected SIP2 integrations where external systems needed to specify exact
due dates for checkouts, particularly for offline circulation scenarios.

**Root Cause:**
The `ProcessOfflineIssue` function in C4::Circulation was only returning a
success/failure message, but the SIP checkout transaction needed access to the
resulting checkout object to properly set the due date. The SIP Transaction::Checkout
module was calling `ProcessOfflineIssue` but couldn't retrieve the checkout details
to populate the transaction's due date field.

**Solution:**
1. Modified `ProcessOfflineIssue` to return both the status message AND the checkout
   object: `return ( "Success.", $checkout );`
2. Updated `ProcessOfflineOperation` to handle the new return format by capturing
   only the message: `( $report ) = ProcessOfflineIssue( $operation );`
3. Modified SIP Transaction::Checkout to capture both return values and use the
   checkout object to set the proper due date via `duedatefromissue`
4. Fixed the timestamp parameter to use `dt_from_string` instead of the due date

**Test Plan:**
1. **Unit Tests:**
   - Run `prove t/db_dependent/Circulation/OfflineCirculation.t` to verify
     ProcessOfflineIssue returns checkout object and respects due_date parameter
   - Run `prove t/db_dependent/SIP/Transaction.t` to verify SIP checkout honors
     no_block_due_date parameter

2. **Manual SIP Testing:**
   - Connect to SIP server via telnet on port 6001
   - Send checkout message with no block flag and specific due date:
     `111YN20250115    18000120241201    180001AP|AO|AC|AD|AB<barcode>|AA<cardnumber>|`
   - Verify response contains correct due date in AH field
   - Verify checkout record in database has the specified due date

3. **Regression Testing:**
   - Verify normal SIP checkouts (without no block) still work correctly
   - Verify offline circulation operations continue to function
   - Verify ProcessOfflineOperation still processes queued operations correctly

Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 22 Martin Renvoize (ashimema) 2025-07-08 15:28:53 UTC
Rescued
Comment 23 Lucas Gass (lukeg) 2025-07-11 21:29:09 UTC
Nice work everyone!

Pushed to main for 25.11
Comment 24 Paul Derscheid 2025-07-24 08:19:45 UTC
Nice work everyone!

Pushed to 25.05.x
Comment 25 Fridolin Somers 2025-07-31 13:12:11 UTC
Pushed to 24.11.x for 24.11.08