Bug 39345 - Koha must support COUNTER 5.1
Summary: Koha must support COUNTER 5.1
Status: Pushed to stable
Alias: None
Product: Koha
Classification: Unclassified
Component: ERM (show other bugs)
Version: Main
Hardware: All All
: P3 enhancement
Assignee: Pedro Amorim
QA Contact: Testopia
URL: https://github.com/openfifth/koha/com...
Keywords: rel_24_11_candidate, rel_25_05_candidate
Depends on:
Blocks: 40188
  Show dependency treegraph
 
Reported: 2025-03-16 10:55 UTC by Mathieu Saby
Modified: 2025-09-20 08:57 UTC (History)
15 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement adds support to the ERM module for Release 5.1 of the Code of Practice for COUNTER Metrics that came into force in January 2025, with a requirement for reports to be delivered by the 28th of February 2025.
Version(s) released in:
25.11.00,25.05.04
Circulation function:


Attachments
Bug 39345: Preparation: Move original test files (7.20 KB, patch)
2025-08-20 16:08 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 support (46.28 KB, patch)
2025-08-20 16:09 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 Tests (83.01 KB, patch)
2025-08-20 16:09 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Use data provider report release to establish service URL (3.67 KB, patch)
2025-08-20 16:09 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix data provider tests (3.24 KB, patch)
2025-08-21 08:21 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix QA complaints (2.01 KB, patch)
2025-08-21 08:27 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up) Add exception handling for unsupported counter release (9.09 KB, patch)
2025-08-21 09:34 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release (9.09 KB, patch)
2025-08-21 09:35 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Preparation: Move original test files (7.25 KB, patch)
2025-08-21 19:43 UTC, Eric Phetteplace
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 support (46.32 KB, patch)
2025-08-21 19:46 UTC, Eric Phetteplace
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 Tests (83.05 KB, patch)
2025-08-21 19:46 UTC, Eric Phetteplace
Details | Diff | Splinter Review
Bug 39345: Use data provider report release to establish service URL (3.72 KB, patch)
2025-08-21 19:46 UTC, Eric Phetteplace
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix data provider tests (3.29 KB, patch)
2025-08-21 19:46 UTC, Eric Phetteplace
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix QA complaints (2.06 KB, patch)
2025-08-21 19:46 UTC, Eric Phetteplace
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release (9.13 KB, patch)
2025-08-21 19:46 UTC, Eric Phetteplace
Details | Diff | Splinter Review
Bug 39345: Preparation: Move original test files (7.31 KB, patch)
2025-08-29 13:38 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 support (46.38 KB, patch)
2025-08-29 13:38 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 Tests (83.12 KB, patch)
2025-08-29 13:38 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39345: Use data provider report release to establish service URL (3.78 KB, patch)
2025-08-29 13:39 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix data provider tests (3.35 KB, patch)
2025-08-29 13:39 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix QA complaints (2.12 KB, patch)
2025-08-29 13:39 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release (9.20 KB, patch)
2025-08-29 13:39 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
[24.11.x] Bug 39345: 24.11 SQUASH (147.84 KB, patch)
2025-09-01 11:58 UTC, Pedro Amorim
Details | Diff | Splinter Review
[24.11.x] Bug 39345: 24.11 SQUASH (163.24 KB, patch)
2025-09-02 12:53 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix erm_custom_reports.t (15.74 KB, patch)
2025-09-02 13:04 UTC, Pedro Amorim
Details | Diff | Splinter Review
[24.11.x] Bug 39345: 24.11 SQUASH (163.24 KB, patch)
2025-09-02 13:05 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Preparation: Move original test files (7.31 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 support (46.44 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Add COUNTER 5.1 Tests (83.12 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: Use data provider report release to establish service URL (3.78 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix data provider tests (3.35 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix QA complaints (2.12 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release (9.20 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 39345: (QA follow-up): Fix erm_custom_reports.t (15.74 KB, patch)
2025-09-03 09:41 UTC, Pedro Amorim
Details | Diff | Splinter Review
[24.11.x] Bug 39345: 24.11 SQUASH (163.36 KB, patch)
2025-09-03 09:48 UTC, Pedro Amorim
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Saby 2025-03-16 10:55:14 UTC
Release 5.1 of the Code of Practice came into force in January 2025, with a requirement for reports to be delivered by the 28th of February. (https://www.countermetrics.org/live-r51/?searched=r5%20harveste)

See for for the details : 
https://www.countermetrics.org/education/whats-new/
https://cop5.projectcounter.org/en/5.1/appendices/b-changes-from-r5.html
Comment 1 Jessie Zairo 2025-07-29 12:05:18 UTC
+1, yes, there are about 175 publishers, data hosts, and platforms that have migrated to R5.1.
Comment 2 Aleisha Amohia 2025-08-07 02:59:41 UTC
Increasing the importance of this.
Comment 3 Eric Phetteplace 2025-08-14 15:40:05 UTC
Just adding that since some vendors are dropping COUNTER 5 support (EBSCO, for instance) it makes it impossible to pull their usage statistics until Koha supports the 5.1 releases.
Comment 4 Pedro Amorim 2025-08-20 16:08:58 UTC
Created attachment 185593 [details] [review]
Bug 39345: Preparation: Move original test files

We are now considering more than one COUNTER release. Move test files accordingly.

prove t/db_dependent/Koha/ERM/EUsage/*
Comment 5 Pedro Amorim 2025-08-20 16:09:01 UTC
Created attachment 185594 [details] [review]
Bug 39345: Add COUNTER 5.1 support

This moves the original SUSHI COUNTER logic to a new
Koha::ERM::EUsage::COUNTER::5 class that extends
Koha::ERM::EUsage::SushiCounter.

Adds a new Koha::ERM::EUsage::COUNTER::5_1 class with COUNTER 5.1
specific logic.

Keeps common logic in Koha::ERM::EUsage::SushiCounter.

This ensures we support both COUNTER 5.0 and COUNTER 5.1, but should
hopefully help us better maintain COUNTER 5.1 going forward, without having
to worry about COUNTER 5.0, as the code is seperated.
This should also better set us up for adding support for possible future COUNTER releases.

Test plan, k-t-d:
1) Enable ERMModule
2) Add a new usage data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/add
3) Add the required credential parameters e.g. Customer ID and Requestor ID
4) Update the 'Report release' to '5.1', no single quote.
   (If you add anything other than '5', it'll default to '5.1')
5) Visit the list of data providers and click the 'Run now' button for your data provider:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers
6) Click the background job link at the top. Verify the background job runs as expected.
7) After the successful harvest, run a custom report at, ensure the data is all correct:
   <staff_url>/cgi-bin/koha/erm/eusage/reports
8) Visit the data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/<provider_id>
9) Click the 'Import logs' tab, download the COUNTER 5.1 counter file.
   Open the file and verify the data is correct.
Comment 6 Pedro Amorim 2025-08-20 16:09:04 UTC
Created attachment 185595 [details] [review]
Bug 39345: Add COUNTER 5.1 Tests

JSON data files extracted from sample responses at:
https://counter5.cambridge.org/r51/sushi-docs/#/

Additional docs referenced:
https://cop5.projectcounter.org/en/5.1/appendices/g-sample-counter-reports-and-standard-views.html

Run:
prove t/db_dependent/Koha/ERM/EUsage/*
Comment 7 Pedro Amorim 2025-08-20 16:09:07 UTC
Created attachment 185596 [details] [review]
Bug 39345: Use data provider report release to establish service URL
Comment 8 Eric Phetteplace 2025-08-20 17:37:57 UTC
I will test this today but also with steps to verify no regression to COUNTER v5 providers.
Comment 9 Eric Phetteplace 2025-08-20 17:52:26 UTC Comment hidden (obsolete)
Comment 10 Pedro Amorim 2025-08-21 08:21:37 UTC
Created attachment 185618 [details] [review]
Bug 39345: (QA follow-up): Fix data provider tests

Update original tests to consider COUNTER 5 specifically.
Add new test for COUNTER 5.1.
Comment 11 Pedro Amorim 2025-08-21 08:27:42 UTC
Created attachment 185619 [details] [review]
Bug 39345: (QA follow-up): Fix QA complaints

QA script still fails but I believe these are false positives caused by the tests files being moved
Comment 12 Pedro Amorim 2025-08-21 09:34:44 UTC Comment hidden (obsolete)
Comment 13 Pedro Amorim 2025-08-21 09:35:20 UTC
Created attachment 185621 [details] [review]
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release

To test this on the UI, you can add the following line of code at the beginning on 'new' in SushiCounter.pm:
->{response}->{Report_Header}->{Release} = 6;

This will force the release to be '6'. Then you can perform a harvest using the UI. The background job will fail with the error message saying that counter release 6 is not supported

prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t
Comment 14 Eric Phetteplace 2025-08-21 19:43:55 UTC
Created attachment 185658 [details] [review]
Bug 39345: Preparation: Move original test files

We are now considering more than one COUNTER release. Move test files accordingly.

prove t/db_dependent/Koha/ERM/EUsage/*

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Comment 15 Eric Phetteplace 2025-08-21 19:46:17 UTC
Created attachment 185659 [details] [review]
Bug 39345: Add COUNTER 5.1 support

This moves the original SUSHI COUNTER logic to a new
Koha::ERM::EUsage::COUNTER::5 class that extends
Koha::ERM::EUsage::SushiCounter.

Adds a new Koha::ERM::EUsage::COUNTER::5_1 class with COUNTER 5.1
specific logic.

Keeps common logic in Koha::ERM::EUsage::SushiCounter.

This ensures we support both COUNTER 5.0 and COUNTER 5.1, but should
hopefully help us better maintain COUNTER 5.1 going forward, without having
to worry about COUNTER 5.0, as the code is seperated.
This should also better set us up for adding support for possible future COUNTER releases.

Test plan, k-t-d:
1) Enable ERMModule
2) Add a new usage data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/add
3) Add the required credential parameters e.g. Customer ID and Requestor ID
4) Update the 'Report release' to '5.1', no single quote.
   (If you add anything other than '5', it'll default to '5.1')
5) Visit the list of data providers and click the 'Run now' button for your data provider:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers
6) Click the background job link at the top. Verify the background job runs as expected.
7) After the successful harvest, run a custom report at, ensure the data is all correct:
   <staff_url>/cgi-bin/koha/erm/eusage/reports
8) Visit the data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/<provider_id>
9) Click the 'Import logs' tab, download the COUNTER 5.1 counter file.
   Open the file and verify the data is correct.

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Comment 16 Eric Phetteplace 2025-08-21 19:46:23 UTC
Created attachment 185660 [details] [review]
Bug 39345: Add COUNTER 5.1 Tests

JSON data files extracted from sample responses at:
https://counter5.cambridge.org/r51/sushi-docs/#/

Additional docs referenced:
https://cop5.projectcounter.org/en/5.1/appendices/g-sample-counter-reports-and-standard-views.html

Run:
prove t/db_dependent/Koha/ERM/EUsage/*

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Comment 17 Eric Phetteplace 2025-08-21 19:46:27 UTC
Created attachment 185661 [details] [review]
Bug 39345: Use data provider report release to establish service URL

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Comment 18 Eric Phetteplace 2025-08-21 19:46:30 UTC
Created attachment 185662 [details] [review]
Bug 39345: (QA follow-up): Fix data provider tests

Update original tests to consider COUNTER 5 specifically.
Add new test for COUNTER 5.1.

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Comment 19 Eric Phetteplace 2025-08-21 19:46:34 UTC
Created attachment 185663 [details] [review]
Bug 39345: (QA follow-up): Fix QA complaints

QA script still fails but I believe these are false positives caused by the tests files being moved

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Comment 20 Eric Phetteplace 2025-08-21 19:46:38 UTC
Created attachment 185664 [details] [review]
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release

To test this on the UI, you can add the following line of code at the beginning on 'new' in SushiCounter.pm:
->{response}->{Report_Header}->{Release} = 6;

This will force the release to be '6'. Then you can perform a harvest using the UI. The background job will fail with the error message saying that counter release 6 is not supported

prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Comment 21 Nick Clemens (kidclamp) 2025-08-29 13:38:52 UTC
Created attachment 185922 [details] [review]
Bug 39345: Preparation: Move original test files

We are now considering more than one COUNTER release. Move test files accordingly.

prove t/db_dependent/Koha/ERM/EUsage/*

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 22 Nick Clemens (kidclamp) 2025-08-29 13:38:55 UTC
Created attachment 185923 [details] [review]
Bug 39345: Add COUNTER 5.1 support

This moves the original SUSHI COUNTER logic to a new
Koha::ERM::EUsage::COUNTER::5 class that extends
Koha::ERM::EUsage::SushiCounter.

Adds a new Koha::ERM::EUsage::COUNTER::5_1 class with COUNTER 5.1
specific logic.

Keeps common logic in Koha::ERM::EUsage::SushiCounter.

This ensures we support both COUNTER 5.0 and COUNTER 5.1, but should
hopefully help us better maintain COUNTER 5.1 going forward, without having
to worry about COUNTER 5.0, as the code is seperated.
This should also better set us up for adding support for possible future COUNTER releases.

Test plan, k-t-d:
1) Enable ERMModule
2) Add a new usage data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/add
3) Add the required credential parameters e.g. Customer ID and Requestor ID
4) Update the 'Report release' to '5.1', no single quote.
   (If you add anything other than '5', it'll default to '5.1')
5) Visit the list of data providers and click the 'Run now' button for your data provider:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers
6) Click the background job link at the top. Verify the background job runs as expected.
7) After the successful harvest, run a custom report at, ensure the data is all correct:
   <staff_url>/cgi-bin/koha/erm/eusage/reports
8) Visit the data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/<provider_id>
9) Click the 'Import logs' tab, download the COUNTER 5.1 counter file.
   Open the file and verify the data is correct.

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 23 Nick Clemens (kidclamp) 2025-08-29 13:38:59 UTC
Created attachment 185924 [details] [review]
Bug 39345: Add COUNTER 5.1 Tests

JSON data files extracted from sample responses at:
https://counter5.cambridge.org/r51/sushi-docs/#/

Additional docs referenced:
https://cop5.projectcounter.org/en/5.1/appendices/g-sample-counter-reports-and-standard-views.html

Run:
prove t/db_dependent/Koha/ERM/EUsage/*

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 24 Nick Clemens (kidclamp) 2025-08-29 13:39:02 UTC
Created attachment 185925 [details] [review]
Bug 39345: Use data provider report release to establish service URL

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 25 Nick Clemens (kidclamp) 2025-08-29 13:39:05 UTC
Created attachment 185926 [details] [review]
Bug 39345: (QA follow-up): Fix data provider tests

Update original tests to consider COUNTER 5 specifically.
Add new test for COUNTER 5.1.

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 26 Nick Clemens (kidclamp) 2025-08-29 13:39:09 UTC
Created attachment 185927 [details] [review]
Bug 39345: (QA follow-up): Fix QA complaints

QA script still fails but I believe these are false positives caused by the tests files being moved

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 27 Nick Clemens (kidclamp) 2025-08-29 13:39:12 UTC
Created attachment 185928 [details] [review]
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release

To test this on the UI, you can add the following line of code at the beginning on 'new' in SushiCounter.pm:
->{response}->{Report_Header}->{Release} = 6;

This will force the release to be '6'. Then you can perform a harvest using the UI. The background job will fail with the error message saying that counter release 6 is not supported

prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 28 Katrin Fischer 2025-08-29 14:14:50 UTC
This is marked as enhancement, but please consider for backporting to 24.11 LTS at least. Thank you!
Comment 29 Pedro Amorim 2025-09-01 09:06:30 UTC Comment hidden (obsolete)
Comment 30 Pedro Amorim 2025-09-01 11:58:55 UTC
Created attachment 186016 [details] [review]
[24.11.x] Bug 39345: 24.11 SQUASH

prove t/db_dependent/api/v1/erm_usa*
prove t/db_dependent/Koha/ERM/EUsage/*
prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 31 Pedro Amorim 2025-09-02 12:04:22 UTC
This is causing the following tests to fail:
prove t/db_dependent/api/v1/erm_custom_reports.t

I'm working on it.
Comment 32 Pedro Amorim 2025-09-02 12:53:26 UTC Comment hidden (obsolete)
Comment 33 Pedro Amorim 2025-09-02 13:04:00 UTC
Created attachment 186043 [details] [review]
Bug 39345: (QA follow-up): Fix erm_custom_reports.t

prove t/db_dependent/api/v1/erm_custom_reports.t
Comment 34 Pedro Amorim 2025-09-02 13:05:10 UTC
Created attachment 186044 [details] [review]
[24.11.x] Bug 39345: 24.11 SQUASH

prove t/db_dependent/api/v1/erm_usa*
prove t/db_dependent/Koha/ERM/EUsage/*
prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t
prove t/db_dependent/api/v1/erm_custom_reports.t

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 35 Pedro Amorim 2025-09-02 13:05:40 UTC
Submitted follow up patch for erm_custom_reports.t + added it to the squashed 24.11.x patch.
Comment 36 Pedro Amorim 2025-09-03 09:41:31 UTC
Created attachment 186060 [details] [review]
Bug 39345: Preparation: Move original test files

We are now considering more than one COUNTER release. Move test files accordingly.

prove t/db_dependent/Koha/ERM/EUsage/*

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 37 Pedro Amorim 2025-09-03 09:41:34 UTC
Created attachment 186061 [details] [review]
Bug 39345: Add COUNTER 5.1 support

This moves the original SUSHI COUNTER logic to a new
Koha::ERM::EUsage::COUNTER::5 class that extends
Koha::ERM::EUsage::SushiCounter.

Adds a new Koha::ERM::EUsage::COUNTER::5_1 class with COUNTER 5.1
specific logic.

Keeps common logic in Koha::ERM::EUsage::SushiCounter.

This ensures we support both COUNTER 5.0 and COUNTER 5.1, but should
hopefully help us better maintain COUNTER 5.1 going forward, without having
to worry about COUNTER 5.0, as the code is seperated.
This should also better set us up for adding support for possible future COUNTER releases.

Test plan, k-t-d:
1) Enable ERMModule
2) Add a new usage data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/add
3) Add the required credential parameters e.g. Customer ID and Requestor ID
4) Update the 'Report release' to '5.1', no single quote.
   (If you add anything other than '5', it'll default to '5.1')
5) Visit the list of data providers and click the 'Run now' button for your data provider:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers
6) Click the background job link at the top. Verify the background job runs as expected.
7) After the successful harvest, run a custom report at, ensure the data is all correct:
   <staff_url>/cgi-bin/koha/erm/eusage/reports
8) Visit the data provider at:
   <staff_url>/cgi-bin/koha/erm/eusage/usage_data_providers/<provider_id>
9) Click the 'Import logs' tab, download the COUNTER 5.1 counter file.
   Open the file and verify the data is correct.

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 38 Pedro Amorim 2025-09-03 09:41:37 UTC
Created attachment 186062 [details] [review]
Bug 39345: Add COUNTER 5.1 Tests

JSON data files extracted from sample responses at:
https://counter5.cambridge.org/r51/sushi-docs/#/

Additional docs referenced:
https://cop5.projectcounter.org/en/5.1/appendices/g-sample-counter-reports-and-standard-views.html

Run:
prove t/db_dependent/Koha/ERM/EUsage/*

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 39 Pedro Amorim 2025-09-03 09:41:40 UTC
Created attachment 186063 [details] [review]
Bug 39345: Use data provider report release to establish service URL

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 40 Pedro Amorim 2025-09-03 09:41:43 UTC
Created attachment 186064 [details] [review]
Bug 39345: (QA follow-up): Fix data provider tests

Update original tests to consider COUNTER 5 specifically.
Add new test for COUNTER 5.1.

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 41 Pedro Amorim 2025-09-03 09:41:46 UTC
Created attachment 186065 [details] [review]
Bug 39345: (QA follow-up): Fix QA complaints

QA script still fails but I believe these are false positives caused by the tests files being moved

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 42 Pedro Amorim 2025-09-03 09:41:49 UTC
Created attachment 186066 [details] [review]
Bug 39345: (QA follow-up): Add exception handling for unsupported counter release

To test this on the UI, you can add the following line of code at the beginning on 'new' in SushiCounter.pm:
->{response}->{Report_Header}->{Release} = 6;

This will force the release to be '6'. Then you can perform a harvest using the UI. The background job will fail with the error message saying that counter release 6 is not supported

prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 43 Pedro Amorim 2025-09-03 09:41:52 UTC
Created attachment 186067 [details] [review]
Bug 39345: (QA follow-up): Fix erm_custom_reports.t

prove t/db_dependent/api/v1/erm_custom_reports.t
Comment 44 Pedro Amorim 2025-09-03 09:45:04 UTC
Main patches rebased on top bug 40579
Comment 45 Pedro Amorim 2025-09-03 09:48:22 UTC
Created attachment 186068 [details] [review]
[24.11.x] Bug 39345: 24.11 SQUASH

The following tests must pass:

prove t/db_dependent/api/v1/erm_usa*
prove t/db_dependent/Koha/ERM/EUsage/*
prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t
prove t/db_dependent/api/v1/erm_custom_reports.t
prove xt/author/Text_CSV_Various.t

Signed-off-by: Eric Phetteplace <phette23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 46 Pedro Amorim 2025-09-03 09:48:50 UTC
24.11.x patch rebased on top of bug 40579
Comment 47 Lucas Gass (lukeg) 2025-09-03 19:41:03 UTC
Pedro, can you double check this test:

/kohadevbox/koha/t/db_dependent/Koha/ERM/EUsage/CounterFile.t

It's failing for me locally.
Comment 48 Pedro Amorim 2025-09-04 08:26:37 UTC
(In reply to Lucas Gass (lukeg) from comment #47)
> Pedro, can you double check this test:
> 
> /kohadevbox/koha/t/db_dependent/Koha/ERM/EUsage/CounterFile.t
> 
> It's failing for me locally.

Hey, double-checking here, t/db_dependent/Koha/ERM/EUsage/CounterFile.t shouldn't exist anymore, it's been split into 2 different files:
t/db_dependent/Koha/ERM/EUsage/COUNTER_5/CounterFile.t 
t/db_dependent/Koha/ERM/EUsage/COUNTER_5.1/CounterFile.t

After applying the patchset, all relevant tests (that I'm aware of) are passing for me:

prove t/db_dependent/api/v1/erm_usa*
prove t/db_dependent/Koha/ERM/EUsage/*
prove t/db_dependent/Koha/BackgroundJobs/ErmSushiHarvester.t
prove t/db_dependent/api/v1/erm_custom_reports.t
prove xt/author/Text_CSV_Various.t
Comment 49 Lucas Gass (lukeg) 2025-09-05 21:16:17 UTC
Nice work everyone!

Pushed to main for 25.11
Comment 50 Paul Derscheid 2025-09-20 08:57:39 UTC
Nice work everyone!

Pushed to 25.05.x