Bug 34587 - Add a Usage Statistics module to ERM
Summary: Add a Usage Statistics module to ERM
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: ERM (show other bugs)
Version: Main
Hardware: All All
: P5 - low new feature (vote)
Assignee: Matt Blenkinsop
QA Contact:
URL: https://github.com/PTFS-Europe/koha/t...
Keywords:
Depends on: 34448 34468 34497
Blocks: 35229 35392 35418 35120 35218 35219 35258 35374 35375
  Show dependency treegraph
 
Reported: 2023-08-22 13:40 UTC by Matt Blenkinsop
Modified: 2023-12-12 08:44 UTC (History)
22 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Large patch
Documentation contact: Jonathan Field
Documentation submission:
Text to go in the release notes:
This feature adds the ability to create Data Providers and harvest usage data in COUNTER format. Data Providers (or Data Platforms) are the organisations who provide usage statistics for your electronic resources. This could be a vendor or a platform provider. The data provider record contains information about SUSHI credentials and any COUNTER data which has been harvested from the provider. There is also the ability to create custom reports using this COUNTER data.
Version(s) released in:
23.11.00


Attachments
queued-report-fail (131.95 KB, image/png)
2023-10-05 19:33 UTC, Jessie Zairo
Details
screenshot of problem (56.91 KB, image/png)
2023-10-13 13:52 UTC, Stephanie
Details
Bug 34587: (follow-up) Remove unnecessary code (3.06 KB, patch)
2023-11-09 17:34 UTC, Matt Blenkinsop
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Blenkinsop 2023-08-22 13:40:38 UTC
A module has been developed to support the gathering and analysis of usage statistics for e-resources in the ERM module.

Github link is attached, here is also a video to a demonstration of the module for context on how it works: https://www.youtube.com/live/Ev1tJt7yUjc?si=BwgDATEt4JlM6Yzj&t=5658
Comment 1 Matt Blenkinsop 2023-08-22 13:47:58 UTC
https://github.com/PTFS-Europe/koha/tree/bug_34587
Comment 2 Matt Blenkinsop 2023-08-22 13:53:11 UTC
This is ready for testing

Testing notes:
Credentials will be required to harvest data via SUSHI, please check if your company/organisation has any access credentials that can be used for testing
Comment 3 Matt Blenkinsop 2023-08-22 14:02:08 UTC
Unit tests are still in development and will be following shortly
Comment 4 Matt Blenkinsop 2023-09-01 12:49:39 UTC
A sandbox is now live at this link: http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/usage_data_providers

Credentials will be required for testing, please check if your organisation has access to any valid SUSHI credentials in order to test.

All credentials are redacted in the sandbox and will not be visible to other users. In the released version, credentials will be visible in the ERM module.
Comment 5 Katrin Fischer 2023-09-28 08:23:05 UTC
I feel like should advertise this a bit more to get more testing - a mail to the mailing list maybe?
Comment 6 Jonathan Druart 2023-10-03 14:40:35 UTC
(quick note)
test_connection returns 500 if credentials are not correct (I guess it's the problem).
Comment 7 Lucas Gass 2023-10-05 19:23:10 UTC
When I enable ERM and Add a new data provider manually I get a type error:

this.$refs.table.dt is not a function. For me this comes from KohaTable.vue on line 150:

let dt = this.$refs.table.dt()


I have tried restarting everything and rebuilding the js with yarn js:build.
Comment 8 Jessie Zairo 2023-10-05 19:33:20 UTC
Created attachment 156581 [details]
queued-report-fail
Comment 9 Jessie Zairo 2023-10-05 19:33:58 UTC
We successfully created a new data provider via SUSHI. We got a successful test - Harvester connection was successful for usage data provider. When we run the job for report type TR_J1, we get a message that the report goes to queue, but fails.
Comment 10 Jonathan Field 2023-10-05 19:46:32 UTC
(In reply to Jessie Zairo from comment #9)
> We successfully created a new data provider via SUSHI. We got a successful
> test - Harvester connection was successful for usage data provider. When we
> run the job for report type TR_J1, we get a message that the report goes to
> queue, but fails.

Hi Jessie, is this on our sandbox? If so, we found a problem earlier today with the background jobs which we are looking into. I’ll let you know when resolved.
Comment 11 Jonathan Field 2023-10-06 10:51:53 UTC
(In reply to Jessie Zairo from comment #9)
> We successfully created a new data provider via SUSHI. We got a successful
> test - Harvester connection was successful for usage data provider. When we
> run the job for report type TR_J1, we get a message that the report goes to
> queue, but fails.

We've fixed our sandbox now, apologies for that, we were missing some of the very latest code which is there now :)
Comment 12 Jessie Zairo 2023-10-12 12:55:56 UTC
Testing Plan
1.) Make sure the ERMModule system preference is set to Enable 
2.) Navigate to E-resource management --> eUsage --> Data providers
3.) Click + New data provider
4.) Begin typing the data provider name (vendor) that you wish to set up. If a match is found, select the data provider. If it is not found, select the Create manually button to the right. 
5.) Enter any additional information about the data provider in the description. (Not a mandatory field)
6.) Select the Harvester status to Active or Inactive. (Not a mandatory field) 
7.) Select the report type from the drop-down menu. 
8.) Under SUSHI credentials, you will see the Service URL. If the data provider was created from the registry, this field will be filled in. If you entered the data provider manually you will need to enter the Service URL.
9.) If you are connecting via SUSHI you will need to enter the Report Release (this is the counter type). You can find more information at https://www.projectcounter.org/code-of-practice-five-sections/5-delivery-counter-reports/. 5 is the standard.
10.) Enter the customer id and requestor id for the sushi reports
11.) Click submit. 
12.) Under the Actions column click test (if you entered SUSHI credentials). Confirm Harvester connection was successful for the usage data provider
13.) Click Run now. Enter a beginning date and end date. Click Yes, Run. 
14.) You will see a Job for report type (of the report) at the top of the page. Make sure the job has been queued, click the 'click here' to check its progress.
15.) Confirm the report has run. 
16.) Click on the Name of the data provider in the table, if you manually created the data provider start at this step. You will see 7 tabs across the top (detail, titles, platforms, items, databases, manual upload, import logs.
17.) If you manually created a data provider, select the manual upload tab. Upload a counter 5 report. Confirm the titles load. 
18.) On the right-hand side click on Reports under eUsage. 
19.) Click on Create Report. 
20.) Select the required fields and data needed for the report. Select report columns.  Enter the name of your report at the bottom of the page and click save report. Click submit. 
21.) Confirm report runs 
22.) Click on Saved Reports and confirm your report saved.
Comment 13 Jessie Zairo 2023-10-12 12:56:54 UTC
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Comment 14 Stephanie 2023-10-13 13:52:00 UTC
Created attachment 157101 [details]
screenshot of problem
Comment 15 Stephanie 2023-10-13 13:56:30 UTC
I entered our library's Ebsco credentials as they are currently set up in Coral and working correctly.  When I test the connection here, I get a message saying "No connection for usage data provider EBSCO Information Services, please check your credentials and try again".  However, I ran the request anyway, and got the reports I asked for in the Import log.  Everything seems to have worked regardless of the message saying there's no connection.  I also entered our ProQuest credentials, and didn't have the same issue.
Comment 16 Stephanie 2023-10-13 14:44:05 UTC
For uploading data, the current Coral manual imports require that the header information from a report be removed before upload. I am not sure how this will interact with the way files have to be imported for the ERM module.  I believe the goal is for information from Coral to be transferred into the ERM without too much fussing, so just wanting to note this difference.
Comment 17 Lucas Gass 2023-10-13 17:07:56 UTC
(In reply to Lucas Gass from comment #7)
> When I enable ERM and Add a new data provider manually I get a type error:
> 
> this.$refs.table.dt is not a function. For me this comes from KohaTable.vue
> on line 150:
> 
> let dt = this.$refs.table.dt()
> 
> 
> I have tried restarting everything and rebuilding the js with yarn js:build.

I don't think this was addressed and I am still getting this error. 

To recreate:

1. ERM -> Add new data provider
2. Create manually
3. Add a provider manually 
4. Go back to /cgi-bin/koha/erm/agreements
5. Notice the type error
Comment 18 Jessie Zairo 2023-10-13 20:52:24 UTC
Feedback from a library partner 
"I was able to play around a little, and it is looking good."

Data Providers
Adding + New data provider
Created a new provider by finding an existing one.
Service URL was automatically generated.
Added our institution’s information.
Able to save without problems.
Able to edit without problems.
Created a new provider by manually inputting information.
Manually added service URL
Added our institution’s information.
Able to save without problems.
Able to edit without problems.
NOTE: fields are grayed out unless you click “create manually” button to enable them.
Connection Test
Both registry and manual providers connected to the SUSHI server successfully.
Harvest Run
Both registry and manual providers harvested most of the requested reports.
There were a few reports that didn’t return with data.
Reports that returned data were added successfully to the provider.
Manual Upload
I tried a report that I thought was counter 5 but it didn’t upload. I didn’t get a chance to get another report to test further.
Reports
Able to create new reports with no errors.
Able to run saved reports with no errors.
Able to export displayed data from the report.
NOTE: Only visible data rows were downloaded. Must select show all to get the full data downloaded.
Noticed a discrepancy between the harvested data and the report data.
Downloaded the import log data and compared the titles against the exported report data, in my SpringerLink test, I saw a difference of about 115 or so titles.
Comment 19 Michaela Sieber 2023-10-14 17:44:20 UTC
First of all thank you for the work you already did!

I added two of our main providers = Springer and De Gruyter

Testing results:

-- HARVESTING --

Harvesting of about 190.000 rows for SpringerLink was done with some errors, 
e.g.
Building the Cycling City could not be processed - please check the logs
Statistische Datenanalyse mit SPSS für Windows could not be processed - please check the logs
Concise Manual of Hematology and Oncology could not be processed - please check the logs

See http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/background_jobs.pl?op=view&id=40

It would be helpful if you could provide the number of titles successfully added/that could not be processed in the Job report similar to the number of usages added/skipped.

The same with De Gruyter, many titles could not be processed.


-- REPORTS --
Feedback on sorting reports tables:
E-resource management > eUsageReports > View report

I tried to sort by month March but the table was not sorted, just processing....

It would be great to have a 'Reporting_Period_Total' column
Comment 20 Pedro Amorim 2023-10-16 08:46:44 UTC
(In reply to Lucas Gass from comment #17)
> (In reply to Lucas Gass from comment #7)
> > When I enable ERM and Add a new data provider manually I get a type error:
> > 
> > this.$refs.table.dt is not a function. For me this comes from KohaTable.vue
> > on line 150:
> > 
> > let dt = this.$refs.table.dt()
> > 
> > 
> > I have tried restarting everything and rebuilding the js with yarn js:build.
> 
> I don't think this was addressed and I am still getting this error. 
> 
> To recreate:
> 
> 1. ERM -> Add new data provider
> 2. Create manually
> 3. Add a provider manually 
> 4. Go back to /cgi-bin/koha/erm/agreements
> 5. Notice the type error

Hi Lucas, thank you!
I'm unable to reproduce this. I'm assuming /cgi-bin/koha/erm/agreements is a typo but I can't reproduce the error in either agreements or data providers.
I created this manual data provider:
http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/usage_data_providers/18
But I have no error anywhere, are you able to provide additional instructions?

Also, update your branch if you're testing locally. We've pushed quite a few bug fixes since the first submission here.
Comment 21 Matt Blenkinsop 2023-10-16 08:59:10 UTC
(In reply to Michaela Sieber from comment #19)

Hi Michaela, 

Thanks for testing!

> Testing results:
> 
> -- HARVESTING --
> 
> Harvesting of about 190.000 rows for SpringerLink was done with some errors, 
> e.g.
> Building the Cycling City could not be processed - please check the logs
> Statistische Datenanalyse mit SPSS für Windows could not be processed -
> please check the logs
> Concise Manual of Hematology and Oncology could not be processed - please
> check the logs

The fact that the rest of the titles are harvesting successfully leads me to suspect that these failures will be due to the fields in the database not being long enough to support the data being harvested. We recently increased the size of a number of fields across the different data types but there are probably more that need doing. We will investigate this and extend fields where appropriate to give some good margin for error when harvesting. 

> 
> See
> http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/
> background_jobs.pl?op=view&id=40
> 
> It would be helpful if you could provide the number of titles successfully
> added/that could not be processed in the Job report similar to the number of
> usages added/skipped.

That can certainly be looked at - we'll add this to the list but this will probably follow later after any bug fixes are tackled

 
> 
> -- REPORTS --
> Feedback on sorting reports tables:
> E-resource management > eUsageReports > View report
> 
> I tried to sort by month March but the table was not sorted, just
> processing....

This is a known issue that we're looking at, hopefully we can patch this soon!

> 
> It would be great to have a 'Reporting_Period_Total' column

You can choose to do a report "By month with period total" in the custom reports viewer, this will add the column you mention :)
Comment 22 Lucas Gass 2023-10-16 14:23:02 UTC
> Hi Lucas, thank you!
> I'm unable to reproduce this. I'm assuming /cgi-bin/koha/erm/agreements is a
> typo but I can't reproduce the error in either agreements or data providers.
> I created this manual data provider:
> http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/
> usage_data_providers/18
> But I have no error anywhere, are you able to provide additional
> instructions?
> 
> Also, update your branch if you're testing locally. We've pushed quite a few
> bug fixes since the first submission here.

I am testing locally, have updated my branch, ran yarn. However I am not going to hold this up due to something that may very well be an issue I am only experiencing locally. I will reset to 'Signed off'.
Comment 23 Pedro Amorim 2023-10-23 13:20:19 UTC
(In reply to Matt Blenkinsop from comment #21)
> (In reply to Michaela Sieber from comment #19)
> 
> Hi Michaela, 
> 
> Thanks for testing!
> 
> > Testing results:
> > 
> > -- HARVESTING --
> > 
> > Harvesting of about 190.000 rows for SpringerLink was done with some errors, 
> > e.g.
> > Building the Cycling City could not be processed - please check the logs
> > Statistische Datenanalyse mit SPSS für Windows could not be processed -
> > please check the logs
> > Concise Manual of Hematology and Oncology could not be processed - please
> > check the logs
> 
> The fact that the rest of the titles are harvesting successfully leads me to
> suspect that these failures will be due to the fields in the database not
> being long enough to support the data being harvested. We recently increased
> the size of a number of fields across the different data types but there are
> probably more that need doing. We will investigate this and extend fields
> where appropriate to give some good margin for error when harvesting. 

It appears this was indeed the case, fixed by this commit:
https://github.com/PTFS-Europe/koha/commit/f0608a0dcaa2d5fa0875603d08124fb10bcd2fd6

> > 
> > See
> > http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/
> > background_jobs.pl?op=view&id=40
> > 
> > It would be helpful if you could provide the number of titles successfully
> > added/that could not be processed in the Job report similar to the number of
> > usages added/skipped.
> 
> That can certainly be looked at - we'll add this to the list but this will
> probably follow later after any bug fixes are tackled

This is something that I believe was working before, but got broken when I moved everything under EUsage and created a new SushiCounter.pm class.
However, I believe this is now fixed, you should see a "Titles added" row now in the job report if titles, "Databases" for databases, etc. Fixed by this commit:
https://github.com/PTFS-Europe/koha/commit/42fad0948a1318d254e7133925b3121f246a8788

@Michaela would you kindly test again and verify for yourself? Particularly the 190.000 rows provider if possible.
Your sign-off here would be very much appreciated, as others of course.

We've provisioned a new sandbox at:
http://staff-sushi.sandboxes.ptfs-europe.co.uk/

With the most up to date code from:
https://github.com/PTFS-Europe/koha/commits/bug_34587
Comment 24 Michaela Sieber 2023-10-23 22:27:42 UTC
As far as I can see all titles are harvested now.

But the usage per month is not correct in my report. 
First run 01.-31.01.2023
Second run 01.02.-30.04.2023

Example: Nature (Online ISSN 1476-4687) 

normally 3000-4000 Unique_Item_Requests per month.

Unique_Item_Requests should be: 
 Jan23: 3431	
 Feb23: 3159
 Mar23: 3758
 Apr23: 2775

Check Import Logs on Data provider Springer and Download file.
The usage numbers for Nature (=ISSN 1476-4687) for Platform nature.com are correct in the downloaded file .  

Check now the created J1 report 

See that the usage numbers for February and March are wrong
Unique_Item_Requests	
 Feb23: 20	
 Mar23:  6

usage numbers for Jan and April are correct
Comment 25 Michaela Sieber 2023-10-24 06:39:31 UTC
(In reply to Michaela Sieber from comment #24)


In a Third run everything was correct


Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Comment 26 Pedro Amorim 2023-10-24 11:05:12 UTC
(In reply to Michaela Sieber from comment #24)
> As far as I can see all titles are harvested now.
> 
> But the usage per month is not correct in my report. 
> First run 01.-31.01.2023
> Second run 01.02.-30.04.2023
> 
> Example: Nature (Online ISSN 1476-4687) 
> 
> normally 3000-4000 Unique_Item_Requests per month.
> 
> Unique_Item_Requests should be: 
>  Jan23: 3431	
>  Feb23: 3159
>  Mar23: 3758
>  Apr23: 2775
> 
> Check Import Logs on Data provider Springer and Download file.
> The usage numbers for Nature (=ISSN 1476-4687) for Platform nature.com are
> correct in the downloaded file .  
> 
> Check now the created J1 report 
> 
> See that the usage numbers for February and March are wrong
> Unique_Item_Requests	
>  Feb23: 20	
>  Mar23:  6
> 
> usage numbers for Jan and April are correct

Hi Michaela.
Once again, thank you so much for this invaluable testing (and sign-off!). We detected and fixed 2 issues here:

1) The problem seems to have been because the 2 titles titled 'nature' in the first report have the same 1476-4687 ISSN and Proprietary_ID, in fact the only thing that differs between the two is the Platform. This was causing the 'nature' title to match erroneously. This is the first time we've seen this and we improved our matching rules:
https://github.com/PTFS-Europe/koha/commit/a855ea64e5b97262bc9a89826020fc43a601418f

The problem was even bigger for the period of jan to april 2023. Because here we have 3 instances of 'nature' 1476-4687 ISSN, but all 3 are from different platforms.

With the above fix, everything should be correct in the first harvest, as well as all subsequent harvests. These 3 'nature' titles are effectively 3 different titles in Koha and each have their corresponding correct monthly usages.

I've reset the sandbox with these fixes if you're available to test it once again.

2) One issue in the logs detected was 'title' column was too short, we changed it from varchar(255) to mediumtext to address this:
https://github.com/PTFS-Europe/koha/commit/32f6bbf4bc79d07a9815bbaa17756a9ca9938724
Comment 27 Michaela Sieber 2023-10-25 20:56:33 UTC
(In reply to Pedro Amorim from comment #26)
> 
Great, Thanks for fixing and for your explanations. 

Fingers crossed for 23.11
Comment 28 Nick Clemens (kidclamp) 2023-10-27 15:58:15 UTC
Did a run through the code, a little clean up for QA tools - talked with the PFTS team and RM and our testers, all seems to be working well. Standalone module so no worries about the late push. Passing QA! Excellent work all!

My SO lines are added on the remote branch.
Comment 29 Tomás Cohen Arazi 2023-10-31 19:50:11 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 30 Jonathan Druart 2023-11-07 21:19:20 UTC
Why those commented lines? Should we remove?

koha-tmpl/intranet-tmpl/prog/js/vue/components/ERM/UsageStatisticsDataProvidersFormAdd.vue

108                         <!-- <li>
109                             <label for="usage_data_provider_method"
110                                 >{{ $__("Method") }}:
111                             </label>
112                             <input
113                                 id="usage_data_provider_method"
114                                 v-model="usage_data_provider.method"
115                                 :disabled="!selected_provider && !manual_form"
116                             />
117                         </li>
118                         <li>
119                             <label for="usage_data_provider_aggregator"
120                                 >{{ $__("Aggregator") }}:
121                             </label>
122                             <input
123                                 id="usage_data_provider_aggregator"
124                                 v-model="usage_data_provider.aggregator"
125                                 :disabled="!selected_provider && !manual_form"
126                             />
127                         </li> -->
Comment 31 Jonathan Druart 2023-11-07 21:21:54 UTC
koha-tmpl/intranet-tmpl/prog/js/vue/components/ERM/UsageStatisticsDataProvidersFormAdd.vue:            fp_config: flatpickr_defaults,

This is not needed, as well as the flatpickr import.
Comment 32 Matt Blenkinsop 2023-11-09 17:34:11 UTC
Created attachment 158754 [details] [review]
Bug 34587: (follow-up) Remove unnecessary code

This patch removes some commented out lines and some unnecessary imports. No impact to working code - self signoff

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Comment 33 Tomás Cohen Arazi 2023-11-09 17:43:06 UTC
Follow-up pushed.
Comment 34 Jonathan Druart 2023-11-15 13:03:45 UTC
ERM/UsageReports_spec.ts and ERM/DataProviders_spec.ts are failing
Comment 35 Pedro Amorim 2023-11-15 14:13:37 UTC
(In reply to Jonathan Druart from comment #34)
> ERM/UsageReports_spec.ts and ERM/DataProviders_spec.ts are failing

See bug 35229.
Comment 36 Jonathan Druart 2023-12-12 08:44:00 UTC
Trailing space chars at t/cypress/support/e2e.js:161

Looks like we need to improve the QA checks.