Summary: | Add a Usage Statistics module to ERM | ||
---|---|---|---|
Product: | Koha | Reporter: | Matt Blenkinsop <matt.blenkinsop> |
Component: | ERM | Assignee: | Matt Blenkinsop <matt.blenkinsop> |
Status: | RESOLVED FIXED | QA Contact: | |
Severity: | new feature | ||
Priority: | P5 - low | CC: | amit.gupta, amitddng135, aude.charillon, brendan, david, emily.lamancusa, ephetteplace, fridolin.somers, jonathan.druart, jonathan.field, jzairo, kyle, lucas, m.de.rooy, martin.renvoize, matt.blenkinsop, michaela.sieber, nick, paul.poulain, pedro.amorim, stephanie.sopka, tomascohen |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
URL: | https://github.com/PTFS-Europe/koha/tree/bug_34587 | ||
Change sponsored?: | Sponsored | Patch complexity: | Large patch |
Documentation contact: | Jonathan Field | Documentation submission: | https://gitlab.com/koha-community/koha-manual/-/merge_requests/751 |
Text to go in the release notes: |
This feature adds the ability to create Data Providers and harvest usage data in COUNTER format. Data Providers (or Data Platforms) are the organisations who provide usage statistics for your electronic resources. This could be a vendor or a platform provider. The data provider record contains information about SUSHI credentials and any COUNTER data which has been harvested from the provider. There is also the ability to create custom reports using this COUNTER data.
|
Version(s) released in: |
23.11.00
|
Circulation function: | |||
Bug Depends on: | 34448, 34468, 34497 | ||
Bug Blocks: | 37288, 39075, 35120, 35218, 35219, 35229, 35258, 35374, 35375, 35392, 35418 | ||
Attachments: |
queued-report-fail
screenshot of problem Bug 34587: (follow-up) Remove unnecessary code Bug 34587: (follow-up) Fix DB inconsistencies |
Description
Matt Blenkinsop
2023-08-22 13:40:38 UTC
This is ready for testing Testing notes: Credentials will be required to harvest data via SUSHI, please check if your company/organisation has any access credentials that can be used for testing Unit tests are still in development and will be following shortly A sandbox is now live at this link: http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/usage_data_providers Credentials will be required for testing, please check if your organisation has access to any valid SUSHI credentials in order to test. All credentials are redacted in the sandbox and will not be visible to other users. In the released version, credentials will be visible in the ERM module. I feel like should advertise this a bit more to get more testing - a mail to the mailing list maybe? (quick note) test_connection returns 500 if credentials are not correct (I guess it's the problem). When I enable ERM and Add a new data provider manually I get a type error: this.$refs.table.dt is not a function. For me this comes from KohaTable.vue on line 150: let dt = this.$refs.table.dt() I have tried restarting everything and rebuilding the js with yarn js:build. Created attachment 156581 [details]
queued-report-fail
We successfully created a new data provider via SUSHI. We got a successful test - Harvester connection was successful for usage data provider. When we run the job for report type TR_J1, we get a message that the report goes to queue, but fails. (In reply to Jessie Zairo from comment #9) > We successfully created a new data provider via SUSHI. We got a successful > test - Harvester connection was successful for usage data provider. When we > run the job for report type TR_J1, we get a message that the report goes to > queue, but fails. Hi Jessie, is this on our sandbox? If so, we found a problem earlier today with the background jobs which we are looking into. I’ll let you know when resolved. (In reply to Jessie Zairo from comment #9) > We successfully created a new data provider via SUSHI. We got a successful > test - Harvester connection was successful for usage data provider. When we > run the job for report type TR_J1, we get a message that the report goes to > queue, but fails. We've fixed our sandbox now, apologies for that, we were missing some of the very latest code which is there now :) Testing Plan 1.) Make sure the ERMModule system preference is set to Enable 2.) Navigate to E-resource management --> eUsage --> Data providers 3.) Click + New data provider 4.) Begin typing the data provider name (vendor) that you wish to set up. If a match is found, select the data provider. If it is not found, select the Create manually button to the right. 5.) Enter any additional information about the data provider in the description. (Not a mandatory field) 6.) Select the Harvester status to Active or Inactive. (Not a mandatory field) 7.) Select the report type from the drop-down menu. 8.) Under SUSHI credentials, you will see the Service URL. If the data provider was created from the registry, this field will be filled in. If you entered the data provider manually you will need to enter the Service URL. 9.) If you are connecting via SUSHI you will need to enter the Report Release (this is the counter type). You can find more information at https://www.projectcounter.org/code-of-practice-five-sections/5-delivery-counter-reports/. 5 is the standard. 10.) Enter the customer id and requestor id for the sushi reports 11.) Click submit. 12.) Under the Actions column click test (if you entered SUSHI credentials). Confirm Harvester connection was successful for the usage data provider 13.) Click Run now. Enter a beginning date and end date. Click Yes, Run. 14.) You will see a Job for report type (of the report) at the top of the page. Make sure the job has been queued, click the 'click here' to check its progress. 15.) Confirm the report has run. 16.) Click on the Name of the data provider in the table, if you manually created the data provider start at this step. You will see 7 tabs across the top (detail, titles, platforms, items, databases, manual upload, import logs. 17.) If you manually created a data provider, select the manual upload tab. Upload a counter 5 report. Confirm the titles load. 18.) On the right-hand side click on Reports under eUsage. 19.) Click on Create Report. 20.) Select the required fields and data needed for the report. Select report columns. Enter the name of your report at the bottom of the page and click save report. Click submit. 21.) Confirm report runs 22.) Click on Saved Reports and confirm your report saved. Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com> Created attachment 157101 [details]
screenshot of problem
I entered our library's Ebsco credentials as they are currently set up in Coral and working correctly. When I test the connection here, I get a message saying "No connection for usage data provider EBSCO Information Services, please check your credentials and try again". However, I ran the request anyway, and got the reports I asked for in the Import log. Everything seems to have worked regardless of the message saying there's no connection. I also entered our ProQuest credentials, and didn't have the same issue. For uploading data, the current Coral manual imports require that the header information from a report be removed before upload. I am not sure how this will interact with the way files have to be imported for the ERM module. I believe the goal is for information from Coral to be transferred into the ERM without too much fussing, so just wanting to note this difference. (In reply to Lucas Gass from comment #7) > When I enable ERM and Add a new data provider manually I get a type error: > > this.$refs.table.dt is not a function. For me this comes from KohaTable.vue > on line 150: > > let dt = this.$refs.table.dt() > > > I have tried restarting everything and rebuilding the js with yarn js:build. I don't think this was addressed and I am still getting this error. To recreate: 1. ERM -> Add new data provider 2. Create manually 3. Add a provider manually 4. Go back to /cgi-bin/koha/erm/agreements 5. Notice the type error Feedback from a library partner "I was able to play around a little, and it is looking good." Data Providers Adding + New data provider Created a new provider by finding an existing one. Service URL was automatically generated. Added our institution’s information. Able to save without problems. Able to edit without problems. Created a new provider by manually inputting information. Manually added service URL Added our institution’s information. Able to save without problems. Able to edit without problems. NOTE: fields are grayed out unless you click “create manually” button to enable them. Connection Test Both registry and manual providers connected to the SUSHI server successfully. Harvest Run Both registry and manual providers harvested most of the requested reports. There were a few reports that didn’t return with data. Reports that returned data were added successfully to the provider. Manual Upload I tried a report that I thought was counter 5 but it didn’t upload. I didn’t get a chance to get another report to test further. Reports Able to create new reports with no errors. Able to run saved reports with no errors. Able to export displayed data from the report. NOTE: Only visible data rows were downloaded. Must select show all to get the full data downloaded. Noticed a discrepancy between the harvested data and the report data. Downloaded the import log data and compared the titles against the exported report data, in my SpringerLink test, I saw a difference of about 115 or so titles. First of all thank you for the work you already did! I added two of our main providers = Springer and De Gruyter Testing results: -- HARVESTING -- Harvesting of about 190.000 rows for SpringerLink was done with some errors, e.g. Building the Cycling City could not be processed - please check the logs Statistische Datenanalyse mit SPSS für Windows could not be processed - please check the logs Concise Manual of Hematology and Oncology could not be processed - please check the logs See http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/background_jobs.pl?op=view&id=40 It would be helpful if you could provide the number of titles successfully added/that could not be processed in the Job report similar to the number of usages added/skipped. The same with De Gruyter, many titles could not be processed. -- REPORTS -- Feedback on sorting reports tables: E-resource management > eUsageReports > View report I tried to sort by month March but the table was not sorted, just processing.... It would be great to have a 'Reporting_Period_Total' column (In reply to Lucas Gass from comment #17) > (In reply to Lucas Gass from comment #7) > > When I enable ERM and Add a new data provider manually I get a type error: > > > > this.$refs.table.dt is not a function. For me this comes from KohaTable.vue > > on line 150: > > > > let dt = this.$refs.table.dt() > > > > > > I have tried restarting everything and rebuilding the js with yarn js:build. > > I don't think this was addressed and I am still getting this error. > > To recreate: > > 1. ERM -> Add new data provider > 2. Create manually > 3. Add a provider manually > 4. Go back to /cgi-bin/koha/erm/agreements > 5. Notice the type error Hi Lucas, thank you! I'm unable to reproduce this. I'm assuming /cgi-bin/koha/erm/agreements is a typo but I can't reproduce the error in either agreements or data providers. I created this manual data provider: http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/usage_data_providers/18 But I have no error anywhere, are you able to provide additional instructions? Also, update your branch if you're testing locally. We've pushed quite a few bug fixes since the first submission here. (In reply to Michaela Sieber from comment #19) Hi Michaela, Thanks for testing! > Testing results: > > -- HARVESTING -- > > Harvesting of about 190.000 rows for SpringerLink was done with some errors, > e.g. > Building the Cycling City could not be processed - please check the logs > Statistische Datenanalyse mit SPSS für Windows could not be processed - > please check the logs > Concise Manual of Hematology and Oncology could not be processed - please > check the logs The fact that the rest of the titles are harvesting successfully leads me to suspect that these failures will be due to the fields in the database not being long enough to support the data being harvested. We recently increased the size of a number of fields across the different data types but there are probably more that need doing. We will investigate this and extend fields where appropriate to give some good margin for error when harvesting. > > See > http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/ > background_jobs.pl?op=view&id=40 > > It would be helpful if you could provide the number of titles successfully > added/that could not be processed in the Job report similar to the number of > usages added/skipped. That can certainly be looked at - we'll add this to the list but this will probably follow later after any bug fixes are tackled > > -- REPORTS -- > Feedback on sorting reports tables: > E-resource management > eUsageReports > View report > > I tried to sort by month March but the table was not sorted, just > processing.... This is a known issue that we're looking at, hopefully we can patch this soon! > > It would be great to have a 'Reporting_Period_Total' column You can choose to do a report "By month with period total" in the custom reports viewer, this will add the column you mention :) > Hi Lucas, thank you!
> I'm unable to reproduce this. I'm assuming /cgi-bin/koha/erm/agreements is a
> typo but I can't reproduce the error in either agreements or data providers.
> I created this manual data provider:
> http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/erm/eusage/
> usage_data_providers/18
> But I have no error anywhere, are you able to provide additional
> instructions?
>
> Also, update your branch if you're testing locally. We've pushed quite a few
> bug fixes since the first submission here.
I am testing locally, have updated my branch, ran yarn. However I am not going to hold this up due to something that may very well be an issue I am only experiencing locally. I will reset to 'Signed off'.
(In reply to Matt Blenkinsop from comment #21) > (In reply to Michaela Sieber from comment #19) > > Hi Michaela, > > Thanks for testing! > > > Testing results: > > > > -- HARVESTING -- > > > > Harvesting of about 190.000 rows for SpringerLink was done with some errors, > > e.g. > > Building the Cycling City could not be processed - please check the logs > > Statistische Datenanalyse mit SPSS für Windows could not be processed - > > please check the logs > > Concise Manual of Hematology and Oncology could not be processed - please > > check the logs > > The fact that the rest of the titles are harvesting successfully leads me to > suspect that these failures will be due to the fields in the database not > being long enough to support the data being harvested. We recently increased > the size of a number of fields across the different data types but there are > probably more that need doing. We will investigate this and extend fields > where appropriate to give some good margin for error when harvesting. It appears this was indeed the case, fixed by this commit: https://github.com/PTFS-Europe/koha/commit/f0608a0dcaa2d5fa0875603d08124fb10bcd2fd6 > > > > See > > http://staff-usagestats.sandboxes.ptfs-europe.co.uk/cgi-bin/koha/admin/ > > background_jobs.pl?op=view&id=40 > > > > It would be helpful if you could provide the number of titles successfully > > added/that could not be processed in the Job report similar to the number of > > usages added/skipped. > > That can certainly be looked at - we'll add this to the list but this will > probably follow later after any bug fixes are tackled This is something that I believe was working before, but got broken when I moved everything under EUsage and created a new SushiCounter.pm class. However, I believe this is now fixed, you should see a "Titles added" row now in the job report if titles, "Databases" for databases, etc. Fixed by this commit: https://github.com/PTFS-Europe/koha/commit/42fad0948a1318d254e7133925b3121f246a8788 @Michaela would you kindly test again and verify for yourself? Particularly the 190.000 rows provider if possible. Your sign-off here would be very much appreciated, as others of course. We've provisioned a new sandbox at: http://staff-sushi.sandboxes.ptfs-europe.co.uk/ With the most up to date code from: https://github.com/PTFS-Europe/koha/commits/bug_34587 As far as I can see all titles are harvested now. But the usage per month is not correct in my report. First run 01.-31.01.2023 Second run 01.02.-30.04.2023 Example: Nature (Online ISSN 1476-4687) normally 3000-4000 Unique_Item_Requests per month. Unique_Item_Requests should be: Jan23: 3431 Feb23: 3159 Mar23: 3758 Apr23: 2775 Check Import Logs on Data provider Springer and Download file. The usage numbers for Nature (=ISSN 1476-4687) for Platform nature.com are correct in the downloaded file . Check now the created J1 report See that the usage numbers for February and March are wrong Unique_Item_Requests Feb23: 20 Mar23: 6 usage numbers for Jan and April are correct (In reply to Michaela Sieber from comment #24) In a Third run everything was correct Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu> (In reply to Michaela Sieber from comment #24) > As far as I can see all titles are harvested now. > > But the usage per month is not correct in my report. > First run 01.-31.01.2023 > Second run 01.02.-30.04.2023 > > Example: Nature (Online ISSN 1476-4687) > > normally 3000-4000 Unique_Item_Requests per month. > > Unique_Item_Requests should be: > Jan23: 3431 > Feb23: 3159 > Mar23: 3758 > Apr23: 2775 > > Check Import Logs on Data provider Springer and Download file. > The usage numbers for Nature (=ISSN 1476-4687) for Platform nature.com are > correct in the downloaded file . > > Check now the created J1 report > > See that the usage numbers for February and March are wrong > Unique_Item_Requests > Feb23: 20 > Mar23: 6 > > usage numbers for Jan and April are correct Hi Michaela. Once again, thank you so much for this invaluable testing (and sign-off!). We detected and fixed 2 issues here: 1) The problem seems to have been because the 2 titles titled 'nature' in the first report have the same 1476-4687 ISSN and Proprietary_ID, in fact the only thing that differs between the two is the Platform. This was causing the 'nature' title to match erroneously. This is the first time we've seen this and we improved our matching rules: https://github.com/PTFS-Europe/koha/commit/a855ea64e5b97262bc9a89826020fc43a601418f The problem was even bigger for the period of jan to april 2023. Because here we have 3 instances of 'nature' 1476-4687 ISSN, but all 3 are from different platforms. With the above fix, everything should be correct in the first harvest, as well as all subsequent harvests. These 3 'nature' titles are effectively 3 different titles in Koha and each have their corresponding correct monthly usages. I've reset the sandbox with these fixes if you're available to test it once again. 2) One issue in the logs detected was 'title' column was too short, we changed it from varchar(255) to mediumtext to address this: https://github.com/PTFS-Europe/koha/commit/32f6bbf4bc79d07a9815bbaa17756a9ca9938724 (In reply to Pedro Amorim from comment #26) > Great, Thanks for fixing and for your explanations. Fingers crossed for 23.11 Did a run through the code, a little clean up for QA tools - talked with the PFTS team and RM and our testers, all seems to be working well. Standalone module so no worries about the late push. Passing QA! Excellent work all! My SO lines are added on the remote branch. Pushed to master for 23.11. Nice work everyone, thanks! Why those commented lines? Should we remove? koha-tmpl/intranet-tmpl/prog/js/vue/components/ERM/UsageStatisticsDataProvidersFormAdd.vue 108 <!-- <li> 109 <label for="usage_data_provider_method" 110 >{{ $__("Method") }}: 111 </label> 112 <input 113 id="usage_data_provider_method" 114 v-model="usage_data_provider.method" 115 :disabled="!selected_provider && !manual_form" 116 /> 117 </li> 118 <li> 119 <label for="usage_data_provider_aggregator" 120 >{{ $__("Aggregator") }}: 121 </label> 122 <input 123 id="usage_data_provider_aggregator" 124 v-model="usage_data_provider.aggregator" 125 :disabled="!selected_provider && !manual_form" 126 /> 127 </li> --> koha-tmpl/intranet-tmpl/prog/js/vue/components/ERM/UsageStatisticsDataProvidersFormAdd.vue: fp_config: flatpickr_defaults, This is not needed, as well as the flatpickr import. Created attachment 158754 [details] [review] Bug 34587: (follow-up) Remove unnecessary code This patch removes some commented out lines and some unnecessary imports. No impact to working code - self signoff Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> Follow-up pushed. ERM/UsageReports_spec.ts and ERM/DataProviders_spec.ts are failing (In reply to Jonathan Druart from comment #34) > ERM/UsageReports_spec.ts and ERM/DataProviders_spec.ts are failing See bug 35229. Trailing space chars at t/cypress/support/e2e.js:161 Looks like we need to improve the QA checks. Documented by Jonathan Field. It seems that this patch set added two inconsistencies in the db schema: STRUCTURE `date_uploaded` timestamp NULL DEFAULT current_timestamp() COMMENT 'counter file upload date', DBREV 230600056.pl `date_uploaded` timestamp DEFAULT NULL DEFAULT current_timestamp() COMMENT 'counter file upload date', => You see two DEFAULTs in the dbrev. Probably a typo. STRUCTURE `importdate` timestamp NULL DEFAULT current_timestamp() COMMENT 'counter file import date', DBREV 230600056. `importdate` timestamp DEFAULT NULL DEFAULT current_timestamp() COMMENT 'counter file import date', => Same Could the authors give follow-up and correct please? Thanks. Created attachment 177672 [details] [review] Bug 34587: (follow-up) Fix DB inconsistencies (In reply to Matt Blenkinsop from comment #39) > Created attachment 177672 [details] [review] [review] > Bug 34587: (follow-up) Fix DB inconsistencies Hi Matt, can I bother you to put this ona new bug please? We'll need it for the release notes and backporting. Thanks! (In reply to Katrin Fischer from comment #40) > (In reply to Matt Blenkinsop from comment #39) > > Created attachment 177672 [details] [review] [review] [review] > > Bug 34587: (follow-up) Fix DB inconsistencies > > Hi Matt, can I bother you to put this ona new bug please? We'll need it for > the release notes and backporting. Thanks! Done - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39075 (In reply to Matt Blenkinsop from comment #41) > (In reply to Katrin Fischer from comment #40) > > (In reply to Matt Blenkinsop from comment #39) > > > Created attachment 177672 [details] [review] [review] [review] [review] > > > Bug 34587: (follow-up) Fix DB inconsistencies > > > > Hi Matt, can I bother you to put this ona new bug please? We'll need it for > > the release notes and backporting. Thanks! > > Done - https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39075 Thanks! |