Bug 34788 - Add the ability to import KBART files to ERM
Summary: Add the ability to import KBART files to ERM
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: ERM (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement with 10 votes (vote)
Assignee: Matt Blenkinsop
QA Contact:
URL:
Keywords:
Depends on: 34789 34468
Blocks:
  Show dependency treegraph
 
Reported: 2023-09-14 12:10 UTC by Matt Blenkinsop
Modified: 2024-04-26 13:13 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 34788: Add an API endpoint to import a KBART file (7.68 KB, patch)
2024-01-25 09:41 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add Vue components, api client and navigation route (9.49 KB, patch)
2024-01-25 09:41 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add a background job to import a KBART file (14.88 KB, patch)
2024-01-25 09:41 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add unit test (15.31 KB, patch)
2024-01-25 09:41 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add a relationship to a package for the imported file (6.74 KB, patch)
2024-01-25 09:41 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Allow for import of CSV files (37.06 KB, patch)
2024-01-25 09:41 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Allow for import of CSV files (37.86 KB, patch)
2024-01-25 10:58 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Example TSV file (1.87 KB, text/tab-separated-values)
2024-04-19 13:34 UTC, Matt Blenkinsop
Details
Example CSV file (contains the 9 titles from the TSV file) (234.11 KB, text/csv)
2024-04-19 13:34 UTC, Matt Blenkinsop
Details
Bug 34788: Add an API endpoint to import a KBART file (7.67 KB, patch)
2024-04-19 13:36 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add Vue components, api client and navigation route (9.49 KB, patch)
2024-04-19 13:36 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add a background job to import a KBART file (14.87 KB, patch)
2024-04-19 13:36 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add unit test (15.30 KB, patch)
2024-04-19 13:36 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Add a relationship to a package for the imported file (6.73 KB, patch)
2024-04-19 13:36 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Allow for import of CSV files (37.94 KB, patch)
2024-04-19 13:36 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 34788: Make biblio creation optional (12.62 KB, patch)
2024-04-19 13:36 UTC, Matt Blenkinsop
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Blenkinsop 2023-09-14 12:10:31 UTC
We are developing the ability to import local titles directly into ERM from a KBART file

Only KBART Phase II files will be supported
See: https://groups.niso.org/higherlogic/ws/public/download/16900/RP-9-2014_KBART.pdf
Comment 1 Koha Team University Lyon 3 2024-01-22 12:38:47 UTC
Hi,
do you have a schedule to work on this one?
We (several french users from KohaLa) are very interested.

Thanks :)
Sonia
Comment 2 Matt Blenkinsop 2024-01-22 14:02:29 UTC
Hi Sonia,

There is a solution ready which is in testing with one of our customers. When I get a chance I'll upload it to the bug for you to test as well. The more feedback the better!
Comment 3 Matt Blenkinsop 2024-01-25 09:41:45 UTC
Created attachment 161379 [details] [review]
Bug 34788: Add an API endpoint to import a KBART file

This patch adds the endpoint needed to queue an import_from_kbart_file background job
Comment 4 Matt Blenkinsop 2024-01-25 09:41:47 UTC
Created attachment 161380 [details] [review]
Bug 34788: Add Vue components, api client and navigation route

This patch adds a new component to handle the file import, a route to that component and the API client route needed to access the API
Comment 5 Matt Blenkinsop 2024-01-25 09:41:49 UTC
Created attachment 161381 [details] [review]
Bug 34788: Add a background job to import a KBART file

This patch adds a background job that will import a KBART file

Sponsored-by: UKHSA
Comment 6 Matt Blenkinsop 2024-01-25 09:41:52 UTC
Created attachment 161382 [details] [review]
Bug 34788: Add unit test
Comment 7 Matt Blenkinsop 2024-01-25 09:41:55 UTC
Created attachment 161383 [details] [review]
Bug 34788: Add a relationship to a package for the imported file
Comment 8 Matt Blenkinsop 2024-01-25 09:41:58 UTC
Created attachment 161384 [details] [review]
Bug 34788: Allow for import of CSV files

This commit allows CSV files to be imported alongside TSV files.
It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes

Test plan:
1) Enable the ERM module
2) Navigate to E-resource management > eHoldings > Local > Packages
3) Create at least one package
4) Navigate to E-resource management > eHoldings > Local > Titles
5) There should be a button for "Import from KBART file"
6) Click this button
7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button
8) Click Submit
9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file
10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected
11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs)
12) It should display a progress bar followed by a report and any error messages
13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles.
14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Comment 9 Matt Blenkinsop 2024-01-25 10:58:16 UTC
Created attachment 161391 [details] [review]
Bug 34788: Allow for import of CSV files

This commit allows CSV files to be imported alongside TSV files.
It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes

Test plan:
1) Enable the ERM module
2) Navigate to E-resource management > eHoldings > Local > Packages
3) Create at least one package
4) Navigate to E-resource management > eHoldings > Local > Titles
5) There should be a button for "Import from KBART file"
6) Click this button
7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button
8) Click Submit
9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file
10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected
11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs)
12) It should display a progress bar followed by a report and any error messages
13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles.
14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Comment 10 Lucas Gass 2024-01-25 23:28:33 UTC
Hi Matt,

I am not terribly familiar with KBART files. I am attempting to test this and getting a "Something went wrong: SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data"

I'm sure this is my file. Do you have a file we can use to test with? 

Other minor things of note: 

 FAIL	koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/import_from_kbart_file.inc
   FAIL	  valid_template
		: filter not found


 FAIL	t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
   FAIL	  forbidden patterns
		forbidden pattern: tab char (line 84)
		forbidden pattern: tab char (line 85)
		forbidden pattern: tab char (line 86)
		forbidden pattern: tab char (line 124)
		forbidden pattern: tab char (line 125)
		forbidden pattern: tab char (line 126)
		forbidden pattern: tab char (line 212)
		forbidden pattern: tab char (line 213)
		forbidden pattern: tab char (line 214)
Comment 11 Jonathan Field 2024-01-29 09:37:47 UTC
Hi Lucas, 
Springer provide their KBART files in the public domain. You could take a few of these to use as sample files. I'll leave Matt to answer the other point!

https://adminportal.springernature.com/metadata/kbart
Thanks
Jonathan
Comment 12 Koha Team University Lyon 3 2024-03-19 10:41:19 UTC
Hello,
it's also possible to export Kbart file from BACON : 
https://bacon.abes.fr/exporter.html

Only in french I'm afraid.

Sonia
Comment 13 Mathieu Saby 2024-03-19 16:15:17 UTC
For your information, the KBART files used in France and available from https://bacon.abes.fr/exporter.html have an extra field "ppn" after the mandatory KBART fields (it is permitted, as per the specification :
"In order to supply additional information for other purposes not required by the KBART Recommended Practice, content providers are permitted to include any extra data fields after the last KBART utilized position (access_type in position 25)." 

An other import source is https://gokb.org/ , whitch adds 3 fields : zdb_id, 
gokb_tipp_uid, gokb_title_uid

It could be interesting to test how Koha behaves with those kind of "enhanced" Kbart Files
Comment 14 Mathieu Saby 2024-03-26 14:15:53 UTC
I cannot activate the ERMModule preference ("Error") in sandbox. It does not seem related with this bug, but I cannot test it.
Comment 15 Matt Blenkinsop 2024-04-19 13:34:25 UTC
Created attachment 165189 [details]
Example TSV file
Comment 16 Matt Blenkinsop 2024-04-19 13:34:57 UTC
Created attachment 165190 [details]
Example CSV file (contains the 9 titles from the TSV file)
Comment 17 Matt Blenkinsop 2024-04-19 13:36:04 UTC
Created attachment 165191 [details] [review]
Bug 34788: Add an API endpoint to import a KBART file

This patch adds the endpoint needed to queue an import_from_kbart_file background job
Comment 18 Matt Blenkinsop 2024-04-19 13:36:07 UTC
Created attachment 165192 [details] [review]
Bug 34788: Add Vue components, api client and navigation route

This patch adds a new component to handle the file import, a route to that component and the API client route needed to access the API
Comment 19 Matt Blenkinsop 2024-04-19 13:36:09 UTC
Created attachment 165193 [details] [review]
Bug 34788: Add a background job to import a KBART file

This patch adds a background job that will import a KBART file

Sponsored-by: UKHSA
Comment 20 Matt Blenkinsop 2024-04-19 13:36:12 UTC
Created attachment 165194 [details] [review]
Bug 34788: Add unit test
Comment 21 Matt Blenkinsop 2024-04-19 13:36:15 UTC
Created attachment 165195 [details] [review]
Bug 34788: Add a relationship to a package for the imported file
Comment 22 Matt Blenkinsop 2024-04-19 13:36:18 UTC
Created attachment 165196 [details] [review]
Bug 34788: Allow for import of CSV files

This commit allows CSV files to be imported alongside TSV files.
It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes

Test plan:
1) Enable the ERM module
2) Navigate to E-resource management > eHoldings > Local > Packages
3) Create at least one package
4) Navigate to E-resource management > eHoldings > Local > Titles
5) There should be a button for "Import from KBART file"
6) Click this button
7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button. I have attached some example files to the bug but feel free to use your own if you have them.
8) Click Submit
9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file
10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected
11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs)
12) It should display a progress bar followed by a report and any error messages
13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles.
14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Comment 23 Matt Blenkinsop 2024-04-19 13:36:20 UTC
Created attachment 165197 [details] [review]
Bug 34788: Make biblio creation optional

This patch rebases in the changes from bug 36618 to make biblio creation optional
Comment 24 Matt Blenkinsop 2024-04-19 13:38:20 UTC
(In reply to Mathieu Saby from comment #13)
> For your information, the KBART files used in France and available from
> https://bacon.abes.fr/exporter.html have an extra field "ppn" after the
> mandatory KBART fields (it is permitted, as per the specification :
> "In order to supply additional information for other purposes not required
> by the KBART Recommended Practice, content providers are permitted to
> include any extra data fields after the last KBART utilized position
> (access_type in position 25)." 
> 
> An other import source is https://gokb.org/ , whitch adds 3 fields : zdb_id, 
> gokb_tipp_uid, gokb_title_uid
> 
> It could be interesting to test how Koha behaves with those kind of
> "enhanced" Kbart Files

Hi, for these file the system will display a warning to say that those columns are not compatible with the database structure and ask the user to remove them from the document. Local titles are structured in line with the standard KBART fields and so any additional fields are not currently stored as there is no valid database column to store them in
Comment 25 Mathieu Saby 2024-04-26 13:02:31 UTC
Thanks you for the explanation.
By "warning" do you mean that only the standard kbart columns are imported (if there are other columns), or that no columns at all are imported ?
Comment 26 Matt Blenkinsop 2024-04-26 13:04:48 UTC
It will return the warning before the job is queued so no columns will be imported at that stage. Once the file is edited to remove the columns it can then be re-imported and the job will be queued
Comment 27 Mathieu Saby 2024-04-26 13:13:48 UTC
Well, I am not your sponsor but as an average librarian I can say that it not the most effective workflow : when we librarians download valid kbart files from gokb or other sources, we usually wont want to edit the file to remove columns that are not defined in kbart specs. If all mandatory fields are in the file, won't it be possible to import the file with a "simple" warning telling the user that the other (non mandatory) fields were discarded?