We are developing the ability to import local titles directly into ERM from a KBART file Only KBART Phase II files will be supported See: https://groups.niso.org/higherlogic/ws/public/download/16900/RP-9-2014_KBART.pdf
Hi, do you have a schedule to work on this one? We (several french users from KohaLa) are very interested. Thanks :) Sonia
Hi Sonia, There is a solution ready which is in testing with one of our customers. When I get a chance I'll upload it to the bug for you to test as well. The more feedback the better!
Created attachment 161379 [details] [review] Bug 34788: Add an API endpoint to import a KBART file This patch adds the endpoint needed to queue an import_from_kbart_file background job
Created attachment 161380 [details] [review] Bug 34788: Add Vue components, api client and navigation route This patch adds a new component to handle the file import, a route to that component and the API client route needed to access the API
Created attachment 161381 [details] [review] Bug 34788: Add a background job to import a KBART file This patch adds a background job that will import a KBART file Sponsored-by: UKHSA
Created attachment 161382 [details] [review] Bug 34788: Add unit test
Created attachment 161383 [details] [review] Bug 34788: Add a relationship to a package for the imported file
Created attachment 161384 [details] [review] Bug 34788: Allow for import of CSV files This commit allows CSV files to be imported alongside TSV files. It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes Test plan: 1) Enable the ERM module 2) Navigate to E-resource management > eHoldings > Local > Packages 3) Create at least one package 4) Navigate to E-resource management > eHoldings > Local > Titles 5) There should be a button for "Import from KBART file" 6) Click this button 7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button 8) Click Submit 9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file 10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected 11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs) 12) It should display a progress bar followed by a report and any error messages 13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles. 14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Created attachment 161391 [details] [review] Bug 34788: Allow for import of CSV files This commit allows CSV files to be imported alongside TSV files. It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes Test plan: 1) Enable the ERM module 2) Navigate to E-resource management > eHoldings > Local > Packages 3) Create at least one package 4) Navigate to E-resource management > eHoldings > Local > Titles 5) There should be a button for "Import from KBART file" 6) Click this button 7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button 8) Click Submit 9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file 10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected 11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs) 12) It should display a progress bar followed by a report and any error messages 13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles. 14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Hi Matt, I am not terribly familiar with KBART files. I am attempting to test this and getting a "Something went wrong: SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data" I'm sure this is my file. Do you have a file we can use to test with? Other minor things of note: FAIL koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/import_from_kbart_file.inc FAIL valid_template : filter not found FAIL t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t FAIL forbidden patterns forbidden pattern: tab char (line 84) forbidden pattern: tab char (line 85) forbidden pattern: tab char (line 86) forbidden pattern: tab char (line 124) forbidden pattern: tab char (line 125) forbidden pattern: tab char (line 126) forbidden pattern: tab char (line 212) forbidden pattern: tab char (line 213) forbidden pattern: tab char (line 214)
Hi Lucas, Springer provide their KBART files in the public domain. You could take a few of these to use as sample files. I'll leave Matt to answer the other point! https://adminportal.springernature.com/metadata/kbart Thanks Jonathan
Hello, it's also possible to export Kbart file from BACON : https://bacon.abes.fr/exporter.html Only in french I'm afraid. Sonia
For your information, the KBART files used in France and available from https://bacon.abes.fr/exporter.html have an extra field "ppn" after the mandatory KBART fields (it is permitted, as per the specification : "In order to supply additional information for other purposes not required by the KBART Recommended Practice, content providers are permitted to include any extra data fields after the last KBART utilized position (access_type in position 25)." An other import source is https://gokb.org/ , whitch adds 3 fields : zdb_id, gokb_tipp_uid, gokb_title_uid It could be interesting to test how Koha behaves with those kind of "enhanced" Kbart Files
I cannot activate the ERMModule preference ("Error") in sandbox. It does not seem related with this bug, but I cannot test it.
Created attachment 165189 [details] Example TSV file
Created attachment 165190 [details] Example CSV file (contains the 9 titles from the TSV file)
Created attachment 165191 [details] [review] Bug 34788: Add an API endpoint to import a KBART file This patch adds the endpoint needed to queue an import_from_kbart_file background job
Created attachment 165192 [details] [review] Bug 34788: Add Vue components, api client and navigation route This patch adds a new component to handle the file import, a route to that component and the API client route needed to access the API
Created attachment 165193 [details] [review] Bug 34788: Add a background job to import a KBART file This patch adds a background job that will import a KBART file Sponsored-by: UKHSA
Created attachment 165194 [details] [review] Bug 34788: Add unit test
Created attachment 165195 [details] [review] Bug 34788: Add a relationship to a package for the imported file
Created attachment 165196 [details] [review] Bug 34788: Allow for import of CSV files This commit allows CSV files to be imported alongside TSV files. It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes Test plan: 1) Enable the ERM module 2) Navigate to E-resource management > eHoldings > Local > Packages 3) Create at least one package 4) Navigate to E-resource management > eHoldings > Local > Titles 5) There should be a button for "Import from KBART file" 6) Click this button 7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button. I have attached some example files to the bug but feel free to use your own if you have them. 8) Click Submit 9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file 10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected 11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs) 12) It should display a progress bar followed by a report and any error messages 13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles. 14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Created attachment 165197 [details] [review] Bug 34788: Make biblio creation optional This patch rebases in the changes from bug 36618 to make biblio creation optional
(In reply to Mathieu Saby from comment #13) > For your information, the KBART files used in France and available from > https://bacon.abes.fr/exporter.html have an extra field "ppn" after the > mandatory KBART fields (it is permitted, as per the specification : > "In order to supply additional information for other purposes not required > by the KBART Recommended Practice, content providers are permitted to > include any extra data fields after the last KBART utilized position > (access_type in position 25)." > > An other import source is https://gokb.org/ , whitch adds 3 fields : zdb_id, > gokb_tipp_uid, gokb_title_uid > > It could be interesting to test how Koha behaves with those kind of > "enhanced" Kbart Files Hi, for these file the system will display a warning to say that those columns are not compatible with the database structure and ask the user to remove them from the document. Local titles are structured in line with the standard KBART fields and so any additional fields are not currently stored as there is no valid database column to store them in
Thanks you for the explanation. By "warning" do you mean that only the standard kbart columns are imported (if there are other columns), or that no columns at all are imported ?
It will return the warning before the job is queued so no columns will be imported at that stage. Once the file is edited to remove the columns it can then be re-imported and the job will be queued
Well, I am not your sponsor but as an average librarian I can say that it not the most effective workflow : when we librarians download valid kbart files from gokb or other sources, we usually wont want to edit the file to remove columns that are not defined in kbart specs. If all mandatory fields are in the file, won't it be possible to import the file with a "simple" warning telling the user that the other (non mandatory) fields were discarded?