Bug 21749

Summary: Importing MARC frameworks from pre-9701 fails
Product: Koha Reporter: Tomás Cohen Arazi <tomascohen>
Component: MARC Bibliographic data supportAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: major    
Priority: P5 - low CC: fridolin.somers, jonathan.druart, katrin.fischer, m.de.rooy, martin.renvoize, nick, tomascohen
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 9701    
Bug Blocks:    
Attachments: Bug 21749: Add tests for ImportFramework
Bug 21749: Fix import of CSV biblio framework
Bug 21749: Add tests for ImportFramework
Bug 21749: Fix import of CSV biblio framework
Bug 21749: (QA follow-up) Add more tests for ImportFramework
Bug 21749: Add tests for ImportFramework
Bug 21749: Fix import of CSV biblio framework
Bug 21749: (QA follow-up) Add more tests for ImportFramework
Bug 21749: Add tests for ImportFramework
Bug 21749: Fix import of CSV biblio framework
Bug 21749: (QA follow-up) Add more tests for ImportFramework

Description Tomás Cohen Arazi 2018-11-01 15:06:00 UTC
If you import a MARC framework that has been exported from a previous version of Koha (17.11 and below) results in an empty MARC framework, without any error reporting to the end user. This is because of the added columns in 'marc_tag_structure' (ind1_defaultvalue and ind2_defaultvalue) and the fact that we don't have a fallback behaviour when they are not present on the uploaded file.

My opinion is we should check the table header (from CSV or whatever format) and let the DB use the default values if a column is missing.

I mark it as Major, because there's no possible data recovery after the failed import.
Comment 1 Jonathan Druart 2018-11-12 16:16:29 UTC Comment hidden (obsolete)
Comment 2 Jonathan Druart 2018-11-12 16:16:33 UTC Comment hidden (obsolete)
Comment 3 Tomás Cohen Arazi 2018-11-12 18:08:59 UTC Comment hidden (obsolete)
Comment 4 Tomás Cohen Arazi 2018-11-12 18:09:02 UTC Comment hidden (obsolete)
Comment 5 Tomás Cohen Arazi 2018-11-12 18:09:08 UTC Comment hidden (obsolete)
Comment 6 Tomás Cohen Arazi 2018-11-12 19:08:08 UTC
Created attachment 82240 [details] [review]
Bug 21749: Add tests for ImportFramework

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 7 Tomás Cohen Arazi 2018-11-12 19:08:14 UTC
Created attachment 82241 [details] [review]
Bug 21749: Fix import of CSV biblio framework

Bug 9701 added 2 new columns to the table marc_tag_structure which broke
the import of biblio frameworks if the CSV file was generated with a
previous version.

Test plan:
- Export a biblio framework from 17.11
- Import it into 18.05+

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

https://bugs.koha-community.org/show_bug.cgi?id=21740
Comment 8 Tomás Cohen Arazi 2018-11-12 19:08:19 UTC
Created attachment 82242 [details] [review]
Bug 21749: (QA follow-up) Add more tests for ImportFramework

This patch adds tests for:
- Output of ImportFramework on fields removal and error conditions (file
  not found)
- Different file format behaviour (csv, xml and ods are tested).

To test:
- Run
  $ kshell
 k$ prove t/db_dependent/ImportExportFramework.t
=> SUCCESS: All green :-D
- Sign off :-D

Note: the biblio_framework* files are based on the original one on the
patchset, and exported in 17.11 with different formats.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 9 Marcel de Rooy 2018-11-14 14:57:06 UTC
Created attachment 82338 [details] [review]
Bug 21749: Add tests for ImportFramework

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 Marcel de Rooy 2018-11-14 14:57:11 UTC
Created attachment 82339 [details] [review]
Bug 21749: Fix import of CSV biblio framework

Bug 9701 added 2 new columns to the table marc_tag_structure which broke
the import of biblio frameworks if the CSV file was generated with a
previous version.

Test plan:
- Export a biblio framework from 17.11
- Import it into 18.05+

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

https://bugs.koha-community.org/show_bug.cgi?id=21740

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 11 Marcel de Rooy 2018-11-14 14:57:16 UTC
Created attachment 82340 [details] [review]
Bug 21749: (QA follow-up) Add more tests for ImportFramework

This patch adds tests for:
- Output of ImportFramework on fields removal and error conditions (file
  not found)
- Different file format behaviour (csv, xml and ods are tested).

To test:
- Run
  $ kshell
 k$ prove t/db_dependent/ImportExportFramework.t
=> SUCCESS: All green :-D
- Sign off :-D

Note: the biblio_framework* files are based on the original one on the
patchset, and exported in 17.11 with different formats.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 12 Nick Clemens 2018-11-15 12:49:21 UTC
Awesome work all!

Pushed to master for 18.11
Comment 13 Martin Renvoize 2018-11-23 16:11:16 UTC
Pushed to 18.05.x for 18.05.06
Comment 14 Fridolin Somers 2018-11-28 15:29:42 UTC
Depends on Bug 9701 not in 17.11.x