Bug 21749 - Importing MARC frameworks from pre-9701 fails
Summary: Importing MARC frameworks from pre-9701 fails
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Target Milestone: ---
Assignee: Jonathan Druart
QA Contact: Marcel de Rooy
URL:
Whiteboard:
Keywords: rel_18_11_candidate
Depends on: 9701
Blocks:
  Show dependency treegraph
 
Reported: 2018-11-01 15:06 UTC by Tomás Cohen Arazi
Modified: 2018-11-28 15:29 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 21749: Add tests for ImportFramework (4.19 KB, patch)
2018-11-12 16:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 21749: Fix import of CSV biblio framework (3.92 KB, patch)
2018-11-12 16:16 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 21749: Add tests for ImportFramework (4.28 KB, patch)
2018-11-12 18:08 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 21749: Fix import of CSV biblio framework (4.02 KB, patch)
2018-11-12 18:09 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 21749: (QA follow-up) Add more tests for ImportFramework (6.40 KB, patch)
2018-11-12 18:09 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 21749: Add tests for ImportFramework (4.28 KB, patch)
2018-11-12 19:08 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 21749: Fix import of CSV biblio framework (4.02 KB, patch)
2018-11-12 19:08 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 21749: (QA follow-up) Add more tests for ImportFramework (48.78 KB, patch)
2018-11-12 19:08 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 21749: Add tests for ImportFramework (4.37 KB, patch)
2018-11-14 14:57 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 21749: Fix import of CSV biblio framework (4.12 KB, patch)
2018-11-14 14:57 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 21749: (QA follow-up) Add more tests for ImportFramework (48.88 KB, patch)
2018-11-14 14:57 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Tomás Cohen Arazi 2018-11-01 15:06:00 UTC
If you import a MARC framework that has been exported from a previous version of Koha (17.11 and below) results in an empty MARC framework, without any error reporting to the end user. This is because of the added columns in 'marc_tag_structure' (ind1_defaultvalue and ind2_defaultvalue) and the fact that we don't have a fallback behaviour when they are not present on the uploaded file.

My opinion is we should check the table header (from CSV or whatever format) and let the DB use the default values if a column is missing.

I mark it as Major, because there's no possible data recovery after the failed import.
Comment 1 Jonathan Druart 2018-11-12 16:16:29 UTC Comment hidden (obsolete)
Comment 2 Jonathan Druart 2018-11-12 16:16:33 UTC Comment hidden (obsolete)
Comment 3 Tomás Cohen Arazi 2018-11-12 18:08:59 UTC Comment hidden (obsolete)
Comment 4 Tomás Cohen Arazi 2018-11-12 18:09:02 UTC Comment hidden (obsolete)
Comment 5 Tomás Cohen Arazi 2018-11-12 18:09:08 UTC Comment hidden (obsolete)
Comment 6 Tomás Cohen Arazi 2018-11-12 19:08:08 UTC
Created attachment 82240 [details] [review]
Bug 21749: Add tests for ImportFramework

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 7 Tomás Cohen Arazi 2018-11-12 19:08:14 UTC
Created attachment 82241 [details] [review]
Bug 21749: Fix import of CSV biblio framework

Bug 9701 added 2 new columns to the table marc_tag_structure which broke
the import of biblio frameworks if the CSV file was generated with a
previous version.

Test plan:
- Export a biblio framework from 17.11
- Import it into 18.05+

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

https://bugs.koha-community.org/show_bug.cgi?id=21740
Comment 8 Tomás Cohen Arazi 2018-11-12 19:08:19 UTC
Created attachment 82242 [details] [review]
Bug 21749: (QA follow-up) Add more tests for ImportFramework

This patch adds tests for:
- Output of ImportFramework on fields removal and error conditions (file
  not found)
- Different file format behaviour (csv, xml and ods are tested).

To test:
- Run
  $ kshell
 k$ prove t/db_dependent/ImportExportFramework.t
=> SUCCESS: All green :-D
- Sign off :-D

Note: the biblio_framework* files are based on the original one on the
patchset, and exported in 17.11 with different formats.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 9 Marcel de Rooy 2018-11-14 14:57:06 UTC
Created attachment 82338 [details] [review]
Bug 21749: Add tests for ImportFramework

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 Marcel de Rooy 2018-11-14 14:57:11 UTC
Created attachment 82339 [details] [review]
Bug 21749: Fix import of CSV biblio framework

Bug 9701 added 2 new columns to the table marc_tag_structure which broke
the import of biblio frameworks if the CSV file was generated with a
previous version.

Test plan:
- Export a biblio framework from 17.11
- Import it into 18.05+

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

https://bugs.koha-community.org/show_bug.cgi?id=21740

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 11 Marcel de Rooy 2018-11-14 14:57:16 UTC
Created attachment 82340 [details] [review]
Bug 21749: (QA follow-up) Add more tests for ImportFramework

This patch adds tests for:
- Output of ImportFramework on fields removal and error conditions (file
  not found)
- Different file format behaviour (csv, xml and ods are tested).

To test:
- Run
  $ kshell
 k$ prove t/db_dependent/ImportExportFramework.t
=> SUCCESS: All green :-D
- Sign off :-D

Note: the biblio_framework* files are based on the original one on the
patchset, and exported in 17.11 with different formats.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 12 Nick Clemens 2018-11-15 12:49:21 UTC
Awesome work all!

Pushed to master for 18.11
Comment 13 Martin Renvoize 2018-11-23 16:11:16 UTC
Pushed to 18.05.x for 18.05.06
Comment 14 Fridolin SOMERS 2018-11-28 15:29:42 UTC
Depends on Bug 9701 not in 17.11.x