Bug 24137 - Marc21 bibliographic fails to install for ru-Ru and uk-UA
Summary: Marc21 bibliographic fails to install for ru-Ru and uk-UA
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Installation and upgrade (web-based installer) (show other bugs)
Version: master
Hardware: All All
: P3 major (vote)
Assignee: Jonathan Druart
QA Contact: Joonas Kylmälä
URL:
Keywords:
Depends on:
Blocks: 17258
  Show dependency treegraph
 
Reported: 2019-11-29 12:25 UTC by Bohdan
Modified: 2020-01-07 23:13 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
20.05.00, 19.11.02


Attachments
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (3.43 KB, patch)
2019-12-02 10:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (3.48 KB, patch)
2019-12-07 11:12 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (mandatory) (574.44 KB, patch)
2019-12-10 11:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (repeatable) (111.91 KB, patch)
2019-12-10 11:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24137: Manual replacement for hidden (35.21 KB, patch)
2019-12-10 11:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (mandatory) (574.49 KB, patch)
2019-12-13 17:55 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (repeatable) (111.96 KB, patch)
2019-12-13 17:55 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24137: Manual replacement for hidden (36.93 KB, patch)
2019-12-13 17:56 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24137: Same fixes for *_local.sql files (100.38 KB, patch)
2019-12-16 09:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24137: Same fixes for *_local.sql files (100.60 KB, patch)
2019-12-16 19:53 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (mandatory) (574.64 KB, patch)
2019-12-17 12:21 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (repeatable) (112.10 KB, patch)
2019-12-17 12:21 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 24137: Manual replacement for hidden (37.08 KB, patch)
2019-12-17 12:21 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 24137: Same fixes for *_local.sql files (100.75 KB, patch)
2019-12-17 12:21 UTC, Joonas Kylmälä
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Bohdan 2019-11-29 12:25:50 UTC
When i try a fresh install for a language other than english, i get this error

marc21_bibliographic_DEFAULT_general.sql

[Fri Nov 29 13:15:36 2019] install.pl: DBD::mysql::st execute failed: Column 'hidden' cannot be null at /usr/share/perl5/DBIx/RunSQL.pm line 273.

marc21_bibliographic_DEFAULT_local.sql

[Fri Nov 29 13:15:36 2019] install.pl: DBD::mysql::st execute failed: Column 'hidden' cannot be null at /usr/share/perl5/DBIx/RunSQL.pm line 273.

As a result, default bibliographic framework is almost empty and import via web interface is broken.
Comment 1 Jonathan Druart 2019-11-29 13:52:03 UTC
Which one are you trying?
Comment 2 Bohdan 2019-11-29 16:22:12 UTC
I am trying ukrainian version.
Koha version is 19.05.04.
Previous versions of 19.05 did not have this issue.
Comment 3 Jonathan Druart 2019-12-02 10:14:54 UTC
Created attachment 95897 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA

They both failed with
   Column 'hidden' cannot be null

Test plan:
Make sure the 2 sql files can be inserted without errors
Comment 4 Katrin Fischer 2019-12-07 11:12:40 UTC
Created attachment 96078 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA

They both failed with
   Column 'hidden' cannot be null

Test plan:
Make sure the 2 sql files can be inserted without errors

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 5 Joonas Kylmälä 2019-12-09 14:37:23 UTC
I'm working on this as my first QA review task.
Comment 6 Joonas Kylmälä 2019-12-10 08:53:54 UTC
With MariaDB this works but inserting the file still fails with MySQL version 5.7.28-0ubuntu0.18.04.4:

> $ koha-mysql kohadev < installer/data/mysql/ru-RU/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql
> ERROR 1366 (HY000) at line 9: Incorrect integer value: '' for column 'repeatable' at row 1
Comment 7 Jonathan Druart 2019-12-10 11:11:50 UTC
Created attachment 96139 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (mandatory)

Replace mandatory='' with mandatory=0

perl -p -i -e "s/^\s\('', '(\d+)', '', / ('', '\1', 0, /" installer/data/mysql/ru-RU/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql installer/data/mysql/uk-UA/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql
Comment 8 Jonathan Druart 2019-12-10 11:11:54 UTC
Created attachment 96140 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (repeatable)

Replace repeatable='' with repeatable=0

perl -p -i -e "s/^\s\('', '(\d+)', (\d), '', / ('', '\1', \2, 0, /" installer/data/mysql/ru-RU/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql installer/data/mysql/uk-UA/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql
Comment 9 Jonathan Druart 2019-12-10 11:11:58 UTC
Created attachment 96141 [details] [review]
Bug 24137: Manual replacement for hidden

Arbitrary set to -6 when was NULL
Comment 10 Joonas Kylmälä 2019-12-10 14:32:28 UTC
It looks like this bug was caused by the change in Bug 23309. And it seems like the change in Bug 23309 also broke the koha database upgrades for installations that used Russian or Ukranian MARC frameworks. (when running the db upgrade in bug 23309 I get "ERROR 1265 (01000): Data truncated for column 'hidden' at row 2")

Jonathan, any idea what the NULL meant before for the visibility? Now you propose -6 which means showing the field in OPAC and Intranet.
Comment 11 Jonathan Druart 2019-12-10 14:41:51 UTC
All NULL from this patch should be compared with the .sql we have in 'en'. But I cannot find time for that.

Maybe 8 is better.
Comment 12 Bernardo Gonzalez Kriegel 2019-12-13 14:51:15 UTC
(In reply to Jonathan Druart from comment #11)
> All NULL from this patch should be compared with the .sql we have in 'en'.
> But I cannot find time for that.
> 
Checked it out

027_2 not present
048_c not present
091_bcdefgi not present
094 not present
097 not present
777 Ok
786 Ok
810_q not present
811_mor not present
830_qu not present

By 'no present' I mean not in MARC21 and/or not in 'en' version.
Ok is the same as 'en'

Most of the lines in the last patch should be to remove the entries.
Comment 13 Jonathan Druart 2019-12-13 15:06:22 UTC
Gracias Bernardo!

Do you mean we should remove the inserts?
I would not do that, how can we know those guys don't use them?
Comment 14 Bernardo Gonzalez Kriegel 2019-12-13 15:43:59 UTC
(In reply to Jonathan Druart from comment #13)
> Gracias Bernardo!

No hay de qué, Jonathan!
 
> Do you mean we should remove the inserts?
> I would not do that, how can we know those guys don't use them?

We can't know unfortunately.
I agree that we can't remove the inserts, for now each language dir has it's own maintainer (or none), just fixing the files it's Ok

As for '-6' for visibility, represents 'hidden', it's the value that I choose as default when inserting new fields/subfields, I think it's Ok too

If there are interest, I can setup entries for ru and uk in https://translate.koha-community.org/projects/marc21/, then get updated frameworks from https://translate.koha-community.org/files/, and replace these files.
Comment 15 Bernardo Gonzalez Kriegel 2019-12-13 17:55:53 UTC
Created attachment 96280 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (mandatory)

Replace mandatory='' with mandatory=0

perl -p -i -e "s/^\s\('', '(\d+)', '', / ('', '\1', 0, /" installer/data/mysql/ru-RU/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql installer/data/mysql/uk-UA/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 16 Bernardo Gonzalez Kriegel 2019-12-13 17:55:57 UTC
Created attachment 96281 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (repeatable)

Replace repeatable='' with repeatable=0

perl -p -i -e "s/^\s\('', '(\d+)', (\d), '', / ('', '\1', \2, 0, /" installer/data/mysql/ru-RU/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql installer/data/mysql/uk-UA/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 17 Bernardo Gonzalez Kriegel 2019-12-13 17:56:01 UTC
Created attachment 96282 [details] [review]
Bug 24137: Manual replacement for hidden

Arbitrary set to -6 when was NULL

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Fixed 991 in uk-UA marc21_bibliographic_DEFAULT_local.sql

All files load without errors, mandatory and optional
Comment 18 Joonas Kylmälä 2019-12-16 08:53:45 UTC
This still doesn't fix the *_local.sql files insertion for russian or ukranian:

> koha-mysql kohadev < installer/data/mysql/uk-UA/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_local.sql
> ERROR 1366 (HY000) at line 4: Incorrect integer value: '' for column 'mandatory' at row 1

Only *_general.sql is fixed completely. For the *_local.sql files only the hidden column is fixed so the installer would fail here.
Comment 19 Jonathan Druart 2019-12-16 09:34:53 UTC
Created attachment 96306 [details] [review]
Bug 24137: Same fixes for *_local.sql files
Comment 20 Joonas Kylmälä 2019-12-16 10:34:33 UTC
Bernardo, can you sign-off also the last patch?
Comment 21 Bernardo Gonzalez Kriegel 2019-12-16 19:53:55 UTC
Created attachment 96353 [details] [review]
Bug 24137: Same fixes for *_local.sql files

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
tested using

$ export L=uk-UA (and ru-RU)
$ for o in mandatory optional;
 do for a in installer/data/mysql/$L/marcflavour/marc21/$o/*.sql; do
  mysql kohatest < $a;
 done;
done
Comment 22 Joonas Kylmälä 2019-12-17 12:21:46 UTC
Created attachment 96365 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (mandatory)

Replace mandatory='' with mandatory=0

perl -p -i -e "s/^\s\('', '(\d+)', '', / ('', '\1', 0, /" installer/data/mysql/ru-RU/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql installer/data/mysql/uk-UA/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Comment 23 Joonas Kylmälä 2019-12-17 12:21:49 UTC
Created attachment 96366 [details] [review]
Bug 24137: Fix biblio frameworks install for ru-Ru and uk-UA (repeatable)

Replace repeatable='' with repeatable=0

perl -p -i -e "s/^\s\('', '(\d+)', (\d), '', / ('', '\1', \2, 0, /" installer/data/mysql/ru-RU/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql installer/data/mysql/uk-UA/marcflavour/marc21/mandatory/marc21_bibliographic_DEFAULT_general.sql

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Comment 24 Joonas Kylmälä 2019-12-17 12:21:53 UTC
Created attachment 96367 [details] [review]
Bug 24137: Manual replacement for hidden

Arbitrary set to -6 when was NULL

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Fixed 991 in uk-UA marc21_bibliographic_DEFAULT_local.sql

All files load without errors, mandatory and optional
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Comment 25 Joonas Kylmälä 2019-12-17 12:21:57 UTC
Created attachment 96368 [details] [review]
Bug 24137: Same fixes for *_local.sql files

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
tested using

$ export L=uk-UA (and ru-RU)
$ for o in mandatory optional;
 do for a in installer/data/mysql/$L/marcflavour/marc21/$o/*.sql; do
  mysql kohatest < $a;
 done;
done
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Comment 26 Martin Renvoize 2019-12-18 16:06:32 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 27 Martin Renvoize 2019-12-20 12:09:49 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 28 Joy Nelson 2020-01-03 22:13:17 UTC
Pushed to 19.11.x for 19.11.02
Comment 29 Lucas Gass 2020-01-07 23:13:35 UTC
backported to 19.05.x for 19.05.07