Bug 16554 - Web installer fails to load i18n sample data on MySQL 5.6+
Summary: Web installer fails to load i18n sample data on MySQL 5.6+
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Installation and upgrade (web-based installer) (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low critical (vote)
Assignee: Bernardo Gonzalez Kriegel
QA Contact: Testopia
URL:
Keywords:
: 16775 (view as bug list)
Depends on: 16573
Blocks: 16797 17258
  Show dependency treegraph
 
Reported: 2016-05-19 13:51 UTC by Bernardo Gonzalez Kriegel
Modified: 2019-06-27 09:24 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Large patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13669: (alternative) use DBIx::RunSQL (3.33 KB, patch)
2016-05-19 13:58 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: Fix fr-FR sample files (1.35 MB, patch)
2016-05-19 13:58 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: Fix I18N sample files (7.26 KB, patch)
2016-05-19 13:58 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - de-DE (420.33 KB, patch)
2016-05-25 01:21 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - es-ES (438.52 KB, patch)
2016-05-25 01:23 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - it-IT (426.78 KB, patch)
2016-05-25 02:23 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - it-IT (426.87 KB, patch)
2016-05-25 02:28 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - nb-NO (2.88 KB, patch)
2016-05-25 02:31 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - pl-PL (1.87 MB, patch)
2016-05-25 02:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: DO NOT PUSH - test for all sample files (2.70 KB, patch)
2016-05-26 14:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: DO NOT PUSH - (followup) - sorted filenames (1.29 KB, patch)
2016-05-28 03:11 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: More i18n changes - en, es, nb and pl files (2.33 MB, patch)
2016-05-28 16:07 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: More i18n changes - fr-FR files (296.67 KB, application/gzip)
2016-05-28 16:20 UTC, Bernardo Gonzalez Kriegel
Details
Bug 16554: More i18n changes - ru-RU files (308.53 KB, application/gzip)
2016-05-28 16:22 UTC, Bernardo Gonzalez Kriegel
Details
Bug 16554: More i18n changes - uk-UA files (361.39 KB, application/gzip)
2016-05-28 16:24 UTC, Bernardo Gonzalez Kriegel
Details
Bug 16554: rewrite mandatory and sample data - it-IT (427.51 KB, patch)
2016-08-08 00:27 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - de-DE (420.95 KB, patch)
2016-08-08 00:28 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554: Fix encoding on it-IT file (7.11 KB, patch)
2016-08-12 19:32 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 16554 - DO NOT PUSH - Follow up to testing script (2.36 KB, patch)
2016-08-12 23:26 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: Fix fr-FR sample files (1.35 MB, patch)
2016-08-13 00:04 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: Fix I18N sample files (7.31 KB, patch)
2016-08-13 00:04 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - es-ES (438.57 KB, patch)
2016-08-13 00:04 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - nb-NO (2.93 KB, patch)
2016-08-13 00:04 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - pl-PL (1.87 MB, patch)
2016-08-13 00:04 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: DO NOT PUSH - test for all sample files (2.75 KB, patch)
2016-08-13 00:05 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: DO NOT PUSH - (followup) - sorted filenames (1.34 KB, patch)
2016-08-13 00:05 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: More i18n changes - en, es, nb and pl files (2.33 MB, patch)
2016-08-13 00:05 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - it-IT (427.56 KB, patch)
2016-08-13 00:06 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - de-DE (421.00 KB, patch)
2016-08-13 00:06 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: Fix encoding on it-IT file (7.16 KB, patch)
2016-08-13 00:06 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554 - DO NOT PUSH - Follow up to testing script (2.41 KB, patch)
2016-08-13 00:06 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16554: Fix fr-FR sample files (1.35 MB, patch)
2016-09-06 07:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: Fix I18N sample files (7.38 KB, patch)
2016-09-06 07:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - es-ES (438.64 KB, patch)
2016-09-06 07:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - nb-NO (3.00 KB, patch)
2016-09-06 07:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - pl-PL (1.87 MB, patch)
2016-09-06 07:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: More i18n changes - en, es, nb and pl files (2.33 MB, patch)
2016-09-06 07:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - it-IT (427.63 KB, patch)
2016-09-06 07:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: rewrite mandatory and sample data - de-DE (421.07 KB, patch)
2016-09-06 07:58 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: Fix encoding on it-IT file (6.53 KB, patch)
2016-09-06 07:58 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 16554: More i18n changes - fr-FR files (218 bytes, application/gzip)
2016-09-06 08:01 UTC, Jonathan Druart
Details
Bug 16554: More i18n changes - ru-RU files (319.33 KB, application/gzip)
2016-09-06 08:03 UTC, Jonathan Druart
Details
Bug 16554: More i18n changes - uk-UA files (373.03 KB, application/gzip)
2016-09-06 08:03 UTC, Jonathan Druart
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bernardo Gonzalez Kriegel 2016-05-19 13:51:54 UTC
Followup to bug 13669
Comment 1 Bernardo Gonzalez Kriegel 2016-05-19 13:58:14 UTC Comment hidden (obsolete)
Comment 2 Bernardo Gonzalez Kriegel 2016-05-19 13:58:20 UTC Comment hidden (obsolete)
Comment 3 Bernardo Gonzalez Kriegel 2016-05-19 13:58:24 UTC Comment hidden (obsolete)
Comment 4 Jonathan Druart 2016-05-24 19:10:47 UTC
Is it ready to be tested?
Comment 5 Bernardo Gonzalez Kriegel 2016-05-25 01:21:40 UTC Comment hidden (obsolete)
Comment 6 Bernardo Gonzalez Kriegel 2016-05-25 01:23:13 UTC Comment hidden (obsolete)
Comment 7 Bernardo Gonzalez Kriegel 2016-05-25 02:23:27 UTC Comment hidden (obsolete)
Comment 8 Bernardo Gonzalez Kriegel 2016-05-25 02:28:30 UTC Comment hidden (obsolete)
Comment 9 Bernardo Gonzalez Kriegel 2016-05-25 02:31:02 UTC Comment hidden (obsolete)
Comment 10 Bernardo Gonzalez Kriegel 2016-05-25 02:32:33 UTC Comment hidden (obsolete)
Comment 11 Bernardo Gonzalez Kriegel 2016-05-25 02:40:18 UTC
(In reply to Jonathan Druart from comment #4)
> Is it ready to be tested?

Yep, it's ready.

ru-RU and uk-UA not fixed, yet.
Perhaps on other bug after release.

I think we need this ASAP.

TEST PLAN
---------

ON TOP OF: Bug 16573

1) Apply ALL patches
2) Use MySQL 5.7
3) Then test web installer for all langs
 marc21: de-DE, es-ES, fr-CA, pl-PL
 unimarc: fr-FR (complete), it-IT
 normarc: nb-NO

 There must be no errors

4) Enjoy !!!
Comment 12 Jonathan Druart 2016-05-26 14:18:37 UTC Comment hidden (obsolete)
Comment 13 Jonathan Druart 2016-05-26 14:20:42 UTC
(In reply to Jonathan Druart from comment #12)
> Created attachment 51840 [details] [review] [review]
> Bug 16554: DO NOT PUSH - test for all sample files
> 
> You will need to create a koha_test_installer DB before

I did not finish the execution of this script but it should permit to test all sample files without inserting it manually.
Could someone test it on MySQL 5.7 and all patches from this bug report (and from bug 16573 as well).
Comment 14 Jonathan Druart 2016-05-26 14:21:04 UTC
(In reply to Jonathan Druart from comment #13)
> Could someone test it on MySQL 5.7 and all patches from this bug report (and
> from bug 16573 as well).

And make sure it raises warnings/errors without!
Comment 15 Bernardo Gonzalez Kriegel 2016-05-28 03:11:16 UTC Comment hidden (obsolete)
Comment 16 Bernardo Gonzalez Kriegel 2016-05-28 03:18:57 UTC
(In reply to Jonathan Druart from comment #13)
> 
> I did not finish the execution of this script but it should permit to test
> all sample files without inserting it manually.

I did, several times.
Great tool Jonathan, it deserves to stay... somewhere :)

> Could someone test it on MySQL 5.7 and all patches from this bug report (and
> from bug 16573 as well).

I found and fixed files for en, es-ES, fr-*, nb & pl
I'm fixing ru & uk, it's a nightmare.
I'll upload more patches soon.
Comment 17 Bernardo Gonzalez Kriegel 2016-05-28 16:07:10 UTC Comment hidden (obsolete)
Comment 18 Bernardo Gonzalez Kriegel 2016-05-28 16:20:18 UTC Comment hidden (obsolete)
Comment 19 Bernardo Gonzalez Kriegel 2016-05-28 16:22:19 UTC Comment hidden (obsolete)
Comment 20 Bernardo Gonzalez Kriegel 2016-05-28 16:24:07 UTC Comment hidden (obsolete)
Comment 21 Bernardo Gonzalez Kriegel 2016-05-28 16:37:55 UTC
REVISED TEST PLAN
-----------------

Apply all patches from 16573, all patches from this one, 
download, gunzip and apply last three patches

On MySQL 5.7 create koha_test_installer db, then run Jonathan test file :)

Must see no warnings/errors

SAMPLE OUTPUT
-------------
$ mysql --version
mysql  Ver 14.14 Distrib 5.7.12, for Linux (x86_64) using  EditLine wrapper
$ prove -v t/db_dependent/Test_installer.t
t/db_dependent/Test_installer.t .. 
loading kohastructure for de-DE | not marc21
loading kohastructure for de-DE | not unimarc
loading kohastructure for en | not marc21
loading kohastructure for en | not unimarc
loading kohastructure for es-ES | not marc21
loading kohastructure for es-ES | not unimarc
loading kohastructure for fr-CA | not marc21
loading kohastructure for fr-CA | not unimarc
loading kohastructure for fr-FR | not marc21
loading kohastructure for fr-FR | not unimarc
loading kohastructure for it-IT | not marc21
loading kohastructure for it-IT | not unimarc
loading kohastructure for nb-NO | not marc21
loading kohastructure for nb-NO | not unimarc
loading kohastructure for pl-PL | not marc21
loading kohastructure for pl-PL | not unimarc
loading kohastructure for ru-RU | not marc21
loading kohastructure for ru-RU | not unimarc
loading kohastructure for uk-UA | not marc21
loading kohastructure for uk-UA | not unimarc
110/110
No subtests run 

Test Summary Report
-------------------
t/db_dependent/Test_installer.t (Wstat: 0 Tests: 0 Failed: 0)
  Parse errors: No plan found in TAP output
Files=1, Tests=0, 3916 wallclock secs ( 0.06 usr  0.01 sys + 10.68 cusr  1.46 csys = 12.21 CPU)
Result: FAIL

Don't worry about FAIL message, with this output all is Ok.
On my very modest system it took ~ 60 minutes.

This has become a gigantic patchset, which guarantees that Koha
installs on MySQL 5.7, any language/marcflavour, no that it runs. 
But it's a first step.
Comment 22 M. Tompsett 2016-07-21 23:24:08 UTC
Please rebase. I was going to test.
Comment 23 M. Tompsett 2016-07-21 23:30:43 UTC
de-DE and it-IT are the ones that fail.
the zip files still apply, though whitespace issues.
Comment 24 Katrin Fischer 2016-08-01 06:06:07 UTC
Sad to see this stuck - any chance for a rebase?
Comment 25 Bernardo Gonzalez Kriegel 2016-08-08 00:27:56 UTC Comment hidden (obsolete)
Comment 26 Bernardo Gonzalez Kriegel 2016-08-08 00:28:29 UTC Comment hidden (obsolete)
Comment 27 Bernardo Gonzalez Kriegel 2016-08-08 00:34:05 UTC
de-DE and it-IT rebased
prove -v t/db_dependent/Test_installer.t still works for all languages.

Download compressed files using
wget -O fr-FR.gz http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51898
wget -O ru-RU.gz http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51899
wget -O uk-UA.gz http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51900

gunzip and apply using git am -3 xx-YY
Comment 28 M. Tompsett 2016-08-08 03:43:01 UTC
Finally got around to attempting to run it on my shiny new Ubuntu 16.04 LTS server VM.
--- SNIP ---
loading kohastructure for it-IT | not unimarc
C4::Installer::load_sql returned the following errors while attempting to load /home/mtompset/kohaclone/installer/data/mysql/it-IT/necessari/sample_numberpatterns.sql:
DBD::mysql::st execute failed: Incorrect string value: '\xE0 l'an...' for column 'description' at row 5 at /usr/share/perl5/DBIx/RunSQL.pm line 273.
loading kohastructure for nb-NO | not marc21
C4::Installer::load_sql returned the following errors while attempting to load /home/mtompset/kohaclone/installer/data/mysql/it-IT/necessari/sample_numberpatterns.sql:
DBD::mysql::st execute failed: Incorrect string value: '\xE0 l'an...' for column 'description' at row 5 at /usr/share/perl5/DBIx/RunSQL.pm line 273.
--- END SNIP ---

So I added a note line...
# /home/mtompset/kohaclone/installer/data/mysql/it-IT/necessari/sample_numberpatterns.sql
C4::Installer::load_sql returned the following errors while attempting to load /home/mtompset/kohaclone/installer/data/mysql/it-IT/necessari/sample_numberpatterns.sql:
DBD::mysql::st execute failed: Incorrect string value: '\xE0 l'an...' for column 'description' at row 5 at /usr/share/perl5/DBIx/RunSQL.pm line 273.

Fix and I'll retest.
Comment 29 M. Tompsett 2016-08-08 05:25:30 UTC
vi installer/.../subscription_numberpatterns.sql
-- notice the [converted]
:set fileencoding=utf-8
:wq

I believe that is all that is needed to fix it.
Comment 30 M. Tompsett 2016-08-11 23:51:03 UTC
(In reply to M. Tompsett from comment #29)
> vi installer/.../subscription_numberpatterns.sql
> -- notice the [converted]
> :set fileencoding=utf-8
> :wq
> 
> I believe that is all that is needed to fix it.

While that fixes some, still issues with:
installer/data/mysql/fr-FR/marcflavour/unimarc_lecture_pub/Obligatoire/autorites_norme_unimarc.sql -- something about 'CO' duplicate primary key.
installer/data/mysql/fr-FR/marcflavour/unimarc_lecture_pub/Obligatoire/framework_DEFAULT.sql -- something about '000' duplicate primary key.
Also, ru-RU and uk-UA also seem to trigger on mandatory and repeatable fields now. They are rather floody.
Comment 31 Bernardo Gonzalez Kriegel 2016-08-12 19:32:30 UTC Comment hidden (obsolete)
Comment 32 Bernardo Gonzalez Kriegel 2016-08-12 19:34:19 UTC
(In reply to M. Tompsett from comment #30)
...
> While that fixes some, still issues with:
> installer/data/mysql/fr-FR/marcflavour/unimarc_lecture_pub/Obligatoire/
> autorites_norme_unimarc.sql -- something about 'CO' duplicate primary key.
> installer/data/mysql/fr-FR/marcflavour/unimarc_lecture_pub/Obligatoire/
> framework_DEFAULT.sql -- something about '000' duplicate primary key.
> Also, ru-RU and uk-UA also seem to trigger on mandatory and repeatable
> fields now. They are rather floody.

Did you apply the compressed patches? 

Fixed encoding on it-IT file.
Comment 33 M. Tompsett 2016-08-12 21:24:37 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #32)
> Did you apply the compressed patches? 

Oh, I may have forgotten that. Let me recheck with your encoding fix.
Comment 34 M. Tompsett 2016-08-12 23:26:05 UTC Comment hidden (obsolete)
Comment 35 M. Tompsett 2016-08-13 00:04:01 UTC Comment hidden (obsolete)
Comment 36 M. Tompsett 2016-08-13 00:04:07 UTC Comment hidden (obsolete)
Comment 37 M. Tompsett 2016-08-13 00:04:18 UTC Comment hidden (obsolete)
Comment 38 M. Tompsett 2016-08-13 00:04:26 UTC Comment hidden (obsolete)
Comment 39 M. Tompsett 2016-08-13 00:04:57 UTC Comment hidden (obsolete)
Comment 40 M. Tompsett 2016-08-13 00:05:08 UTC
Created attachment 54410 [details] [review]
Bug 16554: DO NOT PUSH - test for all sample files

You will need to create a koha_test_installer DB before

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Comment 41 M. Tompsett 2016-08-13 00:05:13 UTC
Created attachment 54411 [details] [review]
Bug 16554: DO NOT PUSH - (followup) - sorted filenames

Filenames must be loaded in alphabetical order.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Comment 42 M. Tompsett 2016-08-13 00:05:52 UTC Comment hidden (obsolete)
Comment 43 M. Tompsett 2016-08-13 00:06:05 UTC Comment hidden (obsolete)
Comment 44 M. Tompsett 2016-08-13 00:06:16 UTC Comment hidden (obsolete)
Comment 45 M. Tompsett 2016-08-13 00:06:23 UTC Comment hidden (obsolete)
Comment 46 M. Tompsett 2016-08-13 00:06:30 UTC
Created attachment 54416 [details] [review]
Bug 16554 - DO NOT PUSH - Follow up to testing script

Made it so that it really does fail or not based on testing.
Also, added printing of files it is loading as it goes along.
Forced a fail test for the files that don't load so they
will get summarized at the end.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Comment 47 M. Tompsett 2016-08-13 00:07:34 UTC
Those gzip's are a pain to handle. This comment serves as notice that I, Mark Tompsett, sign those off too!
Comment 48 M. Tompsett 2016-08-14 15:52:17 UTC
*** Bug 16775 has been marked as a duplicate of this bug. ***
Comment 49 Jonathan Druart 2016-08-15 13:01:52 UTC
Did the test script pass for you guys?
I get errors on "loading kohastructure for fr-FR | not marc21", stuffs like "Cannot add or update a child row: a foreign key constraint fails" and "Duplicate entry 'CO' for key 'PRIMARY'"
Comment 50 Jonathan Druart 2016-08-15 13:22:08 UTC
(In reply to Jonathan Druart from comment #49)
> Did the test script pass for you guys?
> I get errors on "loading kohastructure for fr-FR | not marc21", stuffs like
> "Cannot add or update a child row: a foreign key constraint fails" and
> "Duplicate entry 'CO' for key 'PRIMARY'"

Forgot the gzip files!
Comment 51 Jonathan Druart 2016-08-15 13:36:03 UTC
Still fails, even with gzip files:
# 21/26: /home/koha/src/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/Obligatoire/autorite_default.sql
C4::Installer::load_sql returned the following errors while attempting to load /home/koha/src/installer/data/mysql/fr-FR/marcflavour/un
imarc_complet/Obligatoire/autorites_norme_unimarc.sql:
not ok 1 - /home/koha/src/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/Obligatoire/autorites_norme_unimarc.sql: DBD::mysql::s
t execute failed: Cannot add or update a child row: a foreign key constraint fails (`koha_test_installer`.`auth_subfield_structure`, CO
NSTRAINT `auth_subfield_structure_ibfk_1` FOREIGN KEY (`authtypecode`) REFERENCES `auth_types` (`authtypecode`) ON DELETE CASCADE ON UP
DATE C) at /usr/local/share/perl/5.22.1/DBIx/RunSQL.pm line 254.
# 
#   Failed test '/home/koha/src/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/Obligatoire/autorites_norme_unimarc.sql: DBD::my
sql::st execute failed: Cannot add or update a child row: a foreign key constraint fails (`koha_test_installer`.`auth_subfield_structur
e`, CONSTRAINT `auth_subfield_structure_ibfk_1` FOREIGN KEY (`authtypecode`) REFERENCES `auth_types` (`authtypecode`) ON DELETE CASCADE ON UPDATE C) at /usr/local/share/perl/5.22.1/DBIx/RunSQL.pm line 254.
# '
#   at t/db_dependent/Test_installer.t line 48.
Comment 52 Bernardo Gonzalez Kriegel 2016-08-22 23:10:38 UTC
With all patches applied, all test succeed with Marc fix :)

git log --oneline | head -16
c1bfb24 Bug 16554: More i18n changes - uk-UA files
23001ea Bug 16554: More i18n changes - ru-RU files
ca09bf0 Bug 16554: More i18n changes - fr-FR files
72ca21d Bug 16554 - DO NOT PUSH - Follow up to testing script
4c32489 Bug 16554: Fix encoding on it-IT file
0644c60 Bug 16554: rewrite mandatory and sample data - de-DE
989c319 Bug 16554: rewrite mandatory and sample data - it-IT
9c6ce1d Bug 16554: More i18n changes - en, es, nb and pl files
c85d6e7 Bug 16554: DO NOT PUSH - (followup) - sorted filenames
79c2f1f Bug 16554: DO NOT PUSH - test for all sample files
1442061 Bug 16554: rewrite mandatory and sample data - pl-PL
596721f Bug 16554: rewrite mandatory and sample data - nb-NO
63afd53 Bug 16554: rewrite mandatory and sample data - es-ES
83b2d0d Bug 16554: Fix I18N sample files
1ba4ed1 Bug 16554: Fix fr-FR sample files
3315a88 Bug 17023: Repair "cancel" link

...
ok
All tests successful.
Files=1, Tests=1, 3975 wallclock secs ( 0.24 usr  0.02 sys + 10.02 cusr  1.48 csys = 11.76 CPU)
Result: PASS

Dropping and re-creating database and running for fr-FR only it's Ok, all success.

$ prove -v t/db_dependent/Test_installer.t
t/db_dependent/Test_installer.t .. DBD::mysql::db do failed: Table 'koha_test_installer.systempreferences' doesn't exist [for Statement "
            SELECT * FROM systempreferences WHERE 1 = 0 "] at /home/bgkriegel/kohaclone/Koha/Database.pm line 93.

loading kohastructure for fr-FR | not marc21
# 01/26: /home/bgkriegel/kohaclone/installer/data/mysql/audio_alerts.sql
...
# 26/26: /home/bgkriegel/kohaclone/installer/data/mysql/fr-FR/marcflavour/unimarc_lecture_pub/Obligatoire/framework_DEFAULT.sql
loading kohastructure for fr-FR | not unimarc
# 01/24: /home/bgkriegel/kohaclone/installer/data/mysql/audio_alerts.sql
...
# 24/24: /home/bgkriegel/kohaclone/installer/data/mysql/fr-FR/marcflavour/marc21/Optionnel/marc21_simple_bib_frameworks.sql
ok 1 - No failures
1..1
ok
All tests successful.
Files=1, Tests=1, 327 wallclock secs ( 0.05 usr  0.00 sys +  2.01 cusr  0.22 csys =  2.28 CPU)
Result: PASS

Don't know what to think :(
Comment 53 M. Tompsett 2016-08-23 02:05:07 UTC
(In reply to Jonathan Druart from comment #51)
> Still fails, even with gzip files:

Could you ls -la the offending files?

$ ls -la /home/mtompset/kohaclone/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/Obligatoire/autorites_norme_unimarc.sql
-rw-rw-r-- 1 mtompset mtompset 323371 Aug 22 21:28 /home/mtompset/kohaclone/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/Obligatoire/autorites_norme_unimarc.sql


I just retested on Debian, which is still <5.7, and Ubuntu 16.04 LTS.

Perhaps your files are older? Try reloading them from scratch?

mkdir ~/16554
cd ~/16554
wget -O fr-FR.gz https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51898
wget -O ru-RU.gz https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51899
wget -O uk-UA.gz https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=51900
gunzip fr-FR.gz
gunzip ru-RU.gz
gunzip uk-UA.gz
cd ~/kohaclone
git checkout -b bug_16554 origin/master
git pull
git am < ~/16554/fr-FR
git am < ~/16554/ru-RU
git am < ~/16554/uk-UA
git bz apply 16554
...

Because I'm just as confused as BGK as to why you might have those issues, JD. Though, I really hate those white space issues with the patches, but that isn't causing me any failures.
Comment 54 M. Tompsett 2016-08-23 04:42:27 UTC
(In reply to Jonathan Druart from comment #51)
> Still fails, even with gzip files:
> # 21/26:
> /home/koha/src/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/
> Obligatoire/autorite_default.sql
> C4::Installer::load_sql returned the following errors while attempting to
> load /home/koha/src/installer/data/mysql/fr-FR/marcflavour/un
> imarc_complet/Obligatoire/autorites_norme_unimarc.sql:
> not ok 1 -
> /home/koha/src/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/
> Obligatoire/autorites_norme_unimarc.sql: DBD::mysql::s
> t execute failed: Cannot add or update a child row: a foreign key constraint
> fails (`koha_test_installer`.`auth_subfield_structure`, CO
> NSTRAINT `auth_subfield_structure_ibfk_1` FOREIGN KEY (`authtypecode`)
> REFERENCES `auth_types` (`authtypecode`) ON DELETE CASCADE ON UP
> DATE C) at /usr/local/share/perl/5.22.1/DBIx/RunSQL.pm line 254.
> # 
> #   Failed test
> '/home/koha/src/installer/data/mysql/fr-FR/marcflavour/unimarc_complet/
> Obligatoire/autorites_norme_unimarc.sql: DBD::my
> sql::st execute failed: Cannot add or update a child row: a foreign key
> constraint fails (`koha_test_installer`.`auth_subfield_structur
> e`, CONSTRAINT `auth_subfield_structure_ibfk_1` FOREIGN KEY (`authtypecode`)
> REFERENCES `auth_types` (`authtypecode`) ON DELETE CASCADE ON UPDATE C) at
> /usr/local/share/perl/5.22.1/DBIx/RunSQL.pm line 254.
> # '
> #   at t/db_dependent/Test_installer.t line 48.

OH! I think I have it. If you recall ALTER IGNORE doesn't work in MySQL 5.7.4, so if you upgraded through 3.19.00.042, you may find the upgrade had issues.

I have a massive head cold. If no one posts a "make all the ALTER IGNORE into ALTER only" patch later, I will. I'm going to bed.
Comment 55 Jonathan Druart 2016-08-23 14:32:58 UTC
(In reply to M. Tompsett from comment #54)
> OH! I think I have it. If you recall ALTER IGNORE doesn't work in MySQL
> 5.7.4, so if you upgraded through 3.19.00.042, you may find the upgrade had
> issues.
> 
> I have a massive head cold. If no one posts a "make all the ALTER IGNORE
> into ALTER only" patch later, I will. I'm going to bed.

I don't understand, I was just using the test script t/db_dependent/Test_installer.t, there is nothing about the updatedatabase script inside.
Comment 56 M. Tompsett 2016-08-23 17:36:46 UTC
(In reply to Jonathan Druart from comment #55)
> (In reply to M. Tompsett from comment #54)
> > OH! I think I have it. If you recall ALTER IGNORE doesn't work in MySQL
> > 5.7.4, so if you upgraded through 3.19.00.042, you may find the upgrade had
> > issues.
> 
> I don't understand, I was just using the test script
> t/db_dependent/Test_installer.t, there is nothing about the updatedatabase
> script inside.

Yes, but is your schema for koha_test_installer current? Did you copy if from an upgraded DB? I'm just trying to think of ways you could get up with the constraint failing. I don't guarantee they are good ideas. :)
Comment 57 Jonathan Druart 2016-08-23 17:56:17 UTC
(In reply to M. Tompsett from comment #56)
> (In reply to Jonathan Druart from comment #55)
> > (In reply to M. Tompsett from comment #54)
> > > OH! I think I have it. If you recall ALTER IGNORE doesn't work in MySQL
> > > 5.7.4, so if you upgraded through 3.19.00.042, you may find the upgrade had
> > > issues.
> > 
> > I don't understand, I was just using the test script
> > t/db_dependent/Test_installer.t, there is nothing about the updatedatabase
> > script inside.
> 
> Yes, but is your schema for koha_test_installer current? Did you copy if
> from an upgraded DB? I'm just trying to think of ways you could get up with
> the constraint failing. I don't guarantee they are good ideas. :)

The script recreate the DB before each run/language.
Comment 58 M. Tompsett 2016-08-23 18:12:43 UTC
(In reply to Jonathan Druart from comment #57)
> The script recreate the DB before each run/language.

No, each table. But that wouldn't explain fr-FR which wasn't first. *sigh* Just trying to think of something.
Comment 59 Jonathan Druart 2016-08-24 07:53:32 UTC
(In reply to M. Tompsett from comment #58)
> (In reply to Jonathan Druart from comment #57)
> > The script recreate the DB before each run/language.
> 
> No, each table. But that wouldn't explain fr-FR which wasn't first. *sigh*
> Just trying to think of something.

It's not important, but the script recreates the table before each new "install" (lang+marcflavour):

 29 for my $lang ( qw ( de-DE en es-ES fr-CA fr-FR it-IT nb-NO pl-PL ru-RU uk-UA ) ) {
 30     for my $marcflavour ( qw( marc21 unimarc ) ) {
 31         say "loading kohastructure for $lang | not $marcflavour";
 32         recreate_db( $db_name );
Comment 60 Jonathan Druart 2016-09-06 07:57:17 UTC
Created attachment 55219 [details] [review]
Bug 16554: Fix fr-FR sample files

Needs Bug 13669

Rewrite autorites_norme_unimarc.sql to
use only one INSERT IGNORE.

New load method complains about duplicate keys,
and both auth files are mandatory!

Also removed blank space from sample_labels.sql

Tested with unimarc_complet and all sample files.

To test
1) Apply the patch
2) Try fr-FR web installer, unimarc_complet + all sample files
There must be no errors

Don't know if fr-FR marc21 and unimarc_lecture_pub are used,
need a look from french users.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 61 Jonathan Druart 2016-09-06 07:57:22 UTC
Created attachment 55220 [details] [review]
Bug 16554: Fix I18N sample files

Needs Bug 13669

This patch fix web installer for
de-DE, es-ES, fr-CA, nb-NO and pl-PL

To test:
1) Apply patch
2) Try web installer for any/all listed languages.
a) de-DE, es-ES, fr-CA and pl-PL
There must be no problems for marc21 + all sample files
b) nb-NO
There must be no problems for normarc + all sample files

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 62 Jonathan Druart 2016-09-06 07:57:28 UTC
Created attachment 55221 [details] [review]
Bug 16554: rewrite mandatory and sample data - es-ES

More changes to es-ES files

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 63 Jonathan Druart 2016-09-06 07:57:32 UTC
Created attachment 55222 [details] [review]
Bug 16554: rewrite mandatory and sample data - nb-NO

More changes to nb-NO files

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 64 Jonathan Druart 2016-09-06 07:57:41 UTC
Created attachment 55223 [details] [review]
Bug 16554: rewrite mandatory and sample data - pl-PL

More changes to pl-PL files

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 65 Jonathan Druart 2016-09-06 07:57:53 UTC
Created attachment 55224 [details] [review]
Bug 16554: More i18n changes - en, es, nb and pl files

Mostly fixes bad values in auth files ( '' -> 0 )

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 66 Jonathan Druart 2016-09-06 07:57:58 UTC
Created attachment 55225 [details] [review]
Bug 16554: rewrite mandatory and sample data - it-IT

More changes to it-IT files

Checked marc21 and unimarc files
unimarc_relatorterms.sql had bad encoding, others minor
errors

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 67 Jonathan Druart 2016-09-06 07:58:04 UTC
Created attachment 55226 [details] [review]
Bug 16554: rewrite mandatory and sample data - de-DE

More changes to de-DE files

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 68 Jonathan Druart 2016-09-06 07:58:08 UTC
Created attachment 55227 [details] [review]
Bug 16554: Fix encoding on it-IT file

This patch only fixes encoding of file
installer/data/mysql/it-IT/necessari/sample_numberpatterns.sql

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 69 Jonathan Druart 2016-09-06 08:01:47 UTC
Created attachment 55228 [details]
Bug 16554: More i18n changes - fr-FR files
Comment 70 Jonathan Druart 2016-09-06 08:03:04 UTC
Created attachment 55229 [details]
Bug 16554: More i18n changes - ru-RU files
Comment 71 Jonathan Druart 2016-09-06 08:03:32 UTC
Created attachment 55230 [details]
Bug 16554: More i18n changes - uk-UA files
Comment 72 Jonathan Druart 2016-09-06 08:07:07 UTC
I have successfully made an install with complet and STRICT_TRANS_TABLES, so I think we could let them go.
We need this patch set pushed ASAP to catch remaining bugs before the next release.
Comment 73 Kyle M Hall 2016-09-15 13:30:55 UTC
Pushed to master for 16.11, thanks Bernardo!
Comment 74 Frédéric Demians 2016-09-15 14:53:24 UTC
Pushed in 16.05. Will be in 16.05.04.
Comment 75 Julian Maurice 2016-09-16 09:50:06 UTC
@RM @Joubu: I think the last 3 patches have been forgotten because they are wrapped inside a .tar.gz file. Can you have a look ?
Comment 76 Julian Maurice 2016-09-16 09:55:24 UTC
All patches but the 3 last ones are pushed to 3.22.x, they will be in 3.22.11
Comment 77 Jonathan Druart 2016-09-19 12:01:59 UTC
ALL patches should have been pushed!
The 3 tar.gz file should have been extracted to apply these 3 patches.
Comment 78 Jonathan Druart 2016-09-27 12:52:35 UTC
> Bug 16554: DO NOT PUSH - test for all sample files
> 
> You will need to create a koha_test_installer DB before

I have moved this script to its own bug report, see bug 17360.