Bug 28281

Summary: Installer doesn't work on some languages (pl-PL) because it double decodes installer data
Product: Koha Reporter: Joonas Kylmälä <joonas.kylmala>
Component: Installation and upgrade (web-based installer)Assignee: Joonas Kylmälä <joonas.kylmala>
Status: CLOSED FIXED QA Contact: Kyle M Hall (khall) <kyle>
Severity: major    
Priority: P5 - low CC: fridolin.somers, gmcharlt, jonathan.druart, kyle, victor
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28154
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.05.00
Circulation function:
Bug Depends on: 22824    
Bug Blocks: 27623    
Attachments: Bug 28281: Remove double decoding when importing installation data
Bug 28281: Add tests
Bug 28281: Remove double decoding when importing installation data
Bug 28281: Add tests
Bug 28281: Remove double decoding when importing installation data
Bug 28281: Add tests

Description Joonas Kylmälä 2021-05-04 08:32:16 UTC
If trying to install Koha with pl-PL language it doesn't work due to double encoding error in C4/Installer.pm:

$ koha-mysql kohadev < installer/data/mysql/kohastructure.sql
$ restart_all
$ ./translate update pl-PL
$ ./translate install pl-PL
$ restart_all
 --> Go to installer page and select pl-PL as installer language, click through it and notice it gives error:

> Cannot decode string with wide characters at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 202

In installer the output of two calls YAML::XS::LoadFile is still decoded another time even though according to YAML::XS documentation the call to Load already decodes it. This causes the above error to happen in the MARC21 installation step.
Comment 1 Jonathan Druart 2021-05-04 09:34:03 UTC
Caused by bug 22824 I assume.
Comment 2 Victor Grousset/tuxayo 2021-05-05 00:00:37 UTC
hi :)
Can't reproduce. Any hint ? I retried in case I did something different when testing bug 27623

The only thing I "improvised" was before the 1st step:
echo "DROP DATABASE koha_kohadev; CREATE DATABASE koha_kohadev;" | koha-mysql kohadev ; restart_all

Which is necessary right?

What is the use of translate update in this case?
Comment 3 Jonathan Druart 2021-05-05 09:10:06 UTC
(In reply to Victor Grousset/tuxayo from comment #2)
> What is the use of translate update in this case?

The encoding issue is coming from there.

Joonas, any news on this?
Comment 4 Joonas Kylmälä 2021-05-05 09:18:43 UTC
Created attachment 120484 [details] [review]
Bug 28281: Remove double decoding when importing installation data

The YAML::XS::Load and YAML::XS::LoadFile functions already decode the
YAML to Perl objects/strings, therefore decoding the output yet
another time is not needed and causes an error to happen.

To test:

1. Go through the following steps and notice without this patch the
error happens and with this patch it doesn't:

$ koha-mysql kohadev < installer/data/mysql/kohastructure.sql
$ restart_all
$ ./translate update pl-PL
$ ./translate install pl-PL
$ restart_all
 --> Go to the Koha web installer page and select pl-PL as the installer
 language, click through it and notice it gives following error at the
 MARC21 step:

 Cannot decode string with wide characters at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 202
Comment 5 Joonas Kylmälä 2021-05-05 09:19:47 UTC
(In reply to Jonathan Druart from comment #3)
> (In reply to Victor Grousset/tuxayo from comment #2)
> > What is the use of translate update in this case?
> 
> The encoding issue is coming from there.
> 
> Joonas, any news on this?

I have the fix only, attached it now but no tests yet. I'd gladly welcome help there or if you can wait probably will be able to make it in the coming days.
Comment 6 Joonas Kylmälä 2021-05-05 09:21:26 UTC
(In reply to Victor Grousset/tuxayo from comment #2)
> Can't reproduce. Any hint ? I retried in case I did something different when
> testing bug 27623

No idea why you don't get it. Maybe for some reason you are using some other translation files compared to me?
Comment 7 Jonathan Druart 2021-05-05 10:40:22 UTC
Created attachment 120486 [details] [review]
Bug 28281: Add tests
Comment 8 Jonathan Druart 2021-05-06 13:01:26 UTC
Created attachment 120626 [details] [review]
Bug 28281: Remove double decoding when importing installation data

The YAML::XS::Load and YAML::XS::LoadFile functions already decode the
YAML to Perl objects/strings, therefore decoding the output yet
another time is not needed and causes an error to happen.

To test:

1. Go through the following steps and notice without this patch the
error happens and with this patch it doesn't:

$ koha-mysql kohadev < installer/data/mysql/kohastructure.sql
$ restart_all
$ ./translate update pl-PL
$ ./translate install pl-PL
$ restart_all
 --> Go to the Koha web installer page and select pl-PL as the installer
 language, click through it and notice it gives following error at the
 MARC21 step:

 Cannot decode string with wide characters at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 202

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 9 Jonathan Druart 2021-05-06 13:01:30 UTC
Created attachment 120627 [details] [review]
Bug 28281: Add tests

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 10 Joonas Kylmälä 2021-05-07 11:11:47 UTC
Nick, in case you are QA'ing this right now please check out also the bug 28291. Depending which translation data you have already you might need it.
Comment 11 Joonas Kylmälä 2021-05-07 11:12:19 UTC
(In reply to Joonas Kylmälä from comment #10)
> Nick, in case you are QA'ing this right now please check out also the bug
> 28291. Depending which translation data you have already you might need it.

Err, meant to say Kyle.
Comment 12 Kyle M Hall (khall) 2021-05-07 11:23:34 UTC
Created attachment 120678 [details] [review]
Bug 28281: Remove double decoding when importing installation data

The YAML::XS::Load and YAML::XS::LoadFile functions already decode the
YAML to Perl objects/strings, therefore decoding the output yet
another time is not needed and causes an error to happen.

To test:

1. Go through the following steps and notice without this patch the
error happens and with this patch it doesn't:

$ koha-mysql kohadev < installer/data/mysql/kohastructure.sql
$ restart_all
$ ./translate update pl-PL
$ ./translate install pl-PL
$ restart_all
 --> Go to the Koha web installer page and select pl-PL as the installer
 language, click through it and notice it gives following error at the
 MARC21 step:

 Cannot decode string with wide characters at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 202

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 13 Kyle M Hall (khall) 2021-05-07 11:23:47 UTC
Created attachment 120679 [details] [review]
Bug 28281: Add tests

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 14 Jonathan Druart 2021-05-07 12:41:50 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 15 Fridolin Somers 2021-05-12 14:07:22 UTC
Depends on Bug 22824 not in 20.11.x