Bug 24593

Summary: Rewrite MARC21 optional data to YAML
Product: Koha Reporter: Bernardo Gonzalez Kriegel <bgkriegel>
Component: I18N/L10NAssignee: Bernardo Gonzalez Kriegel <bgkriegel>
Status: CLOSED FIXED QA Contact: Jonathan Druart <jonathan.druart>
Severity: enhancement    
Priority: P5 - low CC: 1joynelson, jonathan.druart, martin.renvoize
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00
Bug Depends on: 24262    
Bug Blocks:    
Attachments: Bug 24593: Rewrite marc21_default_matching_rules to YAML
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML
Bug 24593: Rewrite marc21_default_matching_rules to YAML
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML
Bug 24593: (followup) fix sql statements
Bug 24593: Rewrite marc21_default_matching_rules to YAML
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML
Bug 24593: (followup) fix sql statements

Description Bernardo Gonzalez Kriegel 2020-02-05 13:04:49 UTC
A complete rewrite of MARC21 optional instalation files from SQL to YAML is presented, one patch for each file, the result could be squashed.

The goal is to allow the translation of the installation files
Comment 1 Bernardo Gonzalez Kriegel 2020-02-05 14:42:40 UTC
Created attachment 98494 [details] [review]
Bug 24593: Rewrite marc21_default_matching_rules to YAML

YAML version of marc21 optional marc21_default_matching_rules.

To test:
1) Do a clean install with optional data,
   dump table marc_matchers, reserve.

2) Apply this patch and it's dependencies

3) Do a clean install, dump again and compare
   No major differences expected

4) Try translation
  a) Go to misc/translator
  b) create files for a new language
     ./translate create xx-YY
  c) Check new file po/xx-YY-instaler-MARC21.po
     Verify strings from this file.
  d) Install new language
     ./translate install xx-YY
  e) Try clean install with new files

NOTE: This patch fails with qa tools!
Reason is that qa tools use YAML.pm module to
check files, but Installer.pm or LangInstaller.pm
are using YAML::Syck, using that module gives
no error, check for example:

perl -e 'use YAML::Syck qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
No error!

perl -e 'use YAML qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
Error!
Comment 2 Bernardo Gonzalez Kriegel 2020-02-05 14:42:43 UTC
Created attachment 98495 [details] [review]
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML

YAML version of marc21_sample_fastadd_framework

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Comment 3 Bernardo Gonzalez Kriegel 2020-02-05 14:42:46 UTC
Created attachment 98496 [details] [review]
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML

YAML version of marc21_simple_bib_frameworks

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Comment 4 Martin Renvoize 2020-02-21 16:42:57 UTC
Created attachment 99416 [details] [review]
Bug 24593: Rewrite marc21_default_matching_rules to YAML

YAML version of marc21 optional marc21_default_matching_rules.

To test:
1) Do a clean install with optional data,
   dump table marc_matchers, reserve.

2) Apply this patch and it's dependencies

3) Do a clean install, dump again and compare
   No major differences expected

4) Try translation
  a) Go to misc/translator
  b) create files for a new language
     ./translate create xx-YY
  c) Check new file po/xx-YY-instaler-MARC21.po
     Verify strings from this file.
  d) Install new language
     ./translate install xx-YY
  e) Try clean install with new files

NOTE: This patch fails with qa tools!
Reason is that qa tools use YAML.pm module to
check files, but Installer.pm or LangInstaller.pm
are using YAML::Syck, using that module gives
no error, check for example:

perl -e 'use YAML::Syck qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
No error!

perl -e 'use YAML qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
Error!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 5 Martin Renvoize 2020-02-21 16:43:00 UTC
Created attachment 99417 [details] [review]
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML

YAML version of marc21_sample_fastadd_framework

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 6 Martin Renvoize 2020-02-21 16:43:03 UTC
Created attachment 99418 [details] [review]
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML

YAML version of marc21_simple_bib_frameworks

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2020-02-21 16:43:58 UTC
Works well.. Signing off.

Side note.. we should really standardise on a single YAML module for our parsing :(.
Comment 8 Bernardo Gonzalez Kriegel 2020-03-04 15:56:53 UTC
Created attachment 100139 [details] [review]
Bug 24593: (followup) fix sql statements

Original inserted values depends on the order of insertion.
This patch rewrites the sql statements to make them order
independent
Comment 9 Jonathan Druart 2020-03-09 12:22:13 UTC
Created attachment 100366 [details] [review]
Bug 24593: Rewrite marc21_default_matching_rules to YAML

YAML version of marc21 optional marc21_default_matching_rules.

To test:
1) Do a clean install with optional data,
   dump table marc_matchers, reserve.

2) Apply this patch and it's dependencies

3) Do a clean install, dump again and compare
   No major differences expected

4) Try translation
  a) Go to misc/translator
  b) create files for a new language
     ./translate create xx-YY
  c) Check new file po/xx-YY-instaler-MARC21.po
     Verify strings from this file.
  d) Install new language
     ./translate install xx-YY
  e) Try clean install with new files

NOTE: This patch fails with qa tools!
Reason is that qa tools use YAML.pm module to
check files, but Installer.pm or LangInstaller.pm
are using YAML::Syck, using that module gives
no error, check for example:

perl -e 'use YAML::Syck qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
No error!

perl -e 'use YAML qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
Error!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 10 Jonathan Druart 2020-03-09 12:22:16 UTC
Created attachment 100367 [details] [review]
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML

YAML version of marc21_sample_fastadd_framework

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 11 Jonathan Druart 2020-03-09 12:22:21 UTC
Created attachment 100368 [details] [review]
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML

YAML version of marc21_simple_bib_frameworks

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 12 Jonathan Druart 2020-03-09 12:22:25 UTC
Created attachment 100369 [details] [review]
Bug 24593: (followup) fix sql statements

Original inserted values depends on the order of insertion.
This patch rewrites the sql statements to make them order
independent

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 13 Martin Renvoize 2020-03-09 14:12:39 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 14 Joy Nelson 2020-04-02 18:51:32 UTC
enhancement not backported to 19.11.x