Bug 24593 - Rewrite MARC21 optional data to YAML
Summary: Rewrite MARC21 optional data to YAML
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bernardo Gonzalez Kriegel
QA Contact: Jonathan Druart
URL:
Keywords:
Depends on: 24262
Blocks:
  Show dependency treegraph
 
Reported: 2020-02-05 13:04 UTC by Bernardo Gonzalez Kriegel
Modified: 2020-11-30 21:48 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00


Attachments
Bug 24593: Rewrite marc21_default_matching_rules to YAML (8.74 KB, patch)
2020-02-05 14:42 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML (22.20 KB, patch)
2020-02-05 14:42 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML (112.95 KB, patch)
2020-02-05 14:42 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_default_matching_rules to YAML (8.80 KB, patch)
2020-02-21 16:42 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML (22.27 KB, patch)
2020-02-21 16:43 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML (113.01 KB, patch)
2020-02-21 16:43 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 24593: (followup) fix sql statements (3.97 KB, patch)
2020-03-04 15:56 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_default_matching_rules to YAML (8.80 KB, patch)
2020-03-09 12:22 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML (22.34 KB, patch)
2020-03-09 12:22 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML (113.08 KB, patch)
2020-03-09 12:22 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24593: (followup) fix sql statements (4.04 KB, patch)
2020-03-09 12:22 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Bernardo Gonzalez Kriegel 2020-02-05 13:04:49 UTC
A complete rewrite of MARC21 optional instalation files from SQL to YAML is presented, one patch for each file, the result could be squashed.

The goal is to allow the translation of the installation files
Comment 1 Bernardo Gonzalez Kriegel 2020-02-05 14:42:40 UTC
Created attachment 98494 [details] [review]
Bug 24593: Rewrite marc21_default_matching_rules to YAML

YAML version of marc21 optional marc21_default_matching_rules.

To test:
1) Do a clean install with optional data,
   dump table marc_matchers, reserve.

2) Apply this patch and it's dependencies

3) Do a clean install, dump again and compare
   No major differences expected

4) Try translation
  a) Go to misc/translator
  b) create files for a new language
     ./translate create xx-YY
  c) Check new file po/xx-YY-instaler-MARC21.po
     Verify strings from this file.
  d) Install new language
     ./translate install xx-YY
  e) Try clean install with new files

NOTE: This patch fails with qa tools!
Reason is that qa tools use YAML.pm module to
check files, but Installer.pm or LangInstaller.pm
are using YAML::Syck, using that module gives
no error, check for example:

perl -e 'use YAML::Syck qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
No error!

perl -e 'use YAML qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
Error!
Comment 2 Bernardo Gonzalez Kriegel 2020-02-05 14:42:43 UTC
Created attachment 98495 [details] [review]
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML

YAML version of marc21_sample_fastadd_framework

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Comment 3 Bernardo Gonzalez Kriegel 2020-02-05 14:42:46 UTC
Created attachment 98496 [details] [review]
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML

YAML version of marc21_simple_bib_frameworks

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Comment 4 Martin Renvoize 2020-02-21 16:42:57 UTC
Created attachment 99416 [details] [review]
Bug 24593: Rewrite marc21_default_matching_rules to YAML

YAML version of marc21 optional marc21_default_matching_rules.

To test:
1) Do a clean install with optional data,
   dump table marc_matchers, reserve.

2) Apply this patch and it's dependencies

3) Do a clean install, dump again and compare
   No major differences expected

4) Try translation
  a) Go to misc/translator
  b) create files for a new language
     ./translate create xx-YY
  c) Check new file po/xx-YY-instaler-MARC21.po
     Verify strings from this file.
  d) Install new language
     ./translate install xx-YY
  e) Try clean install with new files

NOTE: This patch fails with qa tools!
Reason is that qa tools use YAML.pm module to
check files, but Installer.pm or LangInstaller.pm
are using YAML::Syck, using that module gives
no error, check for example:

perl -e 'use YAML::Syck qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
No error!

perl -e 'use YAML qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
Error!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 5 Martin Renvoize 2020-02-21 16:43:00 UTC
Created attachment 99417 [details] [review]
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML

YAML version of marc21_sample_fastadd_framework

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 6 Martin Renvoize 2020-02-21 16:43:03 UTC
Created attachment 99418 [details] [review]
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML

YAML version of marc21_simple_bib_frameworks

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2020-02-21 16:43:58 UTC
Works well.. Signing off.

Side note.. we should really standardise on a single YAML module for our parsing :(.
Comment 8 Bernardo Gonzalez Kriegel 2020-03-04 15:56:53 UTC
Created attachment 100139 [details] [review]
Bug 24593: (followup) fix sql statements

Original inserted values depends on the order of insertion.
This patch rewrites the sql statements to make them order
independent
Comment 9 Jonathan Druart 2020-03-09 12:22:13 UTC
Created attachment 100366 [details] [review]
Bug 24593: Rewrite marc21_default_matching_rules to YAML

YAML version of marc21 optional marc21_default_matching_rules.

To test:
1) Do a clean install with optional data,
   dump table marc_matchers, reserve.

2) Apply this patch and it's dependencies

3) Do a clean install, dump again and compare
   No major differences expected

4) Try translation
  a) Go to misc/translator
  b) create files for a new language
     ./translate create xx-YY
  c) Check new file po/xx-YY-instaler-MARC21.po
     Verify strings from this file.
  d) Install new language
     ./translate install xx-YY
  e) Try clean install with new files

NOTE: This patch fails with qa tools!
Reason is that qa tools use YAML.pm module to
check files, but Installer.pm or LangInstaller.pm
are using YAML::Syck, using that module gives
no error, check for example:

perl -e 'use YAML::Syck qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
No error!

perl -e 'use YAML qw( LoadFile ); $y = LoadFile("marc21_default_matching_rules.yml");'
Error!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 10 Jonathan Druart 2020-03-09 12:22:16 UTC
Created attachment 100367 [details] [review]
Bug 24593: Rewrite marc21_sample_fastadd_framework to YAML

YAML version of marc21_sample_fastadd_framework

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 11 Jonathan Druart 2020-03-09 12:22:21 UTC
Created attachment 100368 [details] [review]
Bug 24593: Rewrite marc21_simple_bib_frameworks to YAML

YAML version of marc21_simple_bib_frameworks

To test:
1) Same test plan of first patch

NOTE: Same problem as first patch with qa tools
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 12 Jonathan Druart 2020-03-09 12:22:25 UTC
Created attachment 100369 [details] [review]
Bug 24593: (followup) fix sql statements

Original inserted values depends on the order of insertion.
This patch rewrites the sql statements to make them order
independent

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 13 Martin Renvoize 2020-03-09 14:12:39 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 14 Joy Nelson 2020-04-02 18:51:32 UTC
enhancement not backported to 19.11.x