Attachment 98507 Details for Bug 13897 – Bug 13897: Process description, multiline values and SQL statements in YAML files

[patch] Bug 13897: Process description, multiline values and SQL statements in YAML files

Bug-13897-Process-description-multiline-values-and.patch (text/plain), 6.04 KB, created by Bernardo Gonzalez Kriegel on 2020-02-05 15:24:13 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Bernardo Gonzalez Kriegel

Created: 2020-02-05 15:24:13 UTC

Size: 6.04 KB

patch

obsolete

>From e3616768257eb4d9905d5a9b4e908b28d4bc73fa Mon Sep 17 00:00:00 2001
>From: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
>Date: Thu, 30 Jan 2020 11:30:30 -0300
>Subject: [PATCH] Bug 13897: Process description, multiline values and SQL
> statements in YAML files
>
>This patch adds 3 features:
>1) Display description of YAML files at install time
>   for frameworks, and fixes it's encoding.
>2) Enable use of multiline values, field required
>3) Process SQL statements declared in YAML files
>
>With this features we can process files with the
>following generic YAML strucure:
>
>  ---
>  description:
>    - "File description"
>
>  tables:
>    - table_name:
>        translatable: [ title, content ]
>        multiline: [ content ]
>        rows:
>          - title: "Example title"
>            content:
>              - "Content:"
>              - ""
>              - "This is the content."
>            id: 1
>            value: ~
>
>  sql_statements:
>    - "UPDATE table_name SET value ='empty' WHERE value IS NULL"
>  ...
>
>* file description is now inside the YAML, can have multiple
>  lines.
>  This attribute is expected in all YAML files.
>* translatable attribute in table declare fields that can be
>  translated
>* multiline attribute in table declare fields that can have
>  multiple lines of content, they are joined using '\r\n'
>  before insert into database.
>  This is useful to express fields like 'news' content, and
>  to simplify it's translation.
>  '\r\n' is used for correct display in Windows clients.
>* sql_statements allows to add multiple SQL sentences, not
>  insertions normally, that are executed in order.
>
>This features are not needed for the example file of this patch,
>but will be used in new bugs.
>
>To test:
>1) Use the same test plan of first patch.
>---
> C4/Installer.pm                               | 29 ++++++++++++++-----
> installer/data/mysql/en/optional/auth_val.yml |  5 ++++
> 2 files changed, 27 insertions(+), 7 deletions(-)
>
>diff --git a/C4/Installer.pm b/C4/Installer.pm
>index 2e799c2fd6..02a72552ed 100644
>--- a/C4/Installer.pm
>+++ b/C4/Installer.pm
>@@ -151,11 +151,17 @@ sub marc_framework_sql_list {
>         my %cell;
>         my @frameworklist;
>         map {
>-            my $name = substr( $_, 0, -4 ); # FIXME: restricted to 3 letter extension
>-            open my $fh, "<:encoding(UTF-8)", "$dir/$requirelevel/$name.txt";
>-            my $line = <$fh>;
>-            $line = Encode::encode('UTF-8', $line) unless ( Encode::is_utf8($line) );
>-            my @lines = split /\n/, $line;
>+            my ( $name, $ext ) = split /\./, $_;
>+            my @lines;
>+            if ( $ext =~ /yml/ ) {
>+                my $yaml = LoadFile("$dir/$requirelevel/$name\.$ext");
>+                @lines = map { Encode::decode('UTF-8', $_) } @{ $yaml->{'description'} };
>+            } else {
>+                open my $fh, "<:encoding(UTF-8)", "$dir/$requirelevel/$name.txt";
>+                my $line = <$fh>;
>+                $line = Encode::encode('UTF-8', $line) unless ( Encode::is_utf8($line) );
>+                @lines = split /\n/, $line;
>+            }
>             my $mandatory = ($requirelevel =~ /(mandatory|requi|oblig|necess)/i);
>             push @frameworklist,
>               {
>@@ -232,7 +238,7 @@ sub sample_data_sql_list {
>             my @lines;
>             if ( $ext =~ /yml/ ) {
>                 my $yaml = LoadFile("$dir/$requirelevel/$name\.$ext");
>-                @lines = @{ $yaml->{'description'} };
>+                @lines = map { Encode::decode('UTF-8', $_) } @{ $yaml->{'description'} };
>             } else {
>                 open my $fh, "<:encoding(UTF-8)", "$dir/$requirelevel/$name.txt";
>                 my $line = <$fh>;
>@@ -485,11 +491,20 @@ sub load_sql {
>                     my $placeholders = join ",", map { "?" } @columns;               # '?,..,?' string
>                     my $query        = "INSERT INTO $table_name ( $fields ) VALUES ( $placeholders )";
>                     my $sth          = $dbh->prepare($query);
>+                    my @multiline    = @{ $table->{$table_name}->{'multiline'} };    # to check multiline values;
>                     foreach my $row ( @rows ) {
>-                        my @values = map { $row->{$_} } @columns;
>+                        my @values = map {
>+                                        my $col = $_;
>+                                        ( @multiline and grep { $_ eq $col } @multiline )
>+                                        ? join "\r\n", @{$row->{$col}}                # join multiline values
>+                                        : $row->{$col};
>+                                     } @columns;
>                         $sth->execute( @values );
>                     }
>                 }
>+                for my $statement ( @{ $yaml->{'sql_statements'} } ) {               # extra SQL statements
>+                    $dbh->do($statement);
>+                }
>             };
>         }
>         if ($@){
>diff --git a/installer/data/mysql/en/optional/auth_val.yml b/installer/data/mysql/en/optional/auth_val.yml
>index d303cfe35f..2c662a343d 100644
>--- a/installer/data/mysql/en/optional/auth_val.yml
>+++ b/installer/data/mysql/en/optional/auth_val.yml
>@@ -25,6 +25,7 @@ description:
> tables:
>   - authorised_values:
>       translatable: [ lib ]
>+      multiline: []
>       rows:
>         # Reasons for acceptance or rejection of suggestions in acquisitions
>         - category: "SUGGEST"
>@@ -45,6 +46,7 @@ tables:
> 
>   - authorised_values:
>       translatable: [ lib, lib_opac ]
>+      multiline: []
>       rows:
>         # Desired formats for requesting new materials
>         - category: "SUGGEST_FORMAT"
>@@ -74,6 +76,7 @@ tables:
> 
>   - authorised_values :
>       translatable: [ lib ]
>+      multiline: []
>       rows:
>         # availability statuses
>         - category: "LOST"
>@@ -181,6 +184,7 @@ tables:
> 
>   - authorised_values:
>       translatable: [ lib, lib_opac ]
>+      multiline: []
>       rows:
>         # OPAC Suggestions reasons
>         - category: "OPAC_SUG"
>@@ -195,6 +199,7 @@ tables:
> 
>   - authorised_values:
>       translatable: [ lib ]
>+      multiline: []
>       rows:
>         # Report groups
>         - category: "REPORT_GROUP"
>-- 
>2.17.1

Actions: View | Diff | Splinter Review

Attachments on bug 13897: 37165 | 75930 | 75931 | 95884 | 96370 | 96397 | 96539 | 96591 | 96592 | 96754 | 96755 | 97068 | 97069 | 98507 | 98741 | 99392 | 99393 | 99394 | 99395 | 100340 | 100341 | 100342 | 100343