Attachment 26320 Details for Bug 9114 – Bug 9114: Make frameworks import/export routines correctly use UTF-8

[patch] Bug 9114: Make frameworks import/export routines correctly use UTF-8

Bug-9114-Make-frameworks-importexport-routines-cor.patch (text/plain), 2.95 KB, created by Tomás Cohen Arazi (tcohen) on 2014-03-13 18:25:20 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Tomás Cohen Arazi (tcohen)

Created: 2014-03-13 18:25:20 UTC

Size: 2.95 KB

patch

obsolete

>From c96512467237bf640522a8916b54932e43536a08 Mon Sep 17 00:00:00 2001
>From: Tomas Cohen Arazi <tomascohen@gmail.com>
>Date: Wed, 22 Jan 2014 15:38:41 -0300
>Subject: [PATCH] Bug 9114: Make frameworks import/export routines correctly
> use UTF-8
>
>Currently both the import_export_framework.pl script outputs data with
>Perl's default encoding, ISO-8859. This patch properly sets
>the binmode to UTF-8 when exporting SQL and CSV files using the PerlIO layer
>(":encoding(UTF-8)") for STDOUT.
>
>To test:
>
>Export step test
>- Use some ASCII character(s) with DIACRITICS in some field description in a chosen framework.
>- Export the framework at Administration > MARC frameworks
>- Run this to check the file is ISO-8859 encoded:
> $ file export_XXX.csv
> export_XXX.csv: ISO-8859 text, with very long lines
> (Note: try SQL and other output formats too. But not ODS)
>- Apply the patch
>- Export the framework again (change the name), and test encoding:
> $ file export_XXX_2.csv
> export_XXX_2.csv: UTF-8 Unicode text
>
>Import step test
>I assume you have two files, export_XXX.csv (ISO-8859 encoded) and export_XXX_2.csv (XXX will depend on
>your framework's code)
>- Reset your testing branch to master
>- Import export_XXX.csv
>- The string with non-ASCII chars is truncated at the first non-ASCII char's position
>  (Note: this is the current behaviour).
>- Import export_XXX_2.csv
>- The non-ASCII chars are broken, the logs show errors on non-UNICODE chars.
>  (Note: even thou UTF-8 is the expected encoding it is treated as ISO-8859).
>- Apply the patch
>- Import the good (UTF-8 as expected) file and check everything worked as expected.
>
>No double encoding should occur with either combination of formats.
>
>Regards
>To+
>
>Sponsored-by: Universidad Nacional de Cordoba
>Signed-off-by: Magnus Enger <digitalutvikling@gmail.com>
>I put some Norwegian and accented letters in a fremawork to test.
>Before the patch, the exported CSV came out as ISO-8859, after the
>patch it came out as UTF-8. ODS and XML (viewed in LibreOffice)
>both looked good, before and after the patch.
>
>Importing the ISO-8859 CSV cut off the strings at the first non-ASCII
>char. Importing the UTF-8 CSV worked as epected.
>---
> admin/import_export_framework.pl | 4 ++++
> 1 file changed, 4 insertions(+)
>
>diff --git a/admin/import_export_framework.pl b/admin/import_export_framework.pl
>index 2a95c45..20729f2 100755
>--- a/admin/import_export_framework.pl
>+++ b/admin/import_export_framework.pl
>@@ -54,8 +54,12 @@ if ($action eq 'export' && $input->request_method() eq 'GET') {
>     my $strXml = '';
>     my $format = $input->param('type_export_' . $frameworkcode);
>     ExportFramework($frameworkcode, \$strXml, $format);
>+
>     if ($format eq 'csv') {
>         # CSV file
>+
>+        # Correctly set the encoding to output plain text in UTF-8
>+        binmode(STDOUT,':encoding(UTF-8)');
>         print $input->header(-type => 'application/vnd.ms-excel', -attachment => 'export_' . $frameworkcode . '.csv');
>         print $strXml;
>     } elsif ($format eq 'excel') {
>-- 
>1.9.0

Actions: View | Diff | Splinter Review

Attachments on bug 9114: 22740 | 24450 | 24514 | 24630 | 26285 | 26320 | 26350