Bug 24827 - MARC preview fails when staged record contains items with UTF-8 characters
Summary: MARC preview fails when staged record contains items with UTF-8 characters
Status: Pushed to stable
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic record staging/import (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Andreas Roussos
QA Contact: Testopia
URL:
Keywords:
Depends on: 6520
Blocks:
  Show dependency treegraph
 
Reported: 2020-03-08 18:32 UTC by Andreas Roussos
Modified: 2020-04-13 16:17 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
20.05.00, 19.11.05


Attachments
MARC preview failure (5.38 KB, image/png)
2020-03-08 18:36 UTC, Andreas Roussos
Details
MARC preview working OK (20.00 KB, image/png)
2020-03-08 18:38 UTC, Andreas Roussos
Details
test record with embedded items containing UTF-8 data (553 bytes, application/octet-stream)
2020-03-08 18:40 UTC, Andreas Roussos
Details
Bug 24827: Specify UTF-8 as encoding in new_from_xml() (10.18 KB, patch)
2020-03-08 22:12 UTC, Andreas Roussos
Details | Diff | Splinter Review
Bug 24827: Specify UTF-8 as encoding in new_from_xml() (10.26 KB, patch)
2020-03-09 17:55 UTC, Owen Leonard
Details | Diff | Splinter Review
Bug 24827: Standardise on 'UTF-8' as the encoding name (5.15 KB, patch)
2020-03-15 07:49 UTC, Andreas Roussos
Details | Diff | Splinter Review
Bug 24827: Specify UTF-8 as encoding in new_from_xml() (5.22 KB, patch)
2020-03-15 07:49 UTC, Andreas Roussos
Details | Diff | Splinter Review
Bug 24827: Standardise on 'UTF-8' as the encoding name (5.27 KB, patch)
2020-03-16 12:03 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24827: Specify UTF-8 as encoding in new_from_xml() (5.34 KB, patch)
2020-03-16 12:03 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24827: Add a test (1.77 KB, patch)
2020-03-16 12:03 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Roussos 2020-03-08 18:32:57 UTC
The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not specify 'UTF-8' as the optional encoding to use.

In intranet-error.log you'll get messages such as:
showmarc.pl: XSLT_Handler: :63: parser error : PCDATA invalid Char value 27
showmarc.pl:     <subfield code="o">889.3 \x1b(SVAU\x1bs</subfield>

I suggest we fix this by explicitly specifying 'UTF-8' as the encoding
to use whenever a call to new_from_xml() is made across the codebase.

For consistency, I also propose to standardise on the encoding name in
direct calls to new_from_xml(), and use 'UTF-8' from now on instead of
'utf8' or 'utf-8'.
Comment 1 Andreas Roussos 2020-03-08 18:36:56 UTC
Created attachment 100313 [details]
MARC preview failure
Comment 2 Andreas Roussos 2020-03-08 18:38:13 UTC
Created attachment 100314 [details]
MARC preview working OK
Comment 3 Andreas Roussos 2020-03-08 18:40:40 UTC
Created attachment 100315 [details]
test record with embedded items containing UTF-8 data
Comment 4 Andreas Roussos 2020-03-08 22:12:48 UTC
Created attachment 100320 [details] [review]
Bug 24827: Specify UTF-8 as encoding in new_from_xml()

The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not explicitly specify 'UTF-8' as the optional encoding to use.

This patch fixes that, and also standardises the encoding name used in
direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'.

Test plan (in a MARC21 instance):
0) Stage for import the test record attached to this bug report (make
   sure that 'Check for embedded item record data?' is set to 'Yes'),
   then go to Home > Tools > Manage staged MARC records and click on
   the filename you've just staged.
1) In the 'Citation' column, click on the hyperlink to launch the MARC
   preview -- notice how it fails to produce any output apart from the
   modal heading.
2) Apply the patch, and restart Plack if necessary.
3) Re-launch the MARC preview: this time the output should include the
   whole record, including the embedded item data in field 952.
4) Run the unit tests to ensure nothing was broken inadvertently:
   $ prove t
   $ prove xt
Comment 5 Owen Leonard 2020-03-09 17:55:17 UTC
Created attachment 100410 [details] [review]
Bug 24827: Specify UTF-8 as encoding in new_from_xml()

The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not explicitly specify 'UTF-8' as the optional encoding to use.

This patch fixes that, and also standardises the encoding name used in
direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'.

Test plan (in a MARC21 instance):
0) Stage for import the test record attached to this bug report (make
   sure that 'Check for embedded item record data?' is set to 'Yes'),
   then go to Home > Tools > Manage staged MARC records and click on
   the filename you've just staged.
1) In the 'Citation' column, click on the hyperlink to launch the MARC
   preview -- notice how it fails to produce any output apart from the
   modal heading.
2) Apply the patch, and restart Plack if necessary.
3) Re-launch the MARC preview: this time the output should include the
   whole record, including the embedded item data in field 952.
4) Run the unit tests to ensure nothing was broken inadvertently:
   $ prove t
   $ prove xt

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 6 Jonathan Druart 2020-03-11 16:04:15 UTC
Andreas, I agree with the changes. But could split the patch into 2 parts: 1 for the bugfix, 1 for the enhancement. That way the backport will not cause conflict (only the bugfix could be backported).

Switch back the status to "Signed Off" when done.
Comment 7 Andreas Roussos 2020-03-15 07:49:40 UTC
Created attachment 100694 [details] [review]
Bug 24827: Standardise on 'UTF-8' as the encoding name

This patch standardises the encoding name used in direct calls
to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'.
Comment 8 Andreas Roussos 2020-03-15 07:49:43 UTC
Created attachment 100695 [details] [review]
Bug 24827: Specify UTF-8 as encoding in new_from_xml()

The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not explicitly specify 'UTF-8' as the optional encoding to use.

This patch fixes that.

Test plan (in a MARC21 instance):
0) Stage for import the test record attached to this bug report (make
   sure that 'Check for embedded item record data?' is set to 'Yes'),
   then go to Home > Tools > Manage staged MARC records and click on
   the filename you've just staged.
1) In the 'Citation' column, click on the hyperlink to launch the MARC
   preview -- notice how it fails to produce any output apart from the
   modal heading.
2) Apply the patch, and restart Plack if necessary.
3) Re-launch the MARC preview: this time the output should include the
   whole record, including the embedded item data in field 952.
4) Run the unit tests to ensure nothing was broken inadvertently:
   $ prove t
   $ prove xt
Comment 9 Andreas Roussos 2020-03-15 07:58:44 UTC
Thanks for your feedback, Jonathan. I've split my initial patch in two,
hopefully I got this right ;-) Setting Status to "Signed Off".
Comment 10 Jonathan Druart 2020-03-16 12:03:34 UTC
Created attachment 100737 [details] [review]
Bug 24827: Standardise on 'UTF-8' as the encoding name

This patch standardises the encoding name used in direct calls
to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 11 Jonathan Druart 2020-03-16 12:03:37 UTC
Created attachment 100738 [details] [review]
Bug 24827: Specify UTF-8 as encoding in new_from_xml()

The MARC preview of staged records fails when a staged record contains
items with UTF-8 characters (see attached screenshots). That's because
in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml()
does not explicitly specify 'UTF-8' as the optional encoding to use.

This patch fixes that.

Test plan (in a MARC21 instance):
0) Stage for import the test record attached to this bug report (make
   sure that 'Check for embedded item record data?' is set to 'Yes'),
   then go to Home > Tools > Manage staged MARC records and click on
   the filename you've just staged.
1) In the 'Citation' column, click on the hyperlink to launch the MARC
   preview -- notice how it fails to produce any output apart from the
   modal heading.
2) Apply the patch, and restart Plack if necessary.
3) Re-launch the MARC preview: this time the output should include the
   whole record, including the embedded item data in field 952.
4) Run the unit tests to ensure nothing was broken inadvertently:
   $ prove t
   $ prove xt

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 12 Jonathan Druart 2020-03-16 12:03:41 UTC
Created attachment 100739 [details] [review]
Bug 24827: Add a test

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 13 Martin Renvoize 2020-03-16 14:56:03 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 14 Joy Nelson 2020-04-03 18:22:00 UTC
backported to 19.11.x branch for 19.11.05
Comment 15 Lucas Gass 2020-04-13 16:17:41 UTC
patchset doesnt apply clean to 19.05.x, please rebase if needed