The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not specify 'UTF-8' as the optional encoding to use. In intranet-error.log you'll get messages such as: showmarc.pl: XSLT_Handler: :63: parser error : PCDATA invalid Char value 27 showmarc.pl: <subfield code="o">889.3 \x1b(SVAU\x1bs</subfield> I suggest we fix this by explicitly specifying 'UTF-8' as the encoding to use whenever a call to new_from_xml() is made across the codebase. For consistency, I also propose to standardise on the encoding name in direct calls to new_from_xml(), and use 'UTF-8' from now on instead of 'utf8' or 'utf-8'.
Created attachment 100313 [details] MARC preview failure
Created attachment 100314 [details] MARC preview working OK
Created attachment 100315 [details] test record with embedded items containing UTF-8 data
Created attachment 100320 [details] [review] Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that, and also standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt
Created attachment 100410 [details] [review] Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that, and also standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Andreas, I agree with the changes. But could split the patch into 2 parts: 1 for the bugfix, 1 for the enhancement. That way the backport will not cause conflict (only the bugfix could be backported). Switch back the status to "Signed Off" when done.
Created attachment 100694 [details] [review] Bug 24827: Standardise on 'UTF-8' as the encoding name This patch standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'.
Created attachment 100695 [details] [review] Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt
Thanks for your feedback, Jonathan. I've split my initial patch in two, hopefully I got this right ;-) Setting Status to "Signed Off".
Created attachment 100737 [details] [review] Bug 24827: Standardise on 'UTF-8' as the encoding name This patch standardises the encoding name used in direct calls to new_from_xml() to 'UTF-8' instead of 'utf8' or 'utf-8'. Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 100738 [details] [review] Bug 24827: Specify UTF-8 as encoding in new_from_xml() The MARC preview of staged records fails when a staged record contains items with UTF-8 characters (see attached screenshots). That's because in EmbedItemsInImportBiblio() the call to MARC::Record->new_from_xml() does not explicitly specify 'UTF-8' as the optional encoding to use. This patch fixes that. Test plan (in a MARC21 instance): 0) Stage for import the test record attached to this bug report (make sure that 'Check for embedded item record data?' is set to 'Yes'), then go to Home > Tools > Manage staged MARC records and click on the filename you've just staged. 1) In the 'Citation' column, click on the hyperlink to launch the MARC preview -- notice how it fails to produce any output apart from the modal heading. 2) Apply the patch, and restart Plack if necessary. 3) Re-launch the MARC preview: this time the output should include the whole record, including the embedded item data in field 952. 4) Run the unit tests to ensure nothing was broken inadvertently: $ prove t $ prove xt Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 100739 [details] [review] Bug 24827: Add a test Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Nice work everyone! Pushed to master for 20.05
backported to 19.11.x branch for 19.11.05
patchset doesnt apply clean to 19.05.x, please rebase if needed