Bugzilla – Attachment 70904 Details for
Bug 18336
Add support for Unicode supplementary characters
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 18336: Full stack tests for supplemental UTF-8 chars
Bug-18336-Full-stack-tests-for-supplemental-UTF-8-.patch (text/plain), 6.52 KB, created by
Mark Tompsett
on 2018-01-24 19:43:56 UTC
(
hide
)
Description:
Bug 18336: Full stack tests for supplemental UTF-8 chars
Filename:
MIME Type:
Creator:
Mark Tompsett
Created:
2018-01-24 19:43:56 UTC
Size:
6.52 KB
patch
obsolete
>From b92a5b377e078aa877f9a007257a3b013abdd2c7 Mon Sep 17 00:00:00 2001 >From: Tomas Cohen Arazi <tomascohen@theke.io> >Date: Mon, 22 Jan 2018 13:16:26 -0300 >Subject: [PATCH] Bug 18336: Full stack tests for supplemental UTF-8 chars > >This patch introduces tests for Koha's support for 4-byte supplemental >UTF-8 chars. encoding/decoding tools handle this gracefuly. The missing >piece is the MySQL DB backend. > >The tests in this patch: >- Adds a couple records for each flavour (MARC21 and UNIMARC) so > search_utf8.t tests 4-byte chars are handled correctly >- Adds emoji testing in auth_values_input_www.t > >To test: >- Apply this patch >- Run: > $ kshell > k$ prove t/db_dependent/www/search_utf8.t \ > t/db_dependent/www/auth_values_input_www.t >=> FAIL: It should fail if the DB hasn't been migrated into using >utf8mb4 > >Sponsored-by: Hotchkiss School > >Signed-off-by: Mark Tompsett <mtompset@hotmail.com> >--- > t/db_dependent/www/auth_values_input_www.t | 7 ++++--- > t/db_dependent/www/data/marc21utf8supprec.mrc | 1 + > t/db_dependent/www/data/unimarcutf8supprec.mrc | 1 + > t/db_dependent/www/search_utf8.t | 18 +++++++++++++++++- > 4 files changed, 23 insertions(+), 4 deletions(-) > create mode 100644 t/db_dependent/www/data/marc21utf8supprec.mrc > create mode 100644 t/db_dependent/www/data/unimarcutf8supprec.mrc > >diff --git a/t/db_dependent/www/auth_values_input_www.t b/t/db_dependent/www/auth_values_input_www.t >index 27e7310..c4f10fc 100644 >--- a/t/db_dependent/www/auth_values_input_www.t >+++ b/t/db_dependent/www/auth_values_input_www.t >@@ -49,7 +49,7 @@ elsif (not defined $intranet) { > plan skip_all => "Tests skip. You must set env. variable KOHA_INTRANET_URL to do tests\n"; > } > else { >- plan tests => 33; >+ plan tests => 34; > } > > my $dbh = C4::Context->dbh; >@@ -71,9 +71,9 @@ $agent->field( 'branch', '' ); > $agent->click_ok( '', 'login to staff client' ); > $agent->get_ok( "$intranet/cgi-bin/koha/mainpage.pl", 'load main page' ); > >-#--------------------------------------------------- Test with corean and greek chars >+#---------------------------------------- Test with corean, greek and emoji chars > >-$category = 'å¦åä¼Î¼Î¼'; >+$category = 'å¦åä¼Î¼Î¼ð'; > $dbh->do(q|DELETE FROM authorised_values WHERE category = ?|, undef, $category); > $dbh->do(q|DELETE FROM authorised_value_categories WHERE category_name = ?|, undef, $category); > >@@ -123,6 +123,7 @@ my $text = $agent->text() ; > ok ( ( length(Encode::encode('UTF-8', $text)) != length($text) ) , 'UTF-8 are multi-byte. Good') ; > ok ($text =~ m/å¦åä¼Î¼Î¼/, 'UTF-8 (Asia) chars are correctly present. Good'); > ok ($text =~ m/εÏιμεq/, 'UTF-8 (Greek) chars are correctly present. Good'); >+ok ($text =~ m/ð/, 'UTF-8 (emoji) chars are correctly present. Good'); > my @links = $agent->links; > my $id_to_del =''; > $delete_re = q|op=delete\&searchfield=| . uri_escape_utf8($category) . '\&id=(\d+)'; >diff --git a/t/db_dependent/www/data/marc21utf8supprec.mrc b/t/db_dependent/www/data/marc21utf8supprec.mrc >new file mode 100644 >index 0000000..9fa2d16 >--- /dev/null >+++ b/t/db_dependent/www/data/marc21utf8supprec.mrc >@@ -0,0 +1 @@ >+00219cam a2200097 a 4500001000800000005001700008008004100025245002500066260002100091650000900112100880920140204215200.0140204m2005 gr a 001 0 gre d02aA tiny record (ð) a𠻺tomasito𠻺 4að00219cam a2200097 a 4500001000800000005001700008008004100025245002500066260002100091650000900112100880920140204215200.0140204m2005 gr a 001 0 gre d02aA tiny record (ð) a𠻺tomasito𠻺 4að >\ No newline at end of file >diff --git a/t/db_dependent/www/data/unimarcutf8supprec.mrc b/t/db_dependent/www/data/unimarcutf8supprec.mrc >new file mode 100644 >index 0000000..6c18392 >--- /dev/null >+++ b/t/db_dependent/www/data/unimarcutf8supprec.mrc >@@ -0,0 +1 @@ >+00650nam0a22002293i 4500001000200000005001700002071002100019090000600040100004100046101000800087200002500095210004300120215002400163600002000187601002000207606000900227702004700236702005200283710005500335801002300390942000700413220141116143328.0 b𠻺tomasito𠻺 a2 a20130409d2005 ||||0itay50 ba aita1 aA tiny record (ð) aAthinacEvraico Mouseio Ellathosd2005 a73 p.cill.d27 cm. 2å¦åä¼. μμ 2å¦åä¼. μμ 2ð 1aMenexiadisb, Alexios3IT\ICCU\LO1V\404551 1aAndrianopouloub, Panayota3IT\ICCU\LO1V\40455202aEvraico Mouseio Ellathos3IT\ICCU\LO1V\4045534070 3aITbICCUc20141104 cBK00650nam0a22002293i 4500001000200000005001700002071002100019090000600040100004100046101000800087200002500095210004300120215002400163600002000187601002000207606000900227702004700236702005200283710005500335801002300390942000700413320141116143328.0 b𠻺tomasito𠻺 a2 a20130409d2005 ||||0itay50 ba aita1 aA tiny record (ð) aAthinacEvraico Mouseio Ellathosd2005 a73 p.cill.d27 cm. 2å¦åä¼. μμ 2å¦åä¼. μμ 2ð 1aMenexiadisb, Alexios3IT\ICCU\LO1V\404551 1aAndrianopouloub, Panayota3IT\ICCU\LO1V\40455202aEvraico Mouseio Ellathos3IT\ICCU\LO1V\4045534070 3aITbICCUc20141104 cBK >\ No newline at end of file >diff --git a/t/db_dependent/www/search_utf8.t b/t/db_dependent/www/search_utf8.t >index 2ad7b93..3b3c22f 100644 >--- a/t/db_dependent/www/search_utf8.t >+++ b/t/db_dependent/www/search_utf8.t >@@ -55,6 +55,11 @@ my $file2 = > ? "$testdir/data/unimarclatin1utf8rec.mrc" > : "$testdir/data/marc21latin1utf8rec.mrc"; > >+my $file3 = >+ $marcflavour eq 'UNIMARC' >+ ? "$testdir/data/unimarcutf8supprec.mrc" >+ : "$testdir/data/marc21utf8supprec.mrc"; >+ > my $user = $ENV{KOHA_USER} || $xml->{config}->{user}; > my $password = $ENV{KOHA_PASS} || $xml->{config}->{pass}; > my $intranet = $ENV{KOHA_INTRANET_URL}; >@@ -70,7 +75,7 @@ elsif ( not defined $opac ) { > plan skip_all => "Tests skip. You must set env. variable KOHA_OPAC_URL to do tests\n"; > } > else { >- plan tests => 66; >+ plan tests => 99; > } > > $intranet =~ s#/$##; >@@ -104,6 +109,17 @@ if ( not defined $indexer_pid ) { > my $utf8_reg2 = qr/Tòmas/; > test_search($file2,'Ramòn', 'Tòmas',$utf8_reg2); > >+#--------------------------------- Test with supplementary utf-8 chars; >+launch_zebra( $datadir, $koha_conf ); >+if ( not defined $zebra_pid ) { >+ plan skip_all => "Tests skip. Error starting Zebra Server to do those tests\n"; >+} >+launch_indexer( ); >+if ( not defined $indexer_pid ) { >+ plan skip_all => "Tests skip. Error starting the indexer daemon to do those tests\n"; >+} >+my $utf8_reg3 = qr/ð/; >+test_search($file3, "𠻺tomasito𠻺", 'A tiny record', $utf8_reg3); > > sub test_search{ > #Params >-- >2.1.4
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 18336
:
70854
|
70855
|
70856
|
70857
|
70858
|
70859
|
70904
|
70905
|
70906
|
70907
|
70908
|
70909
|
71022
|
71210
|
71211
|
71212
|
71213
|
71214
|
71215
|
71216
|
71337
|
71779
|
71780
|
71781
|
71782
|
71783
|
71784
|
71785
|
71786
|
71836
|
71843
|
71844
|
71845
|
71846
|
71847
|
71848
|
71849
|
71850
|
71851
|
71953