Hi,
yesterday I find a problem on opac_utf8.t.
If i undo the import, I'm only clean the reservoir, but I don't clean
the catalog.

The documentation is clear:
http://manual.koha-community.org/3.18/en/catalogtools.html#managestaged

this is a feature of Koha, not a bug.

So I will insert a delete on biblio dato in the catalog.

Bye

The last patch is signed by Emmanuele Somma.
Emmanuele started the work on this bug during the Italian Hackmeeting (12-13/Nov/2014) and he is behind the realisation of this bug as the same level of myself.
So I share with him the authorship of those lines of code.

Did you get this error?

$ export KOHA_OPAC_URL=http://catalogue.koha-qa.vm
$ export KOHA_INTRANET_URL=http://pro.koha-qa.vm
$ prove t/db_dependent/www/batch.t
t/db_dependent/www/batch.t .. Error GETing http://pro.koha-qa.vm/cgi-bin/koha/mainpage.pl: Can't connect to pro.koha-qa.vm:80 at t/db_dependent/www/batch.t line 60.

(In reply to Jonathan Druart from comment #31)
> Did you get this error?
> 
> $ export KOHA_OPAC_URL=http://catalogue.koha-qa.vm
> $ export KOHA_INTRANET_URL=http://pro.koha-qa.vm
> $ prove t/db_dependent/www/batch.t
> t/db_dependent/www/batch.t .. Error GETing
> http://pro.koha-qa.vm/cgi-bin/koha/mainpage.pl: Can't connect to
> pro.koha-qa.vm:80 at t/db_dependent/www/batch.t line 60.

No, I never see it. Attention that the patch doesn't modify t/db_dependent/www/batch.t. And to use it (and batch.t) you need a working installation.

If you go to http://pro.koha-qa.vm you see a intranet login form ?

(In reply to Zeno Tajoli from comment #32)
> No, I never see it. Attention that the patch doesn't modify
> t/db_dependent/www/batch.t. And to use it (and batch.t) you need a working
> installation.
> 
> If you go to http://pro.koha-qa.vm you see a intranet login form ?

Sorry, silly me.
/etc/hosts was not set correctly.

I added 2 followups and sign yours off.

I'd like to mention something you might know, but it is important to test:

Most of our encoding problems are related to Perl thinking stuff is ISO-8859-1. If you have pure UTF-8 characters, Perl notices, and sets the UTF-8 flag.

We need to test using only latin characters with diacritics [1] to tests that ISO-fallback is not happening. So I'd (kind of) repeat some of the authorized values tests, using latin characters with diacritics (only), like in 'Tomás'.

[1] As the representation for those is the same in Latin-1 and UTF-8 there's no way for Perl to recognise UTF-8 and fallbacks to its default.

Zeno, could you confirm/infirm that my new tests pass for you?

Hi,

in fact for there are problems with new format in search_utf8.t, not in auth_values_input_www.t.
The problems:

/t/db_dependent/www$ prove search_utf8.t 
search_utf8.t .. 19/32 
#   Failed test 'Base is like "(?^u:search.pl\?idx=kw&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)"'
#   at search_utf8.t line 218.
#          got: "http://192.168.216.128:8080/cgi-bin/koha/catalogue"...
#       length: 75
#     doesn't match '(?^u:search.pl\?idx=kw&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)'
search_utf8.t .. 26/32 
#   Failed test 'Base is like "(?^u:opac-search.pl\?idx=&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)"'
#   at search_utf8.t line 241.
#          got: "http://192.168.216.128/cgi-bin/koha/opac-detail.pl"...
#       length: 65
#     doesn't match '(?^u:opac-search.pl\?idx=&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)'
search_utf8.t .. 32/32 # Looks like you failed 2 tests of 32.
search_utf8.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/32 subtests 

Motivation: I use a MySQL db completly clear, so when I start test at test 19 I have only 1 record into db and zebra (zebra records are coming fromm db).
So the search finds 1 result and it display a FULL RECORD display, not a MANY RESULTS display.

The full record display has a url base DIFFERENT from a many result display.
The test 19 wants a base as a MANY RESULTS display.

In fact if test 2-3 times WITHOUT purge (because you stop the debug) you have two o more time the same test record uploaded in to db and so the search gives you a MANY RESULTS display.

So the first question is: WHERE we want to testt the display of utf-8 chars ?
In the  FULL RECORD display or in the  MANY RESULTS display ?

My original code was on testing uft-8 chars into FULL RECORD display but we can change anch test on MANY RESULTS display.

To correct the problem of comment #45 I duplicate the records in the files  
data/mar21utf8record.mrc
data/unimarcutf8reco.mrc

So the server shows always the 'Many records' display.

In auth_values_input_www.t I added also test for chars utf-8/latin-1 like 'tòmas'.

To test:
prove search_utf8.t
prove auth_values_input_www.t

Now I have complete tests also for the chars that could be utf-8 and latin-1 i the same time.

There are new test .mrc files in www/data:
unimarclatin1utf8rec.mrc
marc21latin1utf8rec.mrc

In search_utf8.t now the test on search is a routine, so it is more easy to add new records and new test cases.

I insert 'Needs Signoff' because there are many new situations.

To test:
prove search_utf8.t
prove auth_values_input_www.t

Quick notes:

1/ I had to add a 'b7' branches to remove the following error in koha logs:
manage-marc-import.pl: DBD::mysql::st execute failed: Cannot add or update a child row: a foreign key constraint fails (`koha_ut`.`items`, CONSTRAINT `items_ibfk_2` FOREIGN KEY (`homebranch`) REFERENCES `branches` (`branchcode`) ON UPDATE CASCADE) at /home/koha/src/C4/Items.pm line 2171.

2/ With the last patches, the tests don't pass:
t/db_dependent/www/search_utf8.t .. 18/64 Error GETing http://pro.koha-qa.vm/cgi-bin/koha/catalogue/search.pl?idx=kw&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1&idx=kw&idx=kw&sort_by=relevance: Internal Server Error at t/db_dependent/www/search_utf8.t line 245.
# Looks like you planned 64 tests but ran 18.
# Looks like your test exited with 255 just after 18.

To make them pass, I need to (manually) revert the Tomas's patch:

diff --git a/t/db_dependent/www/search_utf8.t b/t/db_dependent/www/search_utf8.t
index be5ac96..12dbc41 100644
--- a/t/db_dependent/www/search_utf8.t
+++ b/t/db_dependent/www/search_utf8.t
@@ -233,7 +233,7 @@ sub test_search{
     $agent->get_ok( "$intranet/cgi-bin/koha/catalogue/search.pl" , "got search on intranet");
     $agent->form_number(1);
     $agent->field('idx', 'kw');
-    $agent->field('q', $search_key);
+    $agent->field('q', Encode::encode('UTF-8', $search_key));
     $agent->click();
     my $intra_text = $agent->text() ;
     like( $intra_text, qr|Publisher: $publisher|, );
@@ -241,12 +241,12 @@ sub test_search{
     $agent->get_ok( "$intranet/cgi-bin/koha/catalogue/search.pl" , "got search on intranet");
     $agent->form_number(1);
     $agent->field('idx', 'kw');
-    $agent->field('q', $publisher);
+    $agent->field('q', Encode::encode('UTF-8', $publisher));
     $agent->click();
     $intra_text = $agent->text();
     
     like( $intra_text, qr|Publisher: $publisher|, );
-    my $expected_base = q|search.pl\?idx=kw&q=| . uri_escape_utf8( $publisher );
+    my $expected_base = q|search.pl\?idx=kw&q=| . uri_escape_utf8( Encode::encode('UTF-8', $publisher ));
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $intra_text)) != length($intra_text) ) , 'UTF-8 are multi-byte. Goog') ;
@@ -269,7 +269,7 @@ sub test_search{
     $opac_text = $agent->text();
     
     like( $opac_text, qr|Publisher: $publisher|, );
-    $expected_base = q|opac-search.pl\?idx=&q=| . uri_escape_utf8( $publisher );
+    $expected_base = q|opac-search.pl\?q=| . uri_escape_utf8( $publisher );
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $opac_text)) != length($opac_text) ) , 'UTF-8 are multi-byte. Goog') ;

(In reply to Jonathan Druart from comment #73)
> 1/ I had to add a 'b7' branches to remove the following error in koha logs:
> manage-marc-import.pl: DBD::mysql::st execute failed: Cannot add or update a
> child row: a foreign key constraint fails (`koha_ut`.`items`, CONSTRAINT
> `items_ibfk_2` FOREIGN KEY (`homebranch`) REFERENCES `branches`
> (`branchcode`) ON UPDATE CASCADE) at /home/koha/src/C4/Items.pm line 2171.

The last commit [ Follow up on record files: deleted fields with item info (952/995)] fixes this problem.

In fact item's info are not useful for this test so I delete the fields with item's info inside.

The fields deleted are: 952 (MARC21) and 995 (UNIMARC)

Jonathan and Zeno, it seems that the tests that run fine on Perl 5.18 are failing for 5.20, and vice-versa. So we need to make a decision. I think it is a sign of the need for this tests (in the sense that they are already catching something), but in order to push the UTF8 ASAP I'd like you to take a look at the situation.
We can get rid of my patch, I don't care, but tests certaintly pass with it, and fail without it on 14.04.
During the next days I'll install a Jesee box to try the behaviour Jonathan talks about.

I tested on v5.18.2 and I did not get any failures caused by double encoded problems.

The following diff makes pass all tests:

diff --git a/t/db_dependent/www/search_utf8.t b/t/db_dependent/www/search_utf8.t
index be5ac96..6136950 100644
--- a/t/db_dependent/www/search_utf8.t
+++ b/t/db_dependent/www/search_utf8.t
@@ -246,7 +246,7 @@ sub test_search{
     $intra_text = $agent->text();
     
     like( $intra_text, qr|Publisher: $publisher|, );
-    my $expected_base = q|search.pl\?idx=kw&q=| . uri_escape_utf8( $publisher );
+    my $expected_base = q|q=| . uri_escape_utf8( $publisher );
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $intra_text)) != length($intra_text) ) , 'UTF-8 are multi-byte. Goog') ;
@@ -269,7 +269,9 @@ sub test_search{
     $opac_text = $agent->text();
     
     like( $opac_text, qr|Publisher: $publisher|, );
-    $expected_base = q|opac-search.pl\?idx=&q=| . uri_escape_utf8( $publisher );
+    $expected_base = q|q=| . uri_escape_utf8( $publisher );
+    my $base = $agent->base;
+    warn $base;
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $opac_text)) != length($opac_text) ) , 'UTF-8 are multi-byte. Goog') ;

Hi,

I'm testing on a Debian stable (Wheezy) with perl 5.14.2.
Laste version of  ft/bug_11944.
All modules present and updated.

With the code now in 'Need Signoff', no problem.
If I insert by hand  the changes of comment 73 and 77 I don't pass the test.

The result is:
search_utf8.t .. 19/64 
#   Failed test at search_utf8.t line 250.
Wide character in print at /usr/local/share/perl/5.14.2/Test/Builder.pm line 1826.
#                   'Koha › Catalog › Results of search for 'kw,wrdl: Αθήνα'CirculationPatronsSearchCartMore ListsCatalogingAcquisitionsAuthoritiesSerialsReportsToolsAdministrationAbout Koha koha |  NO LIBRARY SET   Set libraryLog outHelpYour cart is empty.Enter patron card number or partial name:   Scan a barcode to check in: Enter search keywords: Check outCheck inSearch the catalogHome › Catalog › Search for 'kw,wrdl: Αθήνα' No results found No results match your search for 'kw,wrdl: Αθήνα'.    '
#     doesn't match '(?^u:Publisher: Αθήνα)'
 
#   Failed test 'Base is like "(?^u:q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)"'
#   at search_utf8.t line 254.
#          got: "http://192.168.216.128:8080/cgi-bin/koha/catalogue"...
#       length: 203
#     doesn't match '(?^u:q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)'
 
#   Failed test 'UTF-8 chars are correctly present. Good'
#   at search_utf8.t line 257.
search_utf8.t .. 26/64 http://192.168.216.128/cgi-bin/koha/opac-search.pl?idx=&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1 at search_utf8.t line 279.
search_utf8.t .. 49/64 
#   Failed test at search_utf8.t line 240.
Wide character in print at /usr/local/share/perl/5.14.2/Test/Builder.pm line 1826.
#                   'Koha › Catalog › Results of search for 'kw,wrdl: Tòmas'CirculationPatronsSearchCartMore ListsCatalogingAcquisitionsAuthoritiesSerialsReportsToolsAdministrationAbout Koha koha |  NO LIBRARY SET   Set libraryLog outHelpYour cart is empty.Enter patron card number or partial name:   Scan a barcode to check in: Enter search keywords: Check outCheck inSearch the catalogHome › Catalog › Search for 'kw,wrdl: Tòmas' No results found No results match your search for 'kw,wrdl: Tòmas'.    '
#     doesn't match '(?^u:Publisher: Ramòn)'
search_utf8.t .. 51/64 
#   Failed test at search_utf8.t line 250.
Wide character in print at /usr/local/share/perl/5.14.2/Test/Builder.pm line 1826.
#                   'Koha › Catalog › Results of search for 'kw,wrdl: Ramòn'CirculationPatronsSearchCartMore ListsCatalogingAcquisitionsAuthoritiesSerialsReportsToolsAdministrationAbout Koha koha |  NO LIBRARY SET   Set libraryLog outHelpYour cart is empty.Enter patron card number or partial name:   Scan a barcode to check in: Enter search keywords: Check outCheck inSearch the catalogHome › Catalog › Search for 'kw,wrdl: Ramòn' No results found No results match your search for 'kw,wrdl: Ramòn'.    '
#     doesn't match '(?^u:Publisher: Ramòn)'
 
#   Failed test 'Base is like "(?^u:q=Ram%C3%B2n)"'
#   at search_utf8.t line 254.
#          got: "http://192.168.216.128:8080/cgi-bin/koha/catalogue"...
#       length: 159
#     doesn't match '(?^u:q=Ram%C3%B2n)'
 
#   Failed test 'UTF-8 chars are correctly present. Good'
#   at search_utf8.t line 257.
search_utf8.t .. 58/64 http://192.168.216.128/cgi-bin/koha/opac-search.pl?idx=&q=Ram%C3%B2n at search_utf8.t line 279.
search_utf8.t .. 64/64 # Looks like you failed 7 tests of 64.
search_utf8.t .. Dubious, test returned 7 (wstat 1792, 0x700)
Failed 7/64 subtests 
 
Test Summary Report
-------------------
search_utf8.t (Wstat: 1792 Tests: 64 Failed: 7)
  Failed tests:  19-20, 22, 49, 51-52, 54
  Non-zero exit status: 7
Files=1, Tests=64, 118 wallclock secs ( 0.04 usr  0.02 sys +  2.15 cusr  0.43 csys =  2.64 CPU)
Result: FAIL

There are changes on Unicode in 5.14/5.18/5.20
Are so huge ?

I think there are several changes in 5.20, including the removal of core Koha dependencies (e.g. CGI) that propose further challenges.

I'd go for pushing this patches working on 5.18 so we push the utf patches, and evaluate 5.20 in the future (at least when a stable debian release provides it as default).

Regards

I am having problems testing this. I have 2 branches, one with the branch from biblibre repo and these patches on top and another with only those patches on top of community master. In both branches I see this when trying to run the 2 test files:


$ perl t/db_dependent/www/auth_values_input_www.t
1..28
You tried to plan twice at t/db_dependent/www/auth_values_input_www.t line 47.
# Looks like your test exited with 255 before it could output anything.

Created attachment 35110 [details] [review]
[PASSED QA] Bug 13264: UNIMARC and MARC21 records to test UTF-8

The records shares enough chars to do the same search ('deuteros') to
find them and check the same utf-8 chars.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35111 [details] [review]
[PASSED QA] Bug 13264: Test for utf-8 on Opac search

This test need to setup zebraserver and background indexing.
You need to set also KOHA_INTRANET_URL and KOHA_OPAC_URL
in the user enviroment.

It imports sample records and retrieves them on hte OPAC through
Apache.

To run it: prove opac_utf8.t

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35112 [details] [review]
[PASSED QA] Bug 13264: Test for utf-8 on authorised values input/display/delete

This test check utf-8 support on web intranet interface for authorized values.
It test input, search and delete of utf-8 values.
If all OK it destroy values inserted

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35113 [details] [review]
[PASSED QA] Bug 13264: no need to test for MySQL

This patch removes the test for a running MySQL process.
It is already implied by successfuly loading C4::Context.

It also fixes the license text.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35114 [details] [review]
[PASSED QA] Bug 13264: Make opac_utf8.t launch its own helper processes

This patch makes opac_utf8.t launch its own:
- zebrasrv process
- rebuild_zebra.pl (daemon mode)

It also takes care of killing them when finished. It sets a frecquency
of 5 seconds for the indexer, and waits 10 just in case before looking
for the indexed records.

To test:
- Just launch the tests.

Edit: fixes on the license text.

Regards
Tomas

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35115 [details] [review]
[PASSED QA] Bug 13264: Follow up: in opac_utf8.t insert also delete of biblio

In previous version of the test opac_utf8.t clears import from resevoir
but it doesn't delete record from biblio table in MySQL.
This the standard work of the feature, see:
http://manual.koha-community.org/3.18/en/catalogtools.html#managestaged

Now the script deletes also record on MySQL level.
No necessary on Zebra level because it uses a temporaney Zebra index that it is
clenaned at the end of the script.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35116 [details] [review]
[PASSED QA] Bug 13264: Additional test for search in intranet calatogue

The utf-8 test do: insert a biblio record with tool, search the record in intranet,
delete batch upload and biblio records.

To test: prove intranet_search_utf8.t

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35117 [details] [review]
[PASSED QA] Bug 13264: Refactor search utf8 tests and add some more

t/db_dependent/www/search_utf8.t and
t/db_dependent/www/intranet_search_utf8.t were quite similar, I merged
them into a single file (t/db_dependent/www/search_utf8.t).
On the way, I added some tests for them.

Note that you will need the last patch on branch bug_11944 to see the
tests pass.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35118 [details] [review]
[PASSED QA] Bug 13264: (QA followup) prevent double-encoding on the tests

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35119 [details] [review]
[PASSED QA] Bug 13264: Add more records to the marc21 aand unimarc files

Dupliacate record in marc21 and unimarc files, so the search finds always more than one result.

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35120 [details] [review]
[PASSED QA] Bug 13264: Add tests for Latin-1 vs. UTF-8 deduction

add a test in auth_values_input_www to check chars that could be Latin-1 or UTF-8

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35121 [details] [review]
[PASSED QA] Bug 13264: Add Latin-1 vs UTF-8 test specific records

This patch addes two new files .mrc for marc21 and unimarc to test chars in utf8/latin-1 section only, to be sure that perl doesn't switch between utf-8 nad ISO 8859-1 inside itself.

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35122 [details] [review]
[PASSED QA] Bug 13264: Update of unimarc and marc21 record to search publisher

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35123 [details] [review]
[PASSED QA] Bug 13264: More tests for Latin-1 vs. UTF-8 interpretation

New tests in search_utf8.t, now are tested also records with only utf-8 chars that could see also as latin-1 chars

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 35124 [details] [review]
[PASSED QA] Bug 13264: Follow up on record files: deleted fields (952/995) with items info

To insert items info you need to use correct code of sublibrary, not always the code is correct. Items info are not important for this test, so the fields are deleted (952 in MARC21, 995 in UNIMARC)

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Patches pushed to master.

Thanks everyone!

Pushed to 3.18.x will be in 3.18.3