Bug 13264 - Full stack encoding tests (UTF-8)
Summary: Full stack encoding tests (UTF-8)
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Test Suite (show other bugs)
Version: 3.18
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Zeno Tajoli
QA Contact: Galen Charlton
URL:
Keywords:
Depends on:
Blocks: 11944 13596
  Show dependency treegraph
 
Reported: 2014-11-17 12:59 UTC by Zeno Tajoli
Modified: 2016-06-21 21:36 UTC (History)
4 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
test file plus two iso2709 records (unimarc and marc21) (9.58 KB, patch)
2014-11-17 13:13 UTC, Zeno Tajoli
Details | Diff | Splinter Review
unimarc and marc21 records to do tests (2.84 KB, patch)
2014-11-19 21:03 UTC, Zeno Tajoli
Details | Diff | Splinter Review
2) test for web opac search (7.17 KB, patch)
2014-11-19 21:04 UTC, Zeno Tajoli
Details | Diff | Splinter Review
3) test on utf.8 for authorised values (4.59 KB, patch)
2014-11-19 21:05 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264: no need to test for MySQL (1.50 KB, patch)
2014-11-20 15:08 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 13264: Make opac_utf8.t launch its own helper processes (4.68 KB, patch)
2014-11-20 15:09 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: UNIMARC and MARC21 records to test UTF-8 (2.89 KB, patch)
2014-11-20 18:15 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on Opac search (7.31 KB, patch)
2014-11-20 18:15 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on authorised values input/display/delete (4.65 KB, patch)
2014-11-20 18:15 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 13264: no need to test for MySQL (2.76 KB, patch)
2014-11-20 18:20 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 13264: Make opac_utf8.t launch its own helper processes (5.62 KB, patch)
2014-11-20 18:21 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 13264: UNIMARC and MARC21 records to test UTF-8 (2.89 KB, patch)
2014-11-21 12:45 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: UNIMARC and MARC21 records to test UTF-8 (2.89 KB, application/mbox)
2014-11-21 12:46 UTC, Zeno Tajoli
Details
[SIGNED OFF] Bug 13264: Test for utf-8 on Opac search (7.31 KB, application/mbox)
2014-11-21 12:46 UTC, Zeno Tajoli
Details
[SIGNED OFF] Bug 13264: Test for utf-8 on authorised values input/display/delete (4.65 KB, patch)
2014-11-21 12:47 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: UNIMARC and MARC21 records to test UTF-8 (2.89 KB, patch)
2014-11-21 12:52 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on Opac search (7.31 KB, patch)
2014-11-21 12:53 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on authorised values input/display/delete (4.65 KB, patch)
2014-11-21 12:54 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264: no need to test for MySQL (2.77 KB, patch)
2014-11-21 12:54 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264: Make opac_utf8.t launch its own helper processes (5.63 KB, patch)
2014-11-21 12:55 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[Bug 13264]: Follow up: in opac_utf8.t insert also delete of biblio (2.15 KB, patch)
2014-11-21 12:57 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: UNIMARC and MARC21 records to test UTF-8 (2.89 KB, patch)
2014-11-22 20:39 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on Opac search (7.31 KB, patch)
2014-11-22 20:40 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on authorised values input/display/delete (4.65 KB, patch)
2014-11-22 20:42 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264: no need to test for MySQL (2.77 KB, patch)
2014-11-22 20:42 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264: Make opac_utf8.t launch its own helper processes (5.63 KB, patch)
2014-11-22 20:43 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264: Follow up: in opac_utf8.t insert also delete of biblio (2.15 KB, patch)
2014-11-22 20:44 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264 Additional test for search in intranet calatogue (8.80 KB, patch)
2014-11-22 20:45 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 13264: UNIMARC and MARC21 records to test UTF-8 (2.94 KB, patch)
2014-12-04 12:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: Test for utf-8 on Opac search (7.36 KB, patch)
2014-12-04 12:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: Test for utf-8 on authorised values input/display/delete (4.70 KB, patch)
2014-12-04 12:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: no need to test for MySQL (2.82 KB, patch)
2014-12-04 12:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: Make opac_utf8.t launch its own helper processes (5.68 KB, patch)
2014-12-04 12:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: Follow up: in opac_utf8.t insert also delete of biblio (2.20 KB, patch)
2014-12-04 12:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: Additional test for search in intranet calatogue (8.85 KB, patch)
2014-12-04 12:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: Refactor search utf8 tests and add some more (14.91 KB, patch)
2014-12-04 12:22 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 13264: (QA followup) prevent double-encoding on the tests (1.72 KB, patch)
2014-12-26 23:54 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
UNIMARC and MARC21 records to test UTF-8 (2.96 KB, patch)
2014-12-28 22:23 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Test for utf-8 on Opac search (7.37 KB, patch)
2014-12-28 22:24 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Test for utf-8 on authorised values input/display/delete (4.71 KB, patch)
2014-12-28 22:25 UTC, Zeno Tajoli
Details | Diff | Splinter Review
no need to test for MySQL (2.83 KB, patch)
2014-12-28 22:26 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Make opac_utf8.t launch its own helper processes (5.69 KB, patch)
2014-12-28 22:27 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Follow up: in opac_utf8.t insert also delete of biblio (2.21 KB, patch)
2014-12-28 22:27 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Additional test for search in intranet calatogue (8.87 KB, patch)
2014-12-28 22:29 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Refactor search utf8 tests and add some more (29.86 KB, patch)
2014-12-28 22:29 UTC, Zeno Tajoli
Details | Diff | Splinter Review
QA followup - prevent double-encoding on the tests (1.73 KB, patch)
2014-12-28 22:30 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Duplicate record in marc21 and unimarc files to show more than one result (6.21 KB, patch)
2014-12-28 22:32 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Add a test for Auth values to check chars that could be utf-8 or latin-1 (5.02 KB, patch)
2014-12-28 22:34 UTC, Zeno Tajoli
Details | Diff | Splinter Review
UNIMARC and MARC21 records to test UTF-8 (2.96 KB, patch)
2014-12-29 10:49 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Test for utf-8 on Opac search (7.37 KB, patch)
2014-12-29 10:50 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Test for utf-8 on authorised values input/display/delete (4.71 KB, patch)
2014-12-29 10:51 UTC, Zeno Tajoli
Details | Diff | Splinter Review
no need to test for MySQL (2.83 KB, patch)
2014-12-29 10:52 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Make opac_utf8.t launch its own helper processes (5.69 KB, patch)
2014-12-29 10:52 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Follow up: in opac_utf8.t insert also delete of biblio (2.21 KB, patch)
2014-12-29 10:53 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Additional test for search in intranet calatogue (8.87 KB, patch)
2014-12-29 10:54 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Refactor search utf8 tests and add some more (29.86 KB, patch)
2014-12-29 10:54 UTC, Zeno Tajoli
Details | Diff | Splinter Review
QA followup - prevent double-encoding on the tests (1.73 KB, patch)
2014-12-29 10:55 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Duplicate record in marc21 and unimarc files to show more than one result (6.21 KB, patch)
2014-12-29 10:55 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Add a test for Auth values to check chars that could be utf-8 or latin-1 (5.02 KB, patch)
2014-12-29 10:56 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Add two new file .mrc for marc21 and unimarc to test chars utf8/latin1 (3.13 KB, patch)
2014-12-29 10:58 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Update of unimarc and marc21 new test files to seach publisher (4.97 KB, patch)
2014-12-29 10:59 UTC, Zeno Tajoli
Details | Diff | Splinter Review
New test in search_utf8.t to manage chars in utf8/latin1 ambiguity (15.47 KB, patch)
2014-12-29 11:00 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Follow up on record files: deleted fields with item info (952/995) (7.60 KB, patch)
2014-12-29 13:43 UTC, Zeno Tajoli
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: UNIMARC and MARC21 records to test UTF-8 (2.95 KB, patch)
2015-01-09 14:24 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on Opac search (7.37 KB, patch)
2015-01-09 14:25 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Test for utf-8 on authorised values input/display/delete (4.71 KB, patch)
2015-01-09 14:25 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: no need to test for MySQL (2.89 KB, patch)
2015-01-09 14:25 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Make opac_utf8.t launch its own helper processes (5.75 KB, patch)
2015-01-09 14:25 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Follow up: in opac_utf8.t insert also delete of biblio (2.25 KB, patch)
2015-01-09 14:25 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Additional test for search in intranet calatogue (8.92 KB, patch)
2015-01-09 14:26 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Refactor search utf8 tests and add some more (14.98 KB, patch)
2015-01-09 14:26 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: (QA followup) prevent double-encoding on the tests (1.73 KB, patch)
2015-01-09 14:26 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Add more records to the marc21 aand unimarc files (6.35 KB, patch)
2015-01-09 14:26 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Add tests for Latin-1 vs. UTF-8 deduction (5.17 KB, patch)
2015-01-09 14:27 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Add Latin-1 vs UTF-8 test specific records (3.28 KB, patch)
2015-01-09 14:28 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Update of unimarc and marc21 record to search publisher (5.07 KB, patch)
2015-01-09 14:28 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: More tests for Latin-1 vs. UTF-8 interpretation (15.51 KB, patch)
2015-01-09 14:28 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 13264: Follow up on record files: deleted fields (952/995) with items info (7.69 KB, patch)
2015-01-09 14:28 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: UNIMARC and MARC21 records to test UTF-8 (3.01 KB, patch)
2015-01-12 22:07 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Test for utf-8 on Opac search (7.43 KB, patch)
2015-01-12 22:07 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Test for utf-8 on authorised values input/display/delete (4.77 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: no need to test for MySQL (2.95 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Make opac_utf8.t launch its own helper processes (5.81 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Follow up: in opac_utf8.t insert also delete of biblio (2.31 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Additional test for search in intranet calatogue (8.97 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Refactor search utf8 tests and add some more (15.03 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: (QA followup) prevent double-encoding on the tests (1.79 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Add more records to the marc21 aand unimarc files (6.41 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Add tests for Latin-1 vs. UTF-8 deduction (5.23 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Add Latin-1 vs UTF-8 test specific records (3.34 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Update of unimarc and marc21 record to search publisher (5.13 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: More tests for Latin-1 vs. UTF-8 interpretation (15.57 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review
[PASSED QA] Bug 13264: Follow up on record files: deleted fields (952/995) with items info (7.75 KB, patch)
2015-01-12 22:08 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Zeno Tajoli 2014-11-17 12:59:51 UTC
A basic test on opac about utf-8
This is for bug 11944
Comment 1 Zeno Tajoli 2014-11-17 13:13:19 UTC Comment hidden (obsolete)
Comment 2 Zeno Tajoli 2014-11-19 21:03:25 UTC Comment hidden (obsolete)
Comment 3 Zeno Tajoli 2014-11-19 21:04:19 UTC Comment hidden (obsolete)
Comment 4 Zeno Tajoli 2014-11-19 21:05:04 UTC Comment hidden (obsolete)
Comment 5 Tomás Cohen Arazi 2014-11-20 15:08:58 UTC Comment hidden (obsolete)
Comment 6 Tomás Cohen Arazi 2014-11-20 15:09:04 UTC Comment hidden (obsolete)
Comment 7 Tomás Cohen Arazi 2014-11-20 18:15:07 UTC Comment hidden (obsolete)
Comment 8 Tomás Cohen Arazi 2014-11-20 18:15:24 UTC Comment hidden (obsolete)
Comment 9 Tomás Cohen Arazi 2014-11-20 18:15:33 UTC Comment hidden (obsolete)
Comment 10 Tomás Cohen Arazi 2014-11-20 18:20:54 UTC Comment hidden (obsolete)
Comment 11 Tomás Cohen Arazi 2014-11-20 18:21:02 UTC Comment hidden (obsolete)
Comment 12 Zeno Tajoli 2014-11-21 10:06:34 UTC
Hi,
yesterday I find a problem on opac_utf8.t.
If i undo the import, I'm only clean the reservoir, but I don't clean
the catalog.

The documentation is clear:
http://manual.koha-community.org/3.18/en/catalogtools.html#managestaged

this is a feature of Koha, not a bug.

So I will insert a delete on biblio dato in the catalog.

Bye
Comment 13 Zeno Tajoli 2014-11-21 12:45:06 UTC Comment hidden (obsolete)
Comment 14 Zeno Tajoli 2014-11-21 12:46:10 UTC Comment hidden (obsolete)
Comment 15 Zeno Tajoli 2014-11-21 12:46:58 UTC Comment hidden (obsolete)
Comment 16 Zeno Tajoli 2014-11-21 12:47:50 UTC Comment hidden (obsolete)
Comment 17 Zeno Tajoli 2014-11-21 12:52:20 UTC Comment hidden (obsolete)
Comment 18 Zeno Tajoli 2014-11-21 12:53:13 UTC Comment hidden (obsolete)
Comment 19 Zeno Tajoli 2014-11-21 12:54:11 UTC Comment hidden (obsolete)
Comment 20 Zeno Tajoli 2014-11-21 12:54:58 UTC Comment hidden (obsolete)
Comment 21 Zeno Tajoli 2014-11-21 12:55:58 UTC Comment hidden (obsolete)
Comment 22 Zeno Tajoli 2014-11-21 12:57:04 UTC Comment hidden (obsolete)
Comment 23 Zeno Tajoli 2014-11-22 20:39:36 UTC Comment hidden (obsolete)
Comment 24 Zeno Tajoli 2014-11-22 20:40:44 UTC Comment hidden (obsolete)
Comment 25 Zeno Tajoli 2014-11-22 20:42:12 UTC Comment hidden (obsolete)
Comment 26 Zeno Tajoli 2014-11-22 20:42:58 UTC Comment hidden (obsolete)
Comment 27 Zeno Tajoli 2014-11-22 20:43:55 UTC Comment hidden (obsolete)
Comment 28 Zeno Tajoli 2014-11-22 20:44:42 UTC Comment hidden (obsolete)
Comment 29 Zeno Tajoli 2014-11-22 20:45:37 UTC Comment hidden (obsolete)
Comment 30 Zeno Tajoli 2014-11-22 20:48:28 UTC
The last patch is signed by Emmanuele Somma.
Emmanuele started the work on this bug during the Italian Hackmeeting (12-13/Nov/2014) and he is behind the realisation of this bug as the same level of myself.
So I share with him the authorship of those lines of code.
Comment 31 Jonathan Druart 2014-12-03 16:12:58 UTC
Did you get this error?

$ export KOHA_OPAC_URL=http://catalogue.koha-qa.vm
$ export KOHA_INTRANET_URL=http://pro.koha-qa.vm
$ prove t/db_dependent/www/batch.t
t/db_dependent/www/batch.t .. Error GETing http://pro.koha-qa.vm/cgi-bin/koha/mainpage.pl: Can't connect to pro.koha-qa.vm:80 at t/db_dependent/www/batch.t line 60.
Comment 32 Zeno Tajoli 2014-12-03 16:41:23 UTC
(In reply to Jonathan Druart from comment #31)
> Did you get this error?
> 
> $ export KOHA_OPAC_URL=http://catalogue.koha-qa.vm
> $ export KOHA_INTRANET_URL=http://pro.koha-qa.vm
> $ prove t/db_dependent/www/batch.t
> t/db_dependent/www/batch.t .. Error GETing
> http://pro.koha-qa.vm/cgi-bin/koha/mainpage.pl: Can't connect to
> pro.koha-qa.vm:80 at t/db_dependent/www/batch.t line 60.

No, I never see it. Attention that the patch doesn't modify t/db_dependent/www/batch.t. And to use it (and batch.t) you need a working installation.

If you go to http://pro.koha-qa.vm you see a intranet login form ?
Comment 33 Jonathan Druart 2014-12-04 12:21:02 UTC Comment hidden (obsolete)
Comment 34 Jonathan Druart 2014-12-04 12:21:09 UTC Comment hidden (obsolete)
Comment 35 Jonathan Druart 2014-12-04 12:21:16 UTC Comment hidden (obsolete)
Comment 36 Jonathan Druart 2014-12-04 12:21:22 UTC Comment hidden (obsolete)
Comment 37 Jonathan Druart 2014-12-04 12:21:32 UTC Comment hidden (obsolete)
Comment 38 Jonathan Druart 2014-12-04 12:21:44 UTC Comment hidden (obsolete)
Comment 39 Jonathan Druart 2014-12-04 12:21:55 UTC Comment hidden (obsolete)
Comment 40 Jonathan Druart 2014-12-04 12:22:08 UTC Comment hidden (obsolete)
Comment 41 Jonathan Druart 2014-12-04 14:17:44 UTC
(In reply to Zeno Tajoli from comment #32)
> No, I never see it. Attention that the patch doesn't modify
> t/db_dependent/www/batch.t. And to use it (and batch.t) you need a working
> installation.
> 
> If you go to http://pro.koha-qa.vm you see a intranet login form ?

Sorry, silly me.
/etc/hosts was not set correctly.

I added 2 followups and sign yours off.
Comment 42 Tomás Cohen Arazi 2014-12-04 18:27:30 UTC
I'd like to mention something you might know, but it is important to test:

Most of our encoding problems are related to Perl thinking stuff is ISO-8859-1. If you have pure UTF-8 characters, Perl notices, and sets the UTF-8 flag.

We need to test using only latin characters with diacritics [1] to tests that ISO-fallback is not happening. So I'd (kind of) repeat some of the authorized values tests, using latin characters with diacritics (only), like in 'Tomás'.

[1] As the representation for those is the same in Latin-1 and UTF-8 there's no way for Perl to recognise UTF-8 and fallbacks to its default.
Comment 43 Jonathan Druart 2014-12-26 14:57:41 UTC
Zeno, could you confirm/infirm that my new tests pass for you?
Comment 44 Tomás Cohen Arazi 2014-12-26 23:54:02 UTC Comment hidden (obsolete)
Comment 45 Zeno Tajoli 2014-12-28 16:02:23 UTC
Hi,

in fact for there are problems with new format in search_utf8.t, not in auth_values_input_www.t.
The problems:

/t/db_dependent/www$ prove search_utf8.t 
search_utf8.t .. 19/32 
#   Failed test 'Base is like "(?^u:search.pl\?idx=kw&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)"'
#   at search_utf8.t line 218.
#          got: "http://192.168.216.128:8080/cgi-bin/koha/catalogue"...
#       length: 75
#     doesn't match '(?^u:search.pl\?idx=kw&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)'
search_utf8.t .. 26/32 
#   Failed test 'Base is like "(?^u:opac-search.pl\?idx=&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)"'
#   at search_utf8.t line 241.
#          got: "http://192.168.216.128/cgi-bin/koha/opac-detail.pl"...
#       length: 65
#     doesn't match '(?^u:opac-search.pl\?idx=&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)'
search_utf8.t .. 32/32 # Looks like you failed 2 tests of 32.
search_utf8.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/32 subtests 

Motivation: I use a MySQL db completly clear, so when I start test at test 19 I have only 1 record into db and zebra (zebra records are coming fromm db).
So the search finds 1 result and it display a FULL RECORD display, not a MANY RESULTS display.

The full record display has a url base DIFFERENT from a many result display.
The test 19 wants a base as a MANY RESULTS display.

In fact if test 2-3 times WITHOUT purge (because you stop the debug) you have two o more time the same test record uploaded in to db and so the search gives you a MANY RESULTS display.

So the first question is: WHERE we want to testt the display of utf-8 chars ?
In the  FULL RECORD display or in the  MANY RESULTS display ?

My original code was on testing uft-8 chars into FULL RECORD display but we can change anch test on MANY RESULTS display.
Comment 46 Zeno Tajoli 2014-12-28 22:23:13 UTC Comment hidden (obsolete)
Comment 47 Zeno Tajoli 2014-12-28 22:24:38 UTC Comment hidden (obsolete)
Comment 48 Zeno Tajoli 2014-12-28 22:25:19 UTC Comment hidden (obsolete)
Comment 49 Zeno Tajoli 2014-12-28 22:26:04 UTC Comment hidden (obsolete)
Comment 50 Zeno Tajoli 2014-12-28 22:27:07 UTC Comment hidden (obsolete)
Comment 51 Zeno Tajoli 2014-12-28 22:27:56 UTC Comment hidden (obsolete)
Comment 52 Zeno Tajoli 2014-12-28 22:29:03 UTC Comment hidden (obsolete)
Comment 53 Zeno Tajoli 2014-12-28 22:29:42 UTC Comment hidden (obsolete)
Comment 54 Zeno Tajoli 2014-12-28 22:30:40 UTC Comment hidden (obsolete)
Comment 55 Zeno Tajoli 2014-12-28 22:32:41 UTC Comment hidden (obsolete)
Comment 56 Zeno Tajoli 2014-12-28 22:34:59 UTC Comment hidden (obsolete)
Comment 57 Zeno Tajoli 2014-12-28 22:44:37 UTC
To correct the problem of comment #45 I duplicate the records in the files  
data/mar21utf8record.mrc
data/unimarcutf8reco.mrc

So the server shows always the 'Many records' display.

In auth_values_input_www.t I added also test for chars utf-8/latin-1 like 'tòmas'.

To test:
prove search_utf8.t
prove auth_values_input_www.t
Comment 58 Zeno Tajoli 2014-12-29 10:49:58 UTC Comment hidden (obsolete)
Comment 59 Zeno Tajoli 2014-12-29 10:50:56 UTC Comment hidden (obsolete)
Comment 60 Zeno Tajoli 2014-12-29 10:51:57 UTC Comment hidden (obsolete)
Comment 61 Zeno Tajoli 2014-12-29 10:52:24 UTC Comment hidden (obsolete)
Comment 62 Zeno Tajoli 2014-12-29 10:52:55 UTC Comment hidden (obsolete)
Comment 63 Zeno Tajoli 2014-12-29 10:53:24 UTC Comment hidden (obsolete)
Comment 64 Zeno Tajoli 2014-12-29 10:54:03 UTC Comment hidden (obsolete)
Comment 65 Zeno Tajoli 2014-12-29 10:54:36 UTC Comment hidden (obsolete)
Comment 66 Zeno Tajoli 2014-12-29 10:55:02 UTC Comment hidden (obsolete)
Comment 67 Zeno Tajoli 2014-12-29 10:55:41 UTC Comment hidden (obsolete)
Comment 68 Zeno Tajoli 2014-12-29 10:56:29 UTC Comment hidden (obsolete)
Comment 69 Zeno Tajoli 2014-12-29 10:58:42 UTC Comment hidden (obsolete)
Comment 70 Zeno Tajoli 2014-12-29 10:59:39 UTC Comment hidden (obsolete)
Comment 71 Zeno Tajoli 2014-12-29 11:00:51 UTC Comment hidden (obsolete)
Comment 72 Zeno Tajoli 2014-12-29 11:06:15 UTC
Now I have complete tests also for the chars that could be utf-8 and latin-1 i the same time.

There are new test .mrc files in www/data:
unimarclatin1utf8rec.mrc
marc21latin1utf8rec.mrc

In search_utf8.t now the test on search is a routine, so it is more easy to add new records and new test cases.

I insert 'Needs Signoff' because there are many new situations.

To test:
prove search_utf8.t
prove auth_values_input_www.t
Comment 73 Jonathan Druart 2014-12-29 11:39:34 UTC
Quick notes:

1/ I had to add a 'b7' branches to remove the following error in koha logs:
manage-marc-import.pl: DBD::mysql::st execute failed: Cannot add or update a child row: a foreign key constraint fails (`koha_ut`.`items`, CONSTRAINT `items_ibfk_2` FOREIGN KEY (`homebranch`) REFERENCES `branches` (`branchcode`) ON UPDATE CASCADE) at /home/koha/src/C4/Items.pm line 2171.

2/ With the last patches, the tests don't pass:
t/db_dependent/www/search_utf8.t .. 18/64 Error GETing http://pro.koha-qa.vm/cgi-bin/koha/catalogue/search.pl?idx=kw&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1&idx=kw&idx=kw&sort_by=relevance: Internal Server Error at t/db_dependent/www/search_utf8.t line 245.
# Looks like you planned 64 tests but ran 18.
# Looks like your test exited with 255 just after 18.

To make them pass, I need to (manually) revert the Tomas's patch:

diff --git a/t/db_dependent/www/search_utf8.t b/t/db_dependent/www/search_utf8.t
index be5ac96..12dbc41 100644
--- a/t/db_dependent/www/search_utf8.t
+++ b/t/db_dependent/www/search_utf8.t
@@ -233,7 +233,7 @@ sub test_search{
     $agent->get_ok( "$intranet/cgi-bin/koha/catalogue/search.pl" , "got search on intranet");
     $agent->form_number(1);
     $agent->field('idx', 'kw');
-    $agent->field('q', $search_key);
+    $agent->field('q', Encode::encode('UTF-8', $search_key));
     $agent->click();
     my $intra_text = $agent->text() ;
     like( $intra_text, qr|Publisher: $publisher|, );
@@ -241,12 +241,12 @@ sub test_search{
     $agent->get_ok( "$intranet/cgi-bin/koha/catalogue/search.pl" , "got search on intranet");
     $agent->form_number(1);
     $agent->field('idx', 'kw');
-    $agent->field('q', $publisher);
+    $agent->field('q', Encode::encode('UTF-8', $publisher));
     $agent->click();
     $intra_text = $agent->text();
     
     like( $intra_text, qr|Publisher: $publisher|, );
-    my $expected_base = q|search.pl\?idx=kw&q=| . uri_escape_utf8( $publisher );
+    my $expected_base = q|search.pl\?idx=kw&q=| . uri_escape_utf8( Encode::encode('UTF-8', $publisher ));
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $intra_text)) != length($intra_text) ) , 'UTF-8 are multi-byte. Goog') ;
@@ -269,7 +269,7 @@ sub test_search{
     $opac_text = $agent->text();
     
     like( $opac_text, qr|Publisher: $publisher|, );
-    $expected_base = q|opac-search.pl\?idx=&q=| . uri_escape_utf8( $publisher );
+    $expected_base = q|opac-search.pl\?q=| . uri_escape_utf8( $publisher );
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $opac_text)) != length($opac_text) ) , 'UTF-8 are multi-byte. Goog') ;
Comment 74 Zeno Tajoli 2014-12-29 13:43:42 UTC Comment hidden (obsolete)
Comment 75 Zeno Tajoli 2014-12-29 13:47:19 UTC
(In reply to Jonathan Druart from comment #73)
> 1/ I had to add a 'b7' branches to remove the following error in koha logs:
> manage-marc-import.pl: DBD::mysql::st execute failed: Cannot add or update a
> child row: a foreign key constraint fails (`koha_ut`.`items`, CONSTRAINT
> `items_ibfk_2` FOREIGN KEY (`homebranch`) REFERENCES `branches`
> (`branchcode`) ON UPDATE CASCADE) at /home/koha/src/C4/Items.pm line 2171.

The last commit [ Follow up on record files: deleted fields with item info (952/995)] fixes this problem.

In fact item's info are not useful for this test so I delete the fields with item's info inside.

The fields deleted are: 952 (MARC21) and 995 (UNIMARC)
Comment 76 Tomás Cohen Arazi 2014-12-31 20:04:36 UTC
Jonathan and Zeno, it seems that the tests that run fine on Perl 5.18 are failing for 5.20, and vice-versa. So we need to make a decision. I think it is a sign of the need for this tests (in the sense that they are already catching something), but in order to push the UTF8 ASAP I'd like you to take a look at the situation.
We can get rid of my patch, I don't care, but tests certaintly pass with it, and fail without it on 14.04.
During the next days I'll install a Jesee box to try the behaviour Jonathan talks about.
Comment 77 Jonathan Druart 2015-01-02 12:18:01 UTC
I tested on v5.18.2 and I did not get any failures caused by double encoded problems.

The following diff makes pass all tests:

diff --git a/t/db_dependent/www/search_utf8.t b/t/db_dependent/www/search_utf8.t
index be5ac96..6136950 100644
--- a/t/db_dependent/www/search_utf8.t
+++ b/t/db_dependent/www/search_utf8.t
@@ -246,7 +246,7 @@ sub test_search{
     $intra_text = $agent->text();
     
     like( $intra_text, qr|Publisher: $publisher|, );
-    my $expected_base = q|search.pl\?idx=kw&q=| . uri_escape_utf8( $publisher );
+    my $expected_base = q|q=| . uri_escape_utf8( $publisher );
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $intra_text)) != length($intra_text) ) , 'UTF-8 are multi-byte. Goog') ;
@@ -269,7 +269,9 @@ sub test_search{
     $opac_text = $agent->text();
     
     like( $opac_text, qr|Publisher: $publisher|, );
-    $expected_base = q|opac-search.pl\?idx=&q=| . uri_escape_utf8( $publisher );
+    $expected_base = q|q=| . uri_escape_utf8( $publisher );
+    my $base = $agent->base;
+    warn $base;
     $agent->base_like(qr|$expected_base|, );
     
     ok ( ( length(Encode::encode('UTF-8', $opac_text)) != length($opac_text) ) , 'UTF-8 are multi-byte. Goog') ;
Comment 78 Zeno Tajoli 2015-01-06 18:07:04 UTC
Hi,

I'm testing on a Debian stable (Wheezy) with perl 5.14.2.
Laste version of  ft/bug_11944.
All modules present and updated.

With the code now in 'Need Signoff', no problem.
If I insert by hand  the changes of comment 73 and 77 I don't pass the test.

The result is:
search_utf8.t .. 19/64 
#   Failed test at search_utf8.t line 250.
Wide character in print at /usr/local/share/perl/5.14.2/Test/Builder.pm line 1826.
#                   'Koha › Catalog › Results of search for 'kw,wrdl: Αθήνα'CirculationPatronsSearchCartMore ListsCatalogingAcquisitionsAuthoritiesSerialsReportsToolsAdministrationAbout Koha koha |  NO LIBRARY SET   Set libraryLog outHelpYour cart is empty.Enter patron card number or partial name:   Scan a barcode to check in: Enter search keywords: Check outCheck inSearch the catalogHome › Catalog › Search for 'kw,wrdl: Αθήνα' No results found No results match your search for 'kw,wrdl: Αθήνα'.    '
#     doesn't match '(?^u:Publisher: Αθήνα)'
 
#   Failed test 'Base is like "(?^u:q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)"'
#   at search_utf8.t line 254.
#          got: "http://192.168.216.128:8080/cgi-bin/koha/catalogue"...
#       length: 203
#     doesn't match '(?^u:q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)'
 
#   Failed test 'UTF-8 chars are correctly present. Good'
#   at search_utf8.t line 257.
search_utf8.t .. 26/64 http://192.168.216.128/cgi-bin/koha/opac-search.pl?idx=&q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1 at search_utf8.t line 279.
search_utf8.t .. 49/64 
#   Failed test at search_utf8.t line 240.
Wide character in print at /usr/local/share/perl/5.14.2/Test/Builder.pm line 1826.
#                   'Koha › Catalog › Results of search for 'kw,wrdl: Tòmas'CirculationPatronsSearchCartMore ListsCatalogingAcquisitionsAuthoritiesSerialsReportsToolsAdministrationAbout Koha koha |  NO LIBRARY SET   Set libraryLog outHelpYour cart is empty.Enter patron card number or partial name:   Scan a barcode to check in: Enter search keywords: Check outCheck inSearch the catalogHome › Catalog › Search for 'kw,wrdl: Tòmas' No results found No results match your search for 'kw,wrdl: Tòmas'.    '
#     doesn't match '(?^u:Publisher: Ramòn)'
search_utf8.t .. 51/64 
#   Failed test at search_utf8.t line 250.
Wide character in print at /usr/local/share/perl/5.14.2/Test/Builder.pm line 1826.
#                   'Koha › Catalog › Results of search for 'kw,wrdl: Ramòn'CirculationPatronsSearchCartMore ListsCatalogingAcquisitionsAuthoritiesSerialsReportsToolsAdministrationAbout Koha koha |  NO LIBRARY SET   Set libraryLog outHelpYour cart is empty.Enter patron card number or partial name:   Scan a barcode to check in: Enter search keywords: Check outCheck inSearch the catalogHome › Catalog › Search for 'kw,wrdl: Ramòn' No results found No results match your search for 'kw,wrdl: Ramòn'.    '
#     doesn't match '(?^u:Publisher: Ramòn)'
 
#   Failed test 'Base is like "(?^u:q=Ram%C3%B2n)"'
#   at search_utf8.t line 254.
#          got: "http://192.168.216.128:8080/cgi-bin/koha/catalogue"...
#       length: 159
#     doesn't match '(?^u:q=Ram%C3%B2n)'
 
#   Failed test 'UTF-8 chars are correctly present. Good'
#   at search_utf8.t line 257.
search_utf8.t .. 58/64 http://192.168.216.128/cgi-bin/koha/opac-search.pl?idx=&q=Ram%C3%B2n at search_utf8.t line 279.
search_utf8.t .. 64/64 # Looks like you failed 7 tests of 64.
search_utf8.t .. Dubious, test returned 7 (wstat 1792, 0x700)
Failed 7/64 subtests 
 
Test Summary Report
-------------------
search_utf8.t (Wstat: 1792 Tests: 64 Failed: 7)
  Failed tests:  19-20, 22, 49, 51-52, 54
  Non-zero exit status: 7
Files=1, Tests=64, 118 wallclock secs ( 0.04 usr  0.02 sys +  2.15 cusr  0.43 csys =  2.64 CPU)
Result: FAIL

There are changes on Unicode in 5.14/5.18/5.20
Are so huge ?
Comment 79 Tomás Cohen Arazi 2015-01-06 20:40:18 UTC
I think there are several changes in 5.20, including the removal of core Koha dependencies (e.g. CGI) that propose further challenges.

I'd go for pushing this patches working on 5.18 so we push the utf patches, and evaluate 5.20 in the future (at least when a stable debian release provides it as default).

Regards
Comment 80 Tomás Cohen Arazi 2015-01-09 14:24:44 UTC Comment hidden (obsolete)
Comment 81 Tomás Cohen Arazi 2015-01-09 14:25:00 UTC Comment hidden (obsolete)
Comment 82 Tomás Cohen Arazi 2015-01-09 14:25:17 UTC Comment hidden (obsolete)
Comment 83 Tomás Cohen Arazi 2015-01-09 14:25:29 UTC Comment hidden (obsolete)
Comment 84 Tomás Cohen Arazi 2015-01-09 14:25:43 UTC Comment hidden (obsolete)
Comment 85 Tomás Cohen Arazi 2015-01-09 14:25:55 UTC Comment hidden (obsolete)
Comment 86 Tomás Cohen Arazi 2015-01-09 14:26:10 UTC Comment hidden (obsolete)
Comment 87 Tomás Cohen Arazi 2015-01-09 14:26:22 UTC Comment hidden (obsolete)
Comment 88 Tomás Cohen Arazi 2015-01-09 14:26:42 UTC Comment hidden (obsolete)
Comment 89 Tomás Cohen Arazi 2015-01-09 14:26:57 UTC Comment hidden (obsolete)
Comment 90 Tomás Cohen Arazi 2015-01-09 14:27:45 UTC Comment hidden (obsolete)
Comment 91 Tomás Cohen Arazi 2015-01-09 14:28:01 UTC Comment hidden (obsolete)
Comment 92 Tomás Cohen Arazi 2015-01-09 14:28:12 UTC Comment hidden (obsolete)
Comment 93 Tomás Cohen Arazi 2015-01-09 14:28:23 UTC Comment hidden (obsolete)
Comment 94 Tomás Cohen Arazi 2015-01-09 14:28:37 UTC Comment hidden (obsolete)
Comment 95 Katrin Fischer 2015-01-12 20:47:47 UTC
I am having problems testing this. I have 2 branches, one with the branch from biblibre repo and these patches on top and another with only those patches on top of community master. In both branches I see this when trying to run the 2 test files:


$ perl t/db_dependent/www/auth_values_input_www.t
1..28
You tried to plan twice at t/db_dependent/www/auth_values_input_www.t line 47.
# Looks like your test exited with 255 before it could output anything.
Comment 96 Katrin Fischer 2015-01-12 22:07:55 UTC
Created attachment 35110 [details] [review]
[PASSED QA] Bug 13264: UNIMARC and MARC21 records to test UTF-8

The records shares enough chars to do the same search ('deuteros') to
find them and check the same utf-8 chars.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 97 Katrin Fischer 2015-01-12 22:07:59 UTC
Created attachment 35111 [details] [review]
[PASSED QA] Bug 13264: Test for utf-8 on Opac search

This test need to setup zebraserver and background indexing.
You need to set also KOHA_INTRANET_URL and KOHA_OPAC_URL
in the user enviroment.

It imports sample records and retrieves them on hte OPAC through
Apache.

To run it: prove opac_utf8.t

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 98 Katrin Fischer 2015-01-12 22:08:03 UTC
Created attachment 35112 [details] [review]
[PASSED QA] Bug 13264: Test for utf-8 on authorised values input/display/delete

This test check utf-8 support on web intranet interface for authorized values.
It test input, search and delete of utf-8 values.
If all OK it destroy values inserted

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 99 Katrin Fischer 2015-01-12 22:08:07 UTC
Created attachment 35113 [details] [review]
[PASSED QA] Bug 13264: no need to test for MySQL

This patch removes the test for a running MySQL process.
It is already implied by successfuly loading C4::Context.

It also fixes the license text.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 100 Katrin Fischer 2015-01-12 22:08:11 UTC
Created attachment 35114 [details] [review]
[PASSED QA] Bug 13264: Make opac_utf8.t launch its own helper processes

This patch makes opac_utf8.t launch its own:
- zebrasrv process
- rebuild_zebra.pl (daemon mode)

It also takes care of killing them when finished. It sets a frecquency
of 5 seconds for the indexer, and waits 10 just in case before looking
for the indexed records.

To test:
- Just launch the tests.

Edit: fixes on the license text.

Regards
Tomas

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 101 Katrin Fischer 2015-01-12 22:08:15 UTC
Created attachment 35115 [details] [review]
[PASSED QA] Bug 13264: Follow up: in opac_utf8.t insert also delete of biblio

In previous version of the test opac_utf8.t clears import from resevoir
but it doesn't delete record from biblio table in MySQL.
This the standard work of the feature, see:
http://manual.koha-community.org/3.18/en/catalogtools.html#managestaged

Now the script deletes also record on MySQL level.
No necessary on Zebra level because it uses a temporaney Zebra index that it is
clenaned at the end of the script.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 102 Katrin Fischer 2015-01-12 22:08:19 UTC
Created attachment 35116 [details] [review]
[PASSED QA] Bug 13264: Additional test for search in intranet calatogue

The utf-8 test do: insert a biblio record with tool, search the record in intranet,
delete batch upload and biblio records.

To test: prove intranet_search_utf8.t

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 103 Katrin Fischer 2015-01-12 22:08:23 UTC
Created attachment 35117 [details] [review]
[PASSED QA] Bug 13264: Refactor search utf8 tests and add some more

t/db_dependent/www/search_utf8.t and
t/db_dependent/www/intranet_search_utf8.t were quite similar, I merged
them into a single file (t/db_dependent/www/search_utf8.t).
On the way, I added some tests for them.

Note that you will need the last patch on branch bug_11944 to see the
tests pass.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 104 Katrin Fischer 2015-01-12 22:08:27 UTC
Created attachment 35118 [details] [review]
[PASSED QA] Bug 13264: (QA followup) prevent double-encoding on the tests

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 105 Katrin Fischer 2015-01-12 22:08:32 UTC
Created attachment 35119 [details] [review]
[PASSED QA] Bug 13264: Add more records to the marc21 aand unimarc files

Dupliacate record in marc21 and unimarc files, so the search finds always more than one result.

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 106 Katrin Fischer 2015-01-12 22:08:36 UTC
Created attachment 35120 [details] [review]
[PASSED QA] Bug 13264: Add tests for Latin-1 vs. UTF-8 deduction

add a test in auth_values_input_www to check chars that could be Latin-1 or UTF-8

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 107 Katrin Fischer 2015-01-12 22:08:41 UTC
Created attachment 35121 [details] [review]
[PASSED QA] Bug 13264: Add Latin-1 vs UTF-8 test specific records

This patch addes two new files .mrc for marc21 and unimarc to test chars in utf8/latin-1 section only, to be sure that perl doesn't switch between utf-8 nad ISO 8859-1 inside itself.

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 108 Katrin Fischer 2015-01-12 22:08:46 UTC
Created attachment 35122 [details] [review]
[PASSED QA] Bug 13264: Update of unimarc and marc21 record to search publisher

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 109 Katrin Fischer 2015-01-12 22:08:51 UTC
Created attachment 35123 [details] [review]
[PASSED QA] Bug 13264: More tests for Latin-1 vs. UTF-8 interpretation

New tests in search_utf8.t, now are tested also records with only utf-8 chars that could see also as latin-1 chars

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 110 Katrin Fischer 2015-01-12 22:08:55 UTC
Created attachment 35124 [details] [review]
[PASSED QA] Bug 13264: Follow up on record files: deleted fields (952/995) with items info

To insert items info you need to use correct code of sublibrary, not always the code is correct. Items info are not important for this test, so the fields are deleted (952 in MARC21, 995 in UNIMARC)

http://bugs.koha-community.org/show_bug.cgi?id=13264
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 111 Tomás Cohen Arazi 2015-01-13 16:20:59 UTC
Patches pushed to master.

Thanks everyone!
Comment 112 Chris Cormack 2015-01-18 01:49:26 UTC
Pushed to 3.18.x will be in 3.18.3