Bug 6479 - Encoding problem in "recievedlist" when the numbering formula contains utf-8 characters
Summary: Encoding problem in "recievedlist" when the numbering formula contains utf-8 ...
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Serials (show other bugs)
Version: 3.4
Hardware: All All
: PATCH-Sent (DO NOT USE) normal (vote)
Assignee: Frédérick Capovilla
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-06-07 19:10 UTC by Frédérick Capovilla
Modified: 2013-12-05 20:04 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Corrections the encoding problem with "recievedlist" (1.04 KB, patch)
2011-06-07 19:10 UTC, Frédérick Capovilla
Details | Diff | Splinter Review
screenshot (25.32 KB, image/png)
2011-06-15 07:37 UTC, Katrin Fischer
Details
[SIGNED-OFF] Converts all the serialseq variables to UTF-8. (1.49 KB, patch)
2011-06-15 07:43 UTC, Katrin Fischer
Details | Diff | Splinter Review
proposed patch (passed QA) (1.49 KB, patch)
2011-08-11 16:10 UTC, Paul Poulain
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Frédérick Capovilla 2011-06-07 19:10:52 UTC Comment hidden (obsolete)
Comment 1 Katrin Fischer 2011-06-15 07:37:57 UTC
Created attachment 4472 [details]
screenshot

The receive screen is ok, but I can reproduce the problem on the subscription summary and edit pages.
Comment 2 Katrin Fischer 2011-06-15 07:43:42 UTC Comment hidden (obsolete)
Comment 3 Katrin Fischer 2011-06-15 07:44:27 UTC
Comment on attachment 4424 [details] [review]
Corrections the encoding problem with "recievedlist"

>From 4106e035be8ac9886aafccbd5284c3ede7cb34c4 Mon Sep 17 00:00:00 2001
>From: =?utf-8?q?Fr=C3=A9d=C3=A9rick=20Capovilla?= <frederick.capovilla@sys-tech.net>
>Date: Tue, 7 Jun 2011 14:03:08 -0400
>Subject: [PATCH] Converts all the serialseq variables to UTF-8.
>
>Corrects a problem when an UTF-8 character is used in the serial
>numbering formula. The encoding became incorrect when concatenating the
>number in the subscriptionhistory table.
>---
> serials/serials-edit.pl |    5 +++++
> 1 files changed, 5 insertions(+), 0 deletions(-)
>
>diff --git a/serials/serials-edit.pl b/serials/serials-edit.pl
>index aaa0574..b486c04 100755
>--- a/serials/serials-edit.pl
>+++ b/serials/serials-edit.pl
>@@ -196,6 +196,11 @@ $template->param( subscriptions => \@subscriptionloop );
> 
> if ( $op and $op eq 'serialchangestatus' ) {
> 
>+    # Convert serialseqs to UTF-8 to prevent encoding problems
>+    foreach my $seq (@serialseqs) {
>+        utf8::decode($seq) unless utf8::is_utf8($seq);
>+    }
>+
>     my $newserial;
>     for ( my $i = 0 ; $i <= $#serialids ; $i++ ) {
> 
>-- 
>1.5.6.5
>
Comment 4 Paul Poulain 2011-08-11 16:10:32 UTC
Created attachment 4969 [details] [review]
proposed patch (passed QA)

QA comment

* The SIGNED-OFF patch does not apply with a strange message :
fatal: cannot convert from UTF-8utf-8 to UTF-8

Looking at it I can see :

Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="utf-8"

If I remove the 3rd line, patch applies cleanly now. Katrin, it seems the 3rd line has been added by your format patch. Any idea why ?


* The patch fixes a really annoying problem.
* Works as expected. I had some problems to reproduce the bad behaviour due to a missing information = manualhistory must be set to 0 (unchecked) to see it !
* I still don't understand why we must sometimes explicitly decode utf8...

Marking passed QA
Comment 5 Katrin Fischer 2011-08-11 16:15:57 UTC
No idea Paul, but seems to be a one time problem. Other patches signed-off by me seem to be ok.
Comment 6 Chris Cormack 2011-08-12 02:01:17 UTC
The comment here is misleading, we are attempting to decode from UTF-X to 

utf8::decode($string)

           Attempts to convert in-place the octet sequence in UTF-X to the corresponding character
           sequence.  The UTF-8 flag is turned on only if the source string contains multiple-byte
           UTF-X characters.  If $string is invalid as UTF-X, returns false; otherwise returns true.

           Note that this function does not handle arbitrary encodings.  Therefore Encode is
           recommended for the general purposes; see also Encode.

 	
Frédérick are we sure we want to use decode here, not encode?
Comment 7 Paul Poulain 2011-08-12 07:49:34 UTC
(In reply to comment #6)
> The comment here is misleading, we are attempting to decode from UTF-X to 
> 
> utf8::decode($string)

> Frédérick are we sure we want to use decode here, not encode?

That's a part of the problem I have with utf8 encoding/decoding/... = I don't understand why this string is needed, I just can confirm it works. There is another bug (about itemtypes with diacritics in description) that is also related to this utf8:decode. utf8 handling in Perl is still a mystery for me :(
Comment 8 Frédérick Capovilla 2011-08-12 19:42:58 UTC
It's been a while since I created that patch but here is what I understand :

I remember that in the NewIssue subroutine of Serials.pm, The content of 
$serialseq is concatenated with a variable fetched from a SQL query and that's where the problem happen.

I did some tests to check the utf-8 flag on those two variables
$serialseq = variable from the form (is_utf8 = false)
$recievedlist = variable from SQL (is_utf8 = true)

The encoding of the data fetched from SQL differs from the encoding of the data received from the form. If two string with a different encoding gets concatenated together, the encoding of one of the string is automatically changed, and we get an encoding problem on one half of the string.

Using decode on $serialseq adds the utf-8 flag, so we don't get an encoding problem when we concatenate.


We don't get this encoding problem when the first item is added in $recievedlist because $recievedlist is empty and doesn't automatically get the utf-8 flag. I'm guessing Perl DBI automatically addes the utf-8 flag if it finds utf-8 characters in a string returned from a SELECT.

Anyways, that's what I think is happening. Correct me if I'm wrong?
Comment 9 Paul Poulain 2011-09-02 09:05:03 UTC
chris, this has passed QA, the answer from Frédérick sounds logical, I think you can push.
Comment 10 Chris Cormack 2011-09-02 18:40:50 UTC
Pushed, please test
Comment 11 Chris Nighswonger 2011-10-26 18:02:18 UTC
The fix for this bug was published in the 3.4.5 release. If you were the reporter of this bug, please take time to verify the fix and update the status of this bug report accordingly. If the bug is fixed to your satisfaction, please close this report.
Comment 12 Jared Camins-Esakov 2012-12-30 23:58:36 UTC
There have been no further reports of problems so I am marking this bug resolved.