Bug 15290

Summary:	SQL reports encoding problem
Product:	Koha	Reporter:	Vitor Fernandes <vfernandes>
Component:	Reports	Assignee:	Tomás Cohen Arazi (tcohen) <tomascohen>
Status:	CLOSED FIXED	QA Contact:	Testopia <testopia>
Severity:	major
Priority:	P1 - high	CC:	f.demians, hector.hecaxmmx, jonathan.druart, josef.moravec, julian.maurice, kyle, tomascohen
Version:	Main
Hardware:	All
OS:	All
See Also:	http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11944 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22323
Change sponsored?:	---	Patch complexity:	Trivial patch
Documentation contact:		Documentation submission:
Text to go in the release notes:		Version(s) released in:
Circulation function:
Attachments:	Bug 15290: SQL reports headers encoding problem Bug 15290: (follow-up)SQL reports headers encoding problem Bug 15290: (follow-up)SQL reports headers encoding problem Bug 15290: (follow-up)SQL reports headers encoding problem Bug 15290: SQL reports headers encoding problem Bug 15290: (follow-up)SQL reports headers encoding problem Bug 15290: SQL reports headers encoding problem

Description Vitor Fernandes 2015-12-03 13:26:07 UTC

The problem appears when running a SQL report or exporting a SQL report. The column header are in incorrect encoding.

Test plan:

1 - Add a new SQL report using the following SQL for example:

SELECT borrowers.cardnumber 'Número cartão', CONCAT (borrowers.firstname, ' ' , borrowers.surname) Nome, categories.description Grupo, borrowers.branchcode Biblioteca,borrowers.dateenrolled 'Data de inscrição', borrowers.dateexpiry 'Data limite'
FROM borrowers
INNER JOIN categories
ON borrowers.categorycode = categories.categorycode
ORDER BY borrowers.cardnumber ASC

2 - Run the report
3 - The column headers are incorrect.
4 - Export the report to CSV.
5 - The column headers on the file are incorrect.

Comment 1 Tomás Cohen Arazi (tcohen) 2015-12-03 14:29:27 UTC Comment hidden (obsolete)

Created attachment 45353 [details] [review]
Bug 15290: SQL reports headers encoding problem

To test:

- Create an SQL report with column names containing UTF characters
- Run the report
=> FAIL: The column names are double encoded
- Export the report
=> FAIL: The column names are double encoded in the exported file
- Apply the patch
- Re-run the report (reload works)
=> SUCCESS: Encoding correct
- Export the report
=> SUCCESS: Encoding correct
- Sign off :-D

Comment 2 Tomás Cohen Arazi (tcohen) 2015-12-03 14:32:10 UTC

(In reply to Tomás Cohen Arazi from comment #1)
> - Export the report
> => SUCCESS: Encoding correct

It doesn't fix the exported files. Followup needed.

Comment 3 Héctor Eduardo Castro Avalos 2015-12-03 14:51:47 UTC

(In reply to Tomás Cohen Arazi from comment #2)
> (In reply to Tomás Cohen Arazi from comment #1)
> > - Export the report
> > => SUCCESS: Encoding correct
> 
> It doesn't fix the exported files. Followup needed.

Just fail in comma separated in tab separated and open document doesn't fail.

Comment 4 Vitor Fernandes 2015-12-03 14:58:30 UTC

CSV fails.
Tab separated text fails.
Open Document spreadsheet it's fine.

Comment 5 Héctor Eduardo Castro Avalos 2015-12-03 16:18:34 UTC

(In reply to Vitor Fernandes from comment #4)
> CSV fails.
> Tab separated text fails.
> Open Document spreadsheet it's fine.

I'm working on it, I found some issues

Comment 6 Héctor Eduardo Castro Avalos 2015-12-03 19:18:01 UTC Comment hidden (obsolete)

Created attachment 45384 [details] [review]
Bug 15290: (follow-up)SQL reports headers encoding problem

Fix SQL row data and headers enconding problem

To test:
-Apply patches
-Follow previous test plan
-Notice the headers and row data is displayed in UTF-8
 without double encoding

Comment 7 Héctor Eduardo Castro Avalos 2015-12-03 19:23:37 UTC Comment hidden (obsolete)

Created attachment 45385 [details] [review]
Bug 15290: (follow-up)SQL reports headers encoding problem

Fix SQL row data and headers enconding problem

To test:
-Apply patches
-Follow previous test plan
-Notice the headers and row data is displayed in UTF-8
 without double encoding
-Save in Comma separated text, Tab seprated text, and Open Document.
-Verify in the three options the file exported is Ok

Comment 8 Frédéric Demians 2015-12-07 15:57:38 UTC Comment hidden (obsolete)

Created attachment 45456 [details] [review]
Bug 15290: (follow-up)SQL reports headers encoding problem

Fix SQL row data and headers enconding problem

To test:
-Apply patches
-Follow previous test plan
-Notice the headers and row data is displayed in UTF-8
 without double encoding
-Save in Comma separated text, Tab seprated text, and Open Document.
-Verify in the three options the file exported is Ok

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

Comment 9 Frédéric Demians 2015-12-07 15:58:50 UTC Comment hidden (obsolete)

Created attachment 45457 [details] [review]
Bug 15290: SQL reports headers encoding problem

To test:

- Create an SQL report with column names containing UTF characters
- Run the report
=> FAIL: The column names are double encoded
- Export the report
=> FAIL: The column names are double encoded in the exported file
- Apply the patch
- Re-run the report (reload works)
=> SUCCESS: Encoding correct
- Export the report
=> SUCCESS: Encoding correct
- Sign off :-D

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

Comment 10 Jonathan Druart 2015-12-08 15:47:38 UTC

Created attachment 45501 [details] [review]
Bug 15290: (follow-up)SQL reports headers encoding problem

Fix SQL row data and headers enconding problem

To test:
-Apply patches
-Follow previous test plan
-Notice the headers and row data is displayed in UTF-8
 without double encoding
-Save in Comma separated text, Tab seprated text, and Open Document.
-Verify in the three options the file exported is Ok

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Comment 11 Jonathan Druart 2015-12-08 15:47:42 UTC

Created attachment 45502 [details] [review]
Bug 15290: SQL reports headers encoding problem

To test:

- Create an SQL report with column names containing UTF characters
- Run the report
=> FAIL: The column names are double encoded
- Export the report
=> FAIL: The column names are double encoded in the exported file
- Apply the patch
- Re-run the report (reload works)
=> SUCCESS: Encoding correct
- Export the report
=> SUCCESS: Encoding correct
- Sign off :-D

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Comment 12 Jonathan Druart 2015-12-08 15:50:22 UTC

Linked to bug 11944.
Must be backported to 3.22.x and 3.20.x

Comment 13 Kyle M Hall (khall) 2015-12-11 15:20:16 UTC

Patches pushed to master!

Thanks Hector and Tomas!

Comment 14 Julian Maurice 2015-12-11 16:07:40 UTC

Patches pushed to 3.22.x, will be in 3.22.1

Comment 15 Frédéric Demians 2015-12-11 17:12:15 UTC

This patch has been pushed to 3.20.x, will be in 3.20.7.