Bug 15290

Summary: SQL reports encoding problem
Product: Koha Reporter: Vitor Fernandes <vfernandes>
Component: ReportsAssignee: Tomás Cohen Arazi <tomascohen>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P1 - high CC: f.demians, hector.hecaxmmx, jonathan.druart, josef.moravec, julian.maurice, kyle, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11944
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22323
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 15290: SQL reports headers encoding problem
Bug 15290: (follow-up)SQL reports headers encoding problem
Bug 15290: (follow-up)SQL reports headers encoding problem
Bug 15290: (follow-up)SQL reports headers encoding problem
Bug 15290: SQL reports headers encoding problem
Bug 15290: (follow-up)SQL reports headers encoding problem
Bug 15290: SQL reports headers encoding problem

Description Vitor Fernandes 2015-12-03 13:26:07 UTC
The problem appears when running a SQL report or exporting a SQL report. The column header are in incorrect encoding.

Test plan:

1 - Add a new SQL report using the following SQL for example:

SELECT borrowers.cardnumber 'Número cartão', CONCAT (borrowers.firstname, ' ' , borrowers.surname) Nome, categories.description Grupo, borrowers.branchcode Biblioteca,borrowers.dateenrolled 'Data de inscrição', borrowers.dateexpiry 'Data limite'
FROM borrowers
INNER JOIN categories
ON borrowers.categorycode = categories.categorycode
ORDER BY borrowers.cardnumber ASC

2 - Run the report
3 - The column headers are incorrect.
4 - Export the report to CSV.
5 - The column headers on the file are incorrect.
Comment 1 Tomás Cohen Arazi 2015-12-03 14:29:27 UTC Comment hidden (obsolete)
Comment 2 Tomás Cohen Arazi 2015-12-03 14:32:10 UTC
(In reply to Tomás Cohen Arazi from comment #1)
> - Export the report
> => SUCCESS: Encoding correct

It doesn't fix the exported files. Followup needed.
Comment 3 Héctor Eduardo Castro Avalos 2015-12-03 14:51:47 UTC
(In reply to Tomás Cohen Arazi from comment #2)
> (In reply to Tomás Cohen Arazi from comment #1)
> > - Export the report
> > => SUCCESS: Encoding correct
> 
> It doesn't fix the exported files. Followup needed.

Just fail in comma separated in tab separated and open document doesn't fail.
Comment 4 Vitor Fernandes 2015-12-03 14:58:30 UTC
CSV fails.
Tab separated text fails.
Open Document spreadsheet it's fine.
Comment 5 Héctor Eduardo Castro Avalos 2015-12-03 16:18:34 UTC
(In reply to Vitor Fernandes from comment #4)
> CSV fails.
> Tab separated text fails.
> Open Document spreadsheet it's fine.

I'm working on it, I found some issues
Comment 6 Héctor Eduardo Castro Avalos 2015-12-03 19:18:01 UTC Comment hidden (obsolete)
Comment 7 Héctor Eduardo Castro Avalos 2015-12-03 19:23:37 UTC Comment hidden (obsolete)
Comment 8 Frédéric Demians 2015-12-07 15:57:38 UTC Comment hidden (obsolete)
Comment 9 Frédéric Demians 2015-12-07 15:58:50 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2015-12-08 15:47:38 UTC
Created attachment 45501 [details] [review]
Bug 15290: (follow-up)SQL reports headers encoding problem

Fix SQL row data and headers enconding problem

To test:
-Apply patches
-Follow previous test plan
-Notice the headers and row data is displayed in UTF-8
 without double encoding
-Save in Comma separated text, Tab seprated text, and Open Document.
-Verify in the three options the file exported is Ok

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 11 Jonathan Druart 2015-12-08 15:47:42 UTC
Created attachment 45502 [details] [review]
Bug 15290: SQL reports headers encoding problem

To test:

- Create an SQL report with column names containing UTF characters
- Run the report
=> FAIL: The column names are double encoded
- Export the report
=> FAIL: The column names are double encoded in the exported file
- Apply the patch
- Re-run the report (reload works)
=> SUCCESS: Encoding correct
- Export the report
=> SUCCESS: Encoding correct
- Sign off :-D

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 12 Jonathan Druart 2015-12-08 15:50:22 UTC
Linked to bug 11944.
Must be backported to 3.22.x and 3.20.x
Comment 13 Kyle M Hall 2015-12-11 15:20:16 UTC
Patches pushed to master!

Thanks Hector and Tomas!
Comment 14 Julian Maurice 2015-12-11 16:07:40 UTC
Patches pushed to 3.22.x, will be in 3.22.1
Comment 15 Frédéric Demians 2015-12-11 17:12:15 UTC
This patch has been pushed to 3.20.x, will be in 3.20.7.