Bug 39023 - Add utf8 output to CSV output of overdue_notices.pl
Summary: Add utf8 output to CSV output of overdue_notices.pl
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement
Assignee: davewood
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-02-03 20:23 UTC by davewood
Modified: 2025-02-04 13:24 UTC (History)
2 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
patch to set the correct encoding for the CSV output file (1014 bytes, patch)
2025-02-04 10:51 UTC, davewood
Details | Diff | Splinter Review
add utf8 encoding for csv file output (2.07 KB, patch)
2025-02-04 13:24 UTC, davewood
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description davewood 2025-02-03 20:23:38 UTC
the misc/cronjobs/overdue_notices.pl script produces output/files with confusing encoding apparently depending on the content of the lines written. sometimes with characters encoded in latin1 or utf8 in the same file.

https://github.com/Koha-Community/Koha/blob/main/misc/cronjobs/overdue_notices.pl#L422
```
-        open $csv_fh, ">", $csvfilename or die "unable to open $csvfilename: $!";
+        open $csv_fh, ">:encoding(UTF-8)", $csvfilename or die "unable to open $csvfilename: $!";
```
there is possible more needed to fix this then just changing this one line.

```
overdue_notices.pl -v -test -list-all -date="2024-11-01" -csv="overdue.csv" -library="AT-LBST"
```

# examples
# ASCII text (only csv headers present)
# ISO-8859 text (if german umlauts are present)
# Non-ISO extended-ASCII text (if german umlauts AND "č" present)
```
ls 202501*|xargs file
20250101_0310_overdue_test.csv: ASCII text
20250102_0310_overdue_test.csv: ASCII text
20250103_0310_overdue_test.csv: ISO-8859 text
20250104_0310_overdue_test.csv: ASCII text
20250105_0310_overdue_test.csv: ASCII text
20250106_0310_overdue_test.csv: ASCII text
20250107_0310_overdue_test.csv: ISO-8859 text
20250108_0310_overdue_test.csv: ISO-8859 text, with very long lines
20250109_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines
20250110_0310_overdue_test.csv: Non-ISO extended-ASCII text
20250111_0310_overdue_test.csv: ASCII text
20250112_0310_overdue_test.csv: ASCII text
20250113_0310_overdue_test.csv: ISO-8859 text
20250114_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines
20250115_0310_overdue_test.csv: ISO-8859 text, with very long lines
20250116_0310_overdue_test.csv: ISO-8859 text
20250117_0310_overdue_test.csv: ISO-8859 text
20250118_0310_overdue_test.csv: ASCII text
20250119_0310_overdue_test.csv: ASCII text
20250120_0310_overdue_test.csv: ISO-8859 text
20250121_0310_overdue_test.csv: ISO-8859 text
20250122_0310_overdue_test.csv: ISO-8859 text
20250123_0310_overdue_test.csv: ISO-8859 text
20250124_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines
20250125_0310_overdue_test.csv: ASCII text
20250126_0310_overdue_test.csv: ASCII text
20250127_0310_overdue_test.csv: Non-ISO extended-ASCII text
20250128_0310_overdue_test.csv: ISO-8859 text
20250129_0310_overdue_test.csv: ASCII text
20250130_0310_overdue_test.csv: ASCII text
20250131_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines
```

old bugs of the same kind in overdue_notices.pl
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8687
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26601
Comment 1 davewood 2025-02-03 20:49:33 UTC
the change i included in the first bug report at least produces a file where german umlaut "Ö" (UTF-8 hex c3 96) and "č" (UTF-8 hex c4 8d)  seem to be encoded correctly.
Comment 2 Katrin Fischer 2025-02-04 09:58:00 UTC
Hi, do you want to provide the patch file? If you are working with koha-testing docker you can use: git bz attach. If not, a file generated with git format-patch HEAD^ or similar works too. You might want to check the correct name/email address is included in the header lines.
Comment 3 davewood 2025-02-04 10:51:08 UTC
Created attachment 177496 [details] [review]
patch to set the correct encoding for the CSV output file
Comment 4 davewood 2025-02-04 12:54:26 UTC
Testplan:

I) Prepare overdue entry (intranet):
0) Create a patron account with surname Höfler
1) Check if in Patron category administration the overdue column
   is YES for his code
2) In Tools -> Notice triggers define first delay = 7, letter
   Overdue notice. Nothing for second and third
3) Check if Tools -> Notices and slips, code overdue is available
4) checkout a book with a retard more than 7 days.
5) Check if Adminstration -> Defining circulation and fine rules
   for all libraries has fine rules for his patron category,
   item type book.

II) Before applying the patch
0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_old.csv"
1) Validate the characters are in Latin1 encoding. (hexdump -C overdue.csv|less) ö => "f6"

III) After applying the patch
0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_fixed.csv"
1) Validate the characters are in UTF-8 encoding. (hexdump -C overdue_fixed.csv|less) ö => "c3 b6"
Comment 5 davewood 2025-02-04 13:04:36 UTC
Sponsored-by: Styrian State Library
Comment 6 davewood 2025-02-04 13:24:26 UTC
Created attachment 177507 [details] [review]
add utf8 encoding for csv file output

new patch including testplan and sponsored-by statement