the misc/cronjobs/overdue_notices.pl script produces output/files with confusing encoding apparently depending on the content of the lines written. sometimes with characters encoded in latin1 or utf8 in the same file. https://github.com/Koha-Community/Koha/blob/main/misc/cronjobs/overdue_notices.pl#L422 ``` - open $csv_fh, ">", $csvfilename or die "unable to open $csvfilename: $!"; + open $csv_fh, ">:encoding(UTF-8)", $csvfilename or die "unable to open $csvfilename: $!"; ``` there is possible more needed to fix this then just changing this one line. ``` overdue_notices.pl -v -test -list-all -date="2024-11-01" -csv="overdue.csv" -library="AT-LBST" ``` # examples # ASCII text (only csv headers present) # ISO-8859 text (if german umlauts are present) # Non-ISO extended-ASCII text (if german umlauts AND "č" present) ``` ls 202501*|xargs file 20250101_0310_overdue_test.csv: ASCII text 20250102_0310_overdue_test.csv: ASCII text 20250103_0310_overdue_test.csv: ISO-8859 text 20250104_0310_overdue_test.csv: ASCII text 20250105_0310_overdue_test.csv: ASCII text 20250106_0310_overdue_test.csv: ASCII text 20250107_0310_overdue_test.csv: ISO-8859 text 20250108_0310_overdue_test.csv: ISO-8859 text, with very long lines 20250109_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines 20250110_0310_overdue_test.csv: Non-ISO extended-ASCII text 20250111_0310_overdue_test.csv: ASCII text 20250112_0310_overdue_test.csv: ASCII text 20250113_0310_overdue_test.csv: ISO-8859 text 20250114_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines 20250115_0310_overdue_test.csv: ISO-8859 text, with very long lines 20250116_0310_overdue_test.csv: ISO-8859 text 20250117_0310_overdue_test.csv: ISO-8859 text 20250118_0310_overdue_test.csv: ASCII text 20250119_0310_overdue_test.csv: ASCII text 20250120_0310_overdue_test.csv: ISO-8859 text 20250121_0310_overdue_test.csv: ISO-8859 text 20250122_0310_overdue_test.csv: ISO-8859 text 20250123_0310_overdue_test.csv: ISO-8859 text 20250124_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines 20250125_0310_overdue_test.csv: ASCII text 20250126_0310_overdue_test.csv: ASCII text 20250127_0310_overdue_test.csv: Non-ISO extended-ASCII text 20250128_0310_overdue_test.csv: ISO-8859 text 20250129_0310_overdue_test.csv: ASCII text 20250130_0310_overdue_test.csv: ASCII text 20250131_0310_overdue_test.csv: Non-ISO extended-ASCII text, with very long lines ``` old bugs of the same kind in overdue_notices.pl https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8687 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26601
the change i included in the first bug report at least produces a file where german umlaut "Ö" (UTF-8 hex c3 96) and "č" (UTF-8 hex c4 8d) seem to be encoded correctly.
Hi, do you want to provide the patch file? If you are working with koha-testing docker you can use: git bz attach. If not, a file generated with git format-patch HEAD^ or similar works too. You might want to check the correct name/email address is included in the header lines.
Created attachment 177496 [details] [review] patch to set the correct encoding for the CSV output file
Testplan: I) Prepare overdue entry (intranet): 0) Create a patron account with surname Höfler 1) Check if in Patron category administration the overdue column is YES for his code 2) In Tools -> Notice triggers define first delay = 7, letter Overdue notice. Nothing for second and third 3) Check if Tools -> Notices and slips, code overdue is available 4) checkout a book with a retard more than 7 days. 5) Check if Adminstration -> Defining circulation and fine rules for all libraries has fine rules for his patron category, item type book. II) Before applying the patch 0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_old.csv" 1) Validate the characters are in Latin1 encoding. (hexdump -C overdue.csv|less) ö => "f6" III) After applying the patch 0) Run misc/cronjobs/overdue_notices.pl -v -test -list-all -csv="overdue_fixed.csv" 1) Validate the characters are in UTF-8 encoding. (hexdump -C overdue_fixed.csv|less) ö => "c3 b6"
Sponsored-by: Styrian State Library
Created attachment 177507 [details] [review] add utf8 encoding for csv file output new patch including testplan and sponsored-by statement