Bug 2705 - Patron import should warn about invalid characters
Summary: Patron import should warn about invalid characters
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: master
Hardware: PC All
: P3 enhancement (vote)
Assignee: Galen Charlton
QA Contact:
Depends on:
Reported: 2008-10-21 08:17 UTC by Galen Charlton
Modified: 2019-06-28 13:55 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2010-05-21 00:54:55 UTC

---- Reported by gmcharlt@gmail.com 2008-10-21 08:17:15 ----

Per the patch submitted for http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=2627, the patron import now requires the input file to use UTF-8.

If non-UTF-8 characters are present, the tool does not warn about it.  Instead, when the affected field is loaded, MySQL will silently truncate the contents starting with the first invalid character.

The tool should check its input.  If any fields have invalid 8-bit characters, it should provide a warning and refuse to load the record.

---- Additional Comments From joe.atzberger@liblime.com 2009-05-06 17:21:12 ----

I'm not sure how to approach this.  We only access the attachment via Text::CSV in binary mode.  

Do you want a prepass for invalid characters?  Or a per-line check with Encode::Guess?

We may conclude that no additional code is warranted, since the errors for Text::CSV parsing are already well reported.  If the attachment is ASCII, for example, then there is no problem.  What other encodings are we expecting?

It seems like no CSV editor is going to insert characters outside UTF-8 too easily.  Can you attach an example CSV file with data you think should be caught by the script?

--- Bug imported by chris@bigballofwax.co.nz 2010-05-21 00:54 UTC  ---

This bug was previously known as _bug_ 2705 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=2705

Actual time not defined. Setting to 0.0

Comment 1 Katrin Fischer 2019-05-04 18:32:14 UTC
An example where I have seen this would be a file with German umlauts in a non-UTF-8 encoding. The surname Müller would turn into M after import.
Comment 2 Castro B 2019-06-28 13:55:42 UTC
what is the dif. of UTF-8,UTF-16 and ANSI?

Castro B,