It also affects "Item number files" which triggers bug 21141 But it's easier to test with barcodes. And the fix should be the same. == Test plan == 1. Go to Staff › Tools › Batch item modification 2. Upload the file from the attachments (see next comments) named numbers-without-BOM.csv 3. See that it complains about non existing barcodes but the codes look ok 4. Upload the file from the attachments named numbers-with-BOM.csv 5. See that the first line is corrupted. Something like "3131545845784555" If you don't know how to check that my test files are correct. On a Unix-like system do: file numbers-with-BOM.csv file numbers-without-BOM.csv less numbers-with-BOM.csv less numbers-without-BOM.csv
Created attachment 77406 [details] Numbers without BOM
Created attachment 77407 [details] Numbers with BOM
I'm not sure to work on this. Anyone can feel free to take it.
It is just a display problem, right?
Created attachment 77986 [details] [review] Bug 21142: Open uploaded files in utf-8 - batch mod/del Batch item/record modification/deletion tools does not open uploaded files in utf-8 Test plan: Edit and delete records and items using a file which will contain: valid and invalid barcodes with and without utf-8 chars ([U+1F495]) You should see the warning message: "The *** 39999000[U+1F495]001679 does not exist in the database."
Created attachment 77992 [details] [review] Bug 21142: Open uploaded files in utf-8 - batch mod/del Batch item/record modification/deletion tools does not open uploaded files in utf-8 Test plan: Edit and delete records and items using a file which will contain: valid and invalid barcodes with and without utf-8 chars ([U+1F495]) You should see the warning message: "The *** 39999000[U+1F495]001679 does not exist in the database." Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Created attachment 78015 [details] [review] Bug 21142: Open uploaded files in utf-8 - batch mod/del Batch item/record modification/deletion tools does not open uploaded files in utf-8 Test plan: Edit and delete records and items using a file which will contain: valid and invalid barcodes with and without utf-8 chars ([U+1F495]) You should see the warning message: "The *** 39999000[U+1F495]001679 does not exist in the database." Signed-off-by: Owen Leonard <oleonard@myacpl.org> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Awesome work all! Pushed to master for 18.11
Thanks for the fix! [U+1F495][U+1F495][U+1F495] (In reply to Jonathan Druart from comment #4) > It is just a display problem, right? Actually jes and no, because with the BOM test case, it's still there (invisible instead of being mojibake[1]) And when using itemnumbers it's not just a display problem, it will make the first itemnumber invalid. For some reason, with barcodes it works; there must be some code that cleans the beginning and the end of the line. [1] https://en.wikipedia.org/wiki/Mojibake
Pushed to 18.05.x for 18.05.03
Pushed to 17.11.x for 17.11.10