1. Set syspref "delimiter" to "tabs" 2. Create a vendor with minimal info 3. Create a basket with minimal info 4. Go to the basket. URL should be /cgi-bin/koha/acqui/basket.pl?basketno=XXXXX 5. "Export as CSV" 6. You should see that the file contains commas "," This is the bug. 7. Create a basket group with minimal info 8. Close this basket group 9. Go to the "Closed" tab to see your basket group 10. "Export as CSV" 11. You should see that the file contains commas "," This is the bug. 12. Create a basket with minimal info 13. Add a item to the basket 14. Close the basket 15. Go to /cgi-bin/koha/acqui/lateorders.pl 16. "Export as CSV" 17. You should see that the file contains commas "," This is the bug.
Everything come from here: http://git.koha-community.org/gitweb/?p=koha.git;a=tree;f=koha-tmpl/intranet-tmpl/prog/en/modules/acqui/csv;hb=94cd27677d8febd2979f9f618f1e8cd1bd2d68d7 3 templates with commas hardcoded
Created attachment 70421 [details] [review] Bug 19928: Acquisitions' CSV exports now honors syspref "delimiter" Bonus fixes: - Basketgroup: there was an unnecessary space. " RRP tax excluded" replaced by "RRP tax excluded" - Late orders: - leading spaces before every line - surrounding spaces for [% orders.size %] Other changes: - quoted "Total orders in late" to no worry about additional spaces - Late orders: additional spaces which shouldn't be a problem because all the fields are enclosed by quotes. Test plan: 1. Set syspref "delimiter" to "#'s" 2. Change the language e.g. FR-fr 3. Create a vendor with minimal info 4. Create a basket with two items 5. Go to the basket. URL should be /cgi-bin/koha/acqui/basket.pl?basketno=XXXXX 6. Close this basket 7. "Export as CSV" 8. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 9. Create a basket group with two baskets (tick the close this basket group check box) 10. Go to the "Closed" tab to see your basket group 11. "Export as CSV" 12. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 13. Go to /cgi-bin/koha/acqui/lateorders.pl 14. Tick two orders. Of the same vendor, otherwise Koha won't let you tick orders of different vendors. 15. "Export as CSV" 16. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 17. Apply this patch 18. Translation stuff 1. cd misc/translator 2. perl translate update YOUR_LANG 3. manually edit the po file, remove the fuzzy tags. And translate the string. You can just copy the original and replace some parts by gibberish. 4. perl translate install YOUR_LANG 19. Re-export the same CSVs and compare with the original version to check that - delimiter syspref is honored - the headers are translatable - there is no whitespace regression (additional newlines or spaces)
Created attachment 70422 [details] [review] Bug 19928: Acquisitions' CSV exports now honors syspref "delimiter" Bonus fixes: - Basketgroup: there was an unnecessary space. " RRP tax excluded" replaced by "RRP tax excluded" - Late orders: - leading spaces before every line - surrounding spaces for [% orders.size %] Other changes: - quoted "Total orders in late" to no worry about additional spaces - Late orders: additional spaces which shouldn't be a problem because all the fields are enclosed by quotes. Test plan: 1. Set syspref "delimiter" to "#'s" 2. Change the language e.g. FR-fr 3. Create a vendor with minimal info 4. Create a basket with two items 5. Go to the basket. URL should be /cgi-bin/koha/acqui/basket.pl?basketno=XXXXX 6. Close this basket 7. "Export as CSV" 8. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 9. Create a basket group with two baskets (tick the close this basket group check box) 10. Go to the "Closed" tab to see your basket group 11. "Export as CSV" 12. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 13. Go to /cgi-bin/koha/acqui/lateorders.pl 14. Tick two orders. Of the same vendor, otherwise Koha won't let you tick orders of different vendors. 15. "Export as CSV" 16. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 17. Apply this patch 18. Translation stuff 1. cd misc/translator 2. perl translate update YOUR_LANG 3. manually edit the po file, remove the fuzzy tags. And translate the string. You can just copy the original and replace some parts by gibberish. 4. perl translate install YOUR_LANG 19. Re-export the same CSVs and compare with the original version to check that - delimiter syspref is honored - the headers are translatable - there is no whitespace regression (additional newlines or spaces)
Thanks Joubu for this explanation: http://git.koha-community.org/gitweb/?p=koha.git;a=commit;h=8403799c7212349bee70090249e3425afd62423b About the part "in order to make it a have-to-read for next changes." How would the next person find this? As it won't be the last commit.
(In reply to Victor Grousset/tuxayo from comment #4) > Thanks Joubu for this explanation: > http://git.koha-community.org/gitweb/?p=koha.git;a=commit; > h=8403799c7212349bee70090249e3425afd62423b > About the part "in order to make it a have-to-read for next changes." > How would the next person find this? As it won't be the last commit. (It could be a have-to-read if I fix all the grammar errors!) At least you, Katrin and me are aware of this commit and the problem. So we can guide people to this commit message when they need it :) Feel free to create a wiki page if you think it is a good idea (and you have time).
About your patches, it should not be needed but it is the behaviour of other occurrences: you do not default to ';', ',' or '\t' if the pref 'delimiter' does not exist. Maybe ',' is better here as it was the value before your changes.
(In reply to Jonathan Druart from comment #6) > [...] you do not default to ';', ',' or '\t' if the pref 'delimiter' > does not exist. > > Maybe ',' is better here as it was the value before your changes. You're right, I though that the pref 'delimiter' always existed. Is giving 'delimiter' a default value in the DB[1] also an option? [1] INSERT in updatedatabase.pl
Created attachment 70426 [details] [review] Bug 19928: Acquisitions' CSV exports now honors syspref "delimiter" Bonus fixes: - Basketgroup: there was an unnecessary space. " RRP tax excluded" replaced by "RRP tax excluded" - Late orders: - leading spaces before every line - surrounding spaces for [% orders.size %] Other changes: - quoted "Total orders in late" to no worry about additional spaces - Late orders: additional spaces which shouldn't be a problem because all the fields are enclosed by quotes. Test plan: 1. Set syspref "delimiter" to "#'s" 2. Change the language e.g. FR-fr 3. Create a vendor with minimal info 4. Create a basket with two items 5. Go to the basket. URL should be /cgi-bin/koha/acqui/basket.pl?basketno=XXXXX 6. Close this basket 7. "Export as CSV" 8. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 9. Create a basket group with two baskets (tick the close this basket group check box) 10. Go to the "Closed" tab to see your basket group 11. "Export as CSV" 12. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 13. Go to /cgi-bin/koha/acqui/lateorders.pl 14. Tick two orders. Of the same vendor, otherwise Koha won't let you tick orders of different vendors. 15. "Export as CSV" 16. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 17. Apply this patch 18. Translation stuff 1. cd misc/translator 2. perl translate update YOUR_LANG 3. manually edit the po file, remove the fuzzy tags. And translate the string. You can just copy the original and replace some parts by gibberish. 4. perl translate install YOUR_LANG 19. Re-export the same CSVs and compare with the original version to check that - delimiter syspref is honored - the headers are translatable - there is no whitespace regression (additional newlines or spaces)
"," is now used by default if the syspref is missing.
basket.tt and basketgroup.tt aren't [%- delimiter -%]. I believe the hyphens are important to the formatting of the file. Please correct.
(In reply to M. Tompsett from comment #10) > basket.tt and basketgroup.tt aren't [%- delimiter -%]. I believe the hyphens > are important to the formatting of the file. Please correct. I remember removing the post/pre-chomp after having checked that the output was still correct. Is there another caveat?
(In reply to Victor Grousset/tuxayo from comment #11) > I remember removing the post/pre-chomp after having checked that the output > was still correct. Is there another caveat? If you say it is the same, I'll test it. But it just didn't eyeball well. It if works, I'll set to signed off. If not, I'll confirm that it didn't.
Failed QA test tools.
Please check that the translated export still works nicely after this change.
See bug 18331 - we have broken the translations of the CSV files more times than I can count - should not repeat that.
(In reply to Katrin Fischer from comment #14) > Please check that the translated export still works nicely after this change. That's his test plan in comment #8.
It's missing steps. - you need to update the po files after the patch was applied - you have to check that the string got not marked as 'fuzzy' - you have to reinstall the language Only then the test gives you a correct result.
(In reply to Katrin Fischer from comment #17) > It's missing steps. > > - you need to update the po files after the patch was applied > - you have to check that the string got not marked as 'fuzzy' > - you have to reinstall the language > > Only then the test gives you a correct result. So in short, the test plan in comment #8 combined with this should work. However, this is still Failed QA for not passing the koha qa test tools.
(In reply to M. Tompsett from comment #13) > Failed QA test tools. It passed for me. Did I make any mistake? ########################################################## kohadev-koha@kohadevbox:/home/vagrant/kohaclone$ git show HEAD commit 50d3119a32ce0f8ecb0ab88404975f9cfab03b43 Author: Victor Grousset <victor.grousset@biblibre.com> Date: Wed Jan 10 16:07:36 2018 +0100 Bug 19928: Acquisitions' CSV exports now honors syspref "delimiter" ################################################################# kohadev-koha@kohadevbox:/home/vagrant/kohaclone$ qa -v 1 -c 1 testing 1 commit(s) (applied to 94cd276 'Bug 19918: Close span tag in opac-reg') Processing files before patches |========================>| 6 / 6 (100.00%) Processing files after patches |========================>| 6 / 6 (100.00%) OK koha-tmpl/intranet-tmpl/prog/en/includes/csv_headers/acqui/basket.tt OK forbidden patterns OK git manipulation OK spelling OK tt_valid OK valid_template OK koha-tmpl/intranet-tmpl/prog/en/includes/csv_headers/acqui/basketgroup.tt OK forbidden patterns OK git manipulation OK spelling OK tt_valid OK valid_template OK koha-tmpl/intranet-tmpl/prog/en/includes/csv_headers/acqui/lateorders.tt OK forbidden patterns OK git manipulation OK spelling OK tt_valid OK valid_template OK koha-tmpl/intranet-tmpl/prog/en/modules/acqui/csv/basket.tt OK forbidden patterns OK git manipulation OK spelling OK tt_valid OK valid_template OK koha-tmpl/intranet-tmpl/prog/en/modules/acqui/csv/basketgroup.tt OK forbidden patterns OK git manipulation OK spelling OK tt_valid OK valid_template OK koha-tmpl/intranet-tmpl/prog/en/modules/acqui/csv/lateorders.tt OK forbidden patterns OK git manipulation OK spelling OK tt_valid OK valid_template #################################################################
Tested also with current master without error. commit 129ffee791da9f8ab4ad85a4473a138fc81ccbef Author: Jonathan Druart <jonathan.druart@bugs.koha-community.org> Date: Mon Jan 15 10:10:41 2018 -0300 Bug 19968: Add missing use Date::Calc statement
(In reply to Katrin Fischer from comment #17) > It's missing steps. > > - you need to update the po files after the patch was applied > - you have to check that the string got not marked as 'fuzzy' > - you have to reinstall the language > > Only then the test gives you a correct result. I don't get, isn't this covered in the steps 18?
(In reply to Victor Grousset/tuxayo from comment #19) > (In reply to M. Tompsett from comment #13) > > Failed QA test tools. > > It passed for me. Did I make any mistake? Nope. It would seem I did. It passed after: sudo koha-shell -c bash kohadev ~vagrant/qa-test-tools/koha-qa.pl -v 2 -c 1 But running it from vagrant directly does not exit qa -v 2 -c 1 As for comment #21, I will leave QA to pass or fail it. I'm signing off.
Created attachment 70698 [details] [review] Bug 19928: Acquisitions' CSV exports now honors syspref "delimiter" Bonus fixes: - Basketgroup: there was an unnecessary space. " RRP tax excluded" replaced by "RRP tax excluded" - Late orders: - leading spaces before every line - surrounding spaces for [% orders.size %] Other changes: - quoted "Total orders in late" to no worry about additional spaces - Late orders: additional spaces which shouldn't be a problem because all the fields are enclosed by quotes. Test plan: 1. Set syspref "delimiter" to "#'s" 2. Change the language e.g. FR-fr 3. Create a vendor with minimal info 4. Create a basket with two items 5. Go to the basket. URL should be /cgi-bin/koha/acqui/basket.pl?basketno=XXXXX 6. Close this basket 7. "Export as CSV" 8. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 9. Create a basket group with two baskets (tick the close this basket group check box) 10. Go to the "Closed" tab to see your basket group 11. "Export as CSV" 12. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 13. Go to /cgi-bin/koha/acqui/lateorders.pl 14. Tick two orders. Of the same vendor, otherwise Koha won't let you tick orders of different vendors. 15. "Export as CSV" 16. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 17. Apply this patch 18. Translation stuff 1. cd misc/translator 2. perl translate update YOUR_LANG 3. manually edit the po file, remove the fuzzy tags. And translate the string. You can just copy the original and replace some parts by gibberish. 4. perl translate install YOUR_LANG 19. Re-export the same CSVs and compare with the original version to check that - delimiter syspref is honored - the headers are translatable - there is no whitespace regression (additional newlines or spaces) Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
(In reply to Victor Grousset/tuxayo from comment #21) > (In reply to Katrin Fischer from comment #17) > > It's missing steps. > > > > - you need to update the po files after the patch was applied > > - you have to check that the string got not marked as 'fuzzy' > > - you have to reinstall the language > > > > Only then the test gives you a correct result. > > I don't get, isn't this covered in the steps 18? Sorry Viktor, it appears I didn't read to the end of the test plan...
This works as advertised, however see bug 19910 If we are going to honor the delimiter pref, I think we need to use it in displaying what format will be downloaded. Want another QA opinion here
Created attachment 70750 [details] [review] Bug 19928: Acquisitions' CSV exports now honors syspref "delimiter" Bonus fixes: - Basketgroup: there was an unnecessary space. " RRP tax excluded" replaced by "RRP tax excluded" - Late orders: - leading spaces before every line - surrounding spaces for [% orders.size %] Other changes: - quoted "Total orders in late" to no worry about additional spaces - Late orders: additional spaces which shouldn't be a problem because all the fields are enclosed by quotes. Test plan: 1. Set syspref "delimiter" to "#'s" 2. Change the language e.g. FR-fr 3. Create a vendor with minimal info 4. Create a basket with two items 5. Go to the basket. URL should be /cgi-bin/koha/acqui/basket.pl?basketno=XXXXX 6. Close this basket 7. "Export as CSV" 8. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 9. Create a basket group with two baskets (tick the close this basket group check box) 10. Go to the "Closed" tab to see your basket group 11. "Export as CSV" 12. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 13. Go to /cgi-bin/koha/acqui/lateorders.pl 14. Tick two orders. Of the same vendor, otherwise Koha won't let you tick orders of different vendors. 15. "Export as CSV" 16. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 17. Apply this patch 18. Translation stuff 1. cd misc/translator 2. perl translate update YOUR_LANG 3. manually edit the po file, remove the fuzzy tags. And translate the string. You can just copy the original and replace some parts by gibberish. 4. perl translate install YOUR_LANG 19. Re-export the same CSVs and compare with the original version to check that - delimiter syspref is honored - the headers are translatable - there is no whitespace regression (additional newlines or spaces) Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(In reply to Nick Clemens from comment #25) > This works as advertised, however see bug 19910 > > If we are going to honor the delimiter pref, I think we need to use it in > displaying what format will be downloaded. > > Want another QA opinion here Agreed, please see the changes made on bug 19910 and add the info here too. You can then switch back to Signed Off.
Created attachment 71104 [details] [review] Bug 19928: Acquisitions: show the precise kind of CSV on export buttons That is to say on the 4 "export as CSV" buttons found, now when can read "export as semicolon separated text (.csv)". It follows the "delimiter" syspref.
Created attachment 71105 [details] Screenshot of new buttons Here are how the new buttons looks. I choose values of the syspref that lead to the biggest text in the button to see it it's an issue. The strings are translatable, here is how it looks in the .po files. msgid "" "%s %s %s %s %s %s %s pound (#) separated text (.csv) %s comma separated text " "(.csv) %s slash separated text (.csv) %s semicolon separated text (.csv) %s " "backslash separated text (.csv) %s tab separated text (.csv) %s %s %s " msgid "" "%s %s %s %s %s %s %s pound (#) separated text (.csv) %s comma separated text " "(.csv) %s slash separated text (.csv) %s semicolon separated text (.csv) %s " "backslash separated text (.csv) %s tab separated text (.csv) %s %s %s " msgid "" "Export as %s %s pound (#) separated text (.csv) %s comma separated text (." "csv) %s slash separated text (.csv) %s semicolon separated text (.csv) %s " "backslash separated text (.csv) %s tab separated text (.csv) %s "
Owen, could you take a look at the screenshots please? Bit worried they might get even longer with translations.
Both patches work very nicely, I especially tested for translations. The only thing I am not happy about it the length of the buttons - I think Victor had similar thoughts on posting the screenshots. It's bad in English, but it will be even worse in translation for example to German. Maybe a more generic term would help? "Export file"? If you have a global setting, it might not be necessary to point it out everywhere, because people will know after first time they use it. I am attaching the 2 patches with my sign-off, but at the moment would only prefer the first to be pushed. Not setting to PQA quite yet, waiting for some feedback on the button description issue first.
Created attachment 71271 [details] [review] Bug 19928: Acquisitions' CSV exports now honors syspref "delimiter" Bonus fixes: - Basketgroup: there was an unnecessary space. " RRP tax excluded" replaced by "RRP tax excluded" - Late orders: - leading spaces before every line - surrounding spaces for [% orders.size %] Other changes: - quoted "Total orders in late" to no worry about additional spaces - Late orders: additional spaces which shouldn't be a problem because all the fields are enclosed by quotes. Test plan: 1. Set syspref "delimiter" to "#'s" 2. Change the language e.g. FR-fr 3. Create a vendor with minimal info 4. Create a basket with two items 5. Go to the basket. URL should be /cgi-bin/koha/acqui/basket.pl?basketno=XXXXX 6. Close this basket 7. "Export as CSV" 8. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 9. Create a basket group with two baskets (tick the close this basket group check box) 10. Go to the "Closed" tab to see your basket group 11. "Export as CSV" 12. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 13. Go to /cgi-bin/koha/acqui/lateorders.pl 14. Tick two orders. Of the same vendor, otherwise Koha won't let you tick orders of different vendors. 15. "Export as CSV" 16. You should see that the file contains commas "," This is the bug. Leave the file open for future comparison to ensure that there are no regressions. 17. Apply this patch 18. Translation stuff 1. cd misc/translator 2. perl translate update YOUR_LANG 3. manually edit the po file, remove the fuzzy tags. And translate the string. You can just copy the original and replace some parts by gibberish. 4. perl translate install YOUR_LANG 19. Re-export the same CSVs and compare with the original version to check that - delimiter syspref is honored - the headers are translatable - there is no whitespace regression (additional newlines or spaces) Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Created attachment 71272 [details] [review] Bug 19928: Acquisitions: show the precise kind of CSV on export buttons That is to say on the 4 "export as CSV" buttons found, now when can read "export as semicolon separated text (.csv)". It follows the "delimiter" syspref. Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
I think "CSV" is commonly understood to include more than just comma-separated values. Is it really necessary to be so specific? For instance, an application like phpMyAdmin offers export in "CSV" format, but it doesn't limit you to comma-separated output.
Thx for your comment, Owen! My suggestion is to keep the current descriptions then and make sure the delimiter is used properly everywhere :)
(In reply to Katrin Fischer from comment #33) > Created attachment 71272 [details] [review] [review] > Bug 19928: Acquisitions: show the precise kind of CSV on export buttons The delimiter for customized CSV profiles (tools/csv-profiles.pl) is configurable so it does not make sense to use the sysprefs to adapt the text of these buttons (at least for basket export).
(In reply to Katrin Fischer from comment #31) > The only thing I am not happy about it the length of the buttons > I think Victor had similar thoughts on posting the screenshots. Actually I would be fine with these long descriptive names. (and with short names) But I don't have much experience about UX. (To have a reliable opinion if it's better than shorter but less informative names.) However I'm sure many people won't think such long names are good. Hence the screenshots to not hide the potential issue. It seems that it worked ^^" (In reply to Owen Leonard from comment #34) > I think "CSV" is commonly understood to include more than just > comma-separated values. Is it really necessary to be so specific? For > instance, an application like phpMyAdmin offers export in "CSV" format, but > it doesn't limit you to comma-separated output. +1 it's not necessary (In reply to Katrin Fischer from comment #35) > My suggestion is to keep the current descriptions then and make sure the > delimiter is used properly everywhere :) Current as in master[1][3] or current are in the current patches[2][4] of this bug? If [2] then bug 19910 must be changed as well. [3] this is consistent with Owen's comment [4] this is consistent with you switching this bug to "Passed QA"
You got me a little confused there. Short form was: - Push only first patch, leave button descriptions alone. If this affects another bug, please change accordingly.
Pushed to master for 18.05, thanks to everybody involved!
Thanks for the clarification. Yay this one finishes just in time as can start bug 20171 :D
Created attachment 71389 [details] [review] Bug 19928: Mock delimiter pref in tests
Awesome work all, pushed to stable for 17.11.03