Bug 24163

Summary: Add ability to define a CSV profile for late orders export
Product: Koha Reporter: Jonathan Druart <jonathan.druart>
Component: AcquisitionsAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: 1joynelson, black23, mark.gavillet, martin.renvoize
Version: unspecified   
Hardware: All   
OS: All   
Change sponsored?: Sponsored Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00
Bug Depends on: 24161    
Bug Blocks: 24162    
Attachments: Bug 24163: Allow to define CSV profile for late orders export
Bug 24163: Add the csv into the DB - as an example
Bug 24163: Allow to define CSV profile for late orders export
Bug 24163: Add the csv into the DB - as an example
Bug 24163: Allow to define CSV profile for late orders export
Bug 24163: Add the csv into the DB - as an example
Bug 24163: Allow to define CSV profile for late orders export
Bug 24163: Add the csv into the DB - as an example
Bug 24163: Allow to define CSV profile for late orders export
Bug 24163: Add the csv into the DB - as an example
Bug 24163: Allow to define CSV profile for late orders export
Bug 24163: Add the csv into the DB - as an example
Bug 24163: (QA follow-up) Fix wordings
Bug 24163: Add the new CSV profile to other installer files
Bug 24163: Specify the UTF-8 charset in the header
Bug 24163: UTF-8 encode the output
Bug 24163: Allow to define CSV profile for late orders export
Bug 24163: Add the csv into the DB - as an example
Bug 24163: (QA follow-up) Fix wordings
Bug 24163: Add the new CSV profile to other installer files
Bug 24163: UTF-8 encode the output

Description Jonathan Druart 2019-12-03 19:44:15 UTC
Add the ability to create a CSV profile for the late orders export.

Will allow the inclusion of Title, Author, Date of publication, ISBN, and quantity columns.

Multiple claim dates will also be included (from bug 24162)
Comment 1 Jonathan Druart 2020-01-09 16:47:07 UTC Comment hidden (obsolete)
Comment 2 Jonathan Druart 2020-01-09 16:47:10 UTC Comment hidden (obsolete)
Comment 3 Jonathan Druart 2020-01-09 16:49:02 UTC Comment hidden (obsolete)
Comment 4 Jonathan Druart 2020-01-09 16:49:05 UTC Comment hidden (obsolete)
Comment 5 Jonathan Druart 2020-02-11 08:54:25 UTC
Created attachment 98691 [details] [review]
Bug 24163: Allow to define CSV profile for late orders export

This new enhancement adds the ability to create a CSV profile for the late
orders export.
Prior to this, the CSV profile was hardcoded in a template and not
editable (unless you modify the .tt file of course).

The main difficulty of the request was to make the multiple claims dates
accessible from the CSV profile. So far we only accept columns from the
database. However we would like to use the Koha::Acquisition::Order
object to make things easier (it had a ->claims method to access the
different claims).
To acchieve this, we are going to accept the TT syntax for that CSV profile.
It means that only CSV profiles created for 'late_orders' will have the
capability to use the TT syntax (can be extended later of course if
needed).
The alternative was to use specific placeholders, like %claims_count%,
%claimed_date%, but it sounded more powerful and flexible to allow the
TT syntax instead.

Note that the former export (template based) still exists and is the
default option.

Test plan:
0. Apply the patches and execute the update database entry
1. Create some orders, close the basket
2. Claim some of the orders
3. Note the new button at the bottom of the table that have several
entries.
=> The first entry will generate a CSV file using the previous method
=> The second entry will generate it using the new CSV profile, note
that the number of claims as well as the different claims date are
displayed.

4. Bonus point: Edit the CSV profile (Home › Tools › CSV export
profiles) that is named "Late orders (csv profile)", or create a new
one, and modify it.
Export the late orders using this CSV profile and confirm that the
generated CSV file is the one you expect.

Sponsored-by: Institute of Technology Tralee
Comment 6 Jonathan Druart 2020-02-11 08:54:28 UTC
Created attachment 98692 [details] [review]
Bug 24163: Add the csv into the DB - as an example

Sponsored-by: Institute of Technology Tralee
Comment 7 Jonathan Druart 2020-04-01 09:18:10 UTC
Created attachment 102185 [details] [review]
Bug 24163: Allow to define CSV profile for late orders export

This new enhancement adds the ability to create a CSV profile for the late
orders export.
Prior to this, the CSV profile was hardcoded in a template and not
editable (unless you modify the .tt file of course).

The main difficulty of the request was to make the multiple claims dates
accessible from the CSV profile. So far we only accept columns from the
database. However we would like to use the Koha::Acquisition::Order
object to make things easier (it had a ->claims method to access the
different claims).
To acchieve this, we are going to accept the TT syntax for that CSV profile.
It means that only CSV profiles created for 'late_orders' will have the
capability to use the TT syntax (can be extended later of course if
needed).
The alternative was to use specific placeholders, like %claims_count%,
%claimed_date%, but it sounded more powerful and flexible to allow the
TT syntax instead.

Note that the former export (template based) still exists and is the
default option.

Test plan:
0. Apply the patches and execute the update database entry
1. Create some orders, close the basket
2. Claim some of the orders
3. Note the new button at the bottom of the table that have several
entries.
=> The first entry will generate a CSV file using the previous method
=> The second entry will generate it using the new CSV profile, note
that the number of claims as well as the different claims date are
displayed.

4. Bonus point: Edit the CSV profile (Home › Tools › CSV export
profiles) that is named "Late orders (csv profile)", or create a new
one, and modify it.
Export the late orders using this CSV profile and confirm that the
generated CSV file is the one you expect.

Sponsored-by: Institute of Technology Tralee
Comment 8 Jonathan Druart 2020-04-01 09:18:15 UTC
Created attachment 102186 [details] [review]
Bug 24163: Add the csv into the DB - as an example

Sponsored-by: Institute of Technology Tralee
Comment 9 David Nind 2020-04-02 05:08:50 UTC
Created attachment 102257 [details] [review]
Bug 24163: Allow to define CSV profile for late orders export

This new enhancement adds the ability to create a CSV profile for the late
orders export.
Prior to this, the CSV profile was hardcoded in a template and not
editable (unless you modify the .tt file of course).

The main difficulty of the request was to make the multiple claims dates
accessible from the CSV profile. So far we only accept columns from the
database. However we would like to use the Koha::Acquisition::Order
object to make things easier (it had a ->claims method to access the
different claims).
To acchieve this, we are going to accept the TT syntax for that CSV profile.
It means that only CSV profiles created for 'late_orders' will have the
capability to use the TT syntax (can be extended later of course if
needed).
The alternative was to use specific placeholders, like %claims_count%,
%claimed_date%, but it sounded more powerful and flexible to allow the
TT syntax instead.

Note that the former export (template based) still exists and is the
default option.

Test plan:
0. Apply the patches and execute the update database entry
1. Create some orders, close the basket
2. Claim some of the orders
3. Note the new button at the bottom of the table that have several
entries.
=> The first entry will generate a CSV file using the previous method
=> The second entry will generate it using the new CSV profile, note
that the number of claims as well as the different claims date are
displayed.

4. Bonus point: Edit the CSV profile (Home › Tools › CSV export
profiles) that is named "Late orders (csv profile)", or create a new
one, and modify it.
Export the late orders using this CSV profile and confirm that the
generated CSV file is the one you expect.

Sponsored-by: Institute of Technology Tralee

Signed-off-by: David Nind <david@davidnind.com>
Comment 10 David Nind 2020-04-02 05:08:54 UTC
Created attachment 102258 [details] [review]
Bug 24163: Add the csv into the DB - as an example

Sponsored-by: Institute of Technology Tralee

Signed-off-by: David Nind <david@davidnind.com>
Comment 11 PTFS Europe Sandboxes 2020-04-23 11:55:30 UTC
Created attachment 103585 [details] [review]
Bug 24163: Allow to define CSV profile for late orders export

This new enhancement adds the ability to create a CSV profile for the late
orders export.
Prior to this, the CSV profile was hardcoded in a template and not
editable (unless you modify the .tt file of course).

The main difficulty of the request was to make the multiple claims dates
accessible from the CSV profile. So far we only accept columns from the
database. However we would like to use the Koha::Acquisition::Order
object to make things easier (it had a ->claims method to access the
different claims).
To acchieve this, we are going to accept the TT syntax for that CSV profile.
It means that only CSV profiles created for 'late_orders' will have the
capability to use the TT syntax (can be extended later of course if
needed).
The alternative was to use specific placeholders, like %claims_count%,
%claimed_date%, but it sounded more powerful and flexible to allow the
TT syntax instead.

Note that the former export (template based) still exists and is the
default option.

Test plan:
0. Apply the patches and execute the update database entry
1. Create some orders, close the basket
2. Claim some of the orders
3. Note the new button at the bottom of the table that have several
entries.
=> The first entry will generate a CSV file using the previous method
=> The second entry will generate it using the new CSV profile, note
that the number of claims as well as the different claims date are
displayed.

4. Bonus point: Edit the CSV profile (Home › Tools › CSV export
profiles) that is named "Late orders (csv profile)", or create a new
one, and modify it.
Export the late orders using this CSV profile and confirm that the
generated CSV file is the one you expect.

Sponsored-by: Institute of Technology Tralee

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Angela O'Connor Desmond <angela.oconnordesmond@staff.ittralee.ie>
Comment 12 PTFS Europe Sandboxes 2020-04-23 11:55:33 UTC
Created attachment 103586 [details] [review]
Bug 24163: Add the csv into the DB - as an example

Sponsored-by: Institute of Technology Tralee

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Angela O'Connor Desmond <angela.oconnordesmond@staff.ittralee.ie>
Comment 13 Katrin Fischer 2020-05-02 12:53:09 UTC
1) Typo and rephrase

+                    <p>For late orders exports you must provide a Template Toolkit syntax that will generate the whole CSV file..</p>

I suggest a sight change here: 
For exporting late orders you must provide a profile in Template Toolkit syntax to generate the CSV file.

Late orders (csv profile) => Late orders (CSV profile)
default CSV export for late orders => Default ...


2) Other installers

Please add the profile to the other installer sample files as well.


3) Default

When you look at the pull down of the split button, it says "Default". Is the default different to the new template in significant ways? Wondering if we need to keep it or could replace it with the CSV profile. (question)


4) Diacritics / encoding

I was testing with our sample record: Typical girls? :
Author: Howe, Zoë Street. 

The default is ok, but in the CSV profile export Zoë is broken.
Comment 14 Jonathan Druart 2020-05-02 13:58:23 UTC
Created attachment 104192 [details] [review]
Bug 24163: (QA follow-up) Fix wordings
Comment 15 Jonathan Druart 2020-05-02 13:58:27 UTC
Created attachment 104193 [details] [review]
Bug 24163: Add the new CSV profile to other installer files

export_format is not populated for other languages apparently...
Comment 16 Jonathan Druart 2020-05-02 13:58:31 UTC
Created attachment 104194 [details] [review]
Bug 24163: Specify the UTF-8 charset in the header

To avoid encoding issues
Comment 17 Jonathan Druart 2020-05-02 13:59:34 UTC
(In reply to Katrin Fischer from comment #13)
> 1) Typo and rephrase
> 
> +                    <p>For late orders exports you must provide a Template
> Toolkit syntax that will generate the whole CSV file..</p>
> 
> I suggest a sight change here: 
> For exporting late orders you must provide a profile in Template Toolkit
> syntax to generate the CSV file.
> 
> Late orders (csv profile) => Late orders (CSV profile)
> default CSV export for late orders => Default ...

Fixed!

> 2) Other installers
> 
> Please add the profile to the other installer sample files as well.

Done!

> 3) Default
> 
> When you look at the pull down of the split button, it says "Default". Is
> the default different to the new template in significant ways? Wondering if
> we need to keep it or could replace it with the CSV profile. (question)

Yes it's different, the "Default" has:
ORDER DATE;ESTIMATED DELIVERY DATE;VENDOR;INFORMATION;TOTAL COST;BASKET;CLAIMS COUNT;CLAIMED DATE;INTERNAL NOTE;VENDOR NOTE;ISBN

And has an additional line at the bottom:
;;"Total orders in late";1

When the new one has:
Title,Author,Publication year,ISBN,Quantity,Number of claims

I thought it was better to keep the default one (for those who use it), but we can remove it if needed.

> 4) Diacritics / encoding
> 
> I was testing with our sample record: Typical girls? :
> Author: Howe, Zoë Street. 
> 
> The default is ok, but in the CSV profile export Zoë is broken.

Good catch, fixed!
Comment 18 Katrin Fischer 2020-05-03 20:08:59 UTC
Hi Jonathan,

can you please retest? I am still seeing the same issue with diacritics and the new CSV profiles. Checked twice I have the new patches, restart_all and the Default CSV is working correct.

When I download the CSV and change the encoding form UTF-8 to Latin 3 (ISO 8859-3 ) in the import screen from LibreOffice, Zoë will display correctly.
Comment 19 Jonathan Druart 2020-05-04 09:04:02 UTC
Created attachment 104252 [details] [review]
Bug 24163: UTF-8 encode the output

Specify the UTF-8 charset in the header and Encode the output to avoid encoding issues
Comment 20 Jonathan Druart 2020-05-04 09:04:53 UTC
Sorry, just found local changes I had not embedded in the last patch!
Comment 21 Katrin Fischer 2020-05-05 06:40:14 UTC
Created attachment 104337 [details] [review]
Bug 24163: Allow to define CSV profile for late orders export

This new enhancement adds the ability to create a CSV profile for the late
orders export.
Prior to this, the CSV profile was hardcoded in a template and not
editable (unless you modify the .tt file of course).

The main difficulty of the request was to make the multiple claims dates
accessible from the CSV profile. So far we only accept columns from the
database. However we would like to use the Koha::Acquisition::Order
object to make things easier (it had a ->claims method to access the
different claims).
To acchieve this, we are going to accept the TT syntax for that CSV profile.
It means that only CSV profiles created for 'late_orders' will have the
capability to use the TT syntax (can be extended later of course if
needed).
The alternative was to use specific placeholders, like %claims_count%,
%claimed_date%, but it sounded more powerful and flexible to allow the
TT syntax instead.

Note that the former export (template based) still exists and is the
default option.

Test plan:
0. Apply the patches and execute the update database entry
1. Create some orders, close the basket
2. Claim some of the orders
3. Note the new button at the bottom of the table that have several
entries.
=> The first entry will generate a CSV file using the previous method
=> The second entry will generate it using the new CSV profile, note
that the number of claims as well as the different claims date are
displayed.

4. Bonus point: Edit the CSV profile (Home › Tools › CSV export
profiles) that is named "Late orders (csv profile)", or create a new
one, and modify it.
Export the late orders using this CSV profile and confirm that the
generated CSV file is the one you expect.

Sponsored-by: Institute of Technology Tralee

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Angela O'Connor Desmond <angela.oconnordesmond@staff.ittralee.ie>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 22 Katrin Fischer 2020-05-05 06:40:17 UTC
Created attachment 104338 [details] [review]
Bug 24163: Add the csv into the DB - as an example

Sponsored-by: Institute of Technology Tralee

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Angela O'Connor Desmond <angela.oconnordesmond@staff.ittralee.ie>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 23 Katrin Fischer 2020-05-05 06:40:21 UTC
Created attachment 104339 [details] [review]
Bug 24163: (QA follow-up) Fix wordings

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 24 Katrin Fischer 2020-05-05 06:40:25 UTC
Created attachment 104340 [details] [review]
Bug 24163: Add the new CSV profile to other installer files

export_format is not populated for other languages apparently...

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 25 Katrin Fischer 2020-05-05 06:40:29 UTC
Created attachment 104341 [details] [review]
Bug 24163: UTF-8 encode the output

Specify the UTF-8 charset in the header and Encode the output to avoid encoding issues

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 26 Martin Renvoize 2020-05-05 10:11:33 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 27 Joy Nelson 2020-05-11 21:20:35 UTC
not backported to 19.11.x