Bug 34645 - Add missing fields to MarcItemFieldsToOrder system preference
Summary: Add missing fields to MarcItemFieldsToOrder system preference
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Acquisitions (show other bugs)
Version: Main
Hardware: All All
: P5 - low major (vote)
Assignee: Kyle M Hall
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 34644
  Show dependency treegraph
 
Reported: 2023-08-29 17:29 UTC by Nick Clemens (kidclamp)
Modified: 2023-12-16 13:06 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.11.00,23.05.05,22.11.11


Attachments
Bug 34645: Add support for discount, sort1 and sort2 to MarcItemFieldsToOrder (2.96 KB, patch)
2023-08-29 17:42 UTC, Kyle M Hall
Details | Diff | Splinter Review
Sample file with order lines (6.28 KB, application/xml)
2023-09-07 17:26 UTC, Nick Clemens (kidclamp)
Details
Bug 34645: Add support for discount, sort1 and sort2 to MarcItemFieldsToOrder (3.01 KB, patch)
2023-09-12 17:21 UTC, ByWater Sandboxes
Details | Diff | Splinter Review
Bug 34645: always use discount, sort1, and sort2 from MarcFieldsToOrder (2.51 KB, patch)
2023-09-18 16:35 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 34645: (follow-up) save sort1 and sort2 in order (993 bytes, patch)
2023-09-25 13:12 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 34645: always use discount, sort1, and sort2 from MarcFieldsToOrder (2.56 KB, patch)
2023-09-25 22:04 UTC, David Nind
Details | Diff | Splinter Review
Bug 34645: (follow-up) save sort1 and sort2 in order (1.02 KB, patch)
2023-09-25 22:04 UTC, David Nind
Details | Diff | Splinter Review
Bug 34645: always use discount, sort1, and sort2 from MarcFieldsToOrder (2.63 KB, patch)
2023-09-26 10:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 34645: (follow-up) save sort1 and sort2 in order (1.09 KB, patch)
2023-09-26 10:26 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2023-08-29 17:29:02 UTC
See bug 34644 - MarcItemFieldsToOrder supersedes MarcFieldsToOrder, so the fields discount, sort1, and sort2 won't be populated if using the latter - We should add these fields in to allow the orders to be created correctly
Comment 1 Kyle M Hall 2023-08-29 17:42:46 UTC
Created attachment 154912 [details] [review]
Bug 34645: Add support for discount, sort1 and sort2 to MarcItemFieldsToOrder

MarcItemFieldsToOrder supersedes MarcFieldsToOrder, so the fields discount, sort1, and sort2 won't be populated if using the latter - We should add these fields in to allow the orders to be created correctly

Test Plan:
1) Set up MarcItemFields to order
2) Map the new keys discount, sor1 and sort2 to fields in the order
   record
3) Import an order record into a basket
4) Note those fields were imported!
Comment 2 David Cook 2023-09-06 01:07:46 UTC
(In reply to Kyle M Hall from comment #1)
> Test Plan:
> 1) Set up MarcItemFields to order
> 2) Map the new keys discount, sor1 and sort2 to fields in the order
>    record
> 3) Import an order record into a basket
> 4) Note those fields were imported!

I don't understand any of these steps πŸ˜….

Could we get a more detailed test plan?
Comment 3 David Nind 2023-09-06 21:19:19 UTC
(In reply to David Cook from comment #2)
> (In reply to Kyle M Hall from comment #1)
> 
> I don't understand any of these steps πŸ˜….
> 
> Could we get a more detailed test plan?

This article may help - Automatically Generate Item Information for Order Files https://bywatersolutions.com/education/automatically-generate-item-information-for-order-files

If it was possible to provide a sample for importing, I'd be happy to test.

Otherwise, I think it requires setting up some info on a vendor website (Baker & Taylor, Brodart, Ingram, etc) to identify the MARC item fields with the order information.
Comment 4 Nick Clemens (kidclamp) 2023-09-07 17:25:30 UTC
Setup -- Set systempreferences below

MarcFieldsToOrder:
price: 949$g
quantity: 949$k
budget_code: 949$l
discount: 949$m
sort1: 949$n
sort2: 949$q

MarcItemFieldsToOrder:
homebranch: 949$a
holdingbranch: 949$b
itype: 949$y
nonpublic_note: 949$x
public_note: 949$z
loc: 949$c
ccode: 949$8
notforloan: 949$7
uri: 949$u
copyno: 949$t
price: 949$g
replacementprice: 949$v
itemcallnumber: 949$o
quantity: 949$k
budget_code: 949$l


Stage the attached bib-303.marcxml file
Add to basket from the staged file
Note that discount and sort1 and sort2 are not populated per the Marc
Apply patch, restart all
Stage and add to basket again
Confirm discount and sort1 and sort2 are populated correctly
Comment 5 Nick Clemens (kidclamp) 2023-09-07 17:26:22 UTC
Created attachment 155338 [details]
Sample file with order lines

This is a pretty standard plan and file for testing, it could (should) be added to the wiki for future reference
Comment 6 ByWater Sandboxes 2023-09-12 17:21:23 UTC
Created attachment 155564 [details] [review]
Bug 34645: Add support for discount, sort1 and sort2 to MarcItemFieldsToOrder

MarcItemFieldsToOrder supersedes MarcFieldsToOrder, so the fields discount, sort1, and sort2 won't be populated if using the latter - We should add these fields in to allow the orders to be created correctly

Test Plan:
1) Set up MarcItemFields to order
2) Map the new keys discount, sor1 and sort2 to fields in the order
   record
3) Import an order record into a basket
4) Note those fields were imported!

Signed-off-by: Esther <esther@bywatersolutions.com>
Comment 7 Esther Melander 2023-09-12 17:28:28 UTC
I ran through the test plan and it did appear to work. However the bibliographic record in the example import file was already in the system. I had to delete the existing bib and items records from the system before adding the order from the staged file to the basket so that everything came through as new records.

If the order is placed from the staged file with the bibliographic record already in the system previously, you are redirected to treating each item record individually. In this case, the discounted price does not seem to come through. I'm not 100% sure on the correct workflow for this scenario. Only mentioning because some orders may actually be reorders for existing records.
Comment 8 Marcel de Rooy 2023-09-15 06:25:46 UTC
(In reply to Esther Melander from comment #7)
> I ran through the test plan and it did appear to work. However the
> bibliographic record in the example import file was already in the system. I
> had to delete the existing bib and items records from the system before
> adding the order from the staged file to the basket so that everything came
> through as new records.
> 
> If the order is placed from the staged file with the bibliographic record
> already in the system previously, you are redirected to treating each item
> record individually. In this case, the discounted price does not seem to
> come through. I'm not 100% sure on the correct workflow for this scenario.
> Only mentioning because some orders may actually be reorders for existing
> records.

Does this still need attention, Kyle?

Changing status for feedback.
Comment 9 Nick Clemens (kidclamp) 2023-09-15 11:03:32 UTC
(In reply to Esther Melander from comment #7)
> I ran through the test plan and it did appear to work. However the
> bibliographic record in the example import file was already in the system. I
> had to delete the existing bib and items records from the system before
> adding the order from the staged file to the basket so that everything came
> through as new records.
> 
> If the order is placed from the staged file with the bibliographic record
> already in the system previously, you are redirected to treating each item
> record individually. In this case, the discounted price does not seem to
> come through. I'm not 100% sure on the correct workflow for this scenario.
> Only mentioning because some orders may actually be reorders for existing
> records.

The order process has two places you can set matching - during staging, or when adding orders to the cart. Generally I would say matching should be set during staging and not during adding to basket, or you hit this problem.

This is an existing workflow issue, outside of this bug
Comment 10 Katrin Fischer 2023-09-17 14:09:40 UTC
I've made sure the patch was applied, restart_all, but the statistic fields and discount don't get filled for me. I tried several times, with 'Order from new file' and 'Order from staged file'. 

I can tell the 9xx fields are still present in the staged file.
Comment 11 Nick Clemens (kidclamp) 2023-09-18 16:35:38 UTC
Created attachment 155852 [details] [review]
Bug 34645: always use discount, sort1, and sort2 from MarcFieldsToOrder

These values are stored at the order level, not per item. This patch simply sets the order values from the values retrieved from MarcFieldsToOrder even inf MarcItemFields to order is being used for other fields

To test:
Setup -- Set systempreferences below

MarcFieldsToOrder:
price: 949$g
quantity: 949$k
budget_code: 949$l
discount: 949$m
sort1: 949$n
sort2: 949$q

MarcItemFieldsToOrder:
homebranch: 949$a
holdingbranch: 949$b
itype: 949$y
nonpublic_note: 949$x
public_note: 949$z
loc: 949$c
ccode: 949$8
notforloan: 949$7
uri: 949$u
copyno: 949$t
price: 949$g
replacementprice: 949$v
itemcallnumber: 949$o
quantity: 949$k
budget_code: 949$l

Stage the attached bib-303.marcxml file
Add to basket from the staged file
Note that discount and sort1 and sort2 are not populated per the Marc
Apply patch, restart all
Stage and add to basket again
Confirm discount and sort1 and sort2 are populated correctly
Comment 12 Katrin Fischer 2023-09-18 19:40:16 UTC
Hi Nick, so it wasn't just me? Did you figure out what went wrong?
Comment 13 David Nind 2023-09-18 20:29:24 UTC
I had a go at testing (using KTD):

1. Deleted record 303, including items.

2. Information is now included when adding to the basket from the staged record (discount and sort1 and sort2) + the funds etc for the item records.

3. However, 8 item records are shown, instead of the 4 that are in the file. Item record 1 and 2 are duplicated (from libraries and funds), and so on.

3. Steps:
   3.1 Deleted items and then the record for Pawn structure chess
   3.2 Staged the file provided - didn't change any of the settings
   3.3 Added to basket already setup in KTD - from a staged file
   3.4 Ticked item 'Pawn structure chess'
       - under match heading shows quantity of 8 and correct discount + Sort1 and Sort in Statistics 1 and 2
      - on the right-hand side, it lists Item record 1 to 8

I'm sure it is something I'm doing wrong!
Comment 14 Katrin Fischer 2023-09-18 20:32:41 UTC
(In reply to David Nind from comment #13)

> I'm sure it is something I'm doing wrong!

I am not so sure about this!
Comment 15 Nick Clemens (kidclamp) 2023-09-18 21:55:53 UTC
(In reply to Katrin Fischer from comment #14)
> (In reply to David Nind from comment #13)
> 
> > I'm sure it is something I'm doing wrong!
> 
> I am not so sure about this!

We expect 8 items - each order line has a quantity of 2

If you complete adding them you get three orders, 2 for 2 and one for 4 - because we group the items by budget I believe
Comment 16 David Nind 2023-09-18 22:30:57 UTC
(In reply to Nick Clemens from comment #15)
> 
> We expect 8 items - each order line has a quantity of 2
> 
> If you complete adding them you get three orders, 2 for 2 and one for 4 -
> because we group the items by budget I believe

Thanks Nick! That explains it....(and makes sense now).
Comment 17 David Nind 2023-09-19 01:10:49 UTC
Now I'm getting "Some required item subfields are not set" when I go to save.
Comment 18 Nick Clemens (kidclamp) 2023-09-19 12:11:06 UTC
(In reply to David Nind from comment #17)
> Now I'm getting "Some required item subfields are not set" when I go to save.

That is an preexisting bug, you can workaround by setting the item type on the second tab

I filed the issue here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34827
Comment 19 David Nind 2023-09-20 21:56:18 UTC
Thanks Nick!

I have had another go at testing, and it works now (yeah!). 

I have a couple of questions though:

1. I can successfully test by deleting the existing items and record. Should this work where an existing record with items exists? I tried leaving the existing record and items in place, matched on 999$c. It matched the record, but didn't show the 8 new items. I'm assuming how this works is determined by what is in the file, and that there is no change in behavour here.

2. I found the statistics codes under the new record in 949$n and 949$q - is that the correct place to look for these? I couldn't see them anywhere else.

3. The new record created uses the default framework. I'm assuming that this is what is expected.

Happy to sign off otherwise.

And apologies for the questions, I know the basics for acquisitions, but not much more....

David


Testing notes (so far):

1. Deleted record 303 (Pawn structure chess), including items.

2. Information is now included when adding to the basket from the staged record (discount and sort1 and sort2) + the funds etc., for the item records.

3. 8 item records are shown, with fund details, prices, etc., from the imported file.

4. Steps:
   4.1 Deleted items and then the record for Pawn structure chess.
   4.2 Staged the file provided - didn't change any of the settings.
   4.3 Added to basket already setup in KTD - from a staged file option.
   4.4 Ticked item 'Pawn structure chess'
       - under the 'match heading' it shows quantity of 8 and correct discount + Sort1 and Sort in Statistics 1 and 2
      - on the right-hand side, it lists Item record 1 to 8
   4.5 Changed to the 'Item information' tab, to select the Koha item type (work around for bug 34827).
   4.6 Changed back to the 'Select to import' tab.
   4.7 Clicked 'Save' at the bottom of the page.
   4.8 result: Items are added to the basket:
       - discount correctly shown on the order lines
       - statistics shown in the record (open title in a new tab, edit record, statistics codes shown under 949$n and 949$q)
Comment 20 Nick Clemens (kidclamp) 2023-09-25 13:12:09 UTC
(In reply to David Nind from comment #19)
> 1. I can successfully test by deleting the existing items and record. Should
> this work where an existing record with items exists? I tried leaving the
> existing record and items in place, matched on 999$c. It matched the record,
> but didn't show the 8 new items. I'm assuming how this works is determined
> by what is in the file, and that there is no change in behavour here.

This works for me - the matching rule should be set when staging, and then using the same during adding to basket - I get 12 items on the record. I would make sure the basket/vendor is set to create items on ordering
> 
> 2. I found the statistics codes under the new record in 949$n and 949$q - is
> that the correct place to look for these? I couldn't see them anywhere else.

Attaching a followup - while these are set as variables when using MarcItemFieldsToOrder it looks like they were never saved into the order

> 
> 3. The new record created uses the default framework. I'm assuming that this
> is what is expected.

Yes, at the moment there is no choice, that's a future/separate enhancement
 
> Happy to sign off otherwise.
> And apologies for the questions, I know the basics for acquisitions, but not
> much more....

Questions are good! And thank you for testing (many times)
Comment 21 Nick Clemens (kidclamp) 2023-09-25 13:12:57 UTC
Created attachment 156156 [details] [review]
Bug 34645: (follow-up) save sort1 and sort2 in order
Comment 22 David Nind 2023-09-25 22:03:48 UTC
Thanks Nick!

I was able to successfully run through the test plan with and without deleting the existing record. These are the settings I used when staging the file:
- Record matching rule: KohaBiblio (999$c)
- Action if matching record found: Ignore incoming record (its items may still be processed)
- Check for embedded item record data? Yes + How to process items: Always add items

I can now see the statistics columns in the basket (used Columns to toggle on the display of the Statistics 1 and Statistics 2 columns).

David


Latest testing notes (using KTD):

1. Works successfully without needing to delete the existing record 303 (Pawn structure chess). Also works successfully if the existing record is deleted.

2. With existing record and items, use these settings when staging the file:
   - Record matching rule: KohaBiblio (999$c)
   - Action if matching record found: Ignore incoming record (its items may still be processed)
   - Check for embedded item record data? Yes + How to process items: Always add items

3. Steps:
   3.1 Remember to set up the MarcFieldsToOrder and MarcItemFieldsToOrder system preferences as per the test plan!
   3.2 Stage the file provided - use these settings (as the record and four items already exist):
      - Record matching rule: KohaBiblio (999$c)
      - Action if matching record found: Ignore incoming record (its items may still be processed)
      - Check for embedded item record data? Yes + How to process items: Always add items   
   3.3 Create a new basket for the vendor, for 'Create items when' select "placing an order" .
   3.4 Add to the new basket using the 'From a staged file' option.
   3.5 Tick item 'Pawn structure chess'
       - under the 'match heading' it shows quantity of 8 and correct discount + Sort1 and Sort2 in Statistics 1 and 2
      - on the right-hand side, it lists 8 item records with the correct budgets, prices, etc., from the staged file.
   3.6 Change to the 'Item information' tab, select a Koha item type (work around for bug 34827).
   3.7 Change back to the 'Select to import' tab.
   3.8 Click 'Save' at the bottom of the page.
   3.9 Result ==> Items are added to the basket:
       - discount correctly shown on the order lines
       - use the Columns setting to display the Statistics 1 and Statistics 2 columns.
Comment 23 David Nind 2023-09-25 22:04:36 UTC
Created attachment 156194 [details] [review]
Bug 34645: always use discount, sort1, and sort2 from MarcFieldsToOrder

These values are stored at the order level, not per item. This patch simply sets the order values from the values retrieved from MarcFieldsToOrder even inf MarcItemFields to order is being used for other fields

To test:
Setup -- Set systempreferences below

MarcFieldsToOrder:
price: 949$g
quantity: 949$k
budget_code: 949$l
discount: 949$m
sort1: 949$n
sort2: 949$q

MarcItemFieldsToOrder:
homebranch: 949$a
holdingbranch: 949$b
itype: 949$y
nonpublic_note: 949$x
public_note: 949$z
loc: 949$c
ccode: 949$8
notforloan: 949$7
uri: 949$u
copyno: 949$t
price: 949$g
replacementprice: 949$v
itemcallnumber: 949$o
quantity: 949$k
budget_code: 949$l

Stage the attached bib-303.marcxml file
Add to basket from the staged file
Note that discount and sort1 and sort2 are not populated per the Marc
Apply patch, restart all
Stage and add to basket again
Confirm discount and sort1 and sort2 are populated correctly

Signed-off-by: David Nind <david@davidnind.com>
Comment 24 David Nind 2023-09-25 22:04:39 UTC
Created attachment 156195 [details] [review]
Bug 34645: (follow-up) save sort1 and sort2 in order

Signed-off-by: David Nind <david@davidnind.com>
Comment 25 Jonathan Druart 2023-09-26 10:26:32 UTC
Created attachment 156211 [details] [review]
Bug 34645: always use discount, sort1, and sort2 from MarcFieldsToOrder

These values are stored at the order level, not per item. This patch simply sets the order values from the values retrieved from MarcFieldsToOrder even inf MarcItemFields to order is being used for other fields

To test:
Setup -- Set systempreferences below

MarcFieldsToOrder:
price: 949$g
quantity: 949$k
budget_code: 949$l
discount: 949$m
sort1: 949$n
sort2: 949$q

MarcItemFieldsToOrder:
homebranch: 949$a
holdingbranch: 949$b
itype: 949$y
nonpublic_note: 949$x
public_note: 949$z
loc: 949$c
ccode: 949$8
notforloan: 949$7
uri: 949$u
copyno: 949$t
price: 949$g
replacementprice: 949$v
itemcallnumber: 949$o
quantity: 949$k
budget_code: 949$l

Stage the attached bib-303.marcxml file
Add to basket from the staged file
Note that discount and sort1 and sort2 are not populated per the Marc
Apply patch, restart all
Stage and add to basket again
Confirm discount and sort1 and sort2 are populated correctly

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 26 Jonathan Druart 2023-09-26 10:26:36 UTC
Created attachment 156212 [details] [review]
Bug 34645: (follow-up) save sort1 and sort2 in order

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 27 TomΓ‘s Cohen Arazi 2023-10-04 13:14:54 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 28 Fridolin Somers 2023-10-05 19:43:30 UTC
Pushed to 23.05.x for 23.05.05
Comment 29 Matt Blenkinsop 2023-10-17 16:56:50 UTC
Nice work everyone!

Pushed to oldstable for 22.11.x