Created attachment 104721 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Created attachment 104983 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Created attachment 104985 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Created attachment 104986 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Created attachment 104987 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Also adds syspref to optionally choose to embed item information in record when adding/editing bibliographic and item records.

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [ITEM EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ITEM ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [ITEM DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ITEM ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [ITEM EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [ITEM EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [ITEM DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Created attachment 109560 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Also adds syspref to optionally choose to embed item information in record when adding/editing bibliographic and item records.

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [ITEM EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ITEM ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [ITEM DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ITEM ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [ITEM EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [ITEM EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [ITEM DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Rebased against master

Created attachment 110435 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Also adds syspref to optionally choose to embed item information in record when adding/editing bibliographic and item records.

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [ITEM EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ITEM ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [ITEM DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ITEM ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [ITEM EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [ITEM EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [ITEM DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Signed-off-by: David Nind <david@davidnind.com>

Hi Alex, can we please have a unit test highlighting the change?

Created attachment 110645 [details] [review]
Bug 25460 : Add unit test for change to C4::OAI::Sets->UpdateOAISetsBiblio()

Test plan:

1. Run unit test:
sudo koha-shell <instancename>
prove t/db_dependent/OAI/Sets.t

2. All tests should pass

Sponsored-by: Catalyst IT

Thanks David and Katrin. 

Attached is a unit test patch amending t/db_dependent/OAI/Sets.t to cover changes to C4/OAI/Sets.t

Could you please re-test?

Many thanks,
Alex

Hi Alex.

I retested, but the tests failed for me:

root@kohadevbox:koha(bz25460)$ prove t/db_dependent/OAI/Sets.t
t/db_dependent/OAI/Sets.t .. 1/153 C4::OAI::Sets::AddOAISetsBiblios(): DBI Exception: DBD::mysql::st execute failed: Duplicate entry '444-7' for key 'PRIMARY' [for Statement "
        INSERT INTO oai_sets_biblios (set_id, biblionumber)
        VALUES (?,?)
    " with ParamValues: 0='7', 1=444] at t/db_dependent/OAI/Sets.t line 370
# Looks like your test exited with 255 just after 100.
t/db_dependent/OAI/Sets.t .. Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 53/153 subtests 

Test Summary Report
-------------------
t/db_dependent/OAI/Sets.t (Wstat: 65280 Tests: 100 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 153 tests but ran 100.
Files=1, Tests=100,  2 wallclock secs ( 0.03 usr  0.01 sys +  1.50 cusr  0.15 csys =  1.69 CPU)
Result: FAIL

Note:
1. If I apply the patch on a clean setup and don't enable the OAI-PMH system preferences the tests pass.
2. If I apply the patch on a clean setup and enable the OAI-PMH system preferences the test fail.

Also, a really minor thing - there is an unnecessary " at the end of the description note for the new system preference.

Hi David, 

Ohh, thanks for noting that! I'll look into why that's happening and hopefully update the patches on here today.

Thanks,
Alex

Just noting I've found this problem appears on master too. When you have the  OAI-PMH:AutoUpdateSets  syspref disabled the t/db_dependent/OAI/Sets.t tests pass, and when that syspref is enabled they fail. Thanks for noting it David! 

I'll work on a fix for this.

Created attachment 111051 [details] [review]
Bug 25460 : Add unit test for change to C4::OAI::Sets->UpdateOAISetsBiblio()

Test plan:

1. Run unit test:
sudo koha-shell <instancename>
prove t/db_dependent/OAI/Sets.t

2. All tests should pass

Sponsored-by: Catalyst IT

Hi David, 

I've written a fix for the failing t/db_dependent/OAI/Sets.t unit tests when the OAI-PMH:AutoUpdateSets syspref is enabled. 

As noted in comment #14 this unit test is failing on master when that syspref is enabled, so as this problem is separate from this enhancement I've put the fix on a separate bug report bug 26589

Could you please re-test this patchset after applying 26589 first?

Many thanks,
Alex

Created attachment 111085 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Also adds syspref to optionally choose to embed item information in record when adding/editing bibliographic and item records.

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [ITEM EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ITEM ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [ITEM DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ITEM ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [ITEM EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [ITEM EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [ITEM DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 111086 [details] [review]
Bug 25460 : Add unit test for change to C4::OAI::Sets->UpdateOAISetsBiblio()

Test plan:

1. Run unit test:
sudo koha-shell <instancename>
prove t/db_dependent/OAI/Sets.t

2. All tests should pass

Sponsored-by: Catalyst IT

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 111644 [details] [review]
Bug 25460: Update OAI sets when adding/editing/deleting item records

Also adds syspref to optionally choose to embed item information in record when adding/editing bibliographic and item records.

Test plan:
1. Enable OAI-PMH and OAI-PMH:AutoUpdateSets sysprefs
2. Create a biblio record with 1 item with CCODE of NFIC
3. In Administration > OAI Sets configuration create a set and define
the mapping:
Field: 952, Subfield: 8, Operator: is equal to,  Value: FIC

4. Run the build_oai_sets.pl script:
 cd misc/migration_tools
 sudo koha-shell <instancename>
 ./build_oai_sets.pl -i

5. Check what items are in the OAI set:
sudo koha-mysql <instancename>
select * from oai_sets_biblios;

6. Observe your bib from step 2 is not in the set
7. [ITEM EDIT] Edit the item to have CCODE of FIC repeat step 5 and observe biblio is still not in OAI set
8. [ITEM ADD] Add a new item with CCODE of FIC and repeat step 5 and observe biblio
still not in OAI set
9. [ITEM DELETE] Delete one of your 2 items and repeat step 5 and observe biblio still
not in OAI set

10. Apply patch, update database and restart plack (instruction below on
the latter 2 actions):

cd installer/data/mysql
sudo koha-shell <instancename>
./updatedatabase.pl
sudo koha-plack --restart <instancename>

11. Enable new syspref OAI-PMH:AutoUpdateSetsEmbedItemData
12. [ITEM ADD] Repeat step 8 and observe biblio is now in the OAI set
13. [ITEM EDIT] Edit both items to have a CCODE of NFIC and repeat step 5 and
observe bib now not in set
14. [ITEM EDIT] Edit one of the items to have CCODE of FIC and repeat step 5
and observe bib now in set
15. [ITEM DELETE] Delete all items off bib and repeat step 5 and observe bib
no longer in set

Sponsored-By: Catalyst IT

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 111645 [details] [review]
Bug 25460: Add unit test for change to C4::OAI::Sets->UpdateOAISetsBiblio()

Test plan:

1. Run unit test:
sudo koha-shell <instancename>
prove t/db_dependent/OAI/Sets.t

2. All tests should pass

Sponsored-by: Catalyst IT

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 111646 [details] [review]
Bug 25460: (QA follow-up) Readd ending punctuation to sys pref description

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Thank you David and Katrin!

I wonder if the bug description might be a little misleading. 

This makes it possible to use 952 in your OAI set mappings and it makes the updates of the OAI sets happen nicely when items are changed/added/updated.

But what it not does is embedding item information in the output formats - you still have to set the conf file and flag for marcxml in order to make that happen. So I am changing the title slightly, let me know if you disagree.

Created attachment 111647 [details] [review]
Bug 25460: (QA follow-up) Remove stray quote from system preference descrpition

Hi Katrin, 

Thank you for clarifying that. I agree with your changes. 

This patchset ensures OAI sets are altered when items are added/altered (if 952 subfields are defined in the OAI set mappings). But as you say, it doesn't mean item data will automatically be embedded in the OAI sets output.

Thanks,
Alex

Created attachment 111694 [details] [review]
Bug 25460: Add unit test for change to C4::OAI::Sets->UpdateOAISetsBiblio()

Test plan:

1. Run unit test:
sudo koha-shell <instancename>
prove t/db_dependent/OAI/Sets.t

2. All tests should pass

Sponsored-by: Catalyst IT

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

JD amended patch: Put tests inside a subtest, and perltidy

Created attachment 111695 [details] [review]
Bug 25460: (QA follow-up) Remove stray quote from system preference description

Pushed to master for 20.11, thanks to everybody involved!

Created attachment 111715 [details] [review]
Bug 25460: Remove swp file

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

enhancement will not be backported to 20.05.x