Bug 23696 - build_oai_sets.pl should take biblios from deletedbiblio_metadata too
Summary: build_oai_sets.pl should take biblios from deletedbiblio_metadata too
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Josef Moravec
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-09-30 09:55 UTC by Josef Moravec
Modified: 2021-12-13 21:08 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.03


Attachments
Bug 23696: take deleted biblios into account when building oai sets (1.49 KB, patch)
2019-09-30 10:00 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 23696: take deleted biblios into account when building oai sets (1.54 KB, patch)
2019-09-30 19:32 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 23696: take deleted biblios into account when building oai sets (1.63 KB, patch)
2020-07-24 07:22 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Josef Moravec 2019-09-30 09:55:12 UTC

    
Comment 1 Josef Moravec 2019-09-30 10:00:24 UTC
Created attachment 93229 [details] [review]
Bug 23696: take deleted biblios into account when building oai sets

Test plan:
0) do not apply the patch
1) set up OAI and create at least one set
2) run build_oai_sets.pl
3) remember/write down number of records added
4) delete a biblio, the is included in set
5) run buid_oai_sets.pl again
6) the set is 1 record smaller
7) apply the patch
8) run build_oai_sets.pl
9) the number of record should be the same as in 3)
Comment 2 Josef Moravec 2019-09-30 10:01:12 UTC
1
Comment 3 Michal Denar 2019-09-30 19:32:20 UTC
Created attachment 93334 [details] [review]
Bug 23696: take deleted biblios into account when building oai sets

Test plan:
0) do not apply the patch
1) set up OAI and create at least one set
2) run build_oai_sets.pl
3) remember/write down number of records added
4) delete a biblio, the is included in set
5) run buid_oai_sets.pl again
6) the set is 1 record smaller
7) apply the patch
8) run build_oai_sets.pl
9) the number of record should be the same as in 3)

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 4 Katrin Fischer 2019-10-10 06:37:56 UTC
This patch appears pretty straight-forward and it works for me, but I am having trouble with rerunning the script:

1) On running the script a second time, I get a lot of errors:
DBD::mysql::st execute failed: Duplicate entry '50-1' for key 'PRIMARY' [for Statement "
        INSERT INTO oai_sets_biblios (set_id, biblionumber)
        VALUES (?,?)
    " with ParamValues: 0='1', 1=50] at /home/vagrant/kohaclone/C4/OAI/Sets.pm line 597.
DBD::mysql::st execute failed: Duplicate entry '52-1' for key 'PRIMARY' [for Statement "
        INSERT INTO oai_sets_biblios (set_id, biblionumber)
        VALUES (?,?)
    " with ParamValues: 0='1', 1=52] at /home/vagrant/kohaclone/C4/OAI/Sets.pm line 597.

2) Also the links for Metadata and Set don't appear to work in our HTML representation of the OAI-Server:

/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=TEST

3) Even without the patch and after rerunning the script reporting 45 records (instead of the 46 it should have), I still can see a deleted record in the web output of the set.

Can you please check these things and let me know? 3 might be related to this patch (blocker), the other 2 possibly separate bugs?
Comment 5 Josef Moravec 2019-12-06 08:35:28 UTC
(In reply to Katrin Fischer from comment #4)
> This patch appears pretty straight-forward and it works for me, but I am
> having trouble with rerunning the script:
> 
> 1) On running the script a second time, I get a lot of errors:
> DBD::mysql::st execute failed: Duplicate entry '50-1' for key 'PRIMARY' [for
> Statement "
>         INSERT INTO oai_sets_biblios (set_id, biblionumber)
>         VALUES (?,?)
>     " with ParamValues: 0='1', 1=50] at
> /home/vagrant/kohaclone/C4/OAI/Sets.pm line 597.
> DBD::mysql::st execute failed: Duplicate entry '52-1' for key 'PRIMARY' [for
> Statement "
>         INSERT INTO oai_sets_biblios (set_id, biblionumber)
>         VALUES (?,?)
>     " with ParamValues: 0='1', 1=52] at
> /home/vagrant/kohaclone/C4/OAI/Sets.pm line 597.


Hi Katrin, thanks for testing!

when you are running this script more than once, you need to add -r parameter - it will clean the table before generating it again.


> 
> 2) Also the links for Metadata and Set don't appear to work in our HTML
> representation of the OAI-Server:
> 
> /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=TEST

Can you add more details to this question?

> 
> 3) Even without the patch and after rerunning the script reporting 45
> records (instead of the 46 it should have), I still can see a deleted record
> in the web output of the set.

The access to records are not limited by set, you can use OAI without set parameter in Koha. And it is OK, because set is optional by standard:
http://www.openarchives.org/OAI/openarchivesprotocol.html#Set
Comment 6 Marcel de Rooy 2020-07-24 07:22:47 UTC
Created attachment 107275 [details] [review]
Bug 23696: take deleted biblios into account when building oai sets

Test plan:
0) do not apply the patch
1) set up OAI and create at least one set
2) run build_oai_sets.pl
3) remember/write down number of records added
4) delete a biblio, the is included in set
5) run buid_oai_sets.pl again
6) the set is 1 record smaller
7) apply the patch
8) run build_oai_sets.pl
9) the number of record should be the same as in 3)

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 7 Marcel de Rooy 2020-07-24 07:23:08 UTC
Patch on itself looks good to me. Passing QA. Watch bug 20271.

Katrin's questions in comment4 apply to OAI in general.
The functionality of OAI sets in Koha looks rather poor to me.
The form where you create a set is not really intuitive. When I created a set, the rule was duplicated.
I wanted a set with biblionumber < some value. This is not possible. I tried field 001, but I can only choose == or != which is useless. And I should add a subfield which is NOT there.
Furthermore I am not convinced if we should really build a table like oai_set_biblios or do that on the fly. The latter looks better to me at first.
Comment 8 Jonathan Druart 2020-07-24 12:14:04 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 9 Lucas Gass 2020-07-31 20:10:50 UTC
small enchacement, choosing to backport to 20.05.x for 20.05.03
Comment 10 Aleisha Amohia 2020-08-03 23:39:56 UTC
enhancement, not backported to 19.11.x