Bug 6440 - Koha's OAI-PMH does not support sets
Summary: Koha's OAI-PMH does not support sets
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Z39.50 / SRU / OpenSearch Servers (show other bugs)
Version: 3.8
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Julian Maurice
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-06-01 11:35 UTC by Magnus Enger
Modified: 2012-10-25 23:09 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 6440: Implement OAI-PMH Sets (51.49 KB, patch)
2012-01-23 16:05 UTC, Julian Maurice
Details | Diff | Splinter Review
New patch (63.36 KB, patch)
2012-02-16 13:50 UTC, Julian Maurice
Details | Diff | Splinter Review
Rebased patch (63.34 KB, patch)
2012-02-24 09:22 UTC, Julian Maurice
Details | Diff | Splinter Review
Rebase patch (63.34 KB, patch)
2012-03-01 07:59 UTC, Julian Maurice
Details | Diff | Splinter Review
Original patch rebased on master (63.25 KB, patch)
2012-03-08 10:16 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 6440: Follow-up: Implement OAI-PMH Sets (4.14 KB, patch)
2012-03-08 10:17 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets (63.99 KB, patch)
2012-03-08 11:04 UTC, Magnus Enger
Details | Diff | Splinter Review
Bug 6440: [SIGNED-OFF] Follow-up: Implement OAI-PMH Sets (4.53 KB, patch)
2012-03-08 11:04 UTC, Magnus Enger
Details | Diff | Splinter Review
Implement OAI-PMH Sets (63.38 KB, patch)
2012-03-19 15:46 UTC, Julian Maurice
Details | Diff | Splinter Review
Follow-up: Implement OAI-PMH Sets (4.14 KB, patch)
2012-03-19 15:46 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Enger 2011-06-01 11:35:08 UTC
Currently a request like /cgi-bin/koha/oai.pl?verb=ListSets results in this response: 

<OAI-PMH xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2011-06-01T12:32:26Z</responseDate><request verb="ListSets">http://sksk.bibkat.no/cgi-bin/koha/oai.pl</request><error code="noSetHierarchy">Koha repository doesn't have sets</error></OAI-PMH>

A starting point might be to define sets based on libraries/branches and/or item types. Taking it a step further one could imagine adding a config file that would let sysadmins define more sets with some kind of queries. 

This lack of sets seems to be a stumbling block when trying to integrate Koha with e.g. XC (http://www.extensiblecatalog.org/), which expects to get a list of sets from it's OAI-PMH sources.
Comment 1 Frédéric Demians 2011-10-21 11:23:57 UTC
> A starting point might be to define sets based on libraries/branches
> and/or item types. Taking it a step further one could imagine adding a
> config file that would let sysadmins define more sets with some kind
> of queries. 

Koha OAI server requests directly MySQL database in order to get all
biblio records biblionumbers. So for now, without changing Koha OAI
engine, we would have to limit queries to MySQL queries. We could
imagine something like:

  ALL: SELECT biblionumber FROM biblio;
  PER: SELECT biblionumber FROM biblioitems WHERE itemtype='PER';

> This lack of sets seems to be a stumbling block when trying to
> integrate Koha with e.g. XC (http://www.extensiblecatalog.org/), which
> expects to get a list of sets from it's OAI-PMH sources.

We could very easily define a ALL set returning the whole catalog as
it's already done.

Koha OAI Server doesn't have also incremental harvesting capabilities.
Comment 2 Frédéric Demians 2011-10-21 11:31:26 UTC
> Koha OAI Server doesn't have also incremental harvesting capabilities.

Sorry, I'm wrong here. Biblio records are sent depending on biblio.timestamp
field.
Comment 3 Magnus Enger 2011-10-21 12:05:26 UTC
One set for "ALL" would be a good start, I think!
Comment 4 Julian Maurice 2012-01-23 14:24:01 UTC
BibLibre have worked on OAI-PMH Sets implementation. I'm currently working on rebasing the patch.
Comment 5 Julian Maurice 2012-01-23 16:05:46 UTC Comment hidden (obsolete)
Comment 6 Frédéric Demians 2012-01-23 17:52:13 UTC
As it is implemented, does it mean that OAI sets have to be rebuild
periodically? Wouldn't it be possible that adding/editing a biblio record
via Koha data entry form update the OAI sets? Do you have a functional
specification of sets? It would help a lot to understand how it works
and help also the documentation redactor.
Comment 7 Julian Maurice 2012-01-26 21:33:43 UTC
(In reply to comment #6)
> As it is implemented, does it mean that OAI sets have to be rebuild
> periodically?
Actually, yes.

> Wouldn't it be possible that adding/editing a biblio record
> via Koha data entry form update the OAI sets?
I don't remember why it was done like this (maybe performance concerns), but it would be trivial to add this feature.
I think adding a syspref for this would be useful, since it can increase time of addition/modification with a large number of sets (maybe someone has a name for this syspref :) ?)

> Do you have a functional
> specification of sets? It would help a lot to understand how it works
> and help also the documentation redactor.
Hmm sorry but no :-/

I can add that a hierarchy of set can be defined using the set specs (eg. 'a:b' define the set 'a:b' which is a subset of set 'a', it means that every element of 'a:b' is implicitly in 'a')
Comment 8 Magnus Enger 2012-02-14 15:06:40 UTC
1) updatedatabase.pl

I have not seen this construct in updatadatabase.pl before, and it gave me an error, probably because I'm on a dev install and this would only work for a normal install? 

+$DBversion = "XXX";
+if ( C4::Context->preference("Version") < TransformToNum($DBversion) ) {
+    my $installer = C4::Installer->new();
+    my $full_path = C4::Context->config('intranetdir') . "/installer/data/$installer->{dbms}/atomicupdate/oai_sets.sql";
+    my $error     = $installer->load_sql($full_path);
+    warn $error if $error;
+    print "Upgrade to $DBversion done (Add OAI SETs tables)\n";
+    SetVersion($DBversion);
+}

I guess the RM will take care of this anyway, though...

2) Online documentation

When I'm on admin/oai_sets.pl and click on [?] in the top right corner I get an error saying that ...help/admin/oai_sets.tt does not exist. 

3) Deleting a set does not work

The Delete link does not have a value for the "id" parameter. 

4) How is the value in mappings handled?

I can't see that this has been mentioned in the comments anywhere, but here is what I have found:

Is it case sensitive? Yes
Does it look for substrings within a subfield? No, complete subfields

This would be nice to have in the online documentation.

5) No POD/documentation for misc/migration_tools/build_oai_sets.pl

The script doesn't actually have any options to document, but one comment at the top saying what it does might be good. Being able to run the script with the -h option to get a short description would also be nice. 

This script needs a licence statement too. 

6) Item level data

Will this work with 952 and its subfields? Probably not, since the MARC data is gotten at by doing "SELECT biblionumber, marcxml FROM biblioitems" and as far as I can tell the item data is not pulled in later. I'm thinking there are several subfields to 952 that might be interesting to match against, like which library a record "belongs to" (in the sense that the library owns an item connected to the record), item type etc

7) OAI-PMH verbs

- verb=ListSets - OK
- verb=ListRecords&metadataPrefix=oai_dc&set=<setSpec> - OK (In Firefox I had to view source to see the setSpec)
- verb=ListIdentifiers&metadataPrefix=oai_dc&set=<setSpec> - OK
- verb=GetRecord&metadataPrefix=oai_dc&identifier=<identifier> - OK (In Firefox I had to view source to see the setSpec)

As far as I can tell, the data returned for all these requests looks good. Yay! :-) 

8) Some thoughts on the implementation

In general, I think the implementation method for generating the sets is sound. Having to build the sets with a script might seem cumbersome, but if you have e.g. 300000 records and want to match on some subfield you can't really do it in real time. Especially since you might be getting a lot of requests. 

On the other hand, doing one big request for every record in the databse will be heavy for large collections. Could it be possible to "chunk it", and not do all records at once? I'll leave the judgement of that to cleverer heads than mine, though. 

9) Possible enhancements

- Matching on values in the datafields (000-009), for example "include records that have an "a" in position 7 of field 008" (not sure that exact example makes sense, but that sort of thing, at least)
- Matching against item level data (952 for MARC21/NORMARC at least)
- Add and remove records from sets when records are saved/updated, as suggested by Frederic in Comment 6. 

I'm failing this because of point number 3 above, and probably because of 2 and 5 also. 

Otherwise: good work, this will be a nice feature to have!
Comment 9 Julian Maurice 2012-02-16 13:50:33 UTC Comment hidden (obsolete)
Comment 10 Paul Poulain 2012-02-17 14:39:12 UTC
The script devel/misc/testbugzillapatches.pl has detected that this bug does not apply anymore, so i'm marking this bug as "does not apply".

The script is supposed to handle conflicts related to updatedatabase, but no other cases of merge conflict. So the conflict may be trivial to fix.

If it happens that the patch attached does apply cleanly, please send me an email, there's something wrong in the script. It's possible, as it's a recent script.
If the patch applies does not apply as I think, please rebase and resubmit, thanks
Comment 11 Julian Maurice 2012-02-24 09:22:15 UTC Comment hidden (obsolete)
Comment 12 Gaetan Boisson 2012-02-29 09:14:24 UTC
I have just tried it, and the patch does not apply :

Applying: Bug 6440: Implement OAI-PMH Sets
error: patch failed: installer/data/mysql/updatedatabase.pl:4712
error: installer/data/mysql/updatedatabase.pl: patch does not apply
Patch failed at 0001 Bug 6440: Implement OAI-PMH Sets
When you have resolved this problem run "git am --resolved".
If you would prefer to skip this patch, instead run "git am --skip".
To restore the original branch and stop patching run "git am --abort".
Patch left in /tmp/Rebased-patch-eYwQu8.patch
Comment 13 Julian Maurice 2012-03-01 07:59:56 UTC Comment hidden (obsolete)
Comment 14 Magnus Enger 2012-03-01 12:28:29 UTC
Wow, looks like this went from great to awesome pretty fast! 

1. Hm, no problem this time! 
2. Fixed
3. Fixed
4. Fixed by the online help

5. Fixed, but the -r option is missing from the documentation in print_usage() and so it's not output when you run the script with the -h option. It is in the POD though, so doing perldoc build_oai_sets.pl gives the right information. I think maybe using the POD for the output of -h leaves less room for mistakes like this? Like in e.g. bulkmarcimport.pl, with the pod2usage function from Pod::Usage. 

6. Fixed - yay! 
7. All the verbs still do their job
8. -r -l and -o does it's job, excellent! 

9. Automatically updating sets works for records level data, but as far as I can see it does not work for item level data? 

Failing this patch because of 9 and 5. Julian: if you can do a followup patch that applies on top of the first patch I'll try to sign off both really quickly!
Comment 15 Julian Maurice 2012-03-08 10:16:59 UTC Comment hidden (obsolete)
Comment 16 Julian Maurice 2012-03-08 10:17:39 UTC Comment hidden (obsolete)
Comment 17 Julian Maurice 2012-03-08 10:22:45 UTC
Hi Magnus.
Sorry for keeping you waiting so long, but here is the follow-up! You can now sign off both patches really quickly, as you said ;-)
Comment 18 Magnus Enger 2012-03-08 11:04:16 UTC Comment hidden (obsolete)
Comment 19 Magnus Enger 2012-03-08 11:04:23 UTC Comment hidden (obsolete)
Comment 20 Magnus Enger 2012-03-08 11:07:23 UTC
All my questions have been fixed now, and I'm happy to sign off. Thanks for your work on this Julian!
Comment 21 Ian Walls 2012-03-12 11:52:04 UTC
So, what this patch does is, essentially, allow us to create and maintain automated groups of biblios.  Structurally, this is much like Lists, only instead of being associated with a person and added to manually, it's associated with some kind of attribution in the MARC, and new materials are added automated as they're catalogued.

This seems to have much wider applications than just OAI... I'd advocate looking at a generalized naming scheme, and making OAI just one exemplar of usage.  Not a QA comment, just one of my own personal preference.  QAing will come shortly.
Comment 22 Alexandre Lengereau 2012-03-19 13:16:39 UTC
The sandbox you've requested is now ready. The patches attached to bugzilla 6440 have been applied, and the result is
<h1>Something went wrong !</h1>Applying: Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets<br/>
Using index info to reconstruct a base tree...<br/>
Falling back to patching base and 3-way merge...<br/>
Auto-merging installer/data/mysql/kohastructure.sql<br/>
Auto-merging installer/data/mysql/sysprefs.sql<br/>
CONFLICT (content): Merge conflict in installer/data/mysql/sysprefs.sql<br/>
Auto-merging installer/data/mysql/updatedatabase.pl<br/>
CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl<br/>
Failed to merge in the changes.<br/>
Patch failed at 0001 Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets<br/>
When you have resolved this problem run git am --resolved.<br/>
If you would prefer to skip this patch, instead run git am --skip.<br/>
To restore the original branch and stop patching run git am --abort.<br/>
Bug 6440 - Koha's OAI-PMH does not support sets<br/>
<br/>
Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets<br/>
Apply? [yn] <br/>
Patch left in /tmp/Bug-6440-SIGNED-OFF-Implement-OAI-PMH-Sets-Hlztor.patch<br/>
<br/>=== translation installation ===<br/>Install templates
  Install templates 'opac
    From: /home/koha/src/koha-tmpl/opac-tmpl/prog/en/
    To  : /home/koha/src/koha-tmpl/opac-tmpl/prog/fr-FR
    With: /home/koha/src/misc/translator/po/fr-FR-i-opac-t-prog-v-3006000.po
The install seems to be successful.
  Install templates 'intranet
    From: /home/koha/src/koha-tmpl/intranet-tmpl/prog/en/
    To  : /home/koha/src/koha-tmpl/intranet-tmpl/prog/fr-FR
    With: /home/koha/src/misc/translator/po/fr-FR-i-staff-t-prog-v-3006000.po
The install seems to be successful.

If you use this system for the 1st time, you may find usefull to reach the wiki page http://wiki.koha-community.org/wiki/Sandboxes to have some information about those results, to check if everything went well & if you can test the patch properly
Comment 23 Julian Maurice 2012-03-19 15:46:17 UTC
Created attachment 8302 [details] [review]
Implement OAI-PMH Sets

Rebased patch
Comment 24 Julian Maurice 2012-03-19 15:46:41 UTC
Created attachment 8303 [details] [review]
Follow-up: Implement OAI-PMH Sets

Rebased patch
Comment 25 Julian Maurice 2012-03-19 15:48:23 UTC
Reset status to "Signed off" since the conflicts were only on database files
Comment 26 Alexandre Lengereau 2012-03-20 10:19:08 UTC
Second test work !
Signed off
Comment 27 Paul Poulain 2012-03-20 10:42:46 UTC
Additional QA comments:
 * percritic is OK
 * correct indentation, SQL,
 * works as expected
 * nothing to say about the updatedatabase, work fine (I don't understand comment 8)

passed QA, will push now (switching to rel_3_8, as it's a large ENH)
Comment 28 Paul Poulain 2012-03-20 10:44:09 UTC
(note: I was supposed to be pushing patches that have already "passed QA" status. For an unknown reason, this patch came to my list, and I made a lot of QA & tests before realizing it was not "passed QA". I decided to pass QA myself & push it.
Comment 29 Julian Maurice 2012-03-22 08:02:46 UTC
ok in master, marked as resolved