Currently a request like /cgi-bin/koha/oai.pl?verb=ListSets results in this response: <OAI-PMH xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2011-06-01T12:32:26Z</responseDate><request verb="ListSets">http://sksk.bibkat.no/cgi-bin/koha/oai.pl</request><error code="noSetHierarchy">Koha repository doesn't have sets</error></OAI-PMH> A starting point might be to define sets based on libraries/branches and/or item types. Taking it a step further one could imagine adding a config file that would let sysadmins define more sets with some kind of queries. This lack of sets seems to be a stumbling block when trying to integrate Koha with e.g. XC (http://www.extensiblecatalog.org/), which expects to get a list of sets from it's OAI-PMH sources.
> A starting point might be to define sets based on libraries/branches > and/or item types. Taking it a step further one could imagine adding a > config file that would let sysadmins define more sets with some kind > of queries. Koha OAI server requests directly MySQL database in order to get all biblio records biblionumbers. So for now, without changing Koha OAI engine, we would have to limit queries to MySQL queries. We could imagine something like: ALL: SELECT biblionumber FROM biblio; PER: SELECT biblionumber FROM biblioitems WHERE itemtype='PER'; > This lack of sets seems to be a stumbling block when trying to > integrate Koha with e.g. XC (http://www.extensiblecatalog.org/), which > expects to get a list of sets from it's OAI-PMH sources. We could very easily define a ALL set returning the whole catalog as it's already done. Koha OAI Server doesn't have also incremental harvesting capabilities.
> Koha OAI Server doesn't have also incremental harvesting capabilities. Sorry, I'm wrong here. Biblio records are sent depending on biblio.timestamp field.
One set for "ALL" would be a good start, I think!
BibLibre have worked on OAI-PMH Sets implementation. I'm currently working on rebasing the patch.
Created attachment 7296 [details] [review] Bug 6440: Implement OAI-PMH Sets New sql tables: - oai_sets: contains the list of sets, described by a spec and a name - oai_sets_descriptions: contains a list of descriptions for each set - oai_sets_mappings: conditions on marc fields to match for biblio to be in a set - oai_sets_biblios: list of biblionumbers for each set New admin page: allow to configure sets: - Creation, deletion, modification of spec, name and descriptions - Define mappings which will be used for building oai sets Implements OAI Sets in opac/oai.pl: - ListSets, ListIdentifiers, ListRecords, GetRecord New script misc/migration_tools/build_oai_sets.pl: - Retrieve marcxml from all biblios and test if they belong to defined sets. The oai_sets_biblios table is then updated accordingly Use OPACBaseURL in oai_dc xslt --- To test: - Run updatedatabase.pl - Go to OAI Sets administration page (Administration > OAI Sets Configuration) - Create some sets with some mappings (mappings define if biblios belong to a set or not, they're in the form of conditions FIELD$SUBFIELD = VALUE, multiple conditions can be defined for a set) - Check some biblios are matching the conditions to belong to one set or more - Run misc/migration_tools/build_oai_sets.pl - Now you can test with $OPACBaseUrl/cgi-bin/koha/oai.pl OAI-PMH spec is here http://www.openarchives.org/OAI/openarchivesprotocol.html Some examples: http://mylibrary.org/cgi-bin/koha/oai.pl?verb=ListSets http://mylibrary.org/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=set1 http://mylibrary.org/cgi-bin/koha/oai.pl?verb=ListIdentifiers&set=set2 http://mylibrary.org/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=oai_dc&identifier=KOHA-OAI-TEST:3 The response of ListRecords and GetRecord must include setSpecs in header
As it is implemented, does it mean that OAI sets have to be rebuild periodically? Wouldn't it be possible that adding/editing a biblio record via Koha data entry form update the OAI sets? Do you have a functional specification of sets? It would help a lot to understand how it works and help also the documentation redactor.
(In reply to comment #6) > As it is implemented, does it mean that OAI sets have to be rebuild > periodically? Actually, yes. > Wouldn't it be possible that adding/editing a biblio record > via Koha data entry form update the OAI sets? I don't remember why it was done like this (maybe performance concerns), but it would be trivial to add this feature. I think adding a syspref for this would be useful, since it can increase time of addition/modification with a large number of sets (maybe someone has a name for this syspref :) ?) > Do you have a functional > specification of sets? It would help a lot to understand how it works > and help also the documentation redactor. Hmm sorry but no :-/ I can add that a hierarchy of set can be defined using the set specs (eg. 'a:b' define the set 'a:b' which is a subset of set 'a', it means that every element of 'a:b' is implicitly in 'a')
1) updatedatabase.pl I have not seen this construct in updatadatabase.pl before, and it gave me an error, probably because I'm on a dev install and this would only work for a normal install? +$DBversion = "XXX"; +if ( C4::Context->preference("Version") < TransformToNum($DBversion) ) { + my $installer = C4::Installer->new(); + my $full_path = C4::Context->config('intranetdir') . "/installer/data/$installer->{dbms}/atomicupdate/oai_sets.sql"; + my $error = $installer->load_sql($full_path); + warn $error if $error; + print "Upgrade to $DBversion done (Add OAI SETs tables)\n"; + SetVersion($DBversion); +} I guess the RM will take care of this anyway, though... 2) Online documentation When I'm on admin/oai_sets.pl and click on [?] in the top right corner I get an error saying that ...help/admin/oai_sets.tt does not exist. 3) Deleting a set does not work The Delete link does not have a value for the "id" parameter. 4) How is the value in mappings handled? I can't see that this has been mentioned in the comments anywhere, but here is what I have found: Is it case sensitive? Yes Does it look for substrings within a subfield? No, complete subfields This would be nice to have in the online documentation. 5) No POD/documentation for misc/migration_tools/build_oai_sets.pl The script doesn't actually have any options to document, but one comment at the top saying what it does might be good. Being able to run the script with the -h option to get a short description would also be nice. This script needs a licence statement too. 6) Item level data Will this work with 952 and its subfields? Probably not, since the MARC data is gotten at by doing "SELECT biblionumber, marcxml FROM biblioitems" and as far as I can tell the item data is not pulled in later. I'm thinking there are several subfields to 952 that might be interesting to match against, like which library a record "belongs to" (in the sense that the library owns an item connected to the record), item type etc 7) OAI-PMH verbs - verb=ListSets - OK - verb=ListRecords&metadataPrefix=oai_dc&set=<setSpec> - OK (In Firefox I had to view source to see the setSpec) - verb=ListIdentifiers&metadataPrefix=oai_dc&set=<setSpec> - OK - verb=GetRecord&metadataPrefix=oai_dc&identifier=<identifier> - OK (In Firefox I had to view source to see the setSpec) As far as I can tell, the data returned for all these requests looks good. Yay! :-) 8) Some thoughts on the implementation In general, I think the implementation method for generating the sets is sound. Having to build the sets with a script might seem cumbersome, but if you have e.g. 300000 records and want to match on some subfield you can't really do it in real time. Especially since you might be getting a lot of requests. On the other hand, doing one big request for every record in the databse will be heavy for large collections. Could it be possible to "chunk it", and not do all records at once? I'll leave the judgement of that to cleverer heads than mine, though. 9) Possible enhancements - Matching on values in the datafields (000-009), for example "include records that have an "a" in position 7 of field 008" (not sure that exact example makes sense, but that sort of thing, at least) - Matching against item level data (952 for MARC21/NORMARC at least) - Add and remove records from sets when records are saved/updated, as suggested by Frederic in Comment 6. I'm failing this because of point number 3 above, and probably because of 2 and 5 also. Otherwise: good work, this will be a nice feature to have!
Created attachment 7689 [details] [review] New patch Thanks Magnus for testing! 1) I'm on a dev install too and see no problems. Can you give me error messages you have? (unmodified in the new patch) 2) Added documentation for admin/oai_sets.pl and admin/oai_set_mappings.pl 3) Fixed 4) Added in the doc 5) Added brief POD documentation and help message with -h 6) Added option -i to build_oai_sets.pl to embed items information in the record before processing 8) Added options -r -l and -o -r is for reset, it truncate table oai_sets_biblios before filling it -l is for length, to tell the script how many biblios you want to process -o is for offset, to tell the script where to start processing (-l is required here) So, to build OAI sets by chunks of 5000 for example, use ./build_oai_sets.pl -r -l 5000 the first time, and then ./build_oai_sets.pl -l 5000 -o 5000 ./build_oai_sets.pl -l 5000 -o 10000 ... 9) Added syspref 'OAI-PMH:AutoUpdateSets'. If ON, the sets are updated when a biblio is created or modified I'm sure there is a lot of improvements to do with mappings (case insensitive, other conditions like 'not equal', 'contains', 'do not contains', ...) but I think they can come later.
The script devel/misc/testbugzillapatches.pl has detected that this bug does not apply anymore, so i'm marking this bug as "does not apply". The script is supposed to handle conflicts related to updatedatabase, but no other cases of merge conflict. So the conflict may be trivial to fix. If it happens that the patch attached does apply cleanly, please send me an email, there's something wrong in the script. It's possible, as it's a recent script. If the patch applies does not apply as I think, please rebase and resubmit, thanks
Created attachment 7849 [details] [review] Rebased patch
I have just tried it, and the patch does not apply : Applying: Bug 6440: Implement OAI-PMH Sets error: patch failed: installer/data/mysql/updatedatabase.pl:4712 error: installer/data/mysql/updatedatabase.pl: patch does not apply Patch failed at 0001 Bug 6440: Implement OAI-PMH Sets When you have resolved this problem run "git am --resolved". If you would prefer to skip this patch, instead run "git am --skip". To restore the original branch and stop patching run "git am --abort". Patch left in /tmp/Rebased-patch-eYwQu8.patch
Created attachment 7954 [details] [review] Rebase patch
Wow, looks like this went from great to awesome pretty fast! 1. Hm, no problem this time! 2. Fixed 3. Fixed 4. Fixed by the online help 5. Fixed, but the -r option is missing from the documentation in print_usage() and so it's not output when you run the script with the -h option. It is in the POD though, so doing perldoc build_oai_sets.pl gives the right information. I think maybe using the POD for the output of -h leaves less room for mistakes like this? Like in e.g. bulkmarcimport.pl, with the pod2usage function from Pod::Usage. 6. Fixed - yay! 7. All the verbs still do their job 8. -r -l and -o does it's job, excellent! 9. Automatically updating sets works for records level data, but as far as I can see it does not work for item level data? Failing this patch because of 9 and 5. Julian: if you can do a followup patch that applies on top of the first patch I'll try to sign off both really quickly!
Created attachment 8078 [details] [review] Original patch rebased on master
Created attachment 8079 [details] [review] Bug 6440: Follow-up: Implement OAI-PMH Sets - Use Pod::Usage in build_oai_sets.pl - Update sets at item creation and modification. Modified subroutines are AddItem, ModItem and AddItemBatchFromMarc
Hi Magnus. Sorry for keeping you waiting so long, but here is the follow-up! You can now sign off both patches really quickly, as you said ;-)
Created attachment 8081 [details] [review] Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets New sql tables: - oai_sets: contains the list of sets, described by a spec and a name - oai_sets_descriptions: contains a list of descriptions for each set - oai_sets_mappings: conditions on marc fields to match for biblio to be in a set - oai_sets_biblios: list of biblionumbers for each set New admin page: allow to configure sets: - Creation, deletion, modification of spec, name and descriptions - Define mappings which will be used for building oai sets Implements OAI Sets in opac/oai.pl: - ListSets, ListIdentifiers, ListRecords, GetRecord New script misc/migration_tools/build_oai_sets.pl: - Retrieve marcxml from all biblios and test if they belong to defined sets. The oai_sets_biblios table is then updated accordingly New system preference OAI-PMH:AutoUpdateSets. If on, update sets automatically when a biblio is created or updated. Use OPACBaseURL in oai_dc xslt Signed-off-by: Magnus Enger <magnus@enger.priv.no> I created a set based on a value in 245$a, ran the script and verified that the created st contained the expected records. I repeated this with a value from 952$a, to test item level data. Both were OK. Then i set OAI-PMH:AutoUpdateSets = Enable and created a set based on a value in 245$a, built the set with the script and verified that it looked OK. Then i changed the value in 245$a and verified that the set was updated without running the script. I repeated this with a value for 952$a. This did not work initially, but this has been fixed by the followup patch. All the options to build_oai_sets.pl work as expected. See comments on the bug for things that have been fixed along the way.
Created attachment 8082 [details] [review] Bug 6440: [SIGNED-OFF] Follow-up: Implement OAI-PMH Sets - Use Pod::Usage in build_oai_sets.pl - Update sets at item creation and modification. Modified subroutines are AddItem, ModItem and AddItemBatchFromMarc Signed-off-by: Magnus Enger <magnus@enger.priv.no> With OAI-PMH:AutoUpdateSets = Enable, I created a set based on a value in 952$a, built the set with the script, changed the value in 952$a and checked that the set was updated without running the script. Success! This also fixes my suggestion to use the POD as the content for the message output by running the script with the -h option.
All my questions have been fixed now, and I'm happy to sign off. Thanks for your work on this Julian!
So, what this patch does is, essentially, allow us to create and maintain automated groups of biblios. Structurally, this is much like Lists, only instead of being associated with a person and added to manually, it's associated with some kind of attribution in the MARC, and new materials are added automated as they're catalogued. This seems to have much wider applications than just OAI... I'd advocate looking at a generalized naming scheme, and making OAI just one exemplar of usage. Not a QA comment, just one of my own personal preference. QAing will come shortly.
The sandbox you've requested is now ready. The patches attached to bugzilla 6440 have been applied, and the result is <h1>Something went wrong !</h1>Applying: Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets<br/> Using index info to reconstruct a base tree...<br/> Falling back to patching base and 3-way merge...<br/> Auto-merging installer/data/mysql/kohastructure.sql<br/> Auto-merging installer/data/mysql/sysprefs.sql<br/> CONFLICT (content): Merge conflict in installer/data/mysql/sysprefs.sql<br/> Auto-merging installer/data/mysql/updatedatabase.pl<br/> CONFLICT (content): Merge conflict in installer/data/mysql/updatedatabase.pl<br/> Failed to merge in the changes.<br/> Patch failed at 0001 Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets<br/> When you have resolved this problem run git am --resolved.<br/> If you would prefer to skip this patch, instead run git am --skip.<br/> To restore the original branch and stop patching run git am --abort.<br/> Bug 6440 - Koha's OAI-PMH does not support sets<br/> <br/> Bug 6440: [SIGNED-OFF] Implement OAI-PMH Sets<br/> Apply? [yn] <br/> Patch left in /tmp/Bug-6440-SIGNED-OFF-Implement-OAI-PMH-Sets-Hlztor.patch<br/> <br/>=== translation installation ===<br/>Install templates Install templates 'opac From: /home/koha/src/koha-tmpl/opac-tmpl/prog/en/ To : /home/koha/src/koha-tmpl/opac-tmpl/prog/fr-FR With: /home/koha/src/misc/translator/po/fr-FR-i-opac-t-prog-v-3006000.po The install seems to be successful. Install templates 'intranet From: /home/koha/src/koha-tmpl/intranet-tmpl/prog/en/ To : /home/koha/src/koha-tmpl/intranet-tmpl/prog/fr-FR With: /home/koha/src/misc/translator/po/fr-FR-i-staff-t-prog-v-3006000.po The install seems to be successful. If you use this system for the 1st time, you may find usefull to reach the wiki page http://wiki.koha-community.org/wiki/Sandboxes to have some information about those results, to check if everything went well & if you can test the patch properly
Created attachment 8302 [details] [review] Implement OAI-PMH Sets Rebased patch
Created attachment 8303 [details] [review] Follow-up: Implement OAI-PMH Sets Rebased patch
Reset status to "Signed off" since the conflicts were only on database files
Second test work ! Signed off
Additional QA comments: * percritic is OK * correct indentation, SQL, * works as expected * nothing to say about the updatedatabase, work fine (I don't understand comment 8) passed QA, will push now (switching to rel_3_8, as it's a large ENH)
(note: I was supposed to be pushing patches that have already "passed QA" status. For an unknown reason, this patch came to my list, and I made a lot of QA & tests before realizing it was not "passed QA". I decided to pass QA myself & push it.
ok in master, marked as resolved