Bugzilla – Attachment 173088 Details for
Bug 35659
OAI harvester
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 35659: OAI Harvester
Bug-35659-OAI-Harvester.patch (text/plain), 103.12 KB, created by
David Cook
on 2024-10-22 01:36:05 UTC
(
hide
)
Description:
Bug 35659: OAI Harvester
Filename:
MIME Type:
Creator:
David Cook
Created:
2024-10-22 01:36:05 UTC
Size:
103.12 KB
patch
obsolete
>From 49bed13e28b0776d5952ad41847ed6315a42d25d Mon Sep 17 00:00:00 2001 >From: Matthias Meusburger <matthias.meusburger@biblibre.com> >Date: Thu, 20 Jul 2023 16:29:49 +0200 >Subject: [PATCH] Bug 35659: OAI Harvester > >Test plan: > > - Run updatedatabase.pl > > - Apply the DBIC update patch > > - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) > > If the repository dataformat is in oai_dc, you can use the sample XSLT files: > For Unimarc: > OAIDCtoUNIMARCXML.xsl for bibliographic records > AuthOAIDCtoUNIMARCXML.xsl for authorities > > For Marc21: > OAIDCtoMARC21XML.xsl for bibliographic records > AuthOAIDCtoMARC21XML.xsl for authorities > > If the repository dataformat is in marc-xml, you can use > OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. > > - Some repositories to test with: > > Bibliographic records, oai_dc: > Endpoint: http://staroai.theses.fr/OAIHandler > Set: ddc:260 > XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl > or > XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl > > Authorities, oai_dc: > Endpoint: https://www.idref.fr/OAI/oai.jsp > Set: a > XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl > or > XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl > > Authorities, marc-xml, Unimarc: > Endpoint: https://www.idref.fr/OAI/oai.jsp > Set: a > XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl > > - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx > Version 4.xx is needed. > See Bug 17704 for updating Koha default version to 4.xx > > - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport > syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. > > - start a harvest with the following command: > misc/cronjobs/harvest_oai.pl > > This script starts an OAI Harvest > > This script has the following parameters : > -h --help: this message > -v --verbose (enable output to stdout) > -r --repository: id of the OAI repository > -d --days: number of days to harvest from (optional) > -l --list: list the OAI repositories > -f --force: force harvesting (ignore records datestamps) (optional) > > Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f > > - Check the logs in the Log Viewer (tools/viewlog.pl) > (CronjobLog syspref must be activated) > >Sponsored-by: KohaLa >Signed-off-by: Michal Denar <black23@gmail.com> >Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> >Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com> >Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> >Signed-off-by: David Cook <dcook@prosentient.com.au> >Sponsored-by: Association KohaLa - https://koha-fr.org/ >--- > Koha/Import/Oaipmh/Authorities.pm | 51 +++ > Koha/Import/Oaipmh/Authority.pm | 41 ++ > Koha/Import/Oaipmh/Biblio.pm | 40 ++ > Koha/Import/Oaipmh/Biblios.pm | 51 +++ > Koha/OAI/Client/Harvester.pm | 372 ++++++++++++++++++ > Koha/OaiServer.pm | 45 +++ > Koha/OaiServers.pm | 57 +++ > admin/oaiservers.pl | 122 ++++++ > .../data/mysql/atomicupdate/Bug-35659.pl | 85 ++++ > .../mysql/en/mandatory/sample_notices.yml | 23 ++ > installer/data/mysql/kohastructure.sql | 58 +++ > installer/data/mysql/mandatory/sysprefs.sql | 1 + > .../data/mysql/mandatory/userpermissions.sql | 2 +- > .../prog/en/includes/admin-menu.inc | 1 + > .../prog/en/includes/oai-admin-search.inc | 43 ++ > .../prog/en/includes/permissions.inc | 2 +- > .../prog/en/modules/admin/admin-home.tt | 2 + > .../prog/en/modules/admin/oaiservers.tt | 229 +++++++++++ > .../admin/preferences/web_services.pref | 5 + > .../prog/en/xslt/AuthOAIDCtoMARC21XML.xsl | 35 ++ > .../prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl | 42 ++ > .../prog/en/xslt/OAIDCtoMARC21XML.xsl | 367 +++++++++++++++++ > .../prog/en/xslt/OAIDCtoUNIMARCXML.xsl | 341 ++++++++++++++++ > .../prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl | 23 ++ > misc/cronjobs/harvest_oai.pl | 110 ++++++ > t/db_dependent/Koha/OAIHarvester.t | 78 ++++ > t/db_dependent/Koha/OAIServers.t | 95 +++++ > 27 files changed, 2319 insertions(+), 2 deletions(-) > create mode 100644 Koha/Import/Oaipmh/Authorities.pm > create mode 100644 Koha/Import/Oaipmh/Authority.pm > create mode 100644 Koha/Import/Oaipmh/Biblio.pm > create mode 100644 Koha/Import/Oaipmh/Biblios.pm > create mode 100644 Koha/OAI/Client/Harvester.pm > create mode 100644 Koha/OaiServer.pm > create mode 100644 Koha/OaiServers.pm > create mode 100755 admin/oaiservers.pl > create mode 100755 installer/data/mysql/atomicupdate/Bug-35659.pl > create mode 100644 koha-tmpl/intranet-tmpl/prog/en/includes/oai-admin-search.inc > create mode 100644 koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt > create mode 100644 koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl > create mode 100644 koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl > create mode 100644 koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl > create mode 100644 koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl > create mode 100644 koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl > create mode 100755 misc/cronjobs/harvest_oai.pl > create mode 100755 t/db_dependent/Koha/OAIHarvester.t > create mode 100755 t/db_dependent/Koha/OAIServers.t > >diff --git a/Koha/Import/Oaipmh/Authorities.pm b/Koha/Import/Oaipmh/Authorities.pm >new file mode 100644 >index 0000000000..90372a8609 >--- /dev/null >+++ b/Koha/Import/Oaipmh/Authorities.pm >@@ -0,0 +1,51 @@ >+package Koha::Import::Oaipmh::Authorities; >+ >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+ >+use Modern::Perl; >+ >+use Koha::Import::Oaipmh::Authority; >+ >+use base qw(Koha::Objects); >+ >+=head1 NAME >+ >+Koha::Import::Oaipmh::Authorities >+ >+This object represents a collection of OAI-PMH records being imported as authorities >+ >+=head1 API >+ >+=head2 Methods >+ >+ >+=head3 type >+ >+=cut >+ >+sub _type { >+ return 'ImportOaipmhAuthority'; >+} >+ >+=head3 object_class >+ >+Koha::Object class >+ >+=cut >+ >+sub object_class { >+ return 'Koha::Import::Oaipmh::Authority'; >+} >+1; >diff --git a/Koha/Import/Oaipmh/Authority.pm b/Koha/Import/Oaipmh/Authority.pm >new file mode 100644 >index 0000000000..fe90c432a2 >--- /dev/null >+++ b/Koha/Import/Oaipmh/Authority.pm >@@ -0,0 +1,41 @@ >+package Koha::Import::Oaipmh::Authority; >+ >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+ >+use Modern::Perl; >+ >+use base qw(Koha::Object); >+ >+=head1 NAME >+ >+Koha::Import::Oaipmh::Authority >+ >+This object represents an OAI-PMH record being imported as an authority >+ >+=head1 API >+ >+=head2 Class Methods >+ >+=cut >+ >+=head3 _type >+ >+=cut >+ >+sub _type { >+ return 'ImportOaipmhAuthority'; >+} >+ >+1; >diff --git a/Koha/Import/Oaipmh/Biblio.pm b/Koha/Import/Oaipmh/Biblio.pm >new file mode 100644 >index 0000000000..d59e5a6449 >--- /dev/null >+++ b/Koha/Import/Oaipmh/Biblio.pm >@@ -0,0 +1,40 @@ >+package Koha::Import::Oaipmh::Biblio; >+ >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+ >+use Modern::Perl; >+ >+use base qw(Koha::Object); >+ >+=head1 NAME >+ >+Koha::Import::Oaipmh::Biblio >+ >+This object represents an OAI-PMH record being imported as a bibliographic record >+ >+=head1 API >+ >+=head2 Methods >+ >+ >+=head3 _type >+ >+=cut >+ >+sub _type { >+ return 'ImportOaipmhBiblio'; >+} >+ >+1; >diff --git a/Koha/Import/Oaipmh/Biblios.pm b/Koha/Import/Oaipmh/Biblios.pm >new file mode 100644 >index 0000000000..9ae111e96b >--- /dev/null >+++ b/Koha/Import/Oaipmh/Biblios.pm >@@ -0,0 +1,51 @@ >+package Koha::Import::Oaipmh::Biblios; >+ >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+ >+use Modern::Perl; >+ >+use Koha::Import::Oaipmh::Biblio; >+ >+use base qw(Koha::Objects); >+ >+=head1 NAME >+ >+Koha::Import::Oaipmh::Biblios >+ >+This object represents a collection of OAI-PMH records being imported as bibliographic records >+ >+=head1 API >+ >+=head2 Methods >+ >+ >+=head3 type >+ >+=cut >+ >+sub _type { >+ return 'ImportOaipmhBiblio'; >+} >+ >+=head3 object_class >+ >+Koha::Object class >+ >+=cut >+ >+sub object_class { >+ return 'Koha::Import::Oaipmh::Biblio'; >+} >+1; >diff --git a/Koha/OAI/Client/Harvester.pm b/Koha/OAI/Client/Harvester.pm >new file mode 100644 >index 0000000000..f800fb1dc1 >--- /dev/null >+++ b/Koha/OAI/Client/Harvester.pm >@@ -0,0 +1,372 @@ >+# Copyright BibLibre 2023 >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ >+package Koha::OAI::Client::Harvester; >+ >+=head1 NAME >+ >+Koha::OAI::Client::Harvester - OAI Harvester >+ >+=head1 DESCRIPTION >+ >+Koha::OAI::Client::Harvester contains OAI-PMH harvester main functions >+ >+=cut >+ >+use utf8; >+use open qw( :std :utf8 ); >+ >+use C4::Biblio qw( AddBiblio GetFrameworkCode ModBiblio DelBiblio ); >+use C4::AuthoritiesMarc qw (AddAuthority GuessAuthTypeCode ModAuthority DelAuthority ); >+use C4::Log qw( cronlogaction ); >+use HTTP::OAI; >+use HTTP::OAI::Metadata::OAI_DC; >+use Koha::DateUtils qw( dt_from_string ); >+use Koha::OaiServers; >+use Koha::Import::Oaipmh::Biblio; >+use Koha::Import::Oaipmh::Biblios; >+use Koha::Import::Oaipmh::Authority; >+use Koha::Import::Oaipmh::Authorities; >+use Koha::XSLT::Base; >+use MARC::File::XML; >+use MARC::Record; >+use Modern::Perl; >+use DateTime; >+use DateTime::Format::Strptime; >+use Try::Tiny qw( catch try ); >+use Date::Calc qw( >+ Add_Delta_Days >+ Today >+); >+ >+my $strp = DateTime::Format::Strptime->new( >+ pattern => '%FT%TZ', >+ time_zone => 'floating', >+); >+ >+our $verbose = 0; >+our $server; >+our $force = 0; >+our $days = 0; >+our $xslt_engine = Koha::XSLT::Base->new; >+ >+=head2 new >+ >+$harvester = Koha::OAI::Client::Harvester->new( { server => $server, verbose => $verbose, days => $days, force => $force } ); >+ >+New instance of Koha::OAI::Client::Harvester >+ >+C<$server> An OAI repository (Koha::OaiServer) >+ >+C<$verbose> print log messages to stdout when enabled >+ >+C<$days> number of days to harvest from (optional) >+ >+C<$force> force harvesting (ignore records datestamps) >+ >+=cut >+ >+sub new { >+ >+ my ( $self, $args ) = @_; >+ >+ $server = $args->{'server'}; >+ $verbose = $args->{'verbose'}; >+ $days = $args->{'days'}; >+ $force = $args->{'force'}; >+ return $self; >+} >+ >+=head2 init >+ >+Starts harvesting >+ >+=cut >+ >+sub init { >+ my ( $self, $args ) = @_; >+ >+ printlog("Starting OAI Harvest from repository"); >+ >+ my $h = HTTP::OAI::Harvester->new( >+ repository => HTTP::OAI::Identify->new( >+ baseURL => $server->endpoint, >+ version => '2.0', >+ ) >+ ); >+ >+ printlog( "Using HTTP::OAI::Harvester version " . $HTTP::OAI::Harvester::VERSION ); >+ >+ printlog( "Using endpoint " >+ . $server->endpoint >+ . ", set " >+ . $server->oai_set >+ . ", dataformat " >+ . $server->dataformat >+ . ", recordtype " >+ . $server->recordtype ); >+ >+ try { >+ my $lmdf = $h->ListMetadataFormats(); >+ my $lmdflog = ""; >+ for ( $lmdf->metadataFormat ) { >+ $lmdflog .= $_->metadataPrefix . " "; >+ } >+ printlog("Available metadata formats for this repository: $lmdflog"); >+ } catch { >+ printlog("ListMetadataFormats failed"); >+ }; >+ >+ if ( $server->add_xslt ) { >+ printlog( "Using XSLT file " . $server->add_xslt ); >+ } else { >+ printlog("Not using an XSLT file"); >+ } >+ >+ my $start_date = ''; >+ my $today_date = ''; >+ >+ if ($days) { >+ >+ # Change this to yyyy-mm-dd >+ my $dt_today = dt_from_string(); >+ my ( $nowyear, $nowmonth, $nowday ) = Today(); >+ my @date = Add_Delta_Days( $nowyear, $nowmonth, $nowday, -$days ); >+ my $dt_start = DateTime->new( year => $date[0], month => $date[1], day => $date[2] ); >+ $start_date = $dt_start->ymd(); >+ printlog("Harvesting from $start_date"); >+ } >+ >+ printlog("Asking for records"); >+ my $response = $h->ListRecords( >+ metadataPrefix => $server->dataformat, >+ set => $server->oai_set, >+ from => $start_date, >+ ); >+ if ( $response->is_error ) { >+ printlog( "Error requesting ListRecords: " . $response->code . " " . $response->message ); >+ exit; >+ } >+ printlog( "Request URL: " . $response->requestURL ); >+ >+ my %stats; >+ my @statuses = qw(added updated deleted in_error skipped total); >+ foreach my $status (@statuses) { >+ $stats{$status} = 0; >+ } >+ >+ printlog("Starting processing results"); >+ while ( my $oai_record = $response->next ) { >+ $stats{'total'}++; >+ my $status = $self->processRecord($oai_record); >+ $stats{$status}++; >+ } >+ >+ my $results = ''; >+ foreach my $status (@statuses) { >+ $results .= $stats{$status} . " $status\n"; >+ } >+ printlog( "Harvest results:\n" . $results ); >+ >+ if ( C4::Context->preference("OAI-PMH:HarvestEmailReport") ) { >+ >+ my $letter = C4::Letters::GetPreparedLetter( >+ ( >+ module => 'catalogue', >+ letter_code => 'OAI_HARVEST_REPORT', >+ substitute => { >+ servername => $server->servername, >+ endpoint => $server->endpoint, >+ set => $server->oai_set, >+ dataformat => $server->dataformat, >+ recordtype => $server->recordtype, >+ added => $stats{'added'}, >+ updated => $stats{'updated'}, >+ deleted => $stats{'deleted'}, >+ skipped => $stats{'skipped'}, >+ in_error => $stats{'in_error'}, >+ total => $stats{'total'} >+ }, >+ ) >+ ) or printlog("OAI_HARVEST_REPORT letter not found"); >+ >+ my $success = C4::Letters::EnqueueLetter( >+ { >+ letter => $letter, >+ to_address => C4::Context->preference("OAI-PMH:HarvestEmailReport"), message_transport_type => 'email' >+ } >+ ); >+ if ($success) { >+ printlog("Email report enqueued"); >+ } else { >+ printlog("Unable to enqueue report email"); >+ } >+ } >+ >+ printlog("Ending OAI Harvest from repository"); >+} >+ >+=head2 processRecord >+ >+This method processes an incoming OAI record >+ >+=cut >+ >+sub processRecord { >+ my $self = shift; >+ my $oai_record = shift; >+ my $status = ''; >+ unless ( $oai_record->identifier ) { >+ printlog("No identifier found"); >+ $status = 'skipped'; >+ return $status; >+ } >+ >+ my $to_delete; >+ my $outputUnimarc; >+ my $marcrecord; >+ if ( $oai_record->status && $oai_record->status eq "deleted" ) { >+ $to_delete = 1; >+ } else { >+ if ( $server->add_xslt ) { >+ $outputUnimarc = $xslt_engine->transform( $oai_record->metadata->dom, $server->add_xslt ); >+ my $err = $xslt_engine->err; >+ if ($err) { >+ printlog("XSLT::Base error: $err"); >+ $status = 'in_error'; >+ return $status; >+ } >+ } else { >+ $outputUnimarc = $oai_record->metadata->dom; >+ } >+ $marcrecord = MARC::Record->new_from_xml( $outputUnimarc, 'UTF-8' ); >+ } >+ my $imported_record; >+ if ( $server->recordtype eq "biblio" ) { >+ $imported_record = Koha::Import::Oaipmh::Biblios->find( >+ { >+ repository => $server->endpoint, >+ identifier => $oai_record->identifier, >+ recordtype => $server->recordtype, >+ } >+ ); >+ } else { >+ $imported_record = Koha::Import::Oaipmh::Authorities->find( >+ { >+ repository => $server->endpoint, >+ identifier => $oai_record->identifier, >+ recordtype => $server->recordtype, >+ } >+ ); >+ } >+ >+ if ($imported_record) { >+ if ($to_delete) { >+ if ( $server->recordtype eq "biblio" ) { >+ my $biblionumber = $imported_record->biblionumber; >+ my $error = DelBiblio($biblionumber); >+ if ($error) { >+ printlog( >+ "Record " . $oai_record->identifier . " not deleted, biblionumber: $biblionumber ($error)" ); >+ $status = 'in_error'; >+ } else { >+ $imported_record->delete; >+ printlog( "Record " . $oai_record->identifier . " deleted, biblionumber: $biblionumber" ); >+ $status = 'deleted'; >+ } >+ } else { >+ my $authid = $imported_record->authid; >+ DelAuthority( { authid => $authid } ); >+ $imported_record->delete; >+ printlog( "Record " . $oai_record->identifier . " deleted, authid: $authid" ); >+ $status = 'deleted'; >+ } >+ } else { >+ my $existing_dt = $strp->parse_datetime( $imported_record->datestamp ); >+ my $incoming_dt = $strp->parse_datetime( $oai_record->datestamp ); >+ >+ if ( $force || !$incoming_dt || !$existing_dt || $incoming_dt > $existing_dt ) { >+ if ( $server->recordtype eq "biblio" ) { >+ my $biblionumber = $imported_record->biblionumber; >+ my $result = ModBiblio( $marcrecord, $biblionumber, GetFrameworkCode($biblionumber) ); >+ printlog( "Record " . $oai_record->identifier . " updated, biblionumber: $biblionumber" ); >+ } else { >+ my $authid = $imported_record->authid; >+ my $result = ModAuthority( $authid, $marcrecord, GuessAuthTypeCode($marcrecord) ); >+ printlog( "Record " . $oai_record->identifier . " updated, authid: $authid" ); >+ } >+ $imported_record->update( >+ { >+ datestamp => $imported_record->datestamp, >+ } >+ ); >+ $status = 'updated'; >+ } else { >+ printlog( "Record " >+ . $oai_record->identifier >+ . " skipped (incoming record was not newer than existing record)" ); >+ $status = 'skipped'; >+ } >+ } >+ } elsif ( !$to_delete ) { >+ if ( $server->recordtype eq "biblio" ) { >+ my ( $biblionumber, $biblioitemnumber ) = AddBiblio($marcrecord); >+ printlog( $oai_record->identifier . " added, biblionumber: $biblionumber" ); >+ Koha::Import::Oaipmh::Biblio->new( >+ { >+ repository => $server->endpoint, >+ identifier => $oai_record->identifier, >+ biblionumber => $biblionumber, >+ recordtype => $server->recordtype, >+ datestamp => $oai_record->datestamp, >+ } >+ )->store; >+ } else { >+ my $authid = AddAuthority( $marcrecord, "", GuessAuthTypeCode($marcrecord) ); >+ printlog( $oai_record->identifier . " added, authid: $authid" ); >+ Koha::Import::Oaipmh::Authority->new( >+ { >+ repository => $server->endpoint, >+ identifier => $oai_record->identifier, >+ authid => $authid, >+ recordtype => $server->recordtype, >+ datestamp => $oai_record->datestamp, >+ } >+ )->store; >+ } >+ $status = 'added'; >+ } else { >+ printlog( "Record " . $oai_record->identifier . " skipped (record not present or already deleted)" ); >+ $status = 'skipped'; >+ } >+ return $status; >+} >+ >+=head2 printlog >+ >+This method adds a cronlog and prints to stdout if verbose is enabled >+ >+=cut >+ >+sub printlog { >+ my $message = shift; >+ $message = $server->servername . ": " . $message; >+ print $message . "\n" if ($verbose); >+ cronlogaction( { info => $message } ); >+} >+ >+1; >diff --git a/Koha/OaiServer.pm b/Koha/OaiServer.pm >new file mode 100644 >index 0000000000..fa97c1f0b7 >--- /dev/null >+++ b/Koha/OaiServer.pm >@@ -0,0 +1,45 @@ >+package Koha::OaiServer; >+ >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ >+use Modern::Perl; >+ >+ >+use Koha::Database; >+ >+use base qw(Koha::Object); >+ >+=head1 NAME >+ >+Koha::OaiServer - Koha OaiServer Object class >+ >+=head1 API >+ >+=head2 Class Methods >+ >+=cut >+ >+=head3 _type >+ >+Return type of Object relating to Schema ResultSet >+ >+=cut >+ >+sub _type { >+ return 'Oaiserver'; >+} >+ >+1; >diff --git a/Koha/OaiServers.pm b/Koha/OaiServers.pm >new file mode 100644 >index 0000000000..b752419c5d >--- /dev/null >+++ b/Koha/OaiServers.pm >@@ -0,0 +1,57 @@ >+package Koha::OaiServers; >+ >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ >+use Modern::Perl; >+ >+ >+use Koha::Database; >+ >+use Koha::OaiServer; >+ >+use base qw(Koha::Objects); >+ >+=head1 NAME >+ >+Koha::OaiServers - Koha OaiServer Object set class >+ >+=head1 API >+ >+=head2 Class Methods >+ >+=cut >+ >+=head3 _type >+ >+Return type of object, relating to Schema ResultSet >+ >+=cut >+ >+sub _type { >+ return 'Oaiserver'; >+} >+ >+=head3 object_class >+ >+Return object class >+ >+=cut >+ >+sub object_class { >+ return 'Koha::OaiServer'; >+} >+ >+1; >diff --git a/admin/oaiservers.pl b/admin/oaiservers.pl >new file mode 100755 >index 0000000000..cdbe26dcb3 >--- /dev/null >+++ b/admin/oaiservers.pl >@@ -0,0 +1,122 @@ >+#!/usr/bin/perl >+ >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ >+# This script is used to maintain the OAI servers table. >+# Parameter $op is operation: list, new, edit, add_validated, delete_confirmed. >+# add_validated saves a validated record and goes to list view. >+# delete_confirmed deletes a record and goes to list view. >+ >+use Modern::Perl; >+use CGI qw ( -utf8 ); >+use C4::Context; >+use C4::Auth qw( get_template_and_user ); >+use C4::Output qw( output_html_with_http_headers ); >+use Koha::Database; >+use Koha::OaiServers; >+ >+# Initialize CGI, template, database >+ >+my $input = CGI->new; >+my $op = $input->param('op') || 'list'; >+my $id = $input->param('id') || 0; >+my $type = $input->param('type') || ''; >+my $searchfield = ''; >+ >+my ( $template, $loggedinuser, $cookie ) = get_template_and_user( >+ { >+ template_name => "admin/oaiservers.tt", >+ query => $input, >+ type => "intranet", >+ flagsrequired => { parameters => 'manage_search_targets' }, >+ } >+); >+my $script_name = "/cgi-bin/koha/admin/oaiservers.pl"; >+my $path = C4::Context->config('intrahtdocs') . "/prog/"; >+ >+$template->param( script_name => $script_name, xslt_path => $path ); >+ >+my $schema = Koha::Database->new()->schema(); >+ >+# Main code >+# First process a confirmed delete, or save a validated record >+ >+if ( $op eq 'delete_confirmed' && $id ) { >+ my $server = Koha::OaiServers->find($id); >+ if ($server) { >+ $server->delete; >+ $template->param( msg_deleted => 1, msg_add => $server->servername ); >+ } else { >+ $template->param( msg_notfound => 1, msg_add => $id ); >+ } >+ $id = 0; >+} elsif ( $op eq 'add_validated' ) { >+ my @fields = qw/endpoint oai_set dataformat >+ recordtype servername >+ add_xslt/; >+ my $formdata = _form_data_hashref( $input, \@fields ); >+ if ($id) { >+ my $server = Koha::OaiServers->find($id); >+ if ($server) { >+ $server->set($formdata)->store; >+ $template->param( msg_updated => 1, msg_add => $formdata->{servername} ); >+ } else { >+ $template->param( msg_notfound => 1, msg_add => $id ); >+ } >+ $id = 0; >+ } else { >+ Koha::OaiServer->new($formdata)->store; >+ $template->param( msg_added => 1, msg_add => $formdata->{servername} ); >+ } >+} else { >+ >+ #use searchfield only in remaining operations >+ $searchfield = $input->param('searchfield') || ''; >+} >+ >+# Now list multiple records, or edit one record >+ >+my $data = []; >+if ( $op eq 'add' || $op eq 'edit' ) { >+ $data = ServerSearch( $schema, $id, $searchfield ) if $searchfield || $id; >+ delete $data->[0]->{id} if @$data && $op eq 'add'; #cloning record >+ $template->param( >+ add_form => 1, server => @$data ? $data->[0] : undef, >+ op => $op, type => $op eq 'add' ? lc $type : '' >+ ); >+} else { >+ $data = ServerSearch( $schema, $id, $searchfield ); >+ $template->param( >+ loop => \@$data, searchfield => $searchfield, id => $id, >+ op => 'list' >+ ); >+} >+output_html_with_http_headers $input, $cookie, $template->output; >+ >+# End of main code >+ >+sub ServerSearch { #find server(s) by id or name >+ my ( $schema, $id, $searchstring ) = @_; >+ >+ return Koha::OaiServers->search( >+ $id ? { id => $id } : { servername => { like => $searchstring . '%' } }, >+ )->unblessed; >+} >+ >+sub _form_data_hashref { >+ my ( $input, $fieldref ) = @_; >+ return { map { ( $_ => scalar $input->param($_) // '' ) } @$fieldref }; >+} >diff --git a/installer/data/mysql/atomicupdate/Bug-35659.pl b/installer/data/mysql/atomicupdate/Bug-35659.pl >new file mode 100755 >index 0000000000..96721d3291 >--- /dev/null >+++ b/installer/data/mysql/atomicupdate/Bug-35659.pl >@@ -0,0 +1,85 @@ >+use Modern::Perl; >+ >+return { >+ bug_number => "35659", >+ description => "OAI-PMH Harvester", >+ up => sub { >+ my ($args) = @_; >+ my ( $dbh, $out ) = @$args{qw(dbh out)}; >+ $dbh->do( >+ q{ >+ CREATE TABLE IF NOT EXISTS `oaiservers` ( >+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'unique identifier assigned by Koha', >+ `endpoint` varchar(255) NOT NULL COMMENT 'OAI endpoint (host + port + path)', >+ `oai_set` varchar(255) DEFAULT NULL COMMENT 'OAI set to harvest', >+ `servername` longtext NOT NULL COMMENT 'name given to the target by the library', >+ `dataformat` enum('oai_dc','marc-xml', 'marcxml') NOT NULL DEFAULT 'oai_dc' COMMENT 'data format', >+ `recordtype` enum('authority','biblio') NOT NULL DEFAULT 'biblio' COMMENT 'server contains bibliographic or authority records', >+ `add_xslt` longtext DEFAULT NULL COMMENT 'zero or more paths to XSLT files to be processed on the search results', >+ PRIMARY KEY (`id`) >+ ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; >+ } >+ ); >+ say $out "Added new table 'oaiservers'"; >+ >+ $dbh->do( >+ q{ >+ CREATE TABLE IF NOT EXISTS `import_oaipmh_biblios` ( >+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'unique identifier assigned by Koha', >+ `biblionumber` int(11) NOT NULL COMMENT 'unique identifier assigned to each koha record', >+ `identifier` varchar(255) NOT NULL COMMENT 'OAI record identifier', >+ `repository` varchar(255) NOT NULL COMMENT 'OAI repository', >+ `recordtype` enum('authority','biblio') NOT NULL DEFAULT 'biblio' COMMENT 'is the record bibliographic or authority', >+ `datestamp` varchar(255) DEFAULT NULL COMMENT 'OAI set to harvest', >+ `last_modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, >+ PRIMARY KEY (id), >+ KEY biblionumber (biblionumber), >+ CONSTRAINT FK_import_oaipmh_biblios_1 FOREIGN KEY (biblionumber) REFERENCES biblio (biblionumber) ON DELETE CASCADE ON UPDATE NO ACTION >+ ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; >+ } >+ ); >+ >+ $dbh->do( >+ q{ >+ CREATE TABLE IF NOT EXISTS `import_oaipmh_authorities` ( >+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'unique identifier assigned by Koha', >+ `authid` bigint(20) unsigned NOT NULL COMMENT 'unique identifier assigned to each koha record', >+ `identifier` varchar(255) NOT NULL COMMENT 'OAI record identifier', >+ `repository` varchar(255) NOT NULL COMMENT 'OAI repository', >+ `recordtype` enum('authority','biblio') NOT NULL DEFAULT 'biblio' COMMENT 'is the record bibliographic or authority', >+ `datestamp` varchar(255) DEFAULT NULL COMMENT 'OAI set to harvest', >+ `last_modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, >+ PRIMARY KEY (id), >+ KEY authid (authid), >+ CONSTRAINT FK_import_oaipmh_authorities_1 FOREIGN KEY (authid) REFERENCES auth_header (authid) ON DELETE CASCADE ON UPDATE NO ACTION >+ ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; >+ } >+ ); >+ say $out "Added new tables 'import_oaipmh_biblios' and import_oaipmh_authorities"; >+ >+ $dbh->do( >+ q{ >+ UPDATE `permissions` SET description='Manage Z39.50 and SRU servers, OAI repositories configuration' WHERE code='manage_search_targets'; >+ } >+ ); >+ say $out "Updated manage_search_targets permission description"; >+ >+ $dbh->do( >+ q{ >+ INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type) VALUES >+ ('OAI-PMH:HarvestEmailReport','','','If set, a report email will be sent to this address after an OAI harvest','Free'); >+ } >+ ); >+ say $out "Added OAI-PMH:HarvestEmailReport system preference"; >+ >+ $dbh->do( >+ q{ >+ INSERT IGNORE INTO letter >+ (module, code, branchcode, name, is_html, title, content, message_transport_type, lang) >+ VALUES ('catalogue','OAI_HARVEST_REPORT','','OAI Harvest Report',0,'OAI Harvest report for [% servername %]','OAI Harvest report for [% servername %]:\n\nEndpoint: [% endpoint %]\nSet: [% set %]\nData format: [% dataformat %]\nRecord type: [% recordtype %]\n\n[% added %] records added\n[% updated %] records updated\n[% deleted %] records deleted\n[% skipped %] records skipped\n[% in_error %] records in error\n[% total %] total','email','default'); >+ } >+ ); >+ say $out "Added OAI_HARVEST_REPORT letter"; >+ >+ }, >+}; >diff --git a/installer/data/mysql/en/mandatory/sample_notices.yml b/installer/data/mysql/en/mandatory/sample_notices.yml >index ec0e2386da..5a04e0ed3d 100644 >--- a/installer/data/mysql/en/mandatory/sample_notices.yml >+++ b/installer/data/mysql/en/mandatory/sample_notices.yml >@@ -2535,3 +2535,26 @@ tables: > - "Your hold on [% hold.biblio.title %] ([% hold.biblio.id %]) has been confirmed." > - "You will be notified by the library when your item is available for pickup." > - "Thank you!" >+ >+ - module: catalogue >+ code: OAI_HARVEST_REPORT >+ branchcode: "" >+ name: "OAI Harvest Report" >+ is_html: 0 >+ title: "OAI Harvest report for [% servername %]" >+ message_transport_type: email >+ lang: default >+ content: >+ - "OAI Harvest report for [% servername %]:" >+ - "" >+ - "Endpoint: [% endpoint %]" >+ - "Set: [% set %]" >+ - "Data format: [% dataformat %]" >+ - "Record type: [% recordtype %]" >+ - "" >+ - "[% added %] records added" >+ - "[% updated %] records updated" >+ - "[% deleted %] records deleted" >+ - "[% skipped %] records skipped" >+ - "[% in_error %] records in error" >+ - "[% total %] total" >diff --git a/installer/data/mysql/kohastructure.sql b/installer/data/mysql/kohastructure.sql >index 31f0828a24..21a589e8c0 100644 >--- a/installer/data/mysql/kohastructure.sql >+++ b/installer/data/mysql/kohastructure.sql >@@ -4856,6 +4856,64 @@ CREATE TABLE `oai_sets_mappings` ( > ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > /*!40101 SET character_set_client = @saved_cs_client */; > >+-- >+-- Table structure for table `oaiservers` >+-- >+ >+DROP TABLE IF EXISTS `oaiservers`; >+/*!40101 SET @saved_cs_client = @@character_set_client */; >+/*!40101 SET character_set_client = utf8 */; >+CREATE TABLE `oaiservers` ( >+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'unique identifier assigned by Koha', >+ `endpoint` varchar(255) NOT NULL COMMENT 'OAI endpoint (host + port + path)', >+ `oai_set` varchar(255) DEFAULT NULL COMMENT 'OAI set to harvest', >+ `servername` longtext NOT NULL COMMENT 'name given to the target by the library', >+ `dataformat` enum('oai_dc','marc-xml', 'marcxml') NOT NULL DEFAULT 'oai_dc' COMMENT 'data format', >+ `recordtype` enum('authority','biblio') NOT NULL DEFAULT 'biblio' COMMENT 'server contains bibliographic or authority records', >+ `add_xslt` longtext DEFAULT NULL COMMENT 'zero or more paths to XSLT files to be processed on the search results', >+ PRIMARY KEY (`id`) >+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; >+ >+-- >+-- Table structure for table `import_oaipmh_biblios` >+-- >+ >+DROP TABLE IF EXISTS `import_oaipmh_biblios`; >+/*!40101 SET @saved_cs_client = @@character_set_client */; >+/*!40101 SET character_set_client = utf8 */; >+CREATE TABLE `import_oaipmh_biblios` ( >+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'unique identifier assigned by Koha', >+ `biblionumber` int(11) NOT NULL COMMENT 'unique identifier assigned to each koha record', >+ `identifier` varchar(255) NOT NULL COMMENT 'OAI record identifier', >+ `repository` varchar(255) NOT NULL COMMENT 'OAI repository', >+ `recordtype` enum('authority','biblio') NOT NULL DEFAULT 'biblio' COMMENT 'is the record bibliographic or authority', >+ `datestamp` varchar(255) DEFAULT NULL COMMENT 'OAI set to harvest', >+ `last_modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, >+ PRIMARY KEY (id), >+ KEY biblionumber (biblionumber), >+ CONSTRAINT FK_import_oaipmh_biblios_1 FOREIGN KEY (biblionumber) REFERENCES biblio (biblionumber) ON DELETE CASCADE ON UPDATE NO ACTION >+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; >+ >+-- >+-- Table structure for table `import_oaipmh_authorities` >+-- >+ >+DROP TABLE IF EXISTS `import_oaipmh_authorities`; >+/*!40101 SET @saved_cs_client = @@character_set_client */; >+/*!40101 SET character_set_client = utf8 */; >+CREATE TABLE `import_oaipmh_authorities` ( >+ `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'unique identifier assigned by Koha', >+ `authid` bigint(20) unsigned NOT NULL COMMENT 'unique identifier assigned to each koha record', >+ `identifier` varchar(255) NOT NULL COMMENT 'OAI record identifier', >+ `repository` varchar(255) NOT NULL COMMENT 'OAI repository', >+ `recordtype` enum('authority','biblio') NOT NULL DEFAULT 'biblio' COMMENT 'is the record bibliographic or authority', >+ `datestamp` varchar(255) DEFAULT NULL COMMENT 'OAI set to harvest', >+ `last_modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, >+ PRIMARY KEY (id), >+ KEY authid (authid), >+ CONSTRAINT FK_import_oaipmh_authorities_1 FOREIGN KEY (authid) REFERENCES auth_header (authid) ON DELETE CASCADE ON UPDATE NO ACTION >+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; >+ > -- > -- Table structure for table `oauth_access_tokens` > -- >diff --git a/installer/data/mysql/mandatory/sysprefs.sql b/installer/data/mysql/mandatory/sysprefs.sql >index 47f81a864e..903600a07d 100644 >--- a/installer/data/mysql/mandatory/sysprefs.sql >+++ b/installer/data/mysql/mandatory/sysprefs.sql >@@ -447,6 +447,7 @@ INSERT INTO systempreferences ( `variable`, `value`, `options`, `explanation`, ` > ('OAI-PMH:AutoUpdateSetsEmbedItemData', '0', '', 'Embed item information when automatically updating OAI sets. Requires OAI-PMH:AutoUpdateSets syspref to be enabled', 'YesNo'), > ('OAI-PMH:ConfFile','',NULL,'If empty, Koha OAI Server operates in normal mode, otherwise it operates in extended mode.','File'), > ('OAI-PMH:DeletedRecord','persistent','Koha\'s deletedbiblio table will never be deleted (persistent), might be deleted (transient), or will never have any data in it (no)','transient|persistent|no','Choice'), >+('OAI-PMH:HarvestEmailReport','','','If set, a report email will be sent to this address after an OAI harvest','Free'), > ('OAI-PMH:MaxCount','50',NULL,'OAI-PMH maximum number of records by answer to ListRecords and ListIdentifiers queries','Integer'), > ('OnSiteCheckoutAutoCheck','0','','Enable/Do not enable onsite checkout by default if last checkout was an onsite checkout','YesNo'), > ('OnSiteCheckouts','0','','Enable/Disable the on-site checkouts feature','YesNo'), >diff --git a/installer/data/mysql/mandatory/userpermissions.sql b/installer/data/mysql/mandatory/userpermissions.sql >index 83808b97a2..80bf9dc2d3 100644 >--- a/installer/data/mysql/mandatory/userpermissions.sql >+++ b/installer/data/mysql/mandatory/userpermissions.sql >@@ -29,7 +29,7 @@ INSERT INTO permissions (module_bit, code, description) VALUES > ( 3, 'manage_item_search_fields', 'Manage item search fields'), > ( 3, 'manage_search_engine_config', 'Manage search engine configuration'), > ( 3, 'manage_marc_overlay_rules', 'Manage MARC overlay rules configuration'), >- ( 3, 'manage_search_targets', 'Manage Z39.50 and SRU server configuration'), >+ ( 3, 'manage_search_targets', 'Manage Z39.50 and SRU servers, OAI repositories configuration'), > ( 3, 'manage_didyoumean', 'Manage Did you mean? configuration'), > ( 3, 'manage_column_config', 'Manage column configuration'), > ( 3, 'manage_sms_providers', 'Manage SMS cellular providers'), >diff --git a/koha-tmpl/intranet-tmpl/prog/en/includes/admin-menu.inc b/koha-tmpl/intranet-tmpl/prog/en/includes/admin-menu.inc >index 3aa5b87b95..5f5e089d1c 100644 >--- a/koha-tmpl/intranet-tmpl/prog/en/includes/admin-menu.inc >+++ b/koha-tmpl/intranet-tmpl/prog/en/includes/admin-menu.inc >@@ -152,6 +152,7 @@ > [% END %] > [% IF ( CAN_user_parameters_manage_search_targets ) %] > <li><a href="/cgi-bin/koha/admin/z3950servers.pl">Z39.50/SRU servers</a></li> >+ <li><a href="/cgi-bin/koha/admin/oaiservers.pl">OAI repositories</a></li> > [% END %] > [% IF ( CAN_user_parameters_manage_smtp_servers ) %] > <li><a href="/cgi-bin/koha/admin/smtp_servers.pl">SMTP servers</a></li> >diff --git a/koha-tmpl/intranet-tmpl/prog/en/includes/oai-admin-search.inc b/koha-tmpl/intranet-tmpl/prog/en/includes/oai-admin-search.inc >new file mode 100644 >index 0000000000..49feedd580 >--- /dev/null >+++ b/koha-tmpl/intranet-tmpl/prog/en/includes/oai-admin-search.inc >@@ -0,0 +1,43 @@ >+[% USE Koha %] >+<!-- oai-admin-search.inc --> >+<div id="header_search" role="tablist"> >+ <div class="tab-content"> >+ <div id="oai_search" role="tabpanel" class="tab-pane active"> >+ <form action="/cgi-bin/koha/admin/oaiservers.pl" method="post"> >+ <div class="form-title"> >+ <label class="control-label">Search OAI repositories</label> >+ </div> >+ >+ <div class="form-content"> >+ <input class="head-searchbox form-control" type="text" name="searchfield" id="searchfield" value="[% searchfield | html %]" placeholder="OAI repository search" /> >+ </div> >+ >+ <button type="submit"><i class="fa fa-arrow-right"></i></button> >+ </form> >+ </div> >+ >+ [% IF ( CAN_user_circulate_circulate_remaining_permissions ) %] >+ [% INCLUDE 'patron-search-box.inc' %] >+ [% END %] >+ [% IF ( CAN_user_catalogue ) %] >+ [% INCLUDE 'catalogue-search-box.inc' %] >+ [% END %] >+ >+ </div><!-- /.tab-content --> >+ <ul class="nav nav-tabs" role="tablist"> >+ <li role="presentation" class="active"> >+ <a title="Search OAI repositories" href="#oai_search" aria-controls="z3950_search" role="tab" aria-expanded="true" data-toggle="tab" class="keep_text"><i class="fa fa-search"></i> <span class="tab-title">Search OAI repositories</span></a> >+ </li> >+ [% IF ( CAN_user_circulate_circulate_remaining_permissions ) %] >+ <li role="presentation"> >+ <a title="Check out" href="#circ_search" aria-controls="circ_search" role="tab" data-toggle="tab" class="keep_text"><i class="fa fa-upload"></i> <span class="tab-title">Check out</span></a> >+ </li> >+ [% END %] >+ [% IF ( CAN_user_catalogue ) %] >+ <li role="presentation"> >+ <a title="Search catalog" href="#catalog_search" aria-controls="catalog_search" role="tab" data-toggle="tab" class="keep_text"><i class="fa fa-fw fa-search"></i> <span class="tab-title">Search catalog</span></a> >+ </li> >+ [% END %] >+ </ul> >+</div><!-- /#header_search --> >+<!-- /oai-admin-search.inc --> >diff --git a/koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc b/koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc >index f5396703e7..b976924041 100644 >--- a/koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc >+++ b/koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc >@@ -244,7 +244,7 @@ > <span class="permissioncode">([% name | html %])</span> > [%- CASE 'manage_search_targets' -%] > <span class="sub_permission manage_search_targets_subpermission"> >- Manage Z39.50 and SRU server configuration >+ Manage Z39.50 and SRU servers, OAI repositories configuration > </span> > <span class="permissioncode">([% name | html %])</span> > [%- CASE 'manage_didyoumean' -%] >diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt >index 554b4dc1b9..735122e2d0 100644 >--- a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt >+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt >@@ -261,6 +261,8 @@ > [% IF ( CAN_user_parameters_manage_search_targets ) %] > <dt><a href="/cgi-bin/koha/admin/z3950servers.pl">Z39.50/SRU servers</a></dt> > <dd>Define which external servers to query for MARC data</dd> >+ <dt><a href="/cgi-bin/koha/admin/oaiservers.pl">OAI repositories</a></dt> >+ <dd>Define which OAI repositories to harvest data from</dd> > [% END %] > [% IF ( CAN_user_parameters_manage_smtp_servers ) %] > <dt><a href="/cgi-bin/koha/admin/smtp_servers.pl">SMTP servers</a></dt> >diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt >new file mode 100644 >index 0000000000..7f95726ab9 >--- /dev/null >+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt >@@ -0,0 +1,229 @@ >+[% USE raw %] >+[% USE Asset %] >+[% SET footerjs = 1 %] >+[% INCLUDE 'doc-head-open.inc' %] >+ >+<title> >+ [% IF op == 'edit' %] >+ Modify OAI repository [% server.servername | html %] › [% END %] >+ >+ [% IF op == 'add' %] >+ New OAI repository › [% END %] >+ >+ OAI repositories › Administration › Koha >+</title> >+[% INCLUDE 'doc-head-close.inc' %] >+</head> >+ >+<body id="admin_oaiservers" class="admin"> >+[% WRAPPER 'header.inc' %] >+ [% INCLUDE 'oai-admin-search.inc' %] >+[% END %] >+ >+[% WRAPPER 'sub-header.inc' %] >+ [% WRAPPER breadcrumbs %] >+ [% WRAPPER breadcrumb_item %] >+ <a href="/cgi-bin/koha/admin/admin-home.pl">Administration</a> >+ [% END %] >+ >+ [% IF op == 'edit' || op == 'add' %] >+ [% WRAPPER breadcrumb_item %] >+ <a href="/cgi-bin/koha/admin/oaiservers.pl">OAI repositories</a> >+ [% END %] >+ [% END %] >+ >+ [% IF op == 'edit' %] >+ [% WRAPPER breadcrumb_item bc_active= 1 %] >+ <span>Modify OAI repository [% server.servername | html %]</span> >+ [% END %] >+ >+ [% ELSIF op == 'add' %] >+ [% WRAPPER breadcrumb_item bc_active= 1 %] >+ <span>New OAI repository</span> >+ [% END %] >+ >+ [% ELSE %] >+ [% WRAPPER breadcrumb_item bc_active= 1 %] >+ <span>OAI repositories</span> >+ [% END %] >+ [% END %] >+ [% END #/ WRAPPER breadcrumbs %] >+[% END #/ WRAPPER sub-header.inc %] >+ >+<div class="main container-fluid"> >+ <div class="row"> >+ <div class="col-sm-10 col-sm-push-2"> >+ <main> >+ >+[% IF msg_deleted %] >+ <div class="dialog alert">OAI repository deleted ([% msg_add | html %])</div> >+[% ELSIF msg_updated %] >+ <div class="dialog message">OAI repository updated ([% msg_add | html %])</div> >+[% ELSIF msg_added %] >+ <div class="dialog message">OAI repository added ([% msg_add | html %])</div> >+[% ELSIF msg_notfound %] >+ <div class="dialog alert">Error: Server with ID [% msg_add | html %] not found</div> >+[% END %] >+ >+[% IF ( add_form ) %] >+ <form action="[% script_name | html %]" name="Aform" method="post" id="serverentry"> >+ <input type="hidden" name="op" value="add_validated" /> >+ [% IF op == 'edit' %] >+ <h1>Modify OAI repository</h1> >+ <input type="hidden" name="id" value="[% server.id | html %]" /> >+ [% ELSE %] >+ <h1>New OAI repository</h1> >+ [% END %] >+ <fieldset class="rows"> >+ <ol> >+ <li><label for="name" class="required">Repository name: </label> >+ [% IF server.servername %] >+ <input type="text" name="servername" id="servername" size="65" maxlength="100" value="[% server.servername | html %]" required="required" /> >+ [% ELSE %] >+ <input type="text" name="servername" id="servername" class="focus" size="65" maxlength="100" value="[% server.servername | html %]" required="required" /> >+ [% END %] >+ <span class="required">Required</span> >+ </li> >+ >+ <li><label for="endpoint" class="required">Endpoint: </label> <input type="text" name="endpoint" id="endpoint" size="30" value="[% server.endpoint | html %]" required="required" /> <span class="required">Required</span> >+ </li> >+ <li><label for="oai_set">Set: </label> <input type="text" name="oai_set" id="oai_set" value="[% server.oai_set | html %]" /> >+ </li> >+ <li> >+ <label for="dataformat">Data format: </label> >+ <select name="dataformat" id="dataformat"> >+ <option value="oai_dc">Dublin Core (oai_dc)</option> >+ <option value="marc-xml">Marcxml (marc-xml)</option> >+ <option value="marcxml">Marcxml (marcxml)</option> >+ </select> >+ <div class="hint">OAI can send records in various formats. Choose one.</div> >+ </li> >+ >+ <li><label for="recordtype">Record type: </label> >+ <select name="recordtype" id="recordtype"> >+ <option value="biblio">Bibliographic</option> >+ <option value="authority">Authority</option> >+ </select> >+ </li> >+ <li> >+ <label for="add_xslt">XSLT file for transforming results: </label> >+ <input type="text" name="add_xslt" id="add_xslt" size="100" placeholder="[% xslt_path | html %]" value="[% server.add_xslt | html %]"/> >+ </li> >+ >+ </ol> >+ </fieldset> >+ >+ <fieldset class="action"><input type="submit" class="btn btn-primary" value="Save" /> <a class="cancel" href="/cgi-bin/koha/admin/oaiservers.pl">Cancel</a></fieldset> >+ </form> >+[% END %] >+ >+[% IF op == 'list' %] >+ <div id="toolbar" class="btn-toolbar"> >+ <a id="newserver" class="btn btn-default" href="/cgi-bin/koha/admin/oaiservers.pl?op=add"><i class="fa fa-plus"></i> New OAI repository</a> >+ </div> >+ <h1>OAI repositories administration</h1> >+ [% IF id %] >+ You searched for record [% id | html %] >+ [% ELSIF searchfield %] >+ You searched for [% searchfield | html %] >+ [% END %] >+ >+ <div class="page-section"> >+ <table id="serverst"> >+ >+ <thead><tr><th>Id</th><th>Repository name</th><th>Endpoint</th><th>Set</th><th>Data format</th><th>Record type</th><th class="noExport">Actions</th> >+ </tr></thead> >+ <tbody> >+ [% FOREACH loo IN loop %] >+ <tr> >+ <td>[% loo.id | html %]</td> >+ <td><a href="/cgi-bin/koha/admin/oaiservers.pl?op=edit&id=[% loo.id | uri %]">[% loo.servername | html %]</a></td><td>[% loo.endpoint | html %]</td><td>[% loo.oai_set | html %]</td> >+ <td>[% loo.dataformat | html %]</td> >+ <td>[% IF ( loo.recordtype == 'biblio' ) %] >+ <span>Bibliographic</span> >+ [% ELSIF ( loo.recordtype == 'authority' ) %] >+ <span>Authority</span> >+ [% END %] >+ </td> >+ <td> >+ <div class="btn-group dropup"> >+ <a class="btn btn-default btn-xs dropdown-toggle" id="reportactions[% savedreport.id | html %]" role="button" data-toggle="dropdown" href="#"> >+ Actions <b class="caret"></b> >+ </a> >+ <ul class="dropdown-menu pull-right" role="menu" aria-labelledby="reportactions[% savedreport.id | html %]"> >+ <li><a href="/cgi-bin/koha/admin/oaiservers.pl?op=edit&id=[% loo.id | uri %]"><i class="fa-solid fa-pencil" aria-hidden="true"></i> Edit</a></li> >+ <li><a href="/cgi-bin/koha/admin/oaiservers.pl?op=add&id=[% loo.id | uri %]"><i class="fa fa-copy"></i> Copy</a></li> >+ <li><a href="/cgi-bin/koha/admin/oaiservers.pl?op=delete_confirmed&id=[% loo.id | uri %]" class="delete" data-servername="[% loo.servername | html %]"><i class="fa fa-trash-can"></i> Delete</a></li> >+ </ul> >+ </div> >+ </td> >+ </tr> >+ [% END %] >+ </tbody> >+ </table> >+ </div> <!-- /.page-section --> >+[% END %] >+ >+ </main> >+ </div> <!-- /.col-sm-10.col-sm-push-2 --> >+ >+ <div class="col-sm-2 col-sm-pull-10"> >+ <aside> >+ [% INCLUDE 'admin-menu.inc' %] >+ </aside> >+ </div> <!-- /.col-sm-2.col-sm-pull-10 --> >+ </div> <!-- /.row --> >+ >+[% MACRO jsinclude BLOCK %] >+ [% Asset.js("js/admin-menu.js") | $raw %] >+ [% IF op == 'list' %] >+ [% INCLUDE 'datatables.inc' %] >+ [% END %] >+ >+ <script> >+ [% IF ( add_form ) %] >+ $(document).ready(function(){ >+ // Update selects for dataformat and recordtype >+ [% IF server %] >+ $("#dataformat").val('[% server.dataformat | html %]'); >+ $("#recordtype").val('[% server.recordtype | html %]'); >+ [% END %] >+ $( "#serverentry" ).validate({ >+ rules: { >+ servername: { required: true }, >+ host: { required: true }, >+ port: { >+ number: true >+ }, >+ rank: { number: true }, >+ } >+ }); >+ $("#serverentry").submit(function( event ) { >+ // enable recordtype to include field in post >+ $('#recordtype').prop('disabled',false); >+ }); >+ $("#servername").on("blur",function(){ >+ toUC(this); >+ }); >+ }); >+ [% ELSE %] >+ $(document).ready(function() { >+ $("#serverst").dataTable($.extend(true, {}, dataTablesDefaults, { >+ "aoColumnDefs": [ >+ { "aTargets": [-1], "bSortable": false, "bSearchable": false }, >+ ], >+ "sPaginationType": "full" >+ })); >+ $(".delete").on("click",function(e){ >+ var servername = $(this).data("servername"); >+ if( confirm( _("Are you sure you want to delete repository %s?").format(servername) ) ) { >+ return true; >+ } else { >+ e.preventDefault(); >+ } >+ }); >+ }); >+ [% END %] >+ </script> >+[% END %] >+[% INCLUDE 'intranet-bottom.inc' %] >diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/web_services.pref b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/web_services.pref >index d5c7c1feb2..da14efec0e 100644 >--- a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/web_services.pref >+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/web_services.pref >@@ -91,6 +91,11 @@ Web services: > transient: might be emptied or truncated at some point (transient) > no: will never have any data in it (no) > - "." >+ - >+ - If set, a report email will be sent to this address after an OAI harvest >+ - pref: "OAI-PMH:HarvestEmailReport" >+ class: Text >+ > ILS-DI: > - > - pref: ILS-DI >diff --git a/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl b/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl >new file mode 100644 >index 0000000000..e2dbe4d4d6 >--- /dev/null >+++ b/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl >@@ -0,0 +1,35 @@ >+<?xml version="1.0" encoding="UTF-8"?> >+<xsl:stylesheet version="1.0" >+ xmlns:dc="http://purl.org/dc/elements/1.1/" >+ xmlns:dcterms="http://purl.org/dc/terms/1.1" >+ xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" >+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >+ xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ >+ http://www.openarchives.org/OAI/2.0/oai_dc.xsd" >+ xmlns:xsl="http://www.w3.org/1999/XSL/Transform" >+ xmlns="http://www.loc.gov/MARC21/slim" exclude-result-prefixes="dc dcterms oai_dc"> >+ >+ <xsl:output method="xml" encoding="UTF-8" indent="yes"/> >+ >+ <xsl:template match="/"> >+ <collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" > >+ <xsl:apply-templates /> >+ </collection> >+ </xsl:template> >+ >+ <xsl:template match="oai_dc:dc"> >+ <record> >+ >+ <xsl:variable name="FamilyName" select="dc:FamilyName"/> >+ <xsl:variable name="GivenName" select="dc:GivenName"/> >+ >+ <datafield tag="100" ind1="0" ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="concat($FamilyName,', ',$GivenName)"/> >+ </subfield> >+ </datafield> >+ >+ </record> >+ </xsl:template> >+ >+</xsl:stylesheet> >diff --git a/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl b/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl >new file mode 100644 >index 0000000000..9dcd285c96 >--- /dev/null >+++ b/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl >@@ -0,0 +1,42 @@ >+<?xml version="1.0" encoding="UTF-8"?> >+<xsl:stylesheet version="1.0" >+ xmlns:dc="http://purl.org/dc/elements/1.1/" >+ xmlns:dcterms="http://purl.org/dc/terms/1.1" >+ xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" >+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >+ xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ >+ http://www.openarchives.org/OAI/2.0/oai_dc.xsd" >+ xmlns:xsl="http://www.w3.org/1999/XSL/Transform" >+ xmlns="http://www.loc.gov/MARC21/slim" exclude-result-prefixes="dc dcterms oai_dc"> >+ >+ <xsl:output method="xml" encoding="UTF-8" indent="yes"/> >+ >+ <xsl:template match="/"> >+ <collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" > >+ <xsl:apply-templates /> >+ </collection> >+ </xsl:template> >+ >+ <xsl:template match="oai_dc:dc"> >+ <record> >+ >+ <xsl:for-each select="dc:FamilyName"> >+ <datafield tag="200" ind1="0" ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:GivenName"> >+ <datafield tag="200" ind1="0" ind2=" "> >+ <subfield code="b"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ </record> >+ </xsl:template> >+ >+</xsl:stylesheet> >diff --git a/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl b/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl >new file mode 100644 >index 0000000000..4e3d13a436 >--- /dev/null >+++ b/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl >@@ -0,0 +1,367 @@ >+<?xml version="1.0" encoding="UTF-8"?> >+<xsl:stylesheet version="1.0" >+ xmlns:dc="http://purl.org/dc/elements/1.1/" >+ xmlns:dcterms="http://purl.org/dc/terms/1.1" >+ xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" >+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >+ xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ >+ http://www.openarchives.org/OAI/2.0/oai_dc.xsd" >+ xmlns:xsl="http://www.w3.org/1999/XSL/Transform" >+ xmlns="http://www.loc.gov/MARC21/slim" exclude-result-prefixes="dc dcterms oai_dc"> >+ >+ <xsl:import href="MARC21slimUtils.xsl"/> >+ <xsl:output method="xml" encoding="UTF-8" indent="yes"/> >+ >+ >+ <xsl:template match="/"> >+ <collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" > >+ <xsl:apply-templates /> >+ </collection> >+ </xsl:template> >+ >+ <xsl:template name="OAI-PMH"> >+ <xsl:for-each select = "ListRecords/record/metadata/oai_dc:dc"> >+ <xsl:apply-templates /> >+ </xsl:for-each> >+ <xsl:for-each select = "GetRecord/record/metadata/oai_dc:dc"> >+ <xsl:apply-templates /> >+ </xsl:for-each> >+ </xsl:template> >+ >+ <xsl:template match="text()" /> >+ <xsl:template match="oai_dc:dc"> >+ <record> >+ <xsl:element name="leader"> >+ <xsl:variable name="type" select="dc:type"/> >+ <xsl:variable name="leader06"> >+ <xsl:choose> >+ <xsl:when test="$type='collection'">p</xsl:when> >+ <xsl:when test="$type='dataset'">m</xsl:when> >+ <xsl:when test="$type='event'">r</xsl:when> >+ <xsl:when test="$type='image'">k</xsl:when> >+ <xsl:when test="$type='interactive resource'">m</xsl:when> >+ <xsl:when test="$type='service'">m</xsl:when> >+ <xsl:when test="$type='software'">m</xsl:when> >+ <xsl:when test="$type='sound'">i</xsl:when> >+ <xsl:when test="$type='text'">a</xsl:when> >+ <xsl:otherwise>a</xsl:otherwise> >+ </xsl:choose> >+ </xsl:variable> >+ <xsl:variable name="leader07"> >+ <xsl:choose> >+ <xsl:when test="$type='collection'">c</xsl:when> >+ <xsl:otherwise>m</xsl:otherwise> >+ </xsl:choose> >+ </xsl:variable> >+ <xsl:value-of select="concat(' ',$leader06,$leader07,' 3u ')"/> >+ </xsl:element> >+ >+ <datafield tag="042" ind1=" " ind2=" "> >+ <subfield code="a">dc</subfield> >+ </datafield> >+ >+ >+ >+ <xsl:for-each select="dc:creator"> >+ <xsl:choose> >+ <xsl:when test="(.!='') and (position()=1)"> >+ <xsl:call-template name="persname_template"> >+ <xsl:with-param name="string" select="." /> >+ <xsl:with-param name="field" select="'100'" /> >+ <xsl:with-param name="ind1" select = "'1'" /> >+ <xsl:with-param name="ind2" select = "'0'" /> >+ <xsl:with-param name="type" select="'author'" /> >+ </xsl:call-template> >+ </xsl:when> >+ <xsl:otherwise> >+ <xsl:if test=".!=''"> >+ <xsl:call-template name="persname_template"> >+ <xsl:with-param name="string" select="." /> >+ <xsl:with-param name="field" select="'700'" /> >+ <xsl:with-param name="ind1" select = "'1'" /> >+ <xsl:with-param name="ind2" select = "'0'" /> >+ <xsl:with-param name="type" select="'author'" /> >+ </xsl:call-template> >+ </xsl:if> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:title[1]"> >+ <datafield tag="245" ind1="0" ind2="0"> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:title[position()>1]"> >+ <xsl:if test=".!=''"> >+ <datafield tag="246" ind1="3" ind2="3"> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:if> >+ </xsl:for-each> >+ >+ <xsl:choose> >+ <xsl:when test="dc:publisher"> >+ <xsl:if test="translate(dc:publisher/.,'.,:;','')!=''"> >+ <datafield tag="260" ind1=" " ind2=" "> >+ <xsl:choose> >+ <xsl:when test="dc:date"> >+ <subfield code="b"><xsl:value-of select="dc:publisher[1]"/>, </subfield> >+ <xsl:if test="translate(dc:date[1]/., '.,:;','')!=''"> >+ <subfield code="c"><xsl:value-of select="dc:date[1]" />.</subfield> >+ </xsl:if> >+ </xsl:when> >+ <xsl:otherwise> >+ <subfield code="b"><xsl:value-of select="dc:publisher[1]"/>.</subfield> >+ </xsl:otherwise> >+ </xsl:choose> >+ </datafield> >+ </xsl:if> >+ </xsl:when> >+ <xsl:otherwise> >+ <xsl:if test="translate(dc:date[1],'.,:;','')!=''"> >+ <datafield tag="260" ind1=" " ind2=" "> >+ <subfield code="c"><xsl:value-of select="dc:date[1]" />.</subfield> >+ </datafield> >+ </xsl:if> >+ </xsl:otherwise> >+ </xsl:choose> >+ >+ <xsl:for-each select="dc:coverage"> >+ <xsl:choose> >+ <xsl:when test="translate(., '0123456789-.?','')=''"> >+ <!--Likely;this is a date--> >+ <datafield tag="500" ind1=" " ind2=" "> >+ <subfield code="a"><xsl:value-of select="."/></subfield> >+ </datafield> >+ </xsl:when> >+ <xsl:otherwise> >+ <!--likely a geographic subject, we will print this later--> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:identifier"> >+ <xsl:if test="position()!=last()"> >+ <datafield tag="500" ind1=" " ind2=" "> >+ <subfield code="a"><xsl:value-of select="." /></subfield> >+ </datafield> >+ </xsl:if> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:description"> >+ <datafield tag="520" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="normalize-space(.)"/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:rights"> >+ <datafield tag="540" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:language"> >+ <datafield tag="546" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ >+ <xsl:for-each select="dc:subject"> >+ <datafield tag="690" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:coverage"> >+ <xsl:choose> >+ <xsl:when test="translate(., '0123456789-.?','')=''"> >+ <!--Likely; this is a date--> >+ </xsl:when> >+ <xsl:otherwise> >+ <!--likely a geographic subject--> >+ <datafield tag="691" ind1=" " ind2=" "> >+ <subfield code="a"><xsl:value-of select="." /></subfield> >+ </datafield> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:type"> >+ <datafield tag="655" ind1="7" ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ <subfield code="2">local</subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ >+ <xsl:for-each select="dc:contributer"> >+ <xsl:call-template name="persname_template"> >+ <xsl:with-param name="string" select="." /> >+ <xsl:with-param name="field" select="'100'" /> >+ <xsl:with-param name="ind1" select = "'1'" /> >+ <xsl:with-param name="ind2" select = "'0'" /> >+ <xsl:with-param name="type" select="'contributor'" /> >+ </xsl:call-template> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:source"> >+ <datafield tag="786" ind1="0" ind2=" "> >+ <subfield code="n"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:relation"> >+ <datafield tag="787" ind1="0" ind2=" "> >+ <subfield code="n"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <xsl:if test="dc:identifier"> >+ <datafield tag="856" ind1="4" ind2="1"> >+ <subfield code="u"><xsl:value-of select="dc:identifier[last()]" /></subfield> >+ <subfield code="z">Connect to this object online.</subfield> >+ </datafield> >+ </xsl:if> >+ </record> >+ >+ </xsl:template> >+<xsl:template name="persname_template"> >+ <xsl:param name="string" /> >+ <xsl:param name="field" /> >+ <xsl:param name="ind1" /> >+ <xsl:param name="ind2" /> >+ <xsl:param name="type" /> >+ <datafield> >+ <xsl:attribute name="tag"> >+ <xsl:value-of select="$field" /> >+ </xsl:attribute> >+ <xsl:attribute name="ind1"> >+ <xsl:value-of select="$ind1" /> >+ </xsl:attribute> >+ <xsl:attribute name="ind2"> >+ <xsl:value-of select="$ind2" /> >+ </xsl:attribute> >+ >+ <!-- Sample input: Brightman, Samuel C. (Samuel Charles), 1911-1992 --> >+ <!-- Sample output: $aBrightman, Samuel C. $q(Samuel Charles), $d1911-. --> >+ <!-- will handle names with dashes e.g. Bourke-White, Margaret --> >+ >+ <!-- CAPTURE PRIMARY NAME BY LOOKING FOR A PAREN OR A DASH OR NEITHER --> >+ <xsl:choose> >+ <!-- IF A PAREN, STOP AT AN OPENING PAREN --> >+ <xsl:when test="contains($string, '(')!=0"> >+ <subfield code="a"> >+ <xsl:value-of select="substring-before($string, '(')" /> >+ </subfield> >+ </xsl:when> >+ <!-- IF A DASH, CHECK IF IT'S A DATE OR PART OF THE NAME --> >+ <xsl:when test="contains($string, '-')!=0"> >+ <xsl:variable name="name_1" select="substring-before($string, '-')" /> >+ <xsl:choose> >+ <!-- IF IT'S A DATE REMOVE IT --> >+ <xsl:when test="translate(substring($name_1, (string-length($name_1)), 1), '0123456789', '9999999999') = '9'"> >+ <xsl:variable name="name" select="substring($name_1, 1, (string-length($name_1)-6))" /> >+ <subfield code="a"> >+ <xsl:value-of select="$name" /> >+ </subfield> >+ </xsl:when> >+ <!-- IF IT'S NOT A DATE, CHECK WHETHER THERE IS A DATE LATER --> >+ <xsl:otherwise> >+ <xsl:variable name="remainder" select="substring-after($string, '-')" /> >+ <xsl:choose> >+ <!-- IF THERE'S A DASH, ASSUME IT'S A DATE AND REMOVE IT --> >+ <xsl:when test="contains($remainder, '-')!=0"> >+ <xsl:variable name="tmp" select="substring-before($remainder, '-')" /> >+ <xsl:variable name="name_2" select="substring($tmp, 1, (string-length($tmp)-6))" /> >+ <subfield code="a"> >+ <xsl:value-of select="$name_1" />-<xsl:value-of select="$name_2" /> >+ </subfield> >+ </xsl:when> >+ <!-- IF THERE'S NO DASH IN THE REMAINDER, OUTPUT IT --> >+ <xsl:otherwise> >+ <subfield code="a"> >+ <xsl:value-of select="$string" /> >+ </subfield> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:when> >+ <!-- NO DASHES, NO PARENS, JUST OUTPUT THE NAME --> >+ <xsl:otherwise> >+ <subfield code="a"> >+ <xsl:value-of select="$string" /> >+ </subfield> >+ </xsl:otherwise> >+ </xsl:choose> >+ >+ <!-- CAPTURE SECONDARY NAME IN PARENS FOR SUBFIELD Q --> >+ <xsl:if test="contains($string, '(')!=0"> >+ <xsl:variable name="subq_tmp" select="substring-after($string, '(')" /> >+ <xsl:variable name="subq" select="substring-before($subq_tmp, ')')" /> >+ <subfield code="q"> >+ (<xsl:value-of select="$subq" />) >+ </subfield> >+ </xsl:if> >+ >+ <!-- CAPTURE DATE FOR SUBFIELD D, ASSUME DATE IS LAST ITEM IN FIELD --> >+ <!-- Note: does not work if name has a dash in it --> >+ <xsl:if test="contains($string, '-')!=0"> >+ <xsl:variable name="date_tmp" select="substring-before($string, '-')" /> >+ <xsl:variable name="remainder" select="substring-after($string, '-')" /> >+ <xsl:choose> >+ <!-- CHECK SECOND HALF FOR ANOTHER DASH; IF PRESENT, ASSUME THAT IS DATE --> >+ <xsl:when test="contains($remainder, '-')!=0"> >+ <xsl:variable name="tmp" select="substring-before($remainder, '-')" /> >+ <xsl:variable name="date_1" select="substring($remainder, (string-length($tmp)-3))" /> >+ <!-- CHECK WHETHER IT HAS A NUMBER BEFORE IT AND IF SO, OUTPUT IT AS DATE --> >+ <xsl:if test="translate(substring($date_1, 1, 1), '0123456789', '9999999999') = '9'"> >+ <subfield code="d"> >+ <xsl:value-of select="$date_1" />. >+ </subfield> >+ </xsl:if> >+ </xsl:when> >+ <!-- OTHERWISE THIS IS THE ONLY DASH SO TAKE IT --> >+ <xsl:otherwise> >+ <xsl:variable name="date_2" select="substring($string, (string-length($date_tmp)-3))" /> >+ <!-- CHECK WHETHER IT HAS A NUMBER BEFORE IT AND IF SO, OUTPUT IT AS DATE --> >+ <xsl:if test="translate(substring($date_2, 1, 1), '0123456789', '9999999999') = '9'"> >+ <subfield code="d"> >+ <xsl:value-of select="$date_2" />. >+ </subfield> >+ </xsl:if> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:if> >+ <subfield code="e"><xsl:value-of select="$type" /></subfield> >+ </datafield> >+ </xsl:template> >+ >+</xsl:stylesheet> >diff --git a/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl b/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl >new file mode 100644 >index 0000000000..2abf4f7d53 >--- /dev/null >+++ b/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl >@@ -0,0 +1,341 @@ >+<?xml version="1.0" encoding="UTF-8"?> >+<xsl:stylesheet version="1.0" >+ xmlns:dc="http://purl.org/dc/elements/1.1/" >+ xmlns:dcterms="http://purl.org/dc/terms/1.1" >+ xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" >+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" >+ xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ >+ http://www.openarchives.org/OAI/2.0/oai_dc.xsd" >+ xmlns:xsl="http://www.w3.org/1999/XSL/Transform" >+ xmlns="http://www.loc.gov/MARC21/slim" exclude-result-prefixes="dc dcterms oai_dc"> >+ >+ <xsl:import href="MARC21slimUtils.xsl"/> >+ <xsl:output method="xml" encoding="UTF-8" indent="yes"/> >+ >+ >+ <xsl:template match="/"> >+ <collection xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd" > >+ <xsl:apply-templates /> >+ </collection> >+ </xsl:template> >+ >+ <xsl:template name="OAI-PMH"> >+ <xsl:for-each select = "ListRecords/record/metadata/oai_dc:dc"> >+ <xsl:apply-templates /> >+ </xsl:for-each> >+ <xsl:for-each select = "GetRecord/record/metadata/oai_dc:dc"> >+ <xsl:apply-templates /> >+ </xsl:for-each> >+ </xsl:template> >+ >+ <xsl:template match="text()" /> >+ <xsl:template match="oai_dc:dc"> >+ <record> >+ >+ <xsl:for-each select="dc:creator"> >+ <xsl:call-template name="persname_template"> >+ <xsl:with-param name="string" select="." /> >+ <xsl:with-param name="field" select="'700'" /> >+ <xsl:with-param name="ind1" select = "'1'" /> >+ <xsl:with-param name="ind2" select = "'0'" /> >+ <xsl:with-param name="type" select="'author'" /> >+ </xsl:call-template> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:title[1]"> >+ <datafield tag="200" ind1="0" ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <xsl:for-each select="dc:title[position()>1]"> >+ <xsl:if test=".!=''"> >+ <datafield tag="532" ind1="3" ind2="0"> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:if> >+ </xsl:for-each> >+ >+ <xsl:choose> >+ <xsl:when test="dc:publisher"> >+ <xsl:if test="translate(dc:publisher/.,'.,:;','')!=''"> >+ <datafield tag="214" ind1=" " ind2=" "> >+ <xsl:choose> >+ <xsl:when test="dc:date"> >+ <subfield code="c"><xsl:value-of select="dc:publisher[1]"/>, </subfield> >+ <xsl:if test="translate(dc:date[1]/., '.,:;','')!=''"> >+ <subfield code="d"><xsl:value-of select="dc:date[1]" />.</subfield> >+ </xsl:if> >+ </xsl:when> >+ <xsl:otherwise> >+ <subfield code="c"><xsl:value-of select="dc:publisher[1]"/>.</subfield> >+ </xsl:otherwise> >+ </xsl:choose> >+ </datafield> >+ </xsl:if> >+ </xsl:when> >+ <xsl:otherwise> >+ <xsl:if test="translate(dc:date[1],'.,:;','')!=''"> >+ <datafield tag="214" ind1=" " ind2=" "> >+ <subfield code="d"><xsl:value-of select="dc:date[1]" />.</subfield> >+ </datafield> >+ </xsl:if> >+ </xsl:otherwise> >+ </xsl:choose> >+ >+ >+ <xsl:for-each select="dc:contributor"> >+ <datafield tag="701" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:coverage"> >+ <xsl:choose> >+ <xsl:when test="translate(., '0123456789-.?','')=''"> >+ <!--Likely;this is a date--> >+ <datafield tag="215" ind1=" " ind2=" "> >+ <subfield code="a"><xsl:value-of select="."/></subfield> >+ </datafield> >+ </xsl:when> >+ <xsl:otherwise> >+ <!--likely a geographic subject, we will print this later--> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:description"> >+ <datafield tag="330" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="normalize-space(.)"/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <!-- no match --> >+ <xsl:for-each select="dc:rights"> >+ <datafield tag="371" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:language"> >+ <datafield tag="101" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:subject"> >+ <datafield tag="610" ind1=" " ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <!-- Resource type will most likely need a specific instance-based mapping --> >+ <!-- >+ <xsl:for-each select="dc:type"> >+ <datafield tag="099" ind1=" " ind2=" "> >+ <subfield code="t"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ --> >+ >+ >+ <!-- no match --> >+ <xsl:for-each select="dc:coverage"> >+ <xsl:choose> >+ <xsl:when test="translate(., '0123456789-.?','')=''"> >+ <!--Likely; this is a date--> >+ </xsl:when> >+ <xsl:otherwise> >+ <!--likely a geographic subject--> >+ <datafield tag="691" ind1=" " ind2=" "> >+ <subfield code="a"><xsl:value-of select="." /></subfield> >+ </datafield> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:for-each> >+ >+ >+ <!-- no match --> >+ <xsl:for-each select="dc:type"> >+ <datafield tag="655" ind1="7" ind2=" "> >+ <subfield code="a"> >+ <xsl:value-of select="."/> >+ </subfield> >+ <subfield code="2">local</subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ >+ <xsl:for-each select="dc:contributer"> >+ <xsl:call-template name="persname_template"> >+ <xsl:with-param name="string" select="." /> >+ <xsl:with-param name="field" select="'702'" /> >+ <xsl:with-param name="ind1" select = "'1'" /> >+ <xsl:with-param name="ind2" select = "'0'" /> >+ <xsl:with-param name="type" select="'contributor'" /> >+ </xsl:call-template> >+ </xsl:for-each> >+ >+ <!-- no match --> >+ <xsl:for-each select="dc:source"> >+ <datafield tag="786" ind1="0" ind2=" "> >+ <subfield code="n"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <!-- TODO: 455 456 470 488--> >+ <xsl:for-each select="dc:relation"> >+ <datafield tag="488" ind1="0" ind2=" "> >+ <subfield code="e"> >+ <xsl:value-of select="."/> >+ </subfield> >+ </datafield> >+ </xsl:for-each> >+ >+ <xsl:if test="dc:identifier"> >+ <datafield tag="856" ind1="4" ind2="1"> >+ <subfield code="u"><xsl:value-of select="dc:identifier[last()]" /></subfield> >+ <subfield code="z">Connect to this object online.</subfield> >+ </datafield> >+ </xsl:if> >+ </record> >+ >+ </xsl:template> >+<xsl:template name="persname_template"> >+ <xsl:param name="string" /> >+ <xsl:param name="field" /> >+ <xsl:param name="ind1" /> >+ <xsl:param name="ind2" /> >+ <xsl:param name="type" /> >+ <datafield> >+ <xsl:attribute name="tag"> >+ <xsl:value-of select="$field" /> >+ </xsl:attribute> >+ <xsl:attribute name="ind1"> >+ <xsl:value-of select="$ind1" /> >+ </xsl:attribute> >+ <xsl:attribute name="ind2"> >+ <xsl:value-of select="$ind2" /> >+ </xsl:attribute> >+ >+ <!-- Sample input: Brightman, Samuel C. (Samuel Charles), 1911-1992 --> >+ <!-- Sample output: $aBrightman, Samuel C. $q(Samuel Charles), $d1911-. --> >+ <!-- will handle names with dashes e.g. Bourke-White, Margaret --> >+ >+ <!-- CAPTURE PRIMARY NAME BY LOOKING FOR A PAREN OR A DASH OR NEITHER --> >+ <xsl:choose> >+ <!-- IF A PAREN, STOP AT AN OPENING PAREN --> >+ <xsl:when test="contains($string, '(')!=0"> >+ <subfield code="a"> >+ <xsl:value-of select="substring-before($string, '(')" /> >+ </subfield> >+ </xsl:when> >+ <!-- IF A DASH, CHECK IF IT'S A DATE OR PART OF THE NAME --> >+ <xsl:when test="contains($string, '-')!=0"> >+ <xsl:variable name="name_1" select="substring-before($string, '-')" /> >+ <xsl:choose> >+ <!-- IF IT'S A DATE REMOVE IT --> >+ <xsl:when test="translate(substring($name_1, (string-length($name_1)), 1), '0123456789', '9999999999') = '9'"> >+ <xsl:variable name="name" select="substring($name_1, 1, (string-length($name_1)-6))" /> >+ <subfield code="a"> >+ <xsl:value-of select="$name" /> >+ </subfield> >+ </xsl:when> >+ <!-- IF IT'S NOT A DATE, CHECK WHETHER THERE IS A DATE LATER --> >+ <xsl:otherwise> >+ <xsl:variable name="remainder" select="substring-after($string, '-')" /> >+ <xsl:choose> >+ <!-- IF THERE'S A DASH, ASSUME IT'S A DATE AND REMOVE IT --> >+ <xsl:when test="contains($remainder, '-')!=0"> >+ <xsl:variable name="tmp" select="substring-before($remainder, '-')" /> >+ <xsl:variable name="name_2" select="substring($tmp, 1, (string-length($tmp)-6))" /> >+ <subfield code="a"> >+ <xsl:value-of select="$name_1" />-<xsl:value-of select="$name_2" /> >+ </subfield> >+ </xsl:when> >+ <!-- IF THERE'S NO DASH IN THE REMAINDER, OUTPUT IT --> >+ <xsl:otherwise> >+ <subfield code="a"> >+ <xsl:value-of select="$string" /> >+ </subfield> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:when> >+ <!-- NO DASHES, NO PARENS, JUST OUTPUT THE NAME --> >+ <xsl:otherwise> >+ <subfield code="a"> >+ <xsl:value-of select="$string" /> >+ </subfield> >+ </xsl:otherwise> >+ </xsl:choose> >+ >+ <!-- CAPTURE SECONDARY NAME IN PARENS FOR SUBFIELD Q --> >+ <xsl:if test="contains($string, '(')!=0"> >+ <xsl:variable name="subq_tmp" select="substring-after($string, '(')" /> >+ <xsl:variable name="subq" select="substring-before($subq_tmp, ')')" /> >+ <subfield code="q"> >+ (<xsl:value-of select="$subq" />) >+ </subfield> >+ </xsl:if> >+ >+ <!-- CAPTURE DATE FOR SUBFIELD D, ASSUME DATE IS LAST ITEM IN FIELD --> >+ <!-- Note: does not work if name has a dash in it --> >+ <xsl:if test="contains($string, '-')!=0"> >+ <xsl:variable name="date_tmp" select="substring-before($string, '-')" /> >+ <xsl:variable name="remainder" select="substring-after($string, '-')" /> >+ <xsl:choose> >+ <!-- CHECK SECOND HALF FOR ANOTHER DASH; IF PRESENT, ASSUME THAT IS DATE --> >+ <xsl:when test="contains($remainder, '-')!=0"> >+ <xsl:variable name="tmp" select="substring-before($remainder, '-')" /> >+ <xsl:variable name="date_1" select="substring($remainder, (string-length($tmp)-3))" /> >+ <!-- CHECK WHETHER IT HAS A NUMBER BEFORE IT AND IF SO, OUTPUT IT AS DATE --> >+ <xsl:if test="translate(substring($date_1, 1, 1), '0123456789', '9999999999') = '9'"> >+ <subfield code="d"> >+ <xsl:value-of select="$date_1" />. >+ </subfield> >+ </xsl:if> >+ </xsl:when> >+ <!-- OTHERWISE THIS IS THE ONLY DASH SO TAKE IT --> >+ <xsl:otherwise> >+ <xsl:variable name="date_2" select="substring($string, (string-length($date_tmp)-3))" /> >+ <!-- CHECK WHETHER IT HAS A NUMBER BEFORE IT AND IF SO, OUTPUT IT AS DATE --> >+ <xsl:if test="translate(substring($date_2, 1, 1), '0123456789', '9999999999') = '9'"> >+ <subfield code="d"> >+ <xsl:value-of select="$date_2" />. >+ </subfield> >+ </xsl:if> >+ </xsl:otherwise> >+ </xsl:choose> >+ </xsl:if> >+ <subfield code="e"><xsl:value-of select="$type" /></subfield> >+ </datafield> >+ </xsl:template> >+ >+</xsl:stylesheet> >diff --git a/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl b/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl >new file mode 100644 >index 0000000000..67669afdb0 >--- /dev/null >+++ b/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl >@@ -0,0 +1,23 @@ >+<?xml version="1.0" encoding="UTF-8"?> >+<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:marc="http://www.loc.gov/MARC21/slim"> >+ <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/> >+ >+ <!-- keep comments --> >+ <xsl:template match="*[name()='metadata']"> >+ <xsl:apply-templates /> >+ </xsl:template> >+ >+ <xsl:template match="*"> >+ <!-- remove element prefix --> >+ <xsl:element name="{local-name()}"> >+ <!-- process attributes --> >+ <xsl:for-each select="@*"> >+ <!-- remove attribute prefix --> >+ <xsl:attribute name="{local-name()}"> >+ <xsl:value-of select="."/> >+ </xsl:attribute> >+ </xsl:for-each> >+ <xsl:apply-templates/> >+ </xsl:element> >+ </xsl:template> >+</xsl:stylesheet> >diff --git a/misc/cronjobs/harvest_oai.pl b/misc/cronjobs/harvest_oai.pl >new file mode 100755 >index 0000000000..2cc212652d >--- /dev/null >+++ b/misc/cronjobs/harvest_oai.pl >@@ -0,0 +1,110 @@ >+#!/usr/bin/perl >+# >+# Copyright 2023 BibLibre >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ >+use warnings; >+use strict; >+use utf8; >+ >+use Getopt::Long qw( GetOptions ); >+use Koha::Script -cron; >+use Koha::OAI::Client::Harvester; >+use Koha::OaiServers; >+use C4::Log qw( cronlogaction ); >+use Try::Tiny qw( catch try ); >+ >+my $command_line_options = join( " ", @ARGV ); >+ >+my ( $help, $verbose, $id, $days, $list, $force ); >+ >+GetOptions( >+ 'h|help' => \$help, >+ 'v|verbose' => \$verbose, >+ 'r|repository:i' => \$id, >+ 'd|days:i' => \$days, >+ 'l|list' => \$list, >+ 'f|force' => \$force, >+); >+my $usage = << 'ENDUSAGE'; >+ >+This script starts an OAI Harvest >+ >+This script has the following parameters : >+ -h --help: this message >+ -v --verbose >+ -r --repository: id of the OAI repository >+ -d --days: number of days to harvest from (optional) >+ -l --list: list the OAI repositories >+ -f --force: force harvesting (ignore records datestamps) >+ >+ENDUSAGE >+ >+if ($help) { >+ print $usage; >+ exit; >+} >+ >+if ($list) { >+ my $servers = Koha::OaiServers->search( {}, { order_by => { -asc => 'id' } } )->unblessed; >+ print "The following repositories are available: \n\n"; >+ foreach my $server (@$servers) { >+ print $server->{'id'} . ": " >+ . $server->{'servername'} >+ . ", endpoint: " >+ . $server->{'endpoint'} >+ . ", set: " >+ . $server->{'oai_set'} >+ . ", recordtype: " >+ . $server->{'recordtype'} . "\n"; >+ } >+ print "\n"; >+ exit; >+} >+ >+if ( !$id ) { >+ print "The repository parameter is mandatory.\n"; >+ print $usage . "\n"; >+} >+ >+my $server = Koha::OaiServers->find($id); >+ >+unless ($server) { >+ print "OAI Server $id unknown\n"; >+ exit; >+} >+ >+my $script_handler = Koha::Script->new( { script => $0 } ); >+ >+try { >+ $script_handler->lock_exec; >+} catch { >+ my $message = "Skipping execution of $0"; >+ print "$message\n" if $verbose; >+ cronlogaction( { info => $message } ); >+ exit; >+}; >+ >+cronlogaction( { action => 'Start', info => "Starting OAI Harvest" } ); >+cronlogaction( { info => "Command line: $command_line_options" } ); >+ >+my $harvester = >+ Koha::OAI::Client::Harvester->new( { server => $server, verbose => $verbose, days => $days, force => $force } ); >+$harvester->init(); >+ >+cronlogaction( { action => 'End', info => "Ending OAI Harvest" } ); >+exit(0); >diff --git a/t/db_dependent/Koha/OAIHarvester.t b/t/db_dependent/Koha/OAIHarvester.t >new file mode 100755 >index 0000000000..e42b9a08f4 >--- /dev/null >+++ b/t/db_dependent/Koha/OAIHarvester.t >@@ -0,0 +1,78 @@ >+#!/usr/bin/perl >+ >+# Copyright 2024 BibLibre >+# >+# This file is part of Koha >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ >+use Modern::Perl; >+use Test::More tests => 5; >+use Test::Exception; >+ >+use t::lib::TestBuilder; >+use HTTP::OAI; >+use HTTP::OAI::Metadata::OAI_DC; >+use HTTP::OAI::Record; >+use HTTP::OAI::Encapsulation; >+use Koha::Database; >+use Koha::OaiServer; >+use Koha::OaiServers; >+use Koha::OAI::Client::Harvester; >+ >+my $schema = Koha::Database->new->schema; >+my $builder = t::lib::TestBuilder->new; >+ >+$schema->storage->txn_begin; >+ >+my $new_oai_1 = Koha::OaiServer->new( >+ { >+ endpoint => 'my_host1.org', >+ oai_set => 'set1', >+ servername => 'my_test_1', >+ dataformat => 'oai_dc', >+ recordtype => 'biblio', >+ add_xslt => '', >+ } >+)->store; >+ >+my $harvester = Koha::OAI::Client::Harvester->new( { server => $new_oai_1, verbose => 1, days => 1, force => 1 } ); >+ >+my $record = >+ '<metadata xmlns="http://www.openarchives.org/OAI/2.0/"><oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd"><dc:title>Pièces diverses </dc:title><dc:identifier>ARCH/0320 [cote]</dc:identifier><dc:relation>FR-920509801 [RCR établissement]</dc:relation><dc:relation>Central obrera boliviana [Fonds ou collection]</dc:relation><dc:format>1 carton</dc:format><dc:date>1971/2000</dc:date><dc:type>Archives et manuscrits</dc:type></oai_dc:dc></metadata>'; >+ >+my $r = HTTP::OAI::Record->new(); >+$r->header->datestamp('2017-05-10T09:18:13Z'); >+$r->metadata( HTTP::OAI::Metadata->new( dom => $record ) ); >+ >+my $status = $harvester->processRecord($r); >+is( $status, 'skipped', 'Record with no identifier is skipped' ); >+ >+$r->header->identifier('oai:myarchive.org:oid-233'); >+ >+$status = $harvester->processRecord($r); >+is( $status, 'added', 'Record is added' ); >+ >+$status = $harvester->processRecord($r); >+is( $status, 'updated', 'When force is used, record is updated' ); >+ >+$harvester = Koha::OAI::Client::Harvester->new( { server => $new_oai_1, verbose => 1, days => 1, force => 0 } ); >+$status = $harvester->processRecord($r); >+is( $status, 'skipped', 'When force is not used, record is skipped (already up to date)' ); >+ >+$r->header->datestamp('2018-05-10T09:18:13Z'); >+$status = $harvester->processRecord($r); >+is( $status, 'updated', 'When force is not used, record is updated if newer' ); >+ >+$schema->storage->txn_rollback; >diff --git a/t/db_dependent/Koha/OAIServers.t b/t/db_dependent/Koha/OAIServers.t >new file mode 100755 >index 0000000000..9f135b77db >--- /dev/null >+++ b/t/db_dependent/Koha/OAIServers.t >@@ -0,0 +1,95 @@ >+#!/usr/bin/perl >+ >+# Copyright 2015 Koha Development team >+# >+# This file is part of Koha >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ >+use Modern::Perl; >+use Test::More tests => 2; >+use Test::Exception; >+ >+use t::lib::TestBuilder; >+use Koha::Database; >+use Koha::OaiServer; >+use Koha::OaiServers; >+ >+my $schema = Koha::Database->new->schema; >+my $builder = t::lib::TestBuilder->new; >+ >+subtest 'new, find and delete tests' => sub { >+ plan tests => 4; >+ $schema->storage->txn_begin; >+ my $nb_of_oais = Koha::OaiServers->search->count; >+ my $new_oai_1 = Koha::OaiServer->new( >+ { >+ endpoint => 'my_host1.org', >+ oai_set => 'set1', >+ servername => 'my_test_1', >+ dataformat => 'oai_dc', >+ recordtype => 'biblio', >+ } >+ )->store; >+ my $new_oai_2 = Koha::OaiServer->new( >+ { >+ endpoint => 'my_host2.org', >+ oai_set => 'set2', >+ servername => 'my_test_2', >+ dataformat => 'marcxml', >+ recordtype => 'authority', >+ } >+ )->store; >+ >+ like( $new_oai_1->id, qr|^\d+$|, 'Adding a new oai server should have set the id' ); >+ is( Koha::OaiServers->search->count, $nb_of_oais + 2, 'The 2 oai servers should have been added' ); >+ >+ my $retrieved_oai_1 = Koha::OaiServers->find( $new_oai_1->id ); >+ is( >+ $retrieved_oai_1->servername, $new_oai_1->servername, >+ 'Find a oai server by id should return the correct oai server' >+ ); >+ >+ $retrieved_oai_1->delete; >+ is( Koha::OaiServers->search->count, $nb_of_oais + 1, 'Delete should have deleted the oai server' ); >+ >+ $schema->storage->txn_rollback; >+}; >+ >+subtest 'Check NOT NULL without default values' => sub { >+ plan tests => 5; >+ $schema->storage->txn_begin; >+ local $SIG{__WARN__} = sub { }; # TODO Needed it for suppressing DBIx warns >+ >+ my $server = Koha::OaiServer->new( >+ { >+ oai_set => 'set3', >+ dataformat => 'marcxml', >+ recordtype => 'biblio', >+ } >+ ); >+ >+ throws_ok { $server->store } 'DBIx::Class::Exception', 'Exception on empty endpoint'; >+ like( $@->{msg}, qr/'endpoint' doesn't have a default value/, 'Verified that DBIx blamed endpoint' ); >+ >+ $server->endpoint('endpoint_added'); >+ throws_ok { $server->store } 'DBIx::Class::Exception', 'Exception on empty servername'; >+ like( $@->{msg}, qr/'servername' doesn't have a default value/, 'Verified that DBIx blamed servername' ); >+ >+ $server->servername('servername_added'); >+ >+ lives_ok { $server->store } 'No exceptions anymore'; >+ >+ $schema->storage->txn_rollback; >+}; >-- >2.39.2
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=173088">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 35659
:
160356
|
160357
|
160744
|
161813
|
161960
|
162004
|
162005
|
162006
|
162007
|
162008
|
162009
|
162010
|
162011
|
162527
|
162528
|
162692
|
162693
|
162694
|
163527
|
164586
|
164590
|
164602
|
164603
|
164604
|
164605
|
164606
|
164618
|
164619
|
164620
|
164621
|
164622
|
165199
|
165594
|
165595
|
165596
|
165597
|
165598
|
167106
|
167107
|
167171
|
167172
|
167173
|
167174
|
167175
|
167176
|
168982
|
168983
|
169941
|
171255
|
171258
|
171260
|
171264
|
171265
|
171267
|
171675
|
171676
|
171678
|
171679
|
172072
|
172073
|
172074
|
172075
|
172076
|
172077
|
172078
|
172079
|
172080
|
172081
|
172082
|
172083
|
172202
|
172203
|
172204
|
172205
|
172206
|
172207
|
172208
|
172209
|
172210
|
172211
|
172212
|
172213
| 173088 |
173089
|
173090
|
173091
|
173092
|
173093
|
173094
|
173095
|
173096
|
173097
|
173098
|
173099
|
173100
|
173101
|
173561
|
173599
|
173600
|
173601
|
173614
|
174790