Bugzilla – Attachment 81784 Details for
Bug 10662
Build OAI-PMH Harvesting Client
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 10662: (QA follow-up) addressing qa test tool output
Bug-10662-QA-follow-up-addressing-qa-test-tool-out.patch (text/plain), 87.63 KB, created by
David Cook
on 2018-11-01 05:13:00 UTC
(
hide
)
Description:
Bug 10662: (QA follow-up) addressing qa test tool output
Filename:
MIME Type:
Creator:
David Cook
Created:
2018-11-01 05:13:00 UTC
Size:
87.63 KB
patch
obsolete
>From d267731a50b7b343aa291772f74af2e6a4506d64 Mon Sep 17 00:00:00 2001 >From: David Cook <dcook@prosentient.com.au> >Date: Wed, 31 Oct 2018 03:14:46 +0000 >Subject: [PATCH] Bug 10662: (QA follow-up) addressing qa test tool output > >I've done the following in this follow-up: > >- Added missing POD >- Replaced DBIC use with Koha::Object[s] >- Replaced DataTables with KohaTable >- Fixed database charset and collation >- Moved Javascript to bottom of templates >- Fixed a syntax error in rewrite-config.PL >--- > Koha/Daemon.pm | 58 +++++ > Koha/OAI/Harvester.pm | 117 +++++++-- > Koha/OAI/Harvester/Biblio.pm | 49 ++++ > Koha/OAI/Harvester/Biblios.pm | 60 +++++ > Koha/OAI/Harvester/Client.pm | 55 ++++- > Koha/OAI/Harvester/Downloader.pm | 33 ++- > Koha/OAI/Harvester/Histories.pm | 60 +++++ > Koha/OAI/Harvester/History.pm | 49 ++++ > Koha/OAI/Harvester/Import/MARCXML.pm | 23 ++ > Koha/OAI/Harvester/Import/Record.pm | 69 +++++- > Koha/OAI/Harvester/ImportQueue.pm | 49 ++++ > Koha/OAI/Harvester/ImportQueues.pm | 60 +++++ > Koha/OAI/Harvester/Listener.pm | 56 +++++ > Koha/OAI/Harvester/Request.pm | 19 +- > Koha/OAI/Harvester/Worker.pm | 59 +++++ > Koha/OAI/Harvester/Worker/Download/Stream.pm | 30 +++ > Koha/OAI/Harvester/Worker/Import.pm | 24 ++ > Koha/Schema/Result/Biblio.pm | 19 +- > Koha/Schema/Result/OaiHarvesterBiblio.pm | 4 +- > Koha/Schema/Result/OaiHarvesterHistory.pm | 4 +- > Koha/Schema/Result/OaiHarvesterImportQueue.pm | 4 +- > Koha/Schema/Result/OaiHarvesterRequest.pm | 4 +- > admin/columns_settings.yml | 59 +++++ > installer/data/mysql/atomicupdate/bug_10662.sql | 8 +- > installer/data/mysql/kohastructure.sql | 8 +- > .../modules/tools/oai-pmh-harvester/dashboard.tt | 275 +++++++++++---------- > .../en/modules/tools/oai-pmh-harvester/request.tt | 123 ++++----- > rewrite-config.PL | 2 +- > svc/oai-pmh-harvester/history | 65 +++-- > tools/oai-pmh-harvester/record.pl | 18 +- > 30 files changed, 1155 insertions(+), 308 deletions(-) > create mode 100644 Koha/OAI/Harvester/Biblio.pm > create mode 100644 Koha/OAI/Harvester/Biblios.pm > create mode 100644 Koha/OAI/Harvester/Histories.pm > create mode 100644 Koha/OAI/Harvester/History.pm > create mode 100644 Koha/OAI/Harvester/ImportQueue.pm > create mode 100644 Koha/OAI/Harvester/ImportQueues.pm > >diff --git a/Koha/Daemon.pm b/Koha/Daemon.pm >index cdae369..19010f0 100644 >--- a/Koha/Daemon.pm >+++ b/Koha/Daemon.pm >@@ -22,6 +22,64 @@ use Modern::Perl; > use POSIX; #For daemonizing > use Fcntl qw(:flock); #For pidfile > >+=head1 NAME >+ >+Koha::Daemon >+ >+=head1 SYNOPSIS >+ >+ use Koha::Daemon; >+ my $daemon = Koha::Daemon->new({ >+ pidfile => "/path/to/file.pid", >+ logfile => "/path/to/file.log", >+ daemonize => 1, >+ }); >+ $daemon->run(); >+ >+=head1 METHODS >+ >+=head2 new >+ >+Create object >+ >+=head2 run >+ >+Run the daemon. >+ >+This method calls all the internal methods which >+are do everything necessary to run the daemon. >+ >+=head1 INTERNAL METHODS >+ >+=head2 daemonize >+ >+Internal function for setting object up as a proper daemon >+(e.g. forking, setting permissions, changing directories, >+ closing file handles, etc.) >+ >+=head2 get_pidfile >+ >+Internal function to get a filehandle for the pidfile >+ >+=head2 lock_pidfile >+ >+Internal function to lock the pidfile, so that only one daemon >+can run against this pidfile >+ >+=head2 log_to_file >+ >+Internal function to log to a file >+ >+=head2 make_pidfilehandle >+ >+Internal function to make a filehandle for pidfile >+ >+=head2 write_pidfile >+ >+Internal function to write pid to pidfile >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; >diff --git a/Koha/OAI/Harvester.pm b/Koha/OAI/Harvester.pm >index 86619fa..7a374fe 100644 >--- a/Koha/OAI/Harvester.pm >+++ b/Koha/OAI/Harvester.pm >@@ -30,11 +30,26 @@ use DateTime; > use DateTime::Format::Strptime; > > use C4::Context; >-use Koha::Database; >+use Koha::OAI::Harvester::ImportQueues; > >-=head1 API >+=head1 NAME > >-=head2 Class Methods >+Koha::OAI::Harvester >+ >+=head1 SYNOPSIS >+ >+my $harvester = Koha::OAI::Harvester->spawn({ >+ Downloader => $downloader, >+ DownloaderWorkers => $download_workers, >+ Importer => $importer, >+ ImporterWorkers => $import_workers, >+ ImportQueuePoll => $import_poll, >+ logger => $logger, >+ state_file => $statefile, >+ spooldir => $spooldir, >+}); >+ >+=head2 METHODS > > =cut > >@@ -45,12 +60,24 @@ my $seconds_granularity = DateTime::Format::Strptime->new( > pattern => '%FT%TZ', > ); > >+=head2 new >+ >+Create object >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; > return bless ($args, $class); > } > >+=head2 spawn >+ >+ Creates POE sessions for handling tasks, downloading, and importing >+ >+=cut >+ > sub spawn { > my ($class, $args) = @_; > my $self = $class->new($args); >@@ -114,15 +141,14 @@ sub spawn { > my $active_tasks = $poe_kernel->call("harvester","list_tasks","active"); > my @active_uuids = map { $_->{uuid} } @$active_tasks; > >- my $schema = Koha::Database->new()->schema(); >- my $rs = $schema->resultset('OaiHarvesterImportQueue')->search({ >+ my $rs = Koha::OAI::Harvester::ImportQueues->new->search({ > uuid => \@active_uuids, > status => "new" > },{ > order_by => { -asc => 'id' }, > rows => 1, > }); >- my $result = $rs->first; >+ my $result = $rs->single; > if ($result){ > $result->status("wip"); > $result->update; >@@ -149,6 +175,12 @@ sub spawn { > return; > } > >+=head2 on_start >+ >+ Internal method that starts the harvester >+ >+=cut >+ > sub on_start { > my ($self, $kernel, $heap) = @_[OBJECT, KERNEL,HEAP]; > $kernel->alias_set('harvester'); >@@ -161,13 +193,19 @@ sub on_start { > $kernel->call("harvester","restore_state"); > } > >+=head2 download_postback >+ >+ Internal method >+ >+=cut >+ > #NOTE: This isn't really implemented at the moment, as it's not really necessary. > sub download_postback { > my ($kernel, $request_packet, $response_packet) = @_[KERNEL, ARG0, ARG1]; > my $message = $response_packet->[0]; > } > >-=head3 deregister >+=head2 deregister > > Remove the worker session from the harvester's in-memory scoreboard, > unset the downloading flag if downloading is completed. >@@ -218,7 +256,7 @@ sub deregister { > } > > >-=head3 is_task_finished >+=head2 is_task_finished > > This event handler checks if the task has finished downloading and importing record. > If it is finished downloading and importing, the task is deleted from the harvester. >@@ -241,6 +279,12 @@ sub is_task_finished { > return 0; > } > >+=head2 register >+ >+ Internal method for registering a task on the harvester's scoreboard >+ >+=cut >+ > sub register { > my ($self, $kernel, $heap, $session, $sender, $type, $task_uuid) = @_[OBJECT, KERNEL,HEAP,SESSION,SENDER,ARG0,ARG1]; > my $logger = $self->{logger}; >@@ -288,6 +332,12 @@ sub register { > } > } > >+=head2 does_task_repeat >+ >+ Internal method for checking if a task is supposed to repeat after it finishes >+ >+=cut >+ > sub does_task_repeat { > my ($self, $kernel, $heap, $session, $uuid) = @_[OBJECT, KERNEL,HEAP,SESSION,ARG0]; > my $task = $kernel->call("harvester","get_task",$uuid); >@@ -306,13 +356,16 @@ sub does_task_repeat { > return 0; > } > >+=head2 reset_imports_status > >+ Internal method that resets the import queue from wip to new >+ >+=cut > > sub reset_imports_status { > my ($self, $kernel, $heap, $session) = @_[OBJECT, KERNEL,HEAP,SESSION]; > >- my $schema = Koha::Database->new()->schema(); >- my $rs = $schema->resultset('OaiHarvesterImportQueue')->search({ >+ my $rs = Koha::OAI::Harvester::ImportQueues->new->search({ > status => "wip", > }); > $rs->update({ >@@ -320,6 +373,12 @@ sub reset_imports_status { > }); > } > >+=head2 restore_state >+ >+ Method to restore the harvester to a pre-existing state. >+ >+=cut >+ > sub restore_state { > my ($self, $kernel, $heap, $session) = @_[OBJECT, KERNEL,HEAP,SESSION]; > >@@ -357,6 +416,12 @@ sub restore_state { > } > } > >+=head2 save_state >+ >+ Method to save the existing state of the harvester >+ >+=cut >+ > sub save_state { > my ($self, $kernel, $heap, $session) = @_[OBJECT, KERNEL,HEAP,SESSION]; > my $state_file = $self->{state_file}; >@@ -385,7 +450,7 @@ sub save_state { > return 0; > } > >-=head3 get_task >+=head2 get_task > > This event handler returns a task from a harvester using the task's > uuid as an argument. >@@ -413,7 +478,9 @@ sub get_task { > return 0; > } > >-=head3 get_import_count_for_task >+=head2 get_import_count_for_task >+ >+ This gets the count of the number of imports exist for a certain task > > =cut > >@@ -421,8 +488,7 @@ sub get_import_count_for_task { > my ($self,$uuid) = @_; > my $count = undef; > if ($uuid){ >- my $schema = Koha::Database->new()->schema(); >- my $items = $schema->resultset('OaiHarvesterImportQueue')->search({ >+ my $items = Koha::OAI::Harvester::ImportQueues->new->search({ > uuid => $uuid, > }); > $count = $items->count; >@@ -430,7 +496,7 @@ sub get_import_count_for_task { > return $count; > } > >-=head3 list_tasks >+=head2 list_tasks > > This event handler returns a list of tasks that have been submitted > to the harvester. It returns data like uuid, status, parameters, >@@ -440,11 +506,10 @@ sub get_import_count_for_task { > > sub list_tasks { > my ($self, $kernel, $heap, $session, $status) = @_[OBJECT, KERNEL,HEAP,SESSION, ARG0]; >- my $schema = Koha::Database->new()->schema(); > my @tasks = (); > foreach my $uuid (sort keys %{$heap->{tasks}}){ > my $task = $heap->{tasks}->{$uuid}; >- my $items = $schema->resultset('OaiHarvesterImportQueue')->search({ >+ my $items = Koha::OAI::Harvester::ImportQueues->new->search({ > uuid => $uuid, > }); > my $count = $items->count // 0; >@@ -457,7 +522,7 @@ sub list_tasks { > return \@tasks; > } > >-=head3 create_task >+=head2 create_task > > This event handler creates a spool directory for the task's imports. > It also adds it to the harvester's memory and then saves memory to >@@ -502,7 +567,7 @@ sub create_task { > return 0; > } > >-=head3 start_task >+=head2 start_task > > This event handler marks a task as active in the harvester's memory, > save the memory to a persistent datastore, then enqueues the task, >@@ -536,9 +601,9 @@ sub start_task { > return 0; > } > >-=head3 repeat_task >- >+=head2 repeat_task > >+ This method re-queues a task for downloading > > =cut > >@@ -553,7 +618,7 @@ sub repeat_task { > } > } > >-=head3 stop_task >+=head2 stop_task > > This event handler prevents new workers from spawning, kills > existing workers, and stops pending imports from being imported. >@@ -594,8 +659,7 @@ sub stop_task { > } > > #Step Three: stop pending imports for this task >- my $schema = Koha::Database->new()->schema(); >- my $items = $schema->resultset('OaiHarvesterImportQueue')->search({ >+ my $items = Koha::OAI::Harvester::ImportQueues->new->search({ > uuid => $task_uuid, > }); > my $rows_updated = $items->update({ >@@ -609,7 +673,7 @@ sub stop_task { > return 0; > } > >-=head3 delete_task >+=head2 delete_task > > Deleted tasks are stopped, pending imports are deleted from the > database and file system, and then the task is removed from the harvester. >@@ -625,8 +689,7 @@ sub delete_task { > $kernel->call($session,"stop_task",$task_uuid); > > #Step Two: delete pending imports in database >- my $schema = Koha::Database->new()->schema(); >- my $items = $schema->resultset('OaiHarvesterImportQueue')->search({ >+ my $items = Koha::OAI::Harvester::ImportQueues->new->search({ > uuid => $task_uuid, > }); > if ($items){ >diff --git a/Koha/OAI/Harvester/Biblio.pm b/Koha/OAI/Harvester/Biblio.pm >new file mode 100644 >index 0000000..b84aeca >--- /dev/null >+++ b/Koha/OAI/Harvester/Biblio.pm >@@ -0,0 +1,49 @@ >+package Koha::OAI::Harvester::Biblio; >+ >+# Copyright Prosentient Systems 2018 >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it under the >+# terms of the GNU General Public License as published by the Free Software >+# Foundation; either version 3 of the License, or (at your option) any later >+# version. >+# >+# Koha is distributed in the hope that it will be useful, but WITHOUT ANY >+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR >+# A PARTICULAR PURPOSE. See the GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License along >+# with Koha; if not, write to the Free Software Foundation, Inc., >+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >+ >+use Modern::Perl; >+ >+use Carp; >+ >+use base qw(Koha::Object); >+ >+=head1 NAME >+ >+Koha::OAI::Harvester::Biblio >+ >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::Biblio; >+ my $biblio = Koha::OAI::Harvester::Biblios->find($id); >+ >+=head1 METHODS >+ >+=cut >+ >+sub _type { >+ return 'OaiHarvesterBiblio'; >+} >+ >+=head1 AUTHOR >+ >+David Cook <dcook@prosentient.com.au> >+ >+=cut >+ >+1; >diff --git a/Koha/OAI/Harvester/Biblios.pm b/Koha/OAI/Harvester/Biblios.pm >new file mode 100644 >index 0000000..a1a9aec >--- /dev/null >+++ b/Koha/OAI/Harvester/Biblios.pm >@@ -0,0 +1,60 @@ >+package Koha::OAI::Harvester::Biblios; >+ >+# Copyright Prosentient Systems 2018 >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it under the >+# terms of the GNU General Public License as published by the Free Software >+# Foundation; either version 3 of the License, or (at your option) any later >+# version. >+# >+# Koha is distributed in the hope that it will be useful, but WITHOUT ANY >+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR >+# A PARTICULAR PURPOSE. See the GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License along >+# with Koha; if not, write to the Free Software Foundation, Inc., >+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >+ >+use Modern::Perl; >+ >+use Carp; >+ >+use Koha::Database; >+ >+use Koha::OAI::Harvester::Biblio; >+ >+use base qw(Koha::Objects); >+ >+=head1 NAME >+ >+Koha::OAI::Harvester::Biblios >+ >+=head2 METHODS >+ >+=cut >+ >+=head3 _type >+ >+=cut >+ >+sub _type { >+ return 'OaiHarvesterBiblio'; >+} >+ >+=head3 object_class >+ >+=cut >+ >+sub object_class { >+ return 'Koha::OAI::Harvester::Biblio'; >+} >+ >+=head1 AUTHOR >+ >+David Cook <dcook@prosentient.com.au> >+ >+=cut >+ >+1; >diff --git a/Koha/OAI/Harvester/Client.pm b/Koha/OAI/Harvester/Client.pm >index d5c5351..abfc276 100644 >--- a/Koha/OAI/Harvester/Client.pm >+++ b/Koha/OAI/Harvester/Client.pm >@@ -23,12 +23,35 @@ use IO::Socket::UNIX; > use IO::Select; > use JSON; > >+=head1 NAME >+ >+Koha::OAI::Harvester::Client >+ >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::Client; >+ my $client = Koha::OAI::Harvester::Client->new($client_config); >+ >+=head1 METHODS >+ >+=head2 new >+ >+Create object >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; > return bless ($args, $class); > } > >+=head2 connect >+ >+Connect to Koha::OAI::Harvester::Listener via a Unix socket >+ >+=cut >+ > sub connect { > my ($self) = @_; > my $socket_uri = $self->{socket_uri}; >@@ -62,6 +85,12 @@ sub connect { > return 0; > } > >+=head2 create >+ >+ Create a task on the harvester >+ >+=cut >+ > sub create { > my ($self,$task) = @_; > my $message = { >@@ -74,6 +103,12 @@ sub create { > return $status; > } > >+=head2 start >+ >+ Start a task on the harvester >+ >+=cut >+ > sub start { > my ($self,$uuid) = @_; > my $message = { >@@ -88,6 +123,12 @@ sub start { > return $status; > } > >+=head2 stop >+ >+ Stop a task on the harvester >+ >+=cut >+ > sub stop { > my ($self,$uuid) = @_; > my $message = { >@@ -102,6 +143,12 @@ sub stop { > return $status; > } > >+=head2 delete >+ >+ Delete a task on the harvester >+ >+=cut >+ > sub delete { > my ($self,$uuid) = @_; > my $message = { >@@ -116,6 +163,12 @@ sub delete { > return $status; > } > >+=head2 list >+ >+ List all tasks on the harvester >+ >+=cut >+ > sub list { > my ($self) = @_; > my $message = { >@@ -174,4 +227,4 @@ sub _read { > } > } > >-1; >\ No newline at end of file >+1; >diff --git a/Koha/OAI/Harvester/Downloader.pm b/Koha/OAI/Harvester/Downloader.pm >index 4ca8be6..3137b10 100644 >--- a/Koha/OAI/Harvester/Downloader.pm >+++ b/Koha/OAI/Harvester/Downloader.pm >@@ -23,9 +23,22 @@ use XML::LibXML::Reader; > use IO::Handle; > use JSON; > >-=head1 API >+=head1 NAME > >-=head2 Class Methods >+Koha::OAI::Harvester::Downloader >+ >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::Downloader; >+ my $oai_downloader = Koha::OAI::Harvester::Downloader->new(); >+ >+ This class is used within a Koha::OAI::Harvester::Work::Download:: module. >+ >+=head1 METHODS >+ >+=head2 new >+ >+ Create object > > =cut > >@@ -96,7 +109,7 @@ sub BuildURL { > } > } > >-=head2 OpenXMLStream >+=head2 GetXMLStream > > Fork a child process to send the HTTP request, which sends chunks > of XML via a pipe to the parent process. >@@ -168,6 +181,13 @@ sub _request { > } > } > >+=head2 ParseXMLStream >+ >+ Parse XML using XML::LibXML::Reader from a file handle (e.g. a stream) >+ >+=cut >+ >+ > sub ParseXMLStream { > my ($self, $args) = @_; > >@@ -245,6 +265,13 @@ sub ParseXMLStream { > } > } > >+=head2 harvest >+ >+ Given a URL and a user-agent, start downloading OAI-PMH records from >+ that URL. >+ >+=cut >+ > sub harvest { > my ($self,$args) = @_; > my $url = $args->{url}; >diff --git a/Koha/OAI/Harvester/Histories.pm b/Koha/OAI/Harvester/Histories.pm >new file mode 100644 >index 0000000..c3f870c >--- /dev/null >+++ b/Koha/OAI/Harvester/Histories.pm >@@ -0,0 +1,60 @@ >+package Koha::OAI::Harvester::Histories; >+ >+# Copyright Prosentient Systems 2018 >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it under the >+# terms of the GNU General Public License as published by the Free Software >+# Foundation; either version 3 of the License, or (at your option) any later >+# version. >+# >+# Koha is distributed in the hope that it will be useful, but WITHOUT ANY >+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR >+# A PARTICULAR PURPOSE. See the GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License along >+# with Koha; if not, write to the Free Software Foundation, Inc., >+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >+ >+use Modern::Perl; >+ >+use Carp; >+ >+use Koha::Database; >+ >+use Koha::OAI::Harvester::History; >+ >+use base qw(Koha::Objects); >+ >+=head1 NAME >+ >+Koha::OAI::Harvester::Histories >+ >+=head2 METHODS >+ >+=cut >+ >+=head3 _type >+ >+=cut >+ >+sub _type { >+ return 'OaiHarvesterHistory'; >+} >+ >+=head3 object_class >+ >+=cut >+ >+sub object_class { >+ return 'Koha::OAI::Harvester::History'; >+} >+ >+=head1 AUTHOR >+ >+David Cook <dcook@prosentient.com.au> >+ >+=cut >+ >+1; >diff --git a/Koha/OAI/Harvester/History.pm b/Koha/OAI/Harvester/History.pm >new file mode 100644 >index 0000000..5ad490c >--- /dev/null >+++ b/Koha/OAI/Harvester/History.pm >@@ -0,0 +1,49 @@ >+package Koha::OAI::Harvester::History; >+ >+# Copyright Prosentient Systems 2018 >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it under the >+# terms of the GNU General Public License as published by the Free Software >+# Foundation; either version 3 of the License, or (at your option) any later >+# version. >+# >+# Koha is distributed in the hope that it will be useful, but WITHOUT ANY >+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR >+# A PARTICULAR PURPOSE. See the GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License along >+# with Koha; if not, write to the Free Software Foundation, Inc., >+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >+ >+use Modern::Perl; >+ >+use Carp; >+ >+use base qw(Koha::Object); >+ >+=head1 NAME >+ >+Koha::OAI::Harvester::History >+ >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::History; >+ my $request = Koha::OAI::Harvester::History->find($id); >+ >+=head1 METHODS >+ >+=cut >+ >+sub _type { >+ return 'OaiHarvesterHistory'; >+} >+ >+=head1 AUTHOR >+ >+David Cook <dcook@prosentient.com.au> >+ >+=cut >+ >+1; >diff --git a/Koha/OAI/Harvester/Import/MARCXML.pm b/Koha/OAI/Harvester/Import/MARCXML.pm >index 6accb78..6eb7c5b 100755 >--- a/Koha/OAI/Harvester/Import/MARCXML.pm >+++ b/Koha/OAI/Harvester/Import/MARCXML.pm >@@ -26,6 +26,23 @@ use C4::Biblio; > > use constant MAX_MATCHES => 99999; #NOTE: This is an arbitrary value. We want to get all matches. > >+=head1 NAME >+ >+Koha::OAI::Harvester::Import::MARCXML >+ >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::Import::MARCXML; >+ my $marcxml = eval { Koha::OAI::Harvester::Import::MARCXML->new({ dom => $results, }) }; >+ >+=head1 METHODS >+ >+=head2 new >+ >+ Create object >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; >@@ -47,6 +64,12 @@ sub new { > return bless ($args, $class); > } > >+=head2 import_record >+ >+ Import a record into Koha >+ >+=cut >+ > sub import_record { > my ($self,$args) = @_; > my $framework = $args->{framework}; >diff --git a/Koha/OAI/Harvester/Import/Record.pm b/Koha/OAI/Harvester/Import/Record.pm >index 9a67eb8..bd2b8b9 100755 >--- a/Koha/OAI/Harvester/Import/Record.pm >+++ b/Koha/OAI/Harvester/Import/Record.pm >@@ -27,16 +27,29 @@ use File::Basename; > use C4::Context; > use C4::Biblio; > >-use Koha::Database; > use Koha::OAI::Harvester::Import::MARCXML; >+use Koha::OAI::Harvester::Biblios; >+use Koha::OAI::Harvester::History; > >-=head1 API >+=head1 NAME > >-=head2 Class Methods >+Koha::OAI::Harvester::Import::Record > >-=cut >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::Import::Record; >+ my $oai_record = Koha::OAI::Harvester::Import::Record->new({ >+ doc => $dom, >+ repository => $repository, >+ }); >+ >+=head1 METHODS > >-my $schema = Koha::Database->new()->schema(); >+=head2 new >+ >+ Create object >+ >+=cut > > sub new { > my ($class, $args) = @_; >@@ -70,6 +83,12 @@ sub new { > return bless ($args, $class); > } > >+=head2 is_deleted_upstream >+ >+ Returns true if OAI-PMH record is deleted upstream >+ >+=cut >+ > sub is_deleted_upstream { > my ($self, $args) = @_; > if ($self->{header_status}){ >@@ -80,6 +99,15 @@ sub is_deleted_upstream { > return 0; > } > >+=head2 set_filter >+ >+ $self->set_filter("/path/to/filter.xsl"); >+ >+ Set a XSLT to use to filter records on import. This >+ takes a full filepath as an argument. >+ >+=cut >+ > sub set_filter { > my ($self, $filter_definition) = @_; > >@@ -121,6 +149,12 @@ sub _parse_filter { > return ($type,$filter); > } > >+=head2 filter >+ >+ Filters the OAI-PMH record using a filter >+ >+=cut >+ > sub filter { > my ($self) = @_; > my $filtered = 0; >@@ -165,7 +199,7 @@ sub _find_koha_link { > my $record_type = $args->{record_type} // "biblio"; > my $link_id; > if ($record_type eq "biblio"){ >- my $link = $schema->resultset('OaiHarvesterBiblio')->find( >+ my $link = Koha::OAI::Harvester::Biblios->new->find( > { > oai_repository => $self->{repository}, > oai_identifier => $self->{header_identifier}, >@@ -173,13 +207,13 @@ sub _find_koha_link { > { key => "oai_record",} > ); > if ($link && $link->biblionumber){ >- $link_id = $link->biblionumber->id; >+ $link_id = $link->biblionumber; > } > } > return $link_id; > } > >-=head3 import_record >+=head2 import_record > > my ($action,$record_id) = $oai_record->import_record({ > filter => $filter, >@@ -246,7 +280,7 @@ sub import_record { > } > > #Log record details to database >- my $importer = $schema->resultset('OaiHarvesterHistory')->create({ >+ Koha::OAI::Harvester::History->new({ > header_identifier => $self->{header_identifier}, > header_datestamp => $self->{header_datestamp}, > header_status => $self->{header_status}, >@@ -257,18 +291,25 @@ sub import_record { > framework => $framework, > record_type => $record_type, > matcher_code => $matcher ? $matcher->code : undef, >- }); >+ })->store(); > > return ($action,$linked_id); > } > >+=head2 link_koha_record >+ >+ Link an OAI-PMH record with a Koha record using >+ the OAI-PMH repository and OAI-PMH identifier >+ >+=cut >+ > sub link_koha_record { > my ($self, $args) = @_; > my $record_type = $args->{record_type} // "biblio"; > my $koha_id = $args->{koha_id}; > if ($koha_id){ > if ($record_type eq "biblio"){ >- my $import_oai_biblio = $schema->resultset('OaiHarvesterBiblio')->find_or_create({ >+ my $import_oai_biblio = Koha::OAI::Harvester::Biblios->new->find_or_create({ > oai_repository => $self->{repository}, > oai_identifier => $self->{header_identifier}, > biblionumber => $koha_id, >@@ -280,6 +321,12 @@ sub link_koha_record { > } > } > >+=head2 delete_koha_record >+ >+ Delete a Koha record >+ >+=cut >+ > sub delete_koha_record { > my ($self, $args) = @_; > my $record_type = $args->{record_type} // "biblio"; >diff --git a/Koha/OAI/Harvester/ImportQueue.pm b/Koha/OAI/Harvester/ImportQueue.pm >new file mode 100644 >index 0000000..065170f >--- /dev/null >+++ b/Koha/OAI/Harvester/ImportQueue.pm >@@ -0,0 +1,49 @@ >+package Koha::OAI::Harvester::ImportQueue; >+ >+# Copyright Prosentient Systems 2018 >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it under the >+# terms of the GNU General Public License as published by the Free Software >+# Foundation; either version 3 of the License, or (at your option) any later >+# version. >+# >+# Koha is distributed in the hope that it will be useful, but WITHOUT ANY >+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR >+# A PARTICULAR PURPOSE. See the GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License along >+# with Koha; if not, write to the Free Software Foundation, Inc., >+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >+ >+use Modern::Perl; >+ >+use Carp; >+ >+use base qw(Koha::Object); >+ >+=head1 NAME >+ >+Koha::OAI::Harvester::ImportQueue >+ >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::ImportQueue; >+ my $request = Koha::OAI::Harvester::ImportQueue->find($id); >+ >+=head1 METHODS >+ >+=cut >+ >+sub _type { >+ return 'OaiHarvesterImportQueue'; >+} >+ >+=head1 AUTHOR >+ >+David Cook <dcook@prosentient.com.au> >+ >+=cut >+ >+1; >diff --git a/Koha/OAI/Harvester/ImportQueues.pm b/Koha/OAI/Harvester/ImportQueues.pm >new file mode 100644 >index 0000000..378027d >--- /dev/null >+++ b/Koha/OAI/Harvester/ImportQueues.pm >@@ -0,0 +1,60 @@ >+package Koha::OAI::Harvester::ImportQueues; >+ >+# Copyright Prosentient Systems 2018 >+# >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it under the >+# terms of the GNU General Public License as published by the Free Software >+# Foundation; either version 3 of the License, or (at your option) any later >+# version. >+# >+# Koha is distributed in the hope that it will be useful, but WITHOUT ANY >+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR >+# A PARTICULAR PURPOSE. See the GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License along >+# with Koha; if not, write to the Free Software Foundation, Inc., >+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >+ >+use Modern::Perl; >+ >+use Carp; >+ >+use Koha::Database; >+ >+use Koha::OAI::Harvester::ImportQueue; >+ >+use base qw(Koha::Objects); >+ >+=head1 NAME >+ >+Koha::OAI::Harvester::ImportQueues >+ >+=head2 METHODS >+ >+=cut >+ >+=head3 _type >+ >+=cut >+ >+sub _type { >+ return 'OaiHarvesterImportQueue'; >+} >+ >+=head3 object_class >+ >+=cut >+ >+sub object_class { >+ return 'Koha::OAI::Harvester::ImportQueue'; >+} >+ >+=head1 AUTHOR >+ >+David Cook <dcook@prosentient.com.au> >+ >+=cut >+ >+1; >diff --git a/Koha/OAI/Harvester/Listener.pm b/Koha/OAI/Harvester/Listener.pm >index 03ec391..a40cf70 100644 >--- a/Koha/OAI/Harvester/Listener.pm >+++ b/Koha/OAI/Harvester/Listener.pm >@@ -24,12 +24,38 @@ use IO::Socket qw(AF_UNIX); > use JSON; > use URI; > >+=head1 NAME >+ >+Koha::OAI::Harvester::Listener >+ >+=head1 SYNOPSIS >+ >+ use Koha::OAI::Harvester::Listener; >+ my $listener = Koha::OAI::Harvester::Listener->spawn({ >+ logger => $logger, >+ socket => $socket_addr, >+ }); >+ >+=head1 METHODS >+ >+=head2 new >+ >+ Create object >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; > return bless ($args, $class); > } > >+=head2 spawn >+ >+ Create POE session for listener >+ >+=cut >+ > sub spawn { > my ($class, $args) = @_; > my $self = $class->new($args); >@@ -50,6 +76,12 @@ sub spawn { > ); > } > >+=head2 on_start >+ >+ Internal method for starting listener >+ >+=cut >+ > sub on_start { > my ($kernel,$heap,$socket_uri) = @_[KERNEL,HEAP,ARG0]; > >@@ -69,6 +101,12 @@ sub on_start { > } > } > >+=head2 on_server_success >+ >+ Internal event handler for successful connection to server >+ >+=cut >+ > sub on_server_success { > my ($self, $client_socket, $server_wheel_id, $heap, $session) = @_[OBJECT, ARG0, ARG3, HEAP,SESSION]; > my $logger = $self->{logger}; >@@ -88,6 +126,12 @@ sub on_server_success { > $client_wheel->put("HELLO"); > } > >+=head2 on_server_error >+ >+ Internal event handler for server errors >+ >+=cut >+ > sub on_server_error { > my ($self, $operation, $errnum, $errstr, $heap, $session) = @_[OBJECT, ARG0, ARG1, ARG2,HEAP, SESSION]; > my $logger = $self->{logger}; >@@ -95,6 +139,12 @@ sub on_server_error { > delete $heap->{server}; > } > >+=head2 on_client_error >+ >+ Internal event handler for errors relating to the client connection >+ >+=cut >+ > sub on_client_error { > my ($self, $wheel_id,$heap,$session) = @_[OBJECT, ARG3,HEAP,SESSION]; > my $logger = $self->{logger}; >@@ -102,6 +152,12 @@ sub on_client_error { > delete $heap->{client}->{$wheel_id}; > } > >+=head2 on_client_input >+ >+ Internal event handler for input from clients >+ >+=cut >+ > sub on_client_input { > my ($self, $input, $wheel_id, $session, $kernel, $heap) = @_[OBJECT, ARG0, ARG1, SESSION, KERNEL, HEAP]; > my $logger = $self->{logger}; >diff --git a/Koha/OAI/Harvester/Request.pm b/Koha/OAI/Harvester/Request.pm >index 3824947..f2fdea3 100644 >--- a/Koha/OAI/Harvester/Request.pm >+++ b/Koha/OAI/Harvester/Request.pm >@@ -29,17 +29,14 @@ use HTTP::OAI; > > =head1 NAME > >-Koha::OAI::Harvester::Request - >+Koha::OAI::Harvester::Request > >-=head1 API >+=head1 SYNOPSIS > >-=head2 Class Methods >+ use Koha::OAI::Harvester::Request; >+ my $request = Koha::OAI::Harvester::Requests->find($id); > >-=cut >- >- >- >-=head3 _type >+=head1 METHODS > > =cut > >@@ -47,6 +44,12 @@ sub _type { > return 'OaiHarvesterRequest'; > } > >+=head2 validate >+ >+ Method to validate the parameters of an OAI-PMH request >+ >+=cut >+ > sub validate { > my ($self) = @_; > my $errors = {}; >diff --git a/Koha/OAI/Harvester/Worker.pm b/Koha/OAI/Harvester/Worker.pm >index 5374a29..f65d853 100644 >--- a/Koha/OAI/Harvester/Worker.pm >+++ b/Koha/OAI/Harvester/Worker.pm >@@ -22,6 +22,23 @@ use POE; > use DateTime; > use JSON; > >+=head1 NAME >+ >+Koha::OAI::Harvester::Worker >+ >+=head1 SYNOPSIS >+ >+ This is just a base class to use when writing workers for >+ the OAI-PMH harvester. >+ >+=head1 METHODS >+ >+=head2 new >+ >+ Create object >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; >@@ -29,6 +46,12 @@ sub new { > return bless ($args, $class); > } > >+=head2 run >+ >+ The entry point to getting a worker to process a task >+ >+=cut >+ > sub run { > my ($self,$args) = @_; > my $postback = $args->{postback}; >@@ -53,6 +76,12 @@ sub run { > ); > } > >+=head2 stop_worker >+ >+ Internal method for killing this worker's child processes >+ >+=cut >+ > sub stop_worker { > my ($self,$heap) = @_[OBJECT,HEAP]; > if (my $child_processes = $heap->{children_by_pid}){ >@@ -63,6 +92,12 @@ sub stop_worker { > } > } > >+=head2 on_stop >+ >+ Internal event handler for deregistering the worker session >+ from the harvester's roster of workers >+ >+=cut > > sub on_stop { > my ($self,$kernel) = @_[OBJECT,KERNEL]; >@@ -71,6 +106,12 @@ sub on_stop { > $kernel->call("harvester","deregister",$self->{type}); > } > >+=head2 on_child_stdout >+ >+ Internal event handler for reading output from a child process >+ >+=cut >+ > # Wheel event, including the wheel's ID. > sub on_child_stdout { > my ($self, $stdout_line, $wheel_id) = @_[OBJECT, ARG0, ARG1]; >@@ -92,6 +133,12 @@ sub on_child_stdout { > } > } > >+=head2 on_child_stderr >+ >+ Internal event handler for reading errors from a child process >+ >+=cut >+ > # Wheel event, including the wheel's ID. > sub on_child_stderr { > my ($self,$stderr_line, $wheel_id) = @_[OBJECT, ARG0, ARG1]; >@@ -103,6 +150,12 @@ sub on_child_stderr { > } > } > >+=head2 on_child_close >+ >+ Internal event handler for when a child closes its pipes >+ >+=cut >+ > # Wheel event, including the wheel's ID. > sub on_child_close { > my ($self,$heap,$wheel_id) = @_[OBJECT,HEAP,ARG0]; >@@ -124,6 +177,12 @@ sub on_child_close { > delete $heap->{children_by_pid}->{$child->PID}; > } > >+=head2 on_child_signal >+ >+ Internal event handler for when a child exits >+ >+=cut >+ > sub on_child_signal { > my ($self,$kernel,$pid,$status) = @_[OBJECT,KERNEL,ARG1,ARG2]; > my $type = $self->{type}; >diff --git a/Koha/OAI/Harvester/Worker/Download/Stream.pm b/Koha/OAI/Harvester/Worker/Download/Stream.pm >index c87580a..442511a 100644 >--- a/Koha/OAI/Harvester/Worker/Download/Stream.pm >+++ b/Koha/OAI/Harvester/Worker/Download/Stream.pm >@@ -28,6 +28,24 @@ use C4::Context; > use Koha::OAI::Harvester::Downloader; > use parent 'Koha::OAI::Harvester::Worker'; > >+=head1 NAME >+ >+Koha::OAI::Harvester::Worker::Download::Stream >+ >+=head1 SYNOPSIS >+ >+ This is a module used by the OAI-PMH harvester internally. >+ >+ As a bare minimum, it must define an "on_start" method. >+ >+=head1 METHODS >+ >+=head2 new >+ >+ Create object >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; >@@ -35,6 +53,12 @@ sub new { > return bless ($args, $class); > } > >+=head2 on_start >+ >+ Internal event handler for starting the processing of a task >+ >+=cut >+ > sub on_start { > my ($self, $kernel, $heap, $postback,$task,$session) = @_[OBJECT, KERNEL, HEAP, ARG0,ARG1,SESSION]; > #Save postback into heap so other event handlers can use it >@@ -71,6 +95,12 @@ sub on_start { > } > } > >+=head2 do_work >+ >+ Internal method for processing a download task >+ >+=cut >+ > sub do_work { > my ($self, $task) = @_; > my $batch = ( $self->{batch} && int($self->{batch}) ) ? $self->{batch} : 100; >diff --git a/Koha/OAI/Harvester/Worker/Import.pm b/Koha/OAI/Harvester/Worker/Import.pm >index ec86736..e57279a 100644 >--- a/Koha/OAI/Harvester/Worker/Import.pm >+++ b/Koha/OAI/Harvester/Worker/Import.pm >@@ -28,6 +28,24 @@ use Koha::OAI::Harvester::Import::Record; > > use parent 'Koha::OAI::Harvester::Worker'; > >+=head1 NAME >+ >+Koha::OAI::Harvester::Worker::Import >+ >+=head1 SYNOPSIS >+ >+ This is a module used by the OAI-PMH harvester internally. >+ >+ As a bare minimum, it must define an "on_start" method. >+ >+=head1 METHODS >+ >+=head2 new >+ >+ Create object >+ >+=cut >+ > sub new { > my ($class, $args) = @_; > $args = {} unless defined $args; >@@ -36,6 +54,12 @@ sub new { > return bless ($args, $class); > } > >+=head2 on_start >+ >+ Internal event handler for starting the processing of a task >+ >+=cut >+ > sub on_start { > my ($self, $kernel, $heap, $postback,$task) = @_[OBJECT, KERNEL, HEAP, ARG0,ARG1]; > >diff --git a/Koha/Schema/Result/Biblio.pm b/Koha/Schema/Result/Biblio.pm >index b118011..89bef66 100644 >--- a/Koha/Schema/Result/Biblio.pm >+++ b/Koha/Schema/Result/Biblio.pm >@@ -242,6 +242,21 @@ __PACKAGE__->has_many( > { cascade_copy => 0, cascade_delete => 0 }, > ); > >+=head2 oai_harvester_biblios >+ >+Type: has_many >+ >+Related object: L<Koha::Schema::Result::OaiHarvesterBiblio> >+ >+=cut >+ >+__PACKAGE__->has_many( >+ "oai_harvester_biblios", >+ "Koha::Schema::Result::OaiHarvesterBiblio", >+ { "foreign.biblionumber" => "self.biblionumber" }, >+ { cascade_copy => 0, cascade_delete => 0 }, >+); >+ > =head2 old_reserves > > Type: has_many >@@ -348,7 +363,7 @@ __PACKAGE__->has_many( > ); > > >-# Created by DBIx::Class::Schema::Loader v0.07042 @ 2018-02-16 17:54:53 >-# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:bUv00JjY09Hj2Zj4klqyxA >+# Created by DBIx::Class::Schema::Loader v0.07042 @ 2018-10-31 05:45:15 >+# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:ptt942qUu2GcGqOTRePNWQ > > 1; >diff --git a/Koha/Schema/Result/OaiHarvesterBiblio.pm b/Koha/Schema/Result/OaiHarvesterBiblio.pm >index 85bb64a..90f46e5 100755 >--- a/Koha/Schema/Result/OaiHarvesterBiblio.pm >+++ b/Koha/Schema/Result/OaiHarvesterBiblio.pm >@@ -112,8 +112,8 @@ __PACKAGE__->belongs_to( > ); > > >-# Created by DBIx::Class::Schema::Loader v0.07046 @ 2017-03-29 12:23:43 >-# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:2URn8tPABMKC+JuIMfGeYw >+# Created by DBIx::Class::Schema::Loader v0.07042 @ 2018-10-31 05:53:48 >+# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:awQgyVvOqnfBrOhl6tecBQ > > > # You can replace this text with custom code or comments, and it will be preserved on regeneration >diff --git a/Koha/Schema/Result/OaiHarvesterHistory.pm b/Koha/Schema/Result/OaiHarvesterHistory.pm >index 3e3d1d5..be1d619 100755 >--- a/Koha/Schema/Result/OaiHarvesterHistory.pm >+++ b/Koha/Schema/Result/OaiHarvesterHistory.pm >@@ -155,8 +155,8 @@ __PACKAGE__->add_columns( > __PACKAGE__->set_primary_key("import_oai_id"); > > >-# Created by DBIx::Class::Schema::Loader v0.07046 @ 2017-03-29 12:23:43 >-# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:Rp/ZEZsKVlLo2vaM3M37ow >+# Created by DBIx::Class::Schema::Loader v0.07042 @ 2018-10-31 05:53:48 >+# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:C/ETSPdx0O4xfQnu09kGeQ > > > # You can replace this text with custom code or comments, and it will be preserved on regeneration >diff --git a/Koha/Schema/Result/OaiHarvesterImportQueue.pm b/Koha/Schema/Result/OaiHarvesterImportQueue.pm >index 54188e3..4ff5e18 100755 >--- a/Koha/Schema/Result/OaiHarvesterImportQueue.pm >+++ b/Koha/Schema/Result/OaiHarvesterImportQueue.pm >@@ -98,8 +98,8 @@ __PACKAGE__->add_columns( > __PACKAGE__->set_primary_key("id"); > > >-# Created by DBIx::Class::Schema::Loader v0.07046 @ 2017-03-29 12:23:43 >-# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:zBD+hMawbvu7sonuLRHnCA >+# Created by DBIx::Class::Schema::Loader v0.07042 @ 2018-10-31 05:53:48 >+# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:1+ziAxPsvC8ftKLEAJ2keA > > > # You can replace this text with custom code or comments, and it will be preserved on regeneration >diff --git a/Koha/Schema/Result/OaiHarvesterRequest.pm b/Koha/Schema/Result/OaiHarvesterRequest.pm >index e322281..6ca35e3 100644 >--- a/Koha/Schema/Result/OaiHarvesterRequest.pm >+++ b/Koha/Schema/Result/OaiHarvesterRequest.pm >@@ -201,8 +201,8 @@ __PACKAGE__->add_columns( > __PACKAGE__->set_primary_key("id"); > > >-# Created by DBIx::Class::Schema::Loader v0.07046 @ 2017-04-07 11:26:24 >-# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:Vm/b4yurmr8WF7z+Fo6KEw >+# Created by DBIx::Class::Schema::Loader v0.07042 @ 2018-10-31 05:53:48 >+# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:adGx1Jo2gvt5PH1eX31y2Q > > > # You can replace this text with custom code or comments, and it will be preserved on regeneration >diff --git a/admin/columns_settings.yml b/admin/columns_settings.yml >index 1480019..1dbb701 100644 >--- a/admin/columns_settings.yml >+++ b/admin/columns_settings.yml >@@ -675,6 +675,65 @@ modules: > columnname: spent > > tools: >+ oai-pmh-harvester-dashboard: >+ saved-table: >+ - >+ columnname: name >+ - >+ columnname: url >+ - >+ columnname: set >+ - >+ columnname: from >+ - >+ columnname: until >+ - >+ columnname: interval >+ >+ submitted-table: >+ - >+ columnname: name >+ - >+ columnname: url >+ - >+ columnname: set >+ - >+ columnname: from >+ - >+ columnname: until >+ - >+ columnname: interval >+ - >+ columnname: "effective from" >+ - >+ columnname: "effective until" >+ - >+ columnname: "pending imports" >+ - >+ columnname: status >+ - >+ columnname: error >+ >+ history-table: >+ - >+ columnname: id >+ - >+ columnname: repository >+ - >+ columnname: identifier >+ - >+ columnname: datestamp >+ - >+ columnname: "upstream status" >+ - >+ columnname: "import status" >+ - >+ columnname: "import timestamp" >+ - >+ columnname: "imported record" >+ - >+ columnname: "downloaded record" >+ > notices: > lettert: > - >diff --git a/installer/data/mysql/atomicupdate/bug_10662.sql b/installer/data/mysql/atomicupdate/bug_10662.sql >index f42b732..30f1d18 100644 >--- a/installer/data/mysql/atomicupdate/bug_10662.sql >+++ b/installer/data/mysql/atomicupdate/bug_10662.sql >@@ -11,7 +11,7 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_biblios` ( > UNIQUE KEY `oai_record` (`oai_identifier`,`oai_repository`) USING BTREE, > KEY `FK_import_oai_biblio_1` (`biblionumber`), > CONSTRAINT `FK_import_oai_biblio_1` FOREIGN KEY (`biblionumber`) REFERENCES `biblio` (`biblionumber`) ON DELETE CASCADE ON UPDATE NO ACTION >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > > -- > -- Table structure for table 'oai_harvester_history' >@@ -31,7 +31,7 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_history` ( > `record_type` enum('biblio','auth','holdings') COLLATE utf8_unicode_ci NOT NULL, > `matcher_code` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL, > PRIMARY KEY (`import_oai_id`) >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > > -- > -- Table structure for table 'oai_harvester_import_queue' >@@ -44,7 +44,7 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_import_queue` ( > `result` text CHARACTER SET utf8 NOT NULL, > `result_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, > PRIMARY KEY (`id`) >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > > -- > -- Table structure for table 'oai_harvester_requests' >@@ -70,4 +70,4 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_requests` ( > `interval` int(10) unsigned NOT NULL, > `name` varchar(45) NOT NULL, > PRIMARY KEY (`id`) >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; >diff --git a/installer/data/mysql/kohastructure.sql b/installer/data/mysql/kohastructure.sql >index e66cdfa..87cd359 100644 >--- a/installer/data/mysql/kohastructure.sql >+++ b/installer/data/mysql/kohastructure.sql >@@ -4331,7 +4331,7 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_biblios` ( > UNIQUE KEY `oai_record` (`oai_identifier`,`oai_repository`) USING BTREE, > KEY `FK_import_oai_biblio_1` (`biblionumber`), > CONSTRAINT `FK_import_oai_biblio_1` FOREIGN KEY (`biblionumber`) REFERENCES `biblio` (`biblionumber`) ON DELETE CASCADE ON UPDATE NO ACTION >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > > -- > -- Table structure for table 'oai_harvester_history' >@@ -4351,7 +4351,7 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_history` ( > `record_type` enum('biblio','auth','holdings') COLLATE utf8_unicode_ci NOT NULL, > `matcher_code` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL, > PRIMARY KEY (`import_oai_id`) >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > > -- > -- Table structure for table 'oai_harvester_import_queue' >@@ -4364,7 +4364,7 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_import_queue` ( > `result` text CHARACTER SET utf8 NOT NULL, > `result_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, > PRIMARY KEY (`id`) >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > > -- > -- Table structure for table 'oai_harvester_requests' >@@ -4390,7 +4390,7 @@ CREATE TABLE IF NOT EXISTS `oai_harvester_requests` ( > `interval` int(10) unsigned NOT NULL, > `name` varchar(45) NOT NULL, > PRIMARY KEY (`id`) >-) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; >+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci; > > /*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */; > /*!40101 SET SQL_MODE=@OLD_SQL_MODE */; >diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/dashboard.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/dashboard.tt >index 36fdccf..e939db6 100644 >--- a/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/dashboard.tt >+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/dashboard.tt >@@ -1,107 +1,10 @@ >+[% USE ColumnsSettings %] >+[% SET footerjs = 1 %] > [% INCLUDE 'doc-head-open.inc' %] > <title>Koha › Tools › OAI-PMH harvester</title> > [% INCLUDE 'doc-head-close.inc' %] >-[% INCLUDE 'datatables.inc' %] > [% dashboard_page = '/cgi-bin/koha/tools/oai-pmh-harvester/dashboard.pl' %] > [% request_page = '/cgi-bin/koha/tools/oai-pmh-harvester/request.pl' %] >-<script type="text/javascript"> >-//<![CDATA[ >- $(document).ready(function() { >- $('#dashboard-items').tabs(); >- [% IF ( result.start.defined || result.stop.defined || result.delete.defined ) %] >- $('#dashboard-items').tabs("option","active",1); >- [% END %] >- >- var saved_table = $('#saved-table').DataTable({}); >- var submitted_table = $('#submitted-table').DataTable({}); >- var history_table = $('#history-table').DataTable({ >- serverSide: true, >- searching: true, >- processing: true, >- ajax: { >- "url": '/cgi-bin/koha/svc/oai-pmh-harvester/history', >- contentType: 'application/json', >- type: 'POST', >- data: function ( d ) { >- return JSON.stringify( d ); >- }, >- dataSrc: function (json){ >- var recordsTotal = json.recordsTotal; >- if(recordsTotal){ >- $('#import_count').text( "("+recordsTotal+")" ); >- } >- return json.data; >- } >- }, >- columns: [ >- { data: 'import_oai_id', }, >- { data: 'repository', }, >- { data: 'header_identifier', }, >- { data: 'header_datestamp', }, >- { >- data: 'header_status', render: function (data, type, full, meta){ >- var display_string = _("Active"); >- if (data == "deleted"){ >- display_string = _("Deleted"); >- } >- return display_string; >- } >- }, >- { >- data: 'status', render: function (data, type, full, meta){ >- var display_string = data; >- if (data == "added"){ >- display_string = _("Added"); >- } >- else if (data == "error"){ >- display_string = _("Error"); >- } >- else if (data == "not_found"){ >- display_string = _("Not found"); >- } >- else if (data == "updated"){ >- display_string = _("Updated"); >- } >- return display_string; >- } >- }, >- { data: 'upload_timestamp', }, >- { >- data: 'imported_record', render: function (data, type, full, meta){ >- var display_string = data; >- var record_type = (full.record_type) ? full.record_type : 'biblio'; >- if (data && record_type == 'biblio'){ >- var link_text = _("View biblio record"); >- var link = '<a href="/cgi-bin/koha/catalogue/detail.pl?biblionumber='+data+'">'+link_text+'</a>'; >- display_string = link; >- } >- return display_string; >- }, searchable: false >- >- }, >- { >- data: 'import_oai_id', render: function (data, type, full, meta){ >- var display_string = data; >- var link_text = _("View record"); >- var link = '<a href="/cgi-bin/koha/tools/oai-pmh-harvester/record.pl?import_oai_id='+data+'">'+link_text+'</a>'; >- display_string = link; >- return display_string; >- }, searchable: false >- }, >- ], >- //In theory, it would be nice to sort in descending order, but >- //it causes severe paging issues the data is frequently updated. >- //Caching would make the paging more stable, but the results would be stale. >- order: [ >- [ 0, 'asc' ], >- ] >- }); >- $('#refresh-button').click(function(){ >- history_table.ajax.reload( null, false ); >- }); >- }); >-//]]> >-</script> > <style type="text/css"> > a.paginate_button { > padding: 2px; >@@ -118,7 +21,7 @@ > <div class="yui-b"> > <h1>OAI-PMH harvester</h1> > <div id="toolbar" class="btn-toolbar"> >- <a id="new-request" class="btn btn-default btn-sm" href="[% request_page %]?op=new"><i class="fa fa-plus"></i> New request</a> >+ <a id="new-request" class="btn btn-default btn-sm" href="[% request_page | url %]?op=new"><i class="fa fa-plus"></i> New request</a> > </div> > [% IF ( harvester.offline ) %] > <div class="alert"> >@@ -128,10 +31,10 @@ > <div id="dashboard-items" class="toptabs"> > <ul> > <li> >- <a href="#saved_requests">Saved requests <span id="saved_count">([% saved_requests.size %])</span></a> >+ <a href="#saved_requests">Saved requests <span id="saved_count">([% saved_requests.size | html %])</span></a> > </li> > <li> >- <a href="#submitted_requests">Submitted requests <span id="submitted_count">([% submitted_requests.size %])</span></a> >+ <a href="#submitted_requests">Submitted requests <span id="submitted_count">([% submitted_requests.size | html %])</span></a> > </li> > <li> > <a href="#imports">Import history <span id="import_count">(0)</span></a> >@@ -184,15 +87,15 @@ > [% IF ( submitted_requests ) %] > [% FOREACH submitted_request IN submitted_requests %] > <tr> >- <td>[% submitted_request.name %]</td> >- <td>[% submitted_request.parameters.oai_pmh.baseURL %]</td> >- <td>[% submitted_request.parameters.oai_pmh.set %]</td> >- <td>[% submitted_request.parameters.oai_pmh.from %]</td> >- <td>[% submitted_request.parameters.oai_pmh.until %]</td> >- <td>[% submitted_request.interval %]</td> >- <td>[% submitted_request.effective_from %]</td> >- <td>[% submitted_request.effective_until %]</td> >- <td>[% submitted_request.pending_imports %]</td> >+ <td>[% submitted_request.name | html %]</td> >+ <td>[% submitted_request.parameters.oai_pmh.baseURL | html %]</td> >+ <td>[% submitted_request.parameters.oai_pmh.set | html %]</td> >+ <td>[% submitted_request.parameters.oai_pmh.from | html %]</td> >+ <td>[% submitted_request.parameters.oai_pmh.until | html %]</td> >+ <td>[% submitted_request.interval | html %]</td> >+ <td>[% submitted_request.effective_from | html %]</td> >+ <td>[% submitted_request.effective_until | html %]</td> >+ <td>[% submitted_request.pending_imports |html %]</td> > <td> > [% IF ( submitted_status = submitted_request.status ) %] > [% IF ( submitted_status == "new" ) %] >@@ -214,13 +117,13 @@ > <a class="btn btn-default btn-xs dropdown-toggle" role="button" data-toggle="dropdown" href="#">Actions <span class="caret"></span></a> > <ul class="dropdown-menu pull-right" role="menu"> > <li> >- <a href="[% dashboard_page %]?op=start&uuid=[% submitted_request.uuid %]"><i class="fa fa-play"></i> Start</a> >+ <a href="[% dashboard_page | url %]?op=start&uuid=[% submitted_request.uuid | url %]"><i class="fa fa-play"></i> Start</a> > </li> > <li> >- <a href="[% dashboard_page %]?op=stop&uuid=[% submitted_request.uuid %]"><i class="fa fa-stop"></i> Stop</a> >+ <a href="[% dashboard_page | url %]?op=stop&uuid=[% submitted_request.uuid | url %]"><i class="fa fa-stop"></i> Stop</a> > </li> > <li> >- <a href="[% dashboard_page %]?op=delete&uuid=[% submitted_request.uuid %]"><i class="fa fa-trash"></i> Delete</a> >+ <a href="[% dashboard_page | url %]?op=delete&uuid=[% submitted_request.uuid | url %]"><i class="fa fa-trash"></i> Delete</a> > </li> > </ul> > </div> >@@ -264,20 +167,20 @@ > [% IF ( saved_requests ) %] > [% FOREACH saved_request IN saved_requests %] > <tr> >- <td>[% saved_request.name %]</td> >- <td>[% saved_request.http_url %]</td> >- <!-- <td>[% saved_request.oai_verb %]</td> >- <td>[% saved_request.oai_metadataPrefix %]</td> >- <td>[% saved_request.oai_identifier %]</td> --> >- <td>[% saved_request.oai_set %]</td> >- <td>[% saved_request.oai_from %]</td> >- <td>[% saved_request.oai_until %]</td> >- <td>[% saved_request.interval %]</td> >+ <td>[% saved_request.name | html %]</td> >+ <td>[% saved_request.http_url | html %]</td> >+ <!-- <td>[% saved_request.oai_verb | html %]</td> >+ <td>[% saved_request.oai_metadataPrefix | html %]</td> >+ <td>[% saved_request.oai_identifier | html %]</td> --> >+ <td>[% saved_request.oai_set | html %]</td> >+ <td>[% saved_request.oai_from | html %]</td> >+ <td>[% saved_request.oai_until | html %]</td> >+ <td>[% saved_request.interval | html %]</td> > <!-- <td> > [% IF ( saved_request.import_filter == "default" ) %] > <span>Default</span> > [% ELSE %] >- [% saved_request.import_filter %] >+ [% saved_request.import_filter | html %] > [% END %] > </td> > <td> >@@ -288,16 +191,16 @@ > [% END %] > [% END %] > [% IF ( display_framework ) %] >- [% display_framework.frameworktext %] >+ [% display_framework.frameworktext | html %] > [% ELSE %] >- [% saved_request.import_framework_code %] >+ [% saved_request.import_framework_code | html %] > [% END %] > </td> > <td> > [% IF ( saved_request.import_record_type == "biblio" ) %] > <span>Bibliographic</span> > [% ELSE %] >- [% saved_request.import_record_type %] >+ [% saved_request.import_record_type | html %] > [% END %] > </td> > <td> >@@ -308,9 +211,9 @@ > [% END %] > [% END %] > [% IF ( display_matcher ) %] >- [% display_matcher.description %] >+ [% display_matcher.description | html %] > [% ELSE %] >- [% saved_request.import_matcher_code %] >+ [% saved_request.import_matcher_code | html %] > [% END %] > </td> --> > <td> >@@ -318,16 +221,16 @@ > <a class="btn btn-default btn-xs dropdown-toggle" role="button" data-toggle="dropdown" href="#">Actions <span class="caret"></span></a> > <ul class="dropdown-menu pull-right" role="menu"> > <li> >- <a href="[% request_page %]?op=edit&id=[% saved_request.id %]"><i class="fa fa-pencil"></i> Edit</a> >+ <a href="[% request_page | url %]?op=edit&id=[% saved_request.id | url %]"><i class="fa fa-pencil"></i> Edit</a> > </li> > <li> >- <a href="[% request_page %]?op=new&id=[% saved_request.id %]"><i class="fa fa-copy"></i> Copy</a> >+ <a href="[% request_page | url %]?op=new&id=[% saved_request.id | url %]"><i class="fa fa-copy"></i> Copy</a> > </li> > <li> >- <a href="[% dashboard_page %]?op=send&id=[% saved_request.id %]"><i class="fa fa-send"></i> Submit</a> >+ <a href="[% dashboard_page | url %]?op=send&id=[% saved_request.id | url %]"><i class="fa fa-send"></i> Submit</a> > </li> > <li> >- <a href="[% request_page %]?op=delete&id=[% saved_request.id %]"><i class="fa fa-trash"></i> Delete</a> >+ <a href="[% request_page | url %]?op=delete&id=[% saved_request.id | url %]"><i class="fa fa-trash"></i> Delete</a> > </li> > </ul> > </div> >@@ -366,4 +269,110 @@ > [% INCLUDE 'tools-menu.inc' %] > </div> > </div> >+ >+[% MACRO jsinclude BLOCK %] >+ [% INCLUDE 'datatables.inc' %] >+ [% INCLUDE 'columns_settings.inc' %] >+ <script type="text/javascript"> >+ $(document).ready(function() { >+ $('#dashboard-items').tabs(); >+ [% IF ( result.start.defined || result.stop.defined || result.delete.defined ) %] >+ $('#dashboard-items').tabs("option","active",1); >+ [% END %] >+ >+ var saved_table_columns_settings = [% ColumnsSettings.GetColumns( 'tools','oai-pmh-harvester-dashboard', 'saved-table', 'json' ) | $raw %]; >+ var saved_table = KohaTable("saved-table",{},saved_table_columns_settings); >+ >+ var submitted_table_columns_settings = [% ColumnsSettings.GetColumns( 'tools','oai-pmh-harvester-dashboard', 'submitted-table', 'json' ) | $raw %]; >+ var submitted_table = KohaTable("submitted-table",{},submitted_table_columns_settings); >+ >+ var history_table_columns_settings = [% ColumnsSettings.GetColumns( 'tools','oai-pmh-harvester-dashboard', 'history-table', 'json' ) | $raw %]; >+ var history_table = KohaTable("history-table",{ >+ serverSide: true, >+ searching: true, >+ processing: true, >+ ajax: { >+ "url": '/cgi-bin/koha/svc/oai-pmh-harvester/history', >+ contentType: 'application/json', >+ type: 'POST', >+ data: function ( d ) { >+ return JSON.stringify( d ); >+ }, >+ dataSrc: function (json){ >+ var recordsTotal = json.recordsTotal; >+ if(recordsTotal){ >+ $('#import_count').text( "("+recordsTotal+")" ); >+ } >+ return json.data; >+ } >+ }, >+ columns: [ >+ { data: 'import_oai_id', }, >+ { data: 'repository', }, >+ { data: 'header_identifier', }, >+ { data: 'header_datestamp', }, >+ { >+ data: 'header_status', render: function (data, type, full, meta){ >+ var display_string = _("Active"); >+ if (data == "deleted"){ >+ display_string = _("Deleted"); >+ } >+ return display_string; >+ } >+ }, >+ { >+ data: 'status', render: function (data, type, full, meta){ >+ var display_string = data; >+ if (data == "added"){ >+ display_string = _("Added"); >+ } >+ else if (data == "error"){ >+ display_string = _("Error"); >+ } >+ else if (data == "not_found"){ >+ display_string = _("Not found"); >+ } >+ else if (data == "updated"){ >+ display_string = _("Updated"); >+ } >+ return display_string; >+ } >+ }, >+ { data: 'upload_timestamp', }, >+ { >+ data: 'imported_record', render: function (data, type, full, meta){ >+ var display_string = data; >+ var record_type = (full.record_type) ? full.record_type : 'biblio'; >+ if (data && record_type == 'biblio'){ >+ var link_text = _("View biblio record"); >+ var link = '<a href="/cgi-bin/koha/catalogue/detail.pl?biblionumber='+data+'">'+link_text+'</a>'; >+ display_string = link; >+ } >+ return display_string; >+ }, searchable: false >+ >+ }, >+ { >+ data: 'import_oai_id', render: function (data, type, full, meta){ >+ var display_string = data; >+ var link_text = _("View record"); >+ var link = '<a href="/cgi-bin/koha/tools/oai-pmh-harvester/record.pl?import_oai_id='+data+'">'+link_text+'</a>'; >+ display_string = link; >+ return display_string; >+ }, searchable: false >+ }, >+ ], >+ //In theory, it would be nice to sort in descending order, but >+ //it causes severe paging issues the data is frequently updated. >+ //Caching would make the paging more stable, but the results would be stale. >+ order: [ >+ [ 0, 'asc' ], >+ ] >+ },history_table_columns_settings); >+ $('#refresh-button').click(function(){ >+ history_table.ajax.reload( null, false ); >+ }); >+ }); >+ </script> >+[% END %] > [% INCLUDE 'intranet-bottom.inc' %] >diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/request.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/request.tt >index ca2d98d..9196942 100644 >--- a/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/request.tt >+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/tools/oai-pmh-harvester/request.tt >@@ -1,48 +1,9 @@ >+[% USE Asset %] >+[% USE raw %] >+[% SET footerjs = 1 %] > [% INCLUDE 'doc-head-open.inc' %] > <title>Koha › Tools › OAI-PMH harvester › Request</title> > [% INCLUDE 'doc-head-close.inc' %] >-[% INCLUDE 'calendar.inc' %] >-<script type="text/javascript" src="[% interface %]/lib/jquery/plugins/jquery-ui-timepicker-addon.min.js"></script> >-[% INCLUDE 'timepicker.inc' %] >-<script type="text/javascript"> >-//<![CDATA[ >- $(document).ready(function() { >- toggle_identifier(); >- $("#oai_verb").on("click",toggle_identifier); >- $(".datetime_utc").datetimepicker({ >- separator: "T", >- timeSuffix: 'Z', >- dateFormat: "yy-mm-dd", >- timeFormat: "HH:mm:ss", >- hour: 0, >- minute: 0, >- second: 0, >- showSecond: 1, >- // timezone doesn't work with the "Now" button in v1.4.3 although it appears to in v1.6.1 >- // timezone: '+000' >- }); >- }); >- function toggle_identifier (){ >- var verb = $("#oai_verb").find(":selected").val(); >- var oai_identifier = $("#oai_identifier"); >- var oai_set = $("#oai_set"); >- var oai_from = $("#oai_from"); >- var oai_until = $("#oai_until"); >- if (verb == 'ListRecords'){ >- oai_identifier.prop('disabled', true); >- oai_set.prop('disabled', false); >- oai_from.prop('disabled', false); >- oai_until.prop('disabled', false); >- } >- else if (verb == 'GetRecord'){ >- oai_identifier.prop('disabled', false); >- oai_set.prop('disabled', true); >- oai_from.prop('disabled', true); >- oai_until.prop('disabled', true); >- } >- } >-//]]> >-</script> > <style type="text/css"> > /* Override staff-global.css which hides second, millisecond, and microsecond sliders */ > .ui_tpicker_second { >@@ -76,16 +37,16 @@ > [% ELSIF ( op == "edit" ) %] > <input type="hidden" name="op" value="update" /> > [% ELSE %] >- <input type="hidden" name="op" value="[% op %]" /> >+ <input type="hidden" name="op" value="[% op | html %]" /> > [% END %] > [% IF ( id ) %] >- <input type="hidden" name="id" value="[% id %]" /> >+ <input type="hidden" name="id" value="[% id | html %]" /> > [% END %] > <fieldset class="rows"> > <ol> > <li> > <label for="name">Name:</label> >- <input type="text" size="30" id="name" name="name" value="[% oai_pmh_request.name %]"/> >+ <input type="text" size="30" id="name" name="name" value="[% oai_pmh_request.name | html %]"/> > <span class="help">This is just a short name to help in managing requests.</span> > </li> > </ol> >@@ -95,7 +56,7 @@ > <ol> > <li> > <label for="http_url">URL:</label> >- <input type="text" size="30" id="http_url" name="http_url" value="[% oai_pmh_request.http_url %]"> >+ <input type="text" size="30" id="http_url" name="http_url" value="[% oai_pmh_request.http_url | html %]"> > [% IF (errors.http_url.malformed) %]<span class="error">[This must be a properly formatted HTTP or HTTPS URL.]</span>[% END %] > [% IF (errors.http.404) %]<span class="error">[Cannot find address specified by this URL.]</span>[% END %] > [% IF (errors.http.401) %]<span class="error">[Permission denied to access this URL.]</span>[% END %] >@@ -106,15 +67,15 @@ > <ol> > <li> > <label for="http_username">Username:</label> >- <input type="text" size="30" id="http_username" name="http_username" value="[% oai_pmh_request.http_username %]"> >+ <input type="text" size="30" id="http_username" name="http_username" value="[% oai_pmh_request.http_username | html %]"> > </li> > <li> > <label for="http_password">Password:</label> >- <input type="text" size="30" id="http_password" name="http_password" value="[% oai_pmh_request.http_password %]"> >+ <input type="text" size="30" id="http_password" name="http_password" value="[% oai_pmh_request.http_password | html %]"> > </li> > <li> > <label for="http_realm">Realm:</label> >- <input type="text" size="30" id="http_realm" name="http_realm" value="[% oai_pmh_request.http_realm %]"> >+ <input type="text" size="30" id="http_realm" name="http_realm" value="[% oai_pmh_request.http_realm | html %]"> > </li> > </ol> > </fieldset> >@@ -138,31 +99,31 @@ > </li> > <li> > <label for="oai_metadataPrefix">Metadata prefix:</label> >- <input type="text" size="30" id="oai_metadataPrefix" name="oai_metadataPrefix" value="[% oai_pmh_request.oai_metadataPrefix %]"> >+ <input type="text" size="30" id="oai_metadataPrefix" name="oai_metadataPrefix" value="[% oai_pmh_request.oai_metadataPrefix | html %]"> > [% IF (errors.oai_metadataPrefix.unavailable) %]<span class="error">[This metadataPrefix is unavailable from this OAI-PMH provider.]</span>[% END %] > [% IF (errors.oai_metadataPrefix.missing) %]<span class="error">[metadataPrefix is a required field for an OAI-PMH request.]</span>[% END %] > </li> > <li> > <label for="oai_identifier">Identifier:</label> >- <input type="text" size="30" id="oai_identifier" name="oai_identifier" value="[% oai_pmh_request.oai_identifier %]"> >+ <input type="text" size="30" id="oai_identifier" name="oai_identifier" value="[% oai_pmh_request.oai_identifier | html %]"> > [% IF (errors.oai_identifier.missing) %]<span class="error">[Identifier is a required field when using GetRecord for an OAI-PMH request.]</span>[% END %] > </li> > <li> > <label for="oai_set">Set:</label> >- <input type="text" size="30" id="oai_set" name="oai_set" value="[% oai_pmh_request.oai_set %]"> >+ <input type="text" size="30" id="oai_set" name="oai_set" value="[% oai_pmh_request.oai_set | html %]"> > [% IF (errors.oai_set.unavailable) %]<span class="error">[This set is unavailable from this OAI-PMH provider.]</span>[% END %] > </li> > [% IF (errors.oai.granularity_mismatch) %]<span class="error">[You must specify the same granularity for both From and Until.]</span>[% END %] > <li> > <label for="oai_from">From:</label> >- <input type="text" size="30" class="datetime_utc" id="oai_from" name="oai_from" value="[% oai_pmh_request.oai_from %]"> >+ <input type="text" size="30" class="datetime_utc" id="oai_from" name="oai_from" value="[% oai_pmh_request.oai_from | html %]"> > <span class="help">This value will be treated as UTC time. Note that some repositories only support YYYY-MM-DD datestamps.</span> > [% IF (errors.oai_from.malformed) %]<span class="error">[This must be in YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ format.]</span>[% END %] > [% IF (errors.oai_from.unavailable) %]<span class="error">[This granularity is unsupported by this OAI-PMH provider.]</span>[% END %] > </li> > <li> > <label for="oai_until">Until:</label> >- <input type="text" size="30" class="datetime_utc" id="oai_until" name="oai_until" value="[% oai_pmh_request.oai_until %]"> >+ <input type="text" size="30" class="datetime_utc" id="oai_until" name="oai_until" value="[% oai_pmh_request.oai_until | html %]"> > <span class="help">This value will be treated as UTC time. Note that some repositories only support YYYY-MM-DD datestamps.</span> > [% IF (errors.oai_until.malformed) %]<span class="error">[This must be in YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ format.]</span>[% END %] > [% IF (errors.oai_until.unavailable) %]<span class="error">[This granularity is unsupported by this OAI-PMH provider.]</span>[% END %] >@@ -177,7 +138,7 @@ > [% IF ( oai_pmh_request.import_filter == "default" ) %] > <input type="text" size="30" id="import_filter" name="import_filter" value="default"> > [% ELSE %] >- <input type="text" size="30" id="import_filter" name="import_filter" value="[% oai_pmh_request.import_filter %]"> >+ <input type="text" size="30" id="import_filter" name="import_filter" value="[% oai_pmh_request.import_filter | html %]"> > [% END %] > <span class="help">If no filter is entered, the default filter will be used.</span> > </li> >@@ -187,9 +148,9 @@ > <option value="">Default framework</option> > [% FOREACH framework IN frameworks %] > [% IF ( oai_pmh_request.import_framework_code == framework.frameworkcode ) %] >- <option selected="selected" value="[% framework.frameworkcode %]">[% framework.frameworktext %]</option> >+ <option selected="selected" value="[% framework.frameworkcode | html %]">[% framework.frameworktext | html %]</option> > [% ELSE %] >- <option value="[% framework.frameworkcode %]">[% framework.frameworktext %]</option> >+ <option value="[% framework.frameworkcode | html %]">[% framework.frameworktext | html %]</option> > [% END %] > [% END %] > </select> >@@ -206,9 +167,9 @@ > <option value="">None</option> > [% FOREACH matcher IN matchers %] > [% IF ( oai_pmh_request.import_matcher_code == matcher.code ) %] >- <option value="[% matcher.code %]" selected="selected">[% matcher.description %]</option> >+ <option value="[% matcher.code | html %]" selected="selected">[% matcher.description | html %]</option> > [% ELSE %] >- <option value="[% matcher.code %]">[% matcher.description %]</option> >+ <option value="[% matcher.code | html %]">[% matcher.description | html %]</option> > [% END %] > [% END %] > </select> >@@ -221,7 +182,7 @@ > <ol> > <li> > <label for="interval">Interval (seconds): </label> >- <input type="text" id="interval" name="interval" value="[% oai_pmh_request.interval %]" size="4"> >+ <input type="text" id="interval" name="interval" value="[% oai_pmh_request.interval | html %]" size="4"> > <span class="help">The download request will be repeated in intervals of this many seconds. Enter "0" if you want the task to only happen once.</span> > </li> > </ol> >@@ -238,4 +199,46 @@ > [% INCLUDE 'tools-menu.inc' %] > </div> > </div> >+[% MACRO jsinclude BLOCK %] >+ [% INCLUDE 'calendar.inc' %] >+ [% Asset.js("lib/jquery/plugins/jquery-ui-timepicker-addon.min.js") | $raw %] >+ [% INCLUDE 'timepicker.inc' %] >+ <script type="text/javascript"> >+ $(document).ready(function() { >+ toggle_identifier(); >+ $("#oai_verb").on("click",toggle_identifier); >+ $(".datetime_utc").datetimepicker({ >+ separator: "T", >+ timeSuffix: 'Z', >+ dateFormat: "yy-mm-dd", >+ timeFormat: "HH:mm:ss", >+ hour: 0, >+ minute: 0, >+ second: 0, >+ showSecond: 1, >+ // timezone doesn't work with the "Now" button in v1.4.3 although it appears to in v1.6.1 >+ // timezone: '+000' >+ }); >+ }); >+ function toggle_identifier (){ >+ var verb = $("#oai_verb").find(":selected").val(); >+ var oai_identifier = $("#oai_identifier"); >+ var oai_set = $("#oai_set"); >+ var oai_from = $("#oai_from"); >+ var oai_until = $("#oai_until"); >+ if (verb == 'ListRecords'){ >+ oai_identifier.prop('disabled', true); >+ oai_set.prop('disabled', false); >+ oai_from.prop('disabled', false); >+ oai_until.prop('disabled', false); >+ } >+ else if (verb == 'GetRecord'){ >+ oai_identifier.prop('disabled', false); >+ oai_set.prop('disabled', true); >+ oai_from.prop('disabled', true); >+ oai_until.prop('disabled', true); >+ } >+ } >+ </script> >+[% END %] > [% INCLUDE 'intranet-bottom.inc' %] >diff --git a/rewrite-config.PL b/rewrite-config.PL >index 219811b..0eff8ab 100644 >--- a/rewrite-config.PL >+++ b/rewrite-config.PL >@@ -151,7 +151,7 @@ $prefix = $ENV{'INSTALL_BASE'} || "/usr"; > "__MEMCACHED_SERVERS__" => "", > "__MEMCACHED_NAMESPACE__" => "", > "__FONT_DIR__" => "/usr/share/fonts/truetype/ttf-dejavu", >- "__TEMPLATE_CACHE_DIR__" => "/tmp/koha" >+ "__TEMPLATE_CACHE_DIR__" => "/tmp/koha", > "__OAI_RUN_DIR__" => "", > "__OAI_LIB_DIR__" => "", > "__OAI_SPOOL_DIR__" => "", >diff --git a/svc/oai-pmh-harvester/history b/svc/oai-pmh-harvester/history >index 851e8a5..62ec4d0 100755 >--- a/svc/oai-pmh-harvester/history >+++ b/svc/oai-pmh-harvester/history >@@ -20,7 +20,8 @@ > use Modern::Perl; > use C4::Auth qw(check_cookie_auth haspermission get_session); > use JSON; >-use Koha::Database; >+use Koha::OAI::Harvester::Biblios; >+use Koha::OAI::Harvester::Histories; > > my $input = new CGI; > >@@ -88,40 +89,36 @@ if ($input->request_method eq "POST"){ > } > > my $page = ( $start / $length ) + 1; >-my $schema = Koha::Database->new()->schema(); >-if ($schema){ >- my $rs = $schema->resultset("OaiHarvesterHistory"); >- my $results = $rs->search( >- \@search, >- { >- result_class => 'DBIx::Class::ResultClass::HashRefInflator', >- page => $page, >- rows => $length, >- order_by => \@order_by, >- }, >- ); >- my $count = $rs->count; >- my $filtered_count = $results->pager->total_entries; >- my @rows = (); >- while (my $row = $results->next){ >- $row->{imported_record} = ''; >- if ($row->{record_type} eq "biblio"){ >- my $harvested_biblio = $schema->resultset("OaiHarvesterBiblio")->find( >- { >- oai_repository => $row->{repository}, >- oai_identifier => $row->{header_identifier}, >- }, >- { key => "oai_record" }, >- ); >- $row->{imported_record} = $harvested_biblio->biblionumber->id if $harvested_biblio; >- } >- push(@rows,$row); >- } >- if ($count){ >- $data->{recordsTotal} = $count; >- $data->{recordsFiltered} = $filtered_count; >- $data->{data} = \@rows if @rows; >+my $results = Koha::OAI::Harvester::Histories->new->search( >+ \@search, >+ { >+ page => $page, >+ rows => $length, >+ order_by => \@order_by, >+ }, >+); >+my $count = Koha::OAI::Harvester::Histories->new->count; >+my $filtered_count = $results->pager->total_entries; >+my @rows = (); >+while (my $obj = $results->next){ >+ my $row = $obj->unblessed; >+ $row->{imported_record} = ''; >+ if ($row->{record_type} eq "biblio"){ >+ my $harvested_biblio = Koha::OAI::Harvester::Biblios->new->find( >+ { >+ oai_repository => $row->{repository}, >+ oai_identifier => $row->{header_identifier}, >+ }, >+ { key => "oai_record" }, >+ ); >+ $row->{imported_record} = $harvested_biblio->biblionumber if $harvested_biblio; > } >+ push(@rows,$row); >+} >+if ($count){ >+ $data->{recordsTotal} = $count; >+ $data->{recordsFiltered} = $filtered_count; >+ $data->{data} = \@rows if @rows; > } > > binmode STDOUT, ":encoding(UTF-8)"; >diff --git a/tools/oai-pmh-harvester/record.pl b/tools/oai-pmh-harvester/record.pl >index fd2375c..205ee0c 100755 >--- a/tools/oai-pmh-harvester/record.pl >+++ b/tools/oai-pmh-harvester/record.pl >@@ -21,7 +21,7 @@ use Modern::Perl; > > use C4::Auth; > use C4::Output; >-use Koha::Database; >+use Koha::OAI::Harvester::Histories; > > my $input = new CGI; > >@@ -35,17 +35,11 @@ my ($template, $loggedinuser, $cookie) = > > my $import_oai_id = $input->param('import_oai_id'); > if ($import_oai_id){ >- my $schema = Koha::Database->new()->schema(); >- if ($schema){ >- my $rs = $schema->resultset("OaiHarvesterHistory"); >- if ($rs){ >- my $row = $rs->find($import_oai_id); >- if ($row){ >- my $record = $row->record; >- if ($record){ >- $template->{VARS}->{ record } = $record; >- } >- } >+ my $history = Koha::OAI::Harvester::Histories->new->find($import_oai_id); >+ if ($history){ >+ my $record = $history->record; >+ if ($record){ >+ $template->{VARS}->{ record } = $record; > } > } > } >-- >2.1.4
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 10662
:
20757
|
20758
|
21954
|
42446
|
42447
|
42448
|
44792
|
44793
|
47675
|
48096
|
49832
|
50254
|
50980
|
51510
|
51511
|
51512
|
51514
|
51562
|
51563
|
51564
|
51565
|
51705
|
53258
|
53259
|
53260
|
53361
|
53362
|
64444
|
64445
|
64446
|
64479
|
64480
|
64481
|
65016
|
65017
|
65018
|
65019
|
68508
|
71008
|
71009
|
71010
|
71011
|
71012
|
71013
|
71014
|
71015
|
78569
|
78570
|
78571
|
78572
|
78573
|
78574
|
78575
|
78576
|
78577
|
78583
|
78601
|
78617
|
78754
|
78756
|
78758
|
78760
|
78762
|
78764
|
78767
|
78769
|
78771
|
78859
|
78860
|
78914
|
78956
|
79039
|
79045
|
81783
|
81784
|
81786
|
81789
|
81790
|
81855
|
81856
|
81861
|
81862
|
81863
|
81864
|
81902
|
82219
|
84318
|
84319
|
84320
|
84321
|
84322
|
85152
|
85224
|
85225
|
85226
|
85227
|
85228
|
85229
|
99739
|
99740
|
99741
|
99742
|
99743
|
99744
|
99745