Bug 12919 - Automate collecting all records added and deleted from the catalogue
Summary: Automate collecting all records added and deleted from the catalogue
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: master
Hardware: All All
: P5 - low new feature (vote)
Assignee: Chris Cormack
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-09-15 02:47 UTC by Chris Cormack
Modified: 2022-03-18 10:36 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 12919 : Module to facilitate sending new (or updated or deleted) cataloguing records somewhere (9.76 KB, patch)
2014-09-15 02:51 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 12919 : Cronjob to create and email new or updated MARC records (4.69 KB, patch)
2014-09-15 02:59 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 12919 : Script to grab and email deleted records (4.42 KB, patch)
2014-09-15 03:22 UTC, Chris Cormack
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 12919 : Module to facilitate sending new (or updated or deleted) cataloguing records somewhere (9.84 KB, patch)
2015-05-17 22:24 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 12919: Cronjob to create and email new or updated MARC records (4.76 KB, patch)
2015-05-17 22:24 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 12919: Script to grab and email deleted records (4.57 KB, patch)
2015-05-17 22:24 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 12929: (followup) fix license (4.00 KB, patch)
2015-05-17 22:27 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 12919 - perltidy and consistent sub naming (8.20 KB, patch)
2015-06-23 03:43 UTC, Liz Rea
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2014-09-15 02:47:09 UTC
In NZ almost all libraries participate in the National Union Catalogue, which means that libraries have to update their holdings periodically.

So we created a system that runs as a cronjob and will generate a list of records added  or updated in the catalogue, and a list of isbn of deleted records.

These can be run as cron jobs with the output being emailed to a person or thing.

Although this was initially designed to support the needs of NZ libraries, it is general enough that it can be used by anyone.
Comment 1 Chris Cormack 2014-09-15 02:51:16 UTC Comment hidden (obsolete)
Comment 2 Chris Cormack 2014-09-15 02:59:40 UTC Comment hidden (obsolete)
Comment 3 Chris Cormack 2014-09-15 03:22:35 UTC Comment hidden (obsolete)
Comment 4 Liz Rea 2015-03-11 01:05:05 UTC
This is such a handy tool, it could really use a signoff.
Comment 5 Bernardo Gonzalez Kriegel 2015-05-02 02:50:38 UTC
It works.

Just a couple of things:

1) If you have no new or deleted records, an email is sent anyway
with an empty Koha.mrc file

2) License needs to be updated

FAIL   misc/cronjobs/newly_deleted_records.pl
   OK     critic
   FAIL   forbidden patterns
                forbidden pattern: Koha is now under the GPLv3 license (line 9)
   OK     pod
   OK     valid

 FAIL   misc/cronjobs/newly_added_records.pl
   OK     critic
   FAIL   forbidden patterns
                forbidden pattern: Koha is now under the GPLv3 license (line 9)
   OK     pod
   OK     valid
Comment 6 Bernardo Gonzalez Kriegel 2015-05-17 22:24:41 UTC
Created attachment 39260 [details] [review]
[SIGNED-OFF] Bug 12919 : Module to facilitate sending new (or updated or deleted) cataloguing records somewhere

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 7 Bernardo Gonzalez Kriegel 2015-05-17 22:24:49 UTC
Created attachment 39261 [details] [review]
[SIGNED-OFF] Bug 12919: Cronjob to create and email new or updated MARC records

To test

1/ Add some new records to your Koha
2/ if you are using koha from packages
run

sudo koha-shell <kohainstancename>

otherwise

export PERL5LIB=/path/to/koha
export KOHA_CONF=/path/to/koha-conf.xml

3/ run

perl misc/cronjobs/newly_added_records --days 1 --address your@email.goes.here --format marc

4/ Check your email and see if you got some marc records

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 8 Bernardo Gonzalez Kriegel 2015-05-17 22:24:56 UTC
Created attachment 39262 [details] [review]
[SIGNED-OFF] Bug 12919: Script to grab and email deleted records

To test

1/ Delete some records from your Koha (with isbns)
  OR mark all items attached to a biblio as lost

2/ if you are using koha from packages
run

sudo koha-shell <kohainstancename>

otherwise

export PERL5LIB=/path/to/koha
export KOHA_CONF=/path/to/koha-conf.xml

3/ run

perl misc/cronjobs/newly_added_records --days 1 --address your@email.goes.here --lost --format isbn

4/ Check your email and see if you got a list of isbn of deleted records

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>

Work as described, both patches
A simple error on license, fixed in followup
Comment 9 Bernardo Gonzalez Kriegel 2015-05-17 22:27:50 UTC
Created attachment 39263 [details] [review]
Bug 12929: (followup) fix license

I fix it myself.

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment 10 Kyle M Hall 2015-05-22 14:25:01 UTC
Feature uses MIME::Lite but does not appear to add it to the dependencies list, can you please add that?

Mime::Lite is already available as a package for Debian so I don't think it will be an issue for Koha packages.
Comment 11 Liz Rea 2015-05-24 23:01:14 UTC
Hi Kyle,

I see this in Master in C4/Installer/PerlDependencies.pm:

 37     'MIME::Lite' => {
 38         'usage'    => 'Core',
 39         'required' => '1',
 40         'min_ver'  => '3'
 41     },

Is this not what is necessary? What more is needed?

Cheers,
Liz
Comment 12 Jonathan Druart 2015-06-18 11:39:19 UTC
QA comments:

1/ perltidy new files
2/ Use pod2usage in pl scripts
3/ Write tests for new subroutines
4/ Be consistent with subroutine names (export_newrecords => export_new_records)

Marked as Failed QA.
Comment 13 Liz Rea 2015-06-23 03:43:35 UTC
Created attachment 40505 [details] [review]
Bug 12919 - perltidy and consistent sub naming
Comment 14 Jonathan Druart 2015-07-09 11:13:36 UTC
(In reply to Liz Rea from comment #13)
> Created attachment 40505 [details] [review] [review]
> Bug 12919 - perltidy and consistent sub naming

Liz, could you confirm this one can leave the FQA queue?
Comment 15 Liz Rea 2015-07-09 22:07:08 UTC
Nope, it still needs unit tests and pod2usage updates as well.


Liz
Comment 16 Jonathan Druart 2015-07-10 07:12:24 UTC
Comment on attachment 39260 [details] [review]
[SIGNED-OFF] Bug 12919 : Module to facilitate sending new (or updated or deleted) cataloguing records somewhere

Review of attachment 39260 [details] [review]:
-----------------------------------------------------------------

Don't you think it would be better to sent the mail to the message queue and let the users define a template letter?

::: C4/RecordExporter.pm
@@ +274,5 @@
> +sub get_isbns {
> +    my ($record) = @_;
> +
> +    my $res = '';
> +    my @fields = $record->field('020');

This is MARC21 specific.
You should use a subroutine from C4::Koha.
Comment 17 Liz Rea 2015-07-13 02:46:50 UTC
Making it use a letter template and the letter queue is a good idea, but a scope creep. Happy to make another bug for that once the dev itself goes in.

Cheers,
Liz
Comment 18 Nicole C. Engard 2016-06-08 14:56:14 UTC
Is anyone still working on this? Is it really 'assigned'?
Comment 19 Cab Vinton 2022-03-18 10:19:50 UTC
ASSIGNED status does indeed appear dubious, but don't feel qualified to update.
Comment 20 Jonathan Druart 2022-03-18 10:36:25 UTC
(In reply to Cab Vinton from comment #19)
> ASSIGNED status does indeed appear dubious, but don't feel qualified to
> update.

Adjusted to NEW.