Created attachment 18678 [details] [review] Bug 10421 - Add cron script to delete deleted records This script will batch delete all records where the leader's record status ( position 5 ) is set to 'd' ( Deleted ). Test Plan: 1) Edit a few records, set the leader 05 to 'd' for deleted 2) Run delete_deleted_records.pl 3) Records that are deletable should be deleted!
Curious about the use case for this -- are there folks who are not actually deleting bibs, but instead setting the Leader/05 to 'd'? If so, why? Or is this somehow arising from record imports?
(In reply to comment #2) > Curious about the use case for this -- are there folks who are not actually > deleting bibs, but instead setting the Leader/05 to 'd'? If so, why? Or is > this somehow arising from record imports? I have seen this with outsourced cataloging departments.
Created attachment 20220 [details] [review] Bug 10421 - [SIGNED-OFF] Add cron script to delete deleted records This script will batch delete all records where the leader's record status ( position 5 ) is set to 'd' ( Deleted ). Test Plan: 1) Edit a few records, set the leader 05 to 'd' for deleted 2) Run delete_deleted_records.pl 3) Records that are deletable should be deleted! Signed-off-by: Magnus Enger <magnus@enger.priv.no> I can confirm that NORMARC uses leader position 5 in the same way as MARC21, and as far as I can see from this document, UNIMARC does too: http://archive.ifla.org/VI/8/unimarc-concise-bibliographic-format-2008.pdf -h and --help as well as running the script without arguments give a nice "help" message -c and --confirm makes the script do its job Records with leader position 5 = d are deleted. If there are items attached to the record, an error is reported: Deleting biblionumber 562 ... ERROR: ( | This Biblio has items attached, please delete them first before deleting this biblio ) Looks good!
QA comment: 3 small things: - FAIL misc/cronjobs/delete_deleted_records.pl FAIL forbidden patterns forbidden pattern: Koha is now under the GPLv3 license (line 78) see http://wiki.koha-community.org/wiki/Coding_Guidelines#Licence - There are some bad indentations: 8 spaces instead of 4 - Maybe it would be great to list the deleted biblios even if the confirm flag is not set. Marked as Failed QA.
(In reply to Galen Charlton from comment #2) > Curious about the use case for this -- are there folks who are not actually > deleting bibs, but instead setting the Leader/05 to 'd'? If so, why? Or is > this somehow arising from record imports? I've wondered about this before as well. I've known cataloguers to switch the Leader/05 to 'd' before, but I've never been 100% certain what is done with the records after that. I think the first step was to suppress them in the OPAC, and that after that...I'm not sure. I wonder if a cronjob is the best idea. After all, if the record has been set to Leader/05 'd' rather than straight up deleting it...perhaps there is a reason why they still want it around?
I could imagine that happens in a union catalog setting or for electronic ressources where you get new records while others are removed from the licensed package. I have recently been working on documentation on when a record can be deleted safely, there are a lot of things you want to take into account there to make sure you don't accidentally lose important data: - Does the record have items? - Are the items checked out? - Are the items on hold? (holds can be cancelled) - Is there an open order in acquisitions for this record? - Are there subscriptions for this record?
(In reply to Katrin Fischer from comment #7) > I have recently been working on documentation on when a > record can be deleted safely, there are a lot of things you want to take > into account there to make sure you don't accidentally lose important data: Once upon a time I tried to collect all bugs related to deleting records below Bug 8149 - Deleting Records/Biblios Omnibus
A relevant discussion on koha-devel - http://lists.koha-community.org/pipermail/koha-devel/2013-September/039662.html
In the past I've come across sites whose databases included a fair propotion of records with deleted status in the header, often imported from some external source. I'm wondering why you would flag recs in the koha db as deleted rather than delete them straight out
(In reply to Colin Campbell from comment #10) > In the past I've come across sites whose databases included a fair propotion > of records with deleted status in the header, often imported from some > external source. I'm wondering why you would flag recs in the koha db as > deleted rather than delete them straight out I decided to do a quick web search on this topic and here is the first thing I stumbled across: "Whether you perform the conversion or we do, consider whether you wish to retain MARC records carrying the delete flag ("d" in Leader/05). You may need to retain these for reporting purposes if your database is part of a union catalog." http://www.itsmarc.com/UpgradingFromITS.asp I imagine that this might be the use case that Katrin was mentioning. -- Perhaps it would be an institutional policy for library techs to mark a record as deleted but only "authorized" librarians are allowed to delete the record? Sort of following along the lines of the ideas mentioned here: http://www.lib.berkeley.edu/asktico/procedures/deletion-bibliographic-record -- In the MARC Authority record context, there are actually several different "deleted" flags such as "s" and "x" which have different reasons attached to them. http://www.loc.gov/marc/authority/adleader.html d - Deleted Used when neither code s nor code x is applicable, or when an organization chooses not to use code s or x. Some level of manual intervention may be needed to effect the change in bibliographic records because the deleted heading may or may not be carried as a 4XX See From Tracing field in other authority records and a 682 field explaining the delete may be present in the record marked for deletion. s - Deleted; heading split into two or more headings Record has been deleted from a file because the heading has been split into two or more headings, requiring a new authority record for each. The heading from the deleted record is included in each of the new authority records as a tracing in a 4XX See From Tracing field. This specialized delete value supports use of automated systems to carry out any necessary reviews, since when a heading is split, a computer cannot automatically replace the old heading in bibliographic records. x - Deleted; heading replaced by another heading Record has been deleted from a file and that a new authority record in which the heading from the deleted record appears as a 4XX See From Tracing field has been added to the file. (In a system where authority control is linked to the bibliographic file, a computer can effect the one-to-one replacement indicated by code x without manual intervention.) Apparently there is also a code "o" for obsolete but not deleted. http://www.itsmarc.com/crs/mergedProjects/helpauth/helpauth/idh_leader_05_auth.htm A person can also use the 682 to provide a reason apparently. http://www.loc.gov/marc/authority/ad682.html Admittedly, the Authority context might not be relevant to this discussion because it talks about automatic/manual updating of Bibliographic records. But if you're deleting Bibliographic records...you wouldn't need to worry about a trickle down effect (except in the case where its linked to another using 7xx linking fields, but that's a whole 'nother can of worms). -- In the end...I wonder if it's best to: 1) Not display "deleted" records in the OPAC 2) Have a label saying "deleted" in the staff client 3) Perhaps have a cronjob to clean out "d" records, but only at the behest of a library administrator. It seems that having this cronjob active by default might cause issues if only at a policy level (which is an extremely important level in libraries)
Mohammad Nashbat's comment on the listserv is also interesting: "I’m trying to upload a deletion batch from ebrary to our Koha LMS, this batch changes the records status in the leader tag to “d” delete, which used to delete the bib records using our old library system “Dynix” after indexing. But now in Koha it do change the status but without deleting the bib records even after indexing, am I missing something?" I think at a bare minimum records with a "d" need to be suppressed in the OPAC. However, actual deletions...I'm not sure what I think of this "batch deletion" idea. I haven't done a lot yet with MARC dumps from vendors like EBSCO, but I could see how re-uploading records with a "d" in the leader to do a batch delete might be an idea. While it's only tangentially related to this discussion, I wonder what sorts of best practices people have for maintaining MARC records in Koha that come from electronic service providers (for services like electronic journals and e-books). We have a homecooked solution here that my boss takes care of but I wonder what the optimal way of doing it would be. I'm sure people in a union catalogue situation/those who use OCLC or other interconnected cataloguing tools must also encounter situations where the original record has changed and now they either need to update or delete their existing records. (I know this is a problem I've also encountered in developing the OAI-PMH harvester. It seems like it could be a pervasive issue.)
(In reply to David Cook from comment #12) > Mohammad Nashbat's comment on the listserv is also interesting: > > "I’m trying to upload a deletion batch from ebrary to our Koha LMS, this > batch changes the records status in the leader tag to “d” delete, which used > to delete the bib records using our old library system “Dynix” after > indexing. But now in Koha it do change the status but without deleting the > bib records even after indexing, am I missing something?" > > I think at a bare minimum records with a "d" need to be suppressed in the > OPAC. > > However, actual deletions...I'm not sure what I think of this "batch > deletion" idea. I haven't done a lot yet with MARC dumps from vendors like > EBSCO, but I could see how re-uploading records with a "d" in the leader to > do a batch delete might be an idea. > > While it's only tangentially related to this discussion, I wonder what sorts > of best practices people have for maintaining MARC records in Koha that come > from electronic service providers (for services like electronic journals and > e-books). > > We have a homecooked solution here that my boss takes care of but I wonder > what the optimal way of doing it would be. > > I'm sure people in a union catalogue situation/those who use OCLC or other > interconnected cataloguing tools must also encounter situations where the > original record has changed and now they either need to update or delete > their existing records. > > (I know this is a problem I've also encountered in developing the OAI-PMH > harvester. It seems like it could be a pervasive issue.) Jared Camins-Esakov has mentioned that using the import/revert features in Koha, a person could revert a batch of MARC records from an electronic subscription, then just import the latest full reload file. That seems to me to be a very reasonable solution. The only problem would be if someone added items to those records, but as he pointed out...with a large enough load (say 50k records), the likelihood of items being added is probably fairly low. That said, I think some of our libraries have items automatically added to some of their electronic holdings. However, perhaps that's a practice that should just be eliminated, since the process Jared describes seems quite robust. In the case where you unsubscribe from a subscription, you'd just revert your records and that's that. So long as you have a full MARC record dump, you're covered for additions/updates/deletes. -- In the union catalog/online cataloging tool scenario, additions/updates shouldn't be much of a problem (provided that there haven't been local changes), but I imagine that deletes are still an issue...and that perhaps you would still want a record marked "d" in your catalogue (although not in your OPAC). Ok. That's it from me. Sorry for spamming all of you!
This is a feature in other systems on which I have relied for keeping my catalog up to date. Chiefly, this is applied to records that are leased and subscription ebook collections that are not static. On a routine basis, the vendor sends or provides access to a batch of records removed from a collection. The librarian loads the records in order to delete them. I have had no less than five partners express a desire for this to me in my short time working with Koha, and I would expect this functionality myself (as in it would be a deal breaker on ILS selections).
Hello unimarc is dealing with deleted record exactly the same way (pos 5 of label = d). I read it in french version of manual. of course it must be checked in english version, but it probably means this dev could and should be made marc agnostic. M. Saby
*** This bug has been marked as a duplicate of bug 11084 ***