Position 05 of the leader in MARC 21 indicates the record's status. A lower case “d” indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). Koha should allow for deletion of bibliographic records that contain a “d” in the LEADER position 05 when by overlaying existing records.
Created attachment 22088 [details] [review] Bug 11084 - Delete biblios on Leader 05 =d
Created attachment 22089 [details] [review] Bug 11084 - Delete biblios on Leader 05 =d Position 05 of the leader in MARC 21 indicates the record's status. A lower case “d” indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/deleted_fixed_field_5.pl -c -v 3) Those records should either be deleted, or an error message should state why the were not.
See also - bug 10421
Note : the label for UNIMARC is buit the same way pos5 = "d" means deleted http://multimedia.bnf.fr/unimarcb_trad/Bsection5-Label_notice-6-2010.pdf So it could be made too for unimarc. Mathieu
Tool needs to be able to delete biblios with items (and the items). Also, it should be able to run in a test mode.
Created attachment 22172 [details] [review] Bug 11084 - Delete biblios on Leader 05 =d Position 05 of the leader in MARC 21 indicates the record's status. A lower case “d” indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/deleted_fixed_field_5.pl -c -v 3) Those records should either be deleted, or an error message should state why the were not.
Patch applied cleanly, go forth and signoff
*** Bug 10421 has been marked as a duplicate of this bug. ***
Created attachment 22634 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Position 05 of the leader in MARC 21 indicates the record's status. A lower case 'd' indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/deleted_fixed_field_5.pl -c -v 3) Those records should either be deleted, or an error message should state why they were not. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Works as advertised. Put some points of attention on Bugzilla comment.
Signing off but with some smaller comments: typo delete_fixec_field_5.pl test mode should be explained (not in usage) name of the script could be confusing: you are obviously not deleting a fixed field, but a record :) encoding: when you are printing the title to stdout, you should probably open stdout as utf8 verbose mode: would it be helpful to say which biblios or items were actually deleted (and count them?) license statement: version 2 or 3? address no longer?
Hm, a few notes from me too: DelBiblio has a few shortcomings that could cause problems: 1) It just deletes attached subscriptions. Subscription data can be created over a long time and is not easily reproducable. There is no backup when we delete it. 428 # We delete attached subscriptions 429 require C4::Serials; 430 my $subscriptions = C4::Serials::GetFullSubscriptionsFromBiblionumber($biblionumber); 431 foreach my $subscription (@$subscriptions) { 432 C4::Serials::DelSubscription( $subscription->{subscriptionid} ); 433 } 2) It will also delete records that are used with orders, as this isn't checked at all. Kyle, could you take a look at Marcel's and my comments?
Hm, bit confused about the help too - is there a way to get it to list all available options? -h didn't seem to do the trick.
(In reply to Kyle M Hall from comment #6) > Position 05 of the leader in MARC 21 indicates the record's status. > A lower case “d” indicates that the record is to be deleted > (http://www.loc.gov/marc/authority/adleader.html). JFTR, that link talks about authority records, but for biblio records is the same: http://www.loc.gov/marc/bibliographic/bdleader.html
Created attachment 36837 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Position 05 of the leader in MARC 21 indicates the record's status. A lower case 'd' indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/deleted_fixed_field_5.pl -c -v 3) Those records should either be deleted, or an error message should state why they were not. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Works as advertised. Put some points of attention on Bugzilla comment.
Created attachment 36838 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Position 05 of the leader in MARC 21 indicates the record's status. A lower case 'd' indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/delete_records_via_leader.pl -c -v 3) Those records should either be deleted, or an error message should state why they were not. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Works as advertised. Put some points of attention on Bugzilla comment.
Created attachment 36840 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Position 05 of the leader in MARC 21 indicates the record's status. A lower case 'd' indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/delete_records_via_leader.pl -c -v 3) Those records should either be deleted, or an error message should state why they were not.
Script now displays all options. (In reply to Katrin Fischer from comment #12) > Hm, bit confused about the help too - is there a way to get it to list all > available options? -h didn't seem to do the trick.
Updated patch should address all these concerns. (In reply to M. de Rooy from comment #10) > Signing off but with some smaller comments: > > typo delete_fixec_field_5.pl > test mode should be explained (not in usage) > name of the script could be confusing: you are obviously not deleting a > fixed field, but a record :) > encoding: when you are printing the title to stdout, you should probably > open stdout as utf8 > verbose mode: would it be helpful to say which biblios or items were > actually deleted (and count them?) > license statement: version 2 or 3? address no longer?
This is a definite issue, but appears to be outside the scope of this bug. It would appear that misc/batchdeletebiblios.pl has the exact same issues. I would suggest we file a separate bug to handle that issue. (In reply to Katrin Fischer from comment #11) > Hm, a few notes from me too: > > DelBiblio has a few shortcomings that could cause problems: > > 1) It just deletes attached subscriptions. Subscription data can be created > over a long time and is not easily reproducable. There is no backup when we > delete it. > > 428 # We delete attached subscriptions > 429 require C4::Serials; > 430 my $subscriptions = > C4::Serials::GetFullSubscriptionsFromBiblionumber($biblionumber); > 431 foreach my $subscription (@$subscriptions) { > 432 C4::Serials::DelSubscription( $subscription->{subscriptionid} ); > 433 } > > 2) It will also delete records that are used with orders, as this isn't > checked at all. > > Kyle, could you take a look at Marcel's and my comments?
Created attachment 37714 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Position 05 of the leader in MARC 21 indicates the record's status. A lower case 'd' indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/delete_records_via_leader.pl -c -v 3) Those records should either be deleted, or an error message should state why they were not. Signed-off-by: Magnus Enger <magnus@enger.priv.no> Works as advertised. Edited a record with an item to have leader05 = d, then ran delete_records_via_leader.pl with -c and got a message that the record could not be deleted because of the item. Removed the item and the script deleted the record. Tried with another record with an item, this time with the -i option and the record was deleted.
Hi Kyle, I am ok with this missing checks for subscriptions and orders - I think we can assume here checks have been made when setting the flag (although i would like a 'save delete' option or something similar :) ) But can you please fix: FAIL misc/cronjobs/delete_records_via_leader.pl FAIL critic I/O layer ":utf8" used at line 24, column 1. Use ":encoding(UTF-8)" to get strict validation.
Created attachment 37954 [details] [review] Bug 11084 [QA Followup]
Hi Kyle, I did some tests with the new patches: Record has 1 item, is not checked out. -t -c ERROR DELETING BIBLIO 24: Test mode enabled I was not sure if this would have been an error outside the test mode (not being deleted) or if that just meant that it was not deleted because of the test mode. -t -i -c ERROR DELETING ITEM 102: Test mode enabled ERROR DELETING BIBLIO 24: Test mode enabled Ok, so both throw an error... so not sure: would those be deleted without -t? -c ERROR DELETING BIBLIO 24: This Biblio has items attached, please delete them first before deleting this biblio This one is good - why not display this in the test mode too? -c -i C4::Context->userenv not defined! at /home/katrin/kohaclone/C4/Items.pm line 2278. No messages, but I didn't set verbose. Item and record appear to have been deleted - can you take a look at the error/warn?
Created attachment 38336 [details] [review] Bug 11084 [QA Followup] - Suppress userenv warning
(In reply to Katrin Fischer from comment #23) > Hi Kyle, > > I did some tests with the new patches: > > Record has 1 item, is not checked out. > > -t -c > ERROR DELETING BIBLIO 24: Test mode enabled > > I was not sure if this would have been an error outside the test mode (not > being deleted) or if that just meant that it was not deleted because of the > test mode. Right now it's not possible to tell if a record is deletable in Koha without trying to delete it! There are simply no library functions to do so. The test mode will tell you what records and items it would attempt to delete, but is not meant to tell you if they would be deleted successfully. > > -t -i -c > ERROR DELETING ITEM 102: Test mode enabled > ERROR DELETING BIBLIO 24: Test mode enabled > > Ok, so both throw an error... so not sure: would those be deleted without -t? See above. > -c > ERROR DELETING BIBLIO 24: This Biblio has items attached, please delete them > first before deleting this biblio > > This one is good - why not display this in the test mode too? See above as well ; ) The errors come from DelBiblio, so the only way to know if the record can't be deleted is to try to delete it! > -c -i > C4::Context->userenv not defined! at /home/katrin/kohaclone/C4/Items.pm line > 2278. > > No messages, but I didn't set verbose. Item and record appear to have been > deleted - can you take a look at the error/warn? I've attached a followup to take care of this. userenv isn't defined for cli scripts, so we just need to check for it's existence before using it.
Can we make that clear in the documentation for the test mode? that it will only list items that it would 'try' to delete?
Created attachment 38337 [details] [review] Bug 11084 [QA Followup] - Document limitations of test mode
Created attachment 38361 [details] [review] [PASSED QA] Bug 11084 - Delete biblios on Leader 05 == d Position 05 of the leader in MARC 21 indicates the record's status. A lower case 'd' indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/delete_records_via_leader.pl -c -v 3) Those records should either be deleted, or an error message should state why they were not. Signed-off-by: Magnus Enger <magnus@enger.priv.no> Works as advertised. Edited a record with an item to have leader05 = d, then ran delete_records_via_leader.pl with -c and got a message that the record could not be deleted because of the item. Removed the item and the script deleted the record. Tried with another record with an item, this time with the -i option and the record was deleted. Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Created attachment 38362 [details] [review] [PASSED QA] Bug 11084 [QA Followup] Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Created attachment 38363 [details] [review] [PASSED QA] Bug 11084 [QA Followup] - Suppress userenv warning Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Created attachment 38364 [details] [review] [PASSED QA] Bug 11084 [QA Followup] - Document limitations of test mode Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Created attachment 46021 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Position 05 of the leader in MARC 21 indicates the record's status. A lower case 'd' indicates that the record is to be deleted (http://www.loc.gov/marc/authority/adleader.html). This patch adds a cronjob script that may be run nightly to delete records where position 05 of the leader is set to 'd'. Test Plan: 1) Edit one or more records, and set the leader position 05 to 'd'. 2) Run misc/cronjobs/delete_records_via_leader.pl -c -v 3) Those records should either be deleted, or an error message should state why they were not. Signed-off-by: Magnus Enger <magnus@enger.priv.no> Works as advertised. Edited a record with an item to have leader05 = d, then ran delete_records_via_leader.pl with -c and got a message that the record could not be deleted because of the item. Removed the item and the script deleted the record. Tried with another record with an item, this time with the -i option and the record was deleted. Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 46022 [details] [review] Bug 11084 [QA Followup] Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 46023 [details] [review] Bug 11084 [QA Followup] - Suppress userenv warning Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Created attachment 46024 [details] [review] Bug 11084 [QA Followup] - Document limitations of test mode Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Pushed to master - Thanks
Comment on attachment 46021 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Review of attachment 46021 [details] [review]: ----------------------------------------------------------------- ::: misc/cronjobs/delete_records_via_leader.pl @@ +67,5 @@ > +} > + > +my $schema = Koha::Database->new()->schema(); > +my @biblioitems = > + $schema->resultset('Biblioitem')->search( { marc => { LIKE => '_____d%' } } ); We're reading the 'marc' field as a string? While that's pretty fast on smaller databases, it takes about 60 seconds on a database with about 1.1 million bib records. I suppose performance doesn't matter too much for a cronjob though - depending on the frequency that it's run. Also, aren't we trying to get rid of the biblioitems.marc field? See: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10455 Wouldn't it be better if we started indexing LDR05 as 'rstatus' or "record_status' or some such?
More over, if we indexed LDR05, we would be able to hide "deleted" records from the OPAC like we do with "suppressed" records. While we had to worry about that OpacSuppression bug, which erroneously returned 0 results when the OpacSuppression index was completely empty, we don't need to worry about that too much since we started using @attr 14=1 http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15198. We could prepend @attr 14=1 at the start of all queries (so that it applied to all query parts) and we wouldn't encounter that problem anymore. -- I desire the above behaviour also because of what David Noe originally described here. It would be nice if records with a LDR05 "d" were deleted during the overlay of existing records during an import. In my case, I'm downloading records via OAI-PMH, but when I get a record with a "deleted" status, I'm not given a metadata record. I just have an identifier and a deleted status. So I can create an empty MARC record, change LDR05 to "d", inject the identifier, and then match to existing records in Koha. However, I don't want to overlay an empty record over a non-empty record in Koha, and then have it show up in search results in the OPAC. Actually, I wouldn't want it to show up in search results in the staff client either, but that might be unavoidable... I suppose I could also add a fake title to the record like "This record has been deleted" or some such, although that's not translation friendly...
Comment on attachment 46021 [details] [review] Bug 11084 - Delete biblios on Leader 05 == d Review of attachment 46021 [details] [review]: ----------------------------------------------------------------- ::: misc/cronjobs/delete_records_via_leader.pl @@ +67,5 @@ > +} > + > +my $schema = Koha::Database->new()->schema(); > +my @biblioitems = > + $schema->resultset('Biblioitem')->search( { marc => { LIKE => '_____d%' } } ); Once Bug 15537 (http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15537) is pushed, this could be replaced with the following query with C4::Search::SimpleSearch(): "Record-status=d"
I have modified this script on bug 10455 with: -my @biblioitems = - $schema->resultset('Biblioitem')->search( { marc => { LIKE => '_____d%' } } ); +my @biblioitems = # Should be replaced by a call to C4::Search on zebra index + # Record-status when bug 15537 will be pushed + $schema->resultset('Biblioitem')->search( { marcxml => { LIKE => '<leader>_____d%' } } ); Could you confirm you agree with that?