Bug 20271 - Merge deletedbiblio* and deleteitems tables with their "alive" cousins
Summary: Merge deletedbiblio* and deleteitems tables with their "alive" cousins
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Database (show other bugs)
Version: master
Hardware: All All
: P1 - high major (vote)
Assignee: Francesco Rivetti
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-22 13:42 UTC by Petter Goksøyr Åsen
Modified: 2018-05-15 21:43 UTC (History)
11 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 20271: merging delete biblio/items (30.19 KB, patch)
2018-03-01 08:53 UTC, Francesco Rivetti
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (34.15 KB, patch)
2018-03-13 15:43 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271 - merge deletedbiblio and -items back - add atomic update (2.39 KB, patch)
2018-03-13 15:43 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (34.15 KB, patch)
2018-03-19 13:44 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271 - merge deletedbiblio and -items back - add atomic update (2.39 KB, patch)
2018-03-19 13:44 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix remaining tests, add checks and report in atomicupdate (9.64 KB, patch)
2018-03-19 13:44 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (33.69 KB, patch)
2018-04-02 22:22 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271 - merge deletedbiblio and -items back - add atomic update (2.39 KB, patch)
2018-04-02 22:22 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix remaining tests, add checks and report in atomicupdate (9.64 KB, patch)
2018-04-02 22:22 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: update kohastructure.sql (13.03 KB, patch)
2018-04-02 22:22 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: delete biblio and biblio_metadata in one go (813 bytes, patch)
2018-04-02 22:22 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: remove comments regarding deleteditems and -biblio* (2.63 KB, patch)
2018-04-02 22:22 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271 - Use a single pass in the OAI-PMH provider (17.99 KB, patch)
2018-04-04 07:50 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (33.69 KB, patch)
2018-04-06 12:24 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271 - merge deletedbiblio and -items back - add atomic update (5.05 KB, patch)
2018-04-06 12:25 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix remaining tests (3.94 KB, patch)
2018-04-06 12:25 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: update kohastructure.sql (13.03 KB, patch)
2018-04-06 12:25 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: delete biblio and biblio_metadata in one go (813 bytes, patch)
2018-04-06 12:25 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: remove comments regarding deleteditems and -biblio* (2.63 KB, patch)
2018-04-06 12:25 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271 - Use a single pass in the OAI-PMH provider (17.99 KB, patch)
2018-04-06 12:25 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (34.79 KB, patch)
2018-04-13 07:17 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix remaining tests (3.94 KB, patch)
2018-04-13 07:17 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: update kohastructure.sql (13.03 KB, patch)
2018-04-13 07:18 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: delete biblio and biblio_metadata in one go (813 bytes, patch)
2018-04-13 07:18 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: remove comments regarding deleteditems and -biblio* (2.63 KB, patch)
2018-04-13 07:18 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: add atomic update (5.17 KB, patch)
2018-04-13 07:18 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: Use a single pass in the OAI-PMH provider (17.99 KB, patch)
2018-04-13 07:19 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix whitespace issues in C4::Items (24.80 KB, patch)
2018-04-13 07:19 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (34.82 KB, patch)
2018-04-13 07:35 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix remaining tests (3.94 KB, patch)
2018-04-13 07:36 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: update kohastructure.sql (13.03 KB, patch)
2018-04-13 07:36 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: delete biblio and biblio_metadata in one go (813 bytes, patch)
2018-04-13 07:36 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: remove comments regarding deleteditems and -biblio* (2.63 KB, patch)
2018-04-13 07:36 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: add atomic update (5.17 KB, patch)
2018-04-13 07:36 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: Use a single pass in the OAI-PMH provider (17.99 KB, patch)
2018-04-13 07:36 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix whitespace issues in C4::Items (24.80 KB, patch)
2018-04-13 07:36 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (34.82 KB, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix remaining tests (3.94 KB, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: update kohastructure.sql (13.03 KB, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: delete biblio and biblio_metadata in one go (813 bytes, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: remove comments regarding deleteditems and -biblio* (2.63 KB, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: add atomic update (5.05 KB, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: Use a single pass in the OAI-PMH provider (17.99 KB, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: fix whitespace issues in C4::Items (24.80 KB, patch)
2018-04-14 10:00 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: print instead of warn unmovable items (5.58 KB, patch)
2018-04-14 17:40 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: atomicupdate - support mysql < 5.7 and address foreign key issues (2.89 KB, patch)
2018-04-16 09:08 UTC, Benjamin Rokseth
Details | Diff | Splinter Review
Bug 20271: merging delete biblio/items (34.89 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: fix remaining tests (4.01 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: update kohastructure.sql (13.10 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: delete biblio and biblio_metadata in one go (879 bytes, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: remove comments regarding deleteditems and -biblio* (2.70 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: add atomic update (5.12 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: Use a single pass in the OAI-PMH provider (18.06 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: fix whitespace issues in C4::Items (24.86 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: print instead of warn unmovable items (5.64 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: atomicupdate - support mysql < 5.7 and address foreign key issues (2.96 KB, patch)
2018-04-18 06:41 UTC, Brendan Gallagher
Details | Diff | Splinter Review
Bug 20271: (follow-up) DBIx schema changes (25.26 KB, patch)
2018-05-11 10:19 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 20271: (QA follow-up): rename to deleted_on and various fixes (42.40 KB, patch)
2018-05-15 21:39 UTC, Benjamin Rokseth
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Petter Goksøyr Åsen 2018-02-22 13:42:11 UTC
In Koha's DB, instead of having a "soft" deletion of entries by flagging the deleted one as so, the rows in the db is actually deleted, but only after a new row in a different table is inserted.

Koha's DB is relational, so there is a cascade of links between each object: a biblio is referred by many items, and each item is referred by issues, holds, messages.

E.g.: when a row in the item table is moved from "items" to "deleted_items", then all the references from holds, transfers and so on will be broken.

The suggested more orthodox approach to this would be to add a "deleted_at" column, which is normally null unless the row is deleted, then it set to the timestamp of when it was deletion.

This means that the row is still there and will still be referred by other tables, but it just won't be shown in the relevant lists.

This would improve the maintenance of Koha in many ways (e.g.: the auto_increment id on those tables caused lots of issues when you add an item and delete it immediately after. the next new item will have the same itemnumber as the previously deleted one, clashing with it in many ways).

This bug is about merging the all these tables:
deletedborrowers       => borrowers
deletedbiblio          => biblio
deletedbiblioitems     => biblioitems
deletedbiblio_metadata => biblio_metadata
deleteditems           => items
old_issues             => issues
old_reserves           => reserves
Comment 1 Katrin Fischer 2018-02-23 20:39:34 UTC
I like the idea of making it a timestamp instead of a boolean as suggested on other bugs (I think last discussed for recalls on bug 19532). Are there any reasons why one should be preferred over the other, for example for performance?
Comment 2 Petter Goksøyr Åsen 2018-02-25 05:51:00 UTC
> Are there any reasons why one should be preferred over the other, for example for performance?

I dont't think so. As long as the column is indexed, querying for one or the other should perform with equivalent speed.

The benefit of using a timestamp is that it carries more information, and mirrors the current behaviour, where the timestamp in the deleted-tables denotes when the row was deleted.
Comment 3 Francesco Rivetti 2018-02-26 09:17:31 UTC
I've started working on this and have migrated items, biblios, biblioitems and biblio_metadata successfully. it works fine on intra and seems ok in the administrative reports.

there is also a wiki page with more info here: https://wiki.koha-community.org/wiki/MergingOfTables
Comment 4 Francesco Rivetti 2018-03-01 08:53:14 UTC
Created attachment 72300 [details] [review]
Bug 20271: merging delete biblio/items

NOTE: deleteditems/biblio on OAI
not sure how to test this, but the changes should be good enough
Comment 5 Josef Moravec 2018-03-02 07:09:29 UTC
Comment on attachment 72300 [details] [review]
Bug 20271: merging delete biblio/items

Review of attachment 72300 [details] [review]:
-----------------------------------------------------------------

Just a quick look:

there is missing atomic database update

also the test suite should be adjusted

I am not sure about the changes in OAI, so need to investigate more later

::: C4/Biblio.pm
@@ -3182,5 @@
> -        my $bkup_sth = $dbh->prepare($query);
> -        $bkup_sth->execute(@bind);
> -        $bkup_sth->finish;
> -
> -        _koha_delete_biblio_metadata( $biblionumber );

this is missing in your patch

::: C4/Items.pm
@@ +1061,4 @@
>       FROM items
>       LEFT JOIN branches AS holding ON items.holdingbranch = holding.branchcode
>       LEFT JOIN branches AS home ON items.homebranch=home.branchcode
> +     LEFT JOIN biblio      ON      biblio.biblionumber     = items.biblionumber AND biblio.deleted_at IS NULL

I don't think this will work, that's condition should be in where clause

::: misc/export_records.pl
@@ +105,5 @@
>          push @record_ids, $_->{biblionumber} for @{
> +            $dbh->selectall_arrayref(q|
> +                SELECT biblionumber
> +                FROM biblioitems
> +             |, { Slice => {} }, ( $timestamp ) x 4 );

Please include the changes made by Bug 19730
Comment 6 Francesco Rivetti 2018-03-05 08:35:03 UTC
(In reply to Josef Moravec from comment #5)
> there is missing atomic database update

It is on its way.

(It's fairly easy to do something reasonable, but there are some corner cases
which I would like to handle somehow better)

I would left the deleted* tables there with any clashing rows and let the
administrator deal with it in case there are problems. thoughts?


> I am not sure about the changes in OAI, so need to investigate more later

Thank you so much, I wish I knew more about OAI

> ::: C4/Biblio.pm
> @@ -3182,5 @@
> > -        my $bkup_sth = $dbh->prepare($query);
> > -        $bkup_sth->execute(@bind);
> > -        $bkup_sth->finish;
> > -
> > -        _koha_delete_biblio_metadata( $biblionumber );
> 
> this is missing in your patch

I might be lagging behind master, because I can't find it.

> ::: C4/Items.pm
> @@ +1061,4 @@
> >       FROM items
> >       LEFT JOIN branches AS holding ON items.holdingbranch = holding.branchcode
> >       LEFT JOIN branches AS home ON items.homebranch=home.branchcode
> > +     LEFT JOIN biblio      ON      biblio.biblionumber     = items.biblionumber AND biblio.deleted_at IS NULL
> 
> I don't think this will work, that's condition should be in where clause

I actually think it will be irrelevant. The FKs should make sure that if there is an item, there must be the biblio which it refers.

Or am I missing something?


> ::: misc/export_records.pl
> @@ +105,5 @@
> >          push @record_ids, $_->{biblionumber} for @{
> > +            $dbh->selectall_arrayref(q|
> > +                SELECT biblionumber
> > +                FROM biblioitems
> > +             |, { Slice => {} }, ( $timestamp ) x 4 );
> 
> Please include the changes made by Bug 19730

Is it in master, right?

TIA
Comment 7 Katrin Fischer 2018-03-05 13:28:42 UTC
I think we should try and provide at least a script to automate merging the two tables. You cannot assume that most admins know Koha well enough to fix the issues themselves. A problem might be the possible existence of duplicated rows. We could check that and then suggest a fix before continuing.
Comment 8 Francesco Rivetti 2018-03-05 13:54:26 UTC
(In reply to Katrin Fischer from comment #7)
> I think we should try and provide at least a script to automate merging the
> two tables. You cannot assume that most admins know Koha well enough to fix
> the issues themselves. A problem might be the possible existence of
> duplicated rows. We could check that and then suggest a fix before
> continuing.

My idea is to move all the rows properly.

if any rows clash (for some unforeseen reason, i found a "orphaned" item referring to a non-existent biblio here, but can be anything!) then they won't be moved.

at the end, if the table is empty, it gets removed, otherwise it stays there.

does it sounds good?
Comment 9 Benjamin Rokseth 2018-03-13 15:43:10 UTC
Created attachment 72799 [details] [review]
Bug 20271: merging delete biblio/items

NOTE: deleteditems/biblio on OAI
not sure how to test this, but the changes should be good enough
Comment 10 Benjamin Rokseth 2018-03-13 15:43:18 UTC
Created attachment 72800 [details] [review]
Bug 20271 - merge deletedbiblio and -items back - add atomic update
Comment 11 Benjamin Rokseth 2018-03-13 15:48:37 UTC
Updated the failing tests, except for the weird GetItemsForInventory test which needs another look.

Ready for testing, we renamed the ticket to only concern tables deletedbiblio, deletedbiblioitems, deletedbiblio_metadata and deleteditems, since they are most crucial to avoid more data loss and corruption.

Suggest to rename deleted* tables temporarily before deleting them, so any merge conflicts can be examined first

Also suggest to run misc/devel/update_dbix_class_files.pl after, so schema/apis etc work as expected
Comment 12 Jon Knight 2018-03-14 19:51:19 UTC
Just a random thought to throw into this: would it be worth creating database views to replicate the old tables based on the new ones?  That way if any installations have pre-existing reports that use the old tables in their queries, they will still continue to work.

For example something like this:

CREATE VIEW old_issues AS SELECT * FROM issues WHERE deleted_at IS NOT NULL;
Comment 13 Francesco Rivetti 2018-03-15 10:41:07 UTC
(In reply to Jon Knight from comment #12)
> Just a random thought to throw into this: would it be worth creating
> database views to replicate the old tables based on the new ones?  That way
> if any installations have pre-existing reports that use the old tables in
> their queries, they will still continue to work.
> 
> For example something like this:
> 
> CREATE VIEW old_issues AS SELECT * FROM issues WHERE deleted_at IS NOT NULL;

I thought about it, but you most likely need the view for "issues" than "old_issues", which means:

RENAME TABLE issues TO all_issues;
CREATE VIEW issues AS SELECT * FROM issues WHERE delted_at IS NULL;

but then, more code will need to be changed for updates/insert/deletes

OTOH, most reports UNION issues with old_issues, might be smart to take this opportunity to simplify them?

(note, this applies to deleteditems/biblios as well)
Comment 14 Benjamin Rokseth 2018-03-19 13:44:39 UTC
Created attachment 73091 [details] [review]
Bug 20271: merging delete biblio/items

NOTE: deleteditems/biblio on OAI
not sure how to test this, but the changes should be good enough
Comment 15 Benjamin Rokseth 2018-03-19 13:44:46 UTC
Created attachment 73092 [details] [review]
Bug 20271 - merge deletedbiblio and -items back - add atomic update
Comment 16 Benjamin Rokseth 2018-03-19 13:44:54 UTC
Created attachment 73093 [details] [review]
Bug 20271: fix remaining tests, add checks and report in atomicupdate
Comment 17 Benjamin Rokseth 2018-03-19 13:52:13 UTC
Ready for testing now, test plan:

1) make sure you have some deleted biblios and items
2) apply patch and run updatedatabase
3) if you have any clashing keys, they should be reported during updatedatabase and be left in renamed corresponding tables with _ (e.g. _deleteditems)
4) run update_dbix_class_files.pl (to make sure schemas and rest api are ok)
5) check that db_dependent tests concerning biblios and items pass OK
6) check that you can still delete items and biblio
7) for any deleted item or biblio, you should now have a timestamp in the 'deleted_at' column.
Comment 18 Benjamin Rokseth 2018-03-20 09:53:32 UTC
raised the importance to critical since it does in fact lead to data loss and possibly data corruption
Comment 19 Francesco Rivetti 2018-03-20 10:27:21 UTC
thanks Benjamin!

(In reply to Benjamin Rokseth from comment #16)
> Created attachment 73093 [details] [review] [review]
> Bug 20271: fix remaining tests, add checks and report in atomicupdate
Comment 20 Ere Maijala 2018-03-27 10:49:34 UTC
I think the OAI-PMH provider should now be changed to return all records in a single pass. It was built to return deleted records separately because that was much faster than joining tables on the fly, but when deleted and non-deleted records are in the same table this complication is not needed anymore. I think the whole ListBase class could be removed in this case.

Please let me know if you'd rather let me do the relevant changes in the OAI classes.
Comment 21 Ere Maijala 2018-03-27 12:13:40 UTC
One more note: I couldn't find any changes to kohastructure.sql. Are they missing from the patches or did I just miss them?
Comment 22 Ere Maijala 2018-03-27 12:44:36 UTC
And one more: there are still comments that refer to the deleted* tables.
Comment 23 Ere Maijala 2018-03-27 12:48:52 UTC
Sorry for the comment spam... one more: the call to _koha_delete_biblio_metadata has been removed from _koha_delete_biblio. As far as I can see nothing takes care of marking the metadata record deleted.
Comment 24 Benjamin Rokseth 2018-04-02 22:22:02 UTC
Created attachment 73535 [details] [review]
Bug 20271: merging delete biblio/items

NOTE: deleteditems/biblio on OAI
not sure how to test this, but the changes should be good enough
Comment 25 Benjamin Rokseth 2018-04-02 22:22:09 UTC
Created attachment 73536 [details] [review]
Bug 20271 - merge deletedbiblio and -items back - add atomic update
Comment 26 Benjamin Rokseth 2018-04-02 22:22:14 UTC
Created attachment 73537 [details] [review]
Bug 20271: fix remaining tests, add checks and report in atomicupdate
Comment 27 Benjamin Rokseth 2018-04-02 22:22:21 UTC
Created attachment 73538 [details] [review]
Bug 20271: update kohastructure.sql
Comment 28 Benjamin Rokseth 2018-04-02 22:22:27 UTC
Created attachment 73539 [details] [review]
Bug 20271: delete biblio and biblio_metadata in one go
Comment 29 Benjamin Rokseth 2018-04-02 22:22:32 UTC
Created attachment 73540 [details] [review]
Bug 20271: remove comments regarding deleteditems and -biblio*
Comment 30 Benjamin Rokseth 2018-04-02 22:24:56 UTC
rebased against master and added fixes to comments except OAI.

Ere, since you voluntered to add the neccessary fixes to OAI, I leave them to you ;)
Comment 31 Ere Maijala 2018-04-04 06:32:09 UTC
I'm working on it now. 


Looks like there may be a typo in the atomic update. I got the following error:

updatedatabase.pl: DBD::mysql::db do failed: Can't DROP FOREIGN KEY `deletedbiblio_metadata_fk_1`; check that it exists [for Statement "ALTER TABLE deletedbiblio_metadata DROP FOREIGN KEY deletedbiblio_metadata_fk_1"] at (eval 1389) line 8.

Looking at the old kohastructure.sql I believe it should have tried to drop 'deletedrecord_metadata_fk_1'.
Comment 32 Benjamin Rokseth 2018-04-04 07:18:49 UTC
(In reply to Ere Maijala from comment #31)
> I'm working on it now. 
> 
> 
> Looks like there may be a typo in the atomic update. I got the following
> error:
> 
> updatedatabase.pl: DBD::mysql::db do failed: Can't DROP FOREIGN KEY
> `deletedbiblio_metadata_fk_1`; check that it exists [for Statement "ALTER
> TABLE deletedbiblio_metadata DROP FOREIGN KEY deletedbiblio_metadata_fk_1"]
> at (eval 1389) line 8.
> 
> Looking at the old kohastructure.sql I believe it should have tried to drop
> 'deletedrecord_metadata_fk_1'.

hmm, sad to say, but no, it's not a typo but db design issues.
this commit seems to have added to the confusion: https://github.com/Koha-Community/Koha/commit/739e2e0c5dc26502e195e2cc4c687ababdcaf381 (Bug 17196)

and following Bug 18284 you can see that most agree on deleted* tables being a bad idea in the first place.

Problem is, kohastructure.sql and updatedatabase.pl is not in sync and libraries having migrated from 16.05 and before would need to delete this foreign key. And this patch is most important for them anyways.

I suppose we could add a delete foreign key for both and ignore errors rather than die in the final version
Comment 33 Ere Maijala 2018-04-04 07:45:12 UTC
Oh, that's a nasty one. But do you even have to drop the constraints to delete the table? I think it's enough to drop the table and all the constraints there would be dropped too, no?
Comment 34 Ere Maijala 2018-04-04 07:50:02 UTC
Created attachment 73597 [details] [review]
Bug 20271 - Use a single pass in the OAI-PMH provider
Comment 35 Ere Maijala 2018-04-04 07:51:06 UTC
Changes for the OAI-PMH provider have been attached. I'm really happy to make it so much more simple. Please feel free to rebase or whatever is necessary.
Comment 36 Benjamin Rokseth 2018-04-04 08:02:53 UTC
(In reply to Benjamin Rokseth from comment #32)
> (In reply to Ere Maijala from comment #31)
> > I'm working on it now. 
> > 
> > 
> > Looks like there may be a typo in the atomic update. I got the following
> > error:
> > 
> > updatedatabase.pl: DBD::mysql::db do failed: Can't DROP FOREIGN KEY
> > `deletedbiblio_metadata_fk_1`; check that it exists [for Statement "ALTER
> > TABLE deletedbiblio_metadata DROP FOREIGN KEY deletedbiblio_metadata_fk_1"]
> > at (eval 1389) line 8.
> > 
> > Looking at the old kohastructure.sql I believe it should have tried to drop
> > 'deletedrecord_metadata_fk_1'.
> 
> hmm, sad to say, but no, it's not a typo but db design issues.
> this commit seems to have added to the confusion:
> https://github.com/Koha-Community/Koha/commit/
> 739e2e0c5dc26502e195e2cc4c687ababdcaf381 (Bug 17196)
> 
> and following Bug 18284 you can see that most agree on deleted* tables being
> a bad idea in the first place.
> 
> Problem is, kohastructure.sql and updatedatabase.pl is not in sync and
> libraries having migrated from 16.05 and before would need to delete this
> foreign key. And this patch is most important for them anyways.
> 
> I suppose we could add a delete foreign key for both and ignore errors
> rather than die in the final version

fail to remember, but probably I put it there because otherwise it complained, perhaps due to the "rename table". the idea was that any leftovers from conflicts should be moved to temporary tables (prepended with _) so that one could check and delete them at convenience

I will remove the dies in the atomicupdate and see if it works for us
Comment 37 Francesco Rivetti 2018-04-04 08:09:23 UTC
(In reply to Benjamin Rokseth from comment #36)
> fail to remember, but probably I put it there because otherwise it
> complained, perhaps due to the "rename table". the idea was that any
> leftovers from conflicts should be moved to temporary tables (prepended with
> _) so that one could check and delete them at convenience

We had to drop the constraint to avoid the delete on cascade. When we clean up the deletedbiblio andt able, we don't want to drop any items which were left behind because not properly merged.
Comment 38 Ere Maijala 2018-04-04 08:10:55 UTC
Ok, fair enough. I think Benjamin's proposal of dropping both without die'ing makes sense.
Comment 39 Benjamin Rokseth 2018-04-06 12:24:59 UTC
Created attachment 73759 [details] [review]
Bug 20271: merging delete biblio/items

NOTE: deleteditems/biblio on OAI
not sure how to test this, but the changes should be good enough
Comment 40 Benjamin Rokseth 2018-04-06 12:25:06 UTC
Created attachment 73760 [details] [review]
Bug 20271 - merge deletedbiblio and -items back - add atomic update
Comment 41 Benjamin Rokseth 2018-04-06 12:25:12 UTC
Created attachment 73761 [details] [review]
Bug 20271: fix remaining tests
Comment 42 Benjamin Rokseth 2018-04-06 12:25:18 UTC
Created attachment 73762 [details] [review]
Bug 20271: update kohastructure.sql
Comment 43 Benjamin Rokseth 2018-04-06 12:25:25 UTC
Created attachment 73763 [details] [review]
Bug 20271: delete biblio and biblio_metadata in one go
Comment 44 Benjamin Rokseth 2018-04-06 12:25:32 UTC
Created attachment 73764 [details] [review]
Bug 20271: remove comments regarding deleteditems and -biblio*
Comment 45 Benjamin Rokseth 2018-04-06 12:25:38 UTC
Created attachment 73765 [details] [review]
Bug 20271 - Use a single pass in the OAI-PMH provider
Comment 46 Benjamin Rokseth 2018-04-06 12:27:33 UTC
Replaced die with warn in atomicupdate, and rebased and squashed a bit.
Should be ready for testing now
Comment 47 Benjamin Rokseth 2018-04-11 11:13:06 UTC
Comment on attachment 73537 [details] [review]
Bug 20271: fix remaining tests, add checks and report in atomicupdate

Forgot to obsolete this
Comment 48 Benjamin Rokseth 2018-04-13 07:17:36 UTC
Created attachment 74127 [details] [review]
Bug 20271: merging delete biblio/items

The present bug merges the following tables:
deletedbiblio          -> biblio
deletedbiblioitems     -> biblioitems
deletedbiblio_metadata -> biblio_metadata
deleteditems           -> items

and adds a column deleted_at to signify time of deletion, if deleted

it replaces all occurrences of the mentioned occurences and uses
of mentioned deleted* tables

upgrade is handled by moving rows from deleted* tables to the live
ones. If all rows are moved, the corresponding tables are dropped.
If there are duplicates or conflicts (duplicate barcodes e.g.) they
reminders are left for scrutiny and table renamed to _deleted*

Test plan:

1) Make sure you have a db with at least a few deleted biblios,
   biblioitems and items.
2) Apply patch and run updatedatabase.pl
3) Make note of the db upgrade messages, if you get a message like:
   There were x deleteditems that could not be moved, please check _deleteditems
   you should check the _deleteditems table and verify that they are
   actual conflicts
4) Click around in the web interface and make sure you can delete items
   and biblios
5) "Undelete" an item by NULLing the deleted_at column for an item
   and verify that it returns in interface (Not neccessary really)
Comment 49 Benjamin Rokseth 2018-04-13 07:17:52 UTC
Created attachment 74128 [details] [review]
Bug 20271: fix remaining tests
Comment 50 Benjamin Rokseth 2018-04-13 07:18:04 UTC
Created attachment 74129 [details] [review]
Bug 20271: update kohastructure.sql
Comment 51 Benjamin Rokseth 2018-04-13 07:18:16 UTC
Created attachment 74130 [details] [review]
Bug 20271: delete biblio and biblio_metadata in one go
Comment 52 Benjamin Rokseth 2018-04-13 07:18:25 UTC
Created attachment 74131 [details] [review]
Bug 20271: remove comments regarding deleteditems and -biblio*
Comment 53 Benjamin Rokseth 2018-04-13 07:18:45 UTC
Created attachment 74132 [details] [review]
Bug 20271: add atomic update
Comment 54 Benjamin Rokseth 2018-04-13 07:19:05 UTC
Created attachment 74133 [details] [review]
Bug 20271: Use a single pass in the OAI-PMH provider
Comment 55 Benjamin Rokseth 2018-04-13 07:19:18 UTC
Created attachment 74134 [details] [review]
Bug 20271: fix whitespace issues in C4::Items
Comment 56 Benjamin Rokseth 2018-04-13 07:35:58 UTC
Created attachment 74135 [details] [review]
Bug 20271: merging delete biblio/items

The present bug merges the following tables:
deletedbiblio          -> biblio
deletedbiblioitems     -> biblioitems
deletedbiblio_metadata -> biblio_metadata
deleteditems           -> items

and adds a column deleted_at to signify time of deletion, if deleted

it replaces all occurrences of the mentioned occurences and uses
of mentioned deleted* tables

upgrade is handled by moving rows from deleted* tables to the live
ones. If all rows are moved, the corresponding tables are dropped.
If there are duplicates or conflicts (duplicate barcodes e.g.) they
reminders are left for scrutiny and table renamed to _deleted*

Test plan:

1) Make sure you have a db with at least a few deleted biblios,
   biblioitems and items.
2) Apply patch and run updatedatabase.pl
3) Make note of the db upgrade messages, if you get a message like:
   There were x deleteditems that could not be moved, please check _deleteditems
   you should check the _deleteditems table and verify that they are
   actual conflicts
4) Click around in the web interface and make sure you can delete items
   and biblios
5) "Undelete" an item by NULLing the deleted_at column for an item
   and verify that it returns in interface (Not neccessary really)
Comment 57 Benjamin Rokseth 2018-04-13 07:36:05 UTC
Created attachment 74136 [details] [review]
Bug 20271: fix remaining tests
Comment 58 Benjamin Rokseth 2018-04-13 07:36:11 UTC
Created attachment 74137 [details] [review]
Bug 20271: update kohastructure.sql
Comment 59 Benjamin Rokseth 2018-04-13 07:36:16 UTC
Created attachment 74138 [details] [review]
Bug 20271: delete biblio and biblio_metadata in one go
Comment 60 Benjamin Rokseth 2018-04-13 07:36:22 UTC
Created attachment 74139 [details] [review]
Bug 20271: remove comments regarding deleteditems and -biblio*
Comment 61 Benjamin Rokseth 2018-04-13 07:36:28 UTC
Created attachment 74140 [details] [review]
Bug 20271: add atomic update
Comment 62 Benjamin Rokseth 2018-04-13 07:36:33 UTC
Created attachment 74141 [details] [review]
Bug 20271: Use a single pass in the OAI-PMH provider
Comment 63 Benjamin Rokseth 2018-04-13 07:36:39 UTC
Created attachment 74142 [details] [review]
Bug 20271: fix whitespace issues in C4::Items
Comment 64 Benjamin Rokseth 2018-04-14 10:00:05 UTC
Created attachment 74182 [details] [review]
Bug 20271: merging delete biblio/items

The present bug merges the following tables:
deletedbiblio          -> biblio
deletedbiblioitems     -> biblioitems
deletedbiblio_metadata -> biblio_metadata
deleteditems           -> items

and adds a column deleted_at to signify time of deletion, if deleted

it replaces all occurrences of the mentioned occurences and uses
of mentioned deleted* tables

upgrade is handled by moving rows from deleted* tables to the live
ones. If all rows are moved, the corresponding tables are dropped.
If there are duplicates or conflicts (duplicate barcodes e.g.) they
reminders are left for scrutiny and table renamed to _deleted*

Test plan:

1) Make sure you have a db with at least a few deleted biblios,
   biblioitems and items.
2) Apply patch and run updatedatabase.pl
3) Make note of the db upgrade messages, if you get a message like:
   There were x deleteditems that could not be moved, please check _deleteditems
   you should check the _deleteditems table and verify that they are
   actual conflicts
4) Click around in the web interface and make sure you can delete items
   and biblios
5) "Undelete" an item by NULLing the deleted_at column for an item
   and verify that it returns in interface (Not neccessary really)
Comment 65 Benjamin Rokseth 2018-04-14 10:00:10 UTC
Created attachment 74183 [details] [review]
Bug 20271: fix remaining tests
Comment 66 Benjamin Rokseth 2018-04-14 10:00:16 UTC
Created attachment 74184 [details] [review]
Bug 20271: update kohastructure.sql
Comment 67 Benjamin Rokseth 2018-04-14 10:00:21 UTC
Created attachment 74185 [details] [review]
Bug 20271: delete biblio and biblio_metadata in one go
Comment 68 Benjamin Rokseth 2018-04-14 10:00:27 UTC
Created attachment 74186 [details] [review]
Bug 20271: remove comments regarding deleteditems and -biblio*
Comment 69 Benjamin Rokseth 2018-04-14 10:00:32 UTC
Created attachment 74187 [details] [review]
Bug 20271: add atomic update

Bug 20271: disable foreign keys on deletedbiblio_metadata
Comment 70 Benjamin Rokseth 2018-04-14 10:00:38 UTC
Created attachment 74188 [details] [review]
Bug 20271: Use a single pass in the OAI-PMH provider
Comment 71 Benjamin Rokseth 2018-04-14 10:00:44 UTC
Created attachment 74189 [details] [review]
Bug 20271: fix whitespace issues in C4::Items
Comment 72 Benjamin Rokseth 2018-04-14 17:40:47 UTC
Created attachment 74190 [details] [review]
Bug 20271: print instead of warn unmovable items
Comment 73 Benjamin Rokseth 2018-04-16 09:08:05 UTC
Created attachment 74223 [details] [review]
Bug 20271: atomicupdate - support mysql < 5.7 and address foreign key issues
Comment 74 Brendan Gallagher 2018-04-18 06:41:10 UTC
Created attachment 74407 [details] [review]
Bug 20271: merging delete biblio/items

The present bug merges the following tables:
deletedbiblio          -> biblio
deletedbiblioitems     -> biblioitems
deletedbiblio_metadata -> biblio_metadata
deleteditems           -> items

and adds a column deleted_at to signify time of deletion, if deleted

it replaces all occurrences of the mentioned occurences and uses
of mentioned deleted* tables

upgrade is handled by moving rows from deleted* tables to the live
ones. If all rows are moved, the corresponding tables are dropped.
If there are duplicates or conflicts (duplicate barcodes e.g.) they
reminders are left for scrutiny and table renamed to _deleted*

Test plan:

1) Make sure you have a db with at least a few deleted biblios,
   biblioitems and items.
2) Apply patch and run updatedatabase.pl
3) Make note of the db upgrade messages, if you get a message like:
   There were x deleteditems that could not be moved, please check _deleteditems
   you should check the _deleteditems table and verify that they are
   actual conflicts
4) Click around in the web interface and make sure you can delete items
   and biblios
5) "Undelete" an item by NULLing the deleted_at column for an item
   and verify that it returns in interface (Not neccessary really)

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Worked on two different databases.
Comment 75 Brendan Gallagher 2018-04-18 06:41:20 UTC
Created attachment 74408 [details] [review]
Bug 20271: fix remaining tests

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 76 Brendan Gallagher 2018-04-18 06:41:24 UTC
Created attachment 74409 [details] [review]
Bug 20271: update kohastructure.sql

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 77 Brendan Gallagher 2018-04-18 06:41:28 UTC
Created attachment 74410 [details] [review]
Bug 20271: delete biblio and biblio_metadata in one go

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 78 Brendan Gallagher 2018-04-18 06:41:32 UTC
Created attachment 74411 [details] [review]
Bug 20271: remove comments regarding deleteditems and -biblio*

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 79 Brendan Gallagher 2018-04-18 06:41:36 UTC
Created attachment 74412 [details] [review]
Bug 20271: add atomic update

Bug 20271: disable foreign keys on deletedbiblio_metadata

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 80 Brendan Gallagher 2018-04-18 06:41:40 UTC
Created attachment 74413 [details] [review]
Bug 20271: Use a single pass in the OAI-PMH provider

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 81 Brendan Gallagher 2018-04-18 06:41:44 UTC
Created attachment 74414 [details] [review]
Bug 20271: fix whitespace issues in C4::Items

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 82 Brendan Gallagher 2018-04-18 06:41:48 UTC
Created attachment 74415 [details] [review]
Bug 20271: print instead of warn unmovable items

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 83 Brendan Gallagher 2018-04-18 06:41:52 UTC
Created attachment 74416 [details] [review]
Bug 20271: atomicupdate - support mysql < 5.7 and address foreign key issues

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Comment 84 Marcel de Rooy 2018-05-11 09:41:16 UTC
Looking here now, but I feel that this patch set will not make it anymore to 18.05.
Comment 85 Marcel de Rooy 2018-05-11 10:19:01 UTC
Created attachment 75265 [details] [review]
Bug 20271: (follow-up) DBIx schema changes
Comment 86 Marcel de Rooy 2018-05-11 10:23:04 UTC
Great job, Francesco
I did not finish my QA now, but want to report some issues I found already. And they unfortunately confirm that this patch set will not make it anymore into 18.05 ;)
But it would be good to have in master early.

qa tools: FAIL   Koha/Patron.pm, FAIL   pod, Spurious text after =cut
deleted_at should probably be better: deleted_on; we also have updated_on etc.
$dbh->do( "ALTER TABLE biblio ADD COLUMN deleted_at datetime DEFAULT NULL" ) or warn $DBI::errstr;
Probably it warns already and you should die here ?
Running OAI/Server.t
ok 12 - use Koha::OAI::Server::ResumptionToken;
DBD::mysql::db do failed: Table 'koha_dev.deletedbiblio' doesn't exist [for Statement "DELETE FROM deletedbiblio"] at t/db_dependent/OAI/Server.t line 68.
DBD::mysql::db do failed: Table 'koha_dev.deletedbiblioitems' doesn't exist [for Statement "DELETE FROM deletedbiblioitems"] at t/db_dependent/OAI/Server.t line 69.
DBD::mysql::db do failed: Table 'koha_dev.deleteditems' doesn't exist [for Statement "DELETE FROM deleteditems"] at t/db_dependent/OAI/Server.t line 70.
git grep deletedbiblio
about.pl:        q|select b.biblionumber from biblio b join deletedbiblio db on b.biblionumber=db.biblionumber|,
koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/web_services.pref (a few other refs too)
git grep deleteditems
about.pl again
koha-tmpl/intranet-tmpl/prog/en/modules/reports/catalogue_stats.tt:                <td><input type="radio" name="Line" value="deleteditems.timestamp" /></td>
might not be a problem?
misc/export_records.pl:            WHERE deleteditems.biblionumber = ? AND deleted_at IS NOT NULL
does not look good
reports/catalogue_stats.pl:    $linefilter[0] = @$filters[15] if ( $line =~ /deleteditems\.timestamp/ );
we should probably get rid of those things; they are confusing now
Comment 87 Marcel de Rooy 2018-05-11 10:25:17 UTC
I am not sure if we should mark this one as critical btw. I would go for major when viewing it as a bugfix. But enh could be argued too.
Not so sure if the RMaints want to backport this one?
Comment 88 Benjamin Rokseth 2018-05-11 10:56:43 UTC
Marcel, thanks for thorough feedback!
will address the issues asap so it gets into master sooner than later.
Downgraded to major, enhancement it is definitely not ;)
Comment 89 Benjamin Rokseth 2018-05-15 21:39:30 UTC
Created attachment 75347 [details] [review]
Bug 20271: (QA follow-up): rename to deleted_on and various fixes

- renamed deleted_at => deleted_on
- removed failing sql in t/db_dependent/OAI/Server.t
- removed deprecated autoincrement checks in about.pl
- fixed errors in catalogue_stats.tt and catalogue_stats.pl
Comment 90 Benjamin Rokseth 2018-05-15 21:43:07 UTC
Adjusted to all qa feedback, except:
- left warn instead of die in atomicupdate - it would be removed by RM when moved to updatedatabase.pl anyways I suppose
- qa tools: FAIL   Koha/Patron.pm, FAIL   pod, Spurious text after =cut
I could not reproduce. Ran qa tools and it passed.