Bug 18265 - Should biblio_metadata.biblionumber be biblioitemnumber instead?
Summary: Should biblio_metadata.biblionumber be biblioitemnumber instead?
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P2 normal
Assignee: Olli-Antti Kivilahti
QA Contact: Testopia
URL:
Keywords:
Depends on: 17196
Blocks:
  Show dependency treegraph
 
Reported: 2017-03-14 14:28 UTC by Olli-Antti Kivilahti
Modified: 2023-09-11 04:47 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. (10.80 KB, patch)
2017-03-14 17:00 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table. (13.46 KB, patch)
2017-03-15 10:31 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table. (14.81 KB, patch)
2017-03-15 10:39 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table. (18.65 KB, patch)
2017-03-15 11:07 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table. (18.76 KB, patch)
2017-03-15 13:05 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review
Bug 18265 - Followup to Buuug 17196. Fix t/db_dependent/Exporter/Record.t (1.50 KB, patch)
2017-03-17 08:01 UTC, Olli-Antti Kivilahti
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Olli-Antti Kivilahti 2017-03-14 14:28:12 UTC
Exhibit 1:

The MARCXML is lost when the biblio is deleted because the values in biblio_metadata-table from biblioitems-table are never moved to deletedbiblio_metadata-table.
Previously the marcxml was preserved in the oldbiblioitems-table
This is a challenge for gathering statistics
For example item has been ordered, received, and removed from collection during the statistics period
Information in marcxml is needed to accurately categorize the item

Exhibit 2:

[15:22:38] <kivilahtio> also. In Koha there once was a great idea that one biblio-entry can have multiple biblioitems
[15:22:45] <kivilahtio> and the marcxml was tied to the biblioitems-table
[15:23:11] <kivilahtio> this is a nice idea, but nobdy really cared for it and now biblionumber and biblioitemnumber are used interchangeably
[15:23:22] <kivilahtio> except when they get out of sync during a data migration
[15:23:41] <kivilahtio> then things start to work really interestingly
[15:23:47] <kivilahtio> but the point is
[15:23:57] <kivilahtio> biblio_metadata has a biblionumber-column
[15:24:15] <kivilahtio> shouldn't it have a biblioitemnumber-column, just for old time's sake?

In the biblioitems-table we had marcxml. primary keyed with biblioitemnumber.
Now the marcxml is moved to biblio_metadata and the foreign key reference was simplified to biblionumber.
This makes the existing FRBR-bug worse.
Comment 1 Olli-Antti Kivilahti 2017-03-14 16:58:22 UTC
Exhibit 3:

Just noticed while doing DB changes, that biblio_metadata is linked to the biblio table instead of the biblioitems-table!

No FRBR there.

Is being fixed.
Comment 2 Olli-Antti Kivilahti 2017-03-14 17:00:34 UTC
Created attachment 61090 [details] [review]
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot.

perl misc/devel/update_dbix_class_files.pl

perl t/db_dependent/Biblio_crud.t

perl t/db_dependent/Holds/HoldFulfillmentPolicy.t
 + other tests from Buug 17196
Comment 3 Olli-Antti Kivilahti 2017-03-14 17:01:14 UTC
WIP. Will fix  tests tomorrow.
Comment 4 Olli-Antti Kivilahti 2017-03-15 10:31:01 UTC
Created attachment 61096 [details] [review]
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table.

Exhibit 1:

The MARCXML is lost when the biblio is deleted because the values in biblio_metadata-table from biblioitems-table are never moved to deletedbiblio_metadata-table.
Previously the marcxml was preserved in the oldbiblioitems-table
This is a challenge for gathering statistics
For example item has been ordered, received, and removed from collection during the statistics period
Information in marcxml is needed to accurately categorize the item

Exhibit 2:

[15:22:38] <kivilahtio> also. In Koha there once was a great idea that one biblio-entry can have multiple biblioitems
[15:22:45] <kivilahtio> and the marcxml was tied to the biblioitems-table
[15:23:11] <kivilahtio> this is a nice idea, but nobdy really cared for it and now biblionumber and biblioitemnumber are used interchangeably
[15:23:22] <kivilahtio> except when they get out of sync during a data migration
[15:23:41] <kivilahtio> then things start to work really interestingly
[15:23:47] <kivilahtio> but the point is
[15:23:57] <kivilahtio> biblio_metadata has a biblionumber-column
[15:24:15] <kivilahtio> shouldn't it have a biblioitemnumber-column, just for old time's sake?

In the biblioitems-table we had marcxml. primary keyed with biblioitemnumber.
Now the marcxml is moved to biblio_metadata and the foreign key reference was simplified to biblionumber.
This makes the existing FRBR-bug worse.

Exhibit 3:

Just noticed while doing DB changes, that biblio_metadata is linked to the biblio table instead of the biblioitems-table!

No FRBR there.

Is being fixed.

REPLICATE ISSUE:

Create biblio
Delete biblio
Where is deletedbiblio_metadata-row?

TEST PLAN:

perl misc/devel/update_dbix_class_files.pl

perl t/db_dependent/Biblio_crud.t

perl t/db_dependent/Holds/HoldFulfillmentPolicy.t
 + other tests from Buug 17196
Comment 5 Olli-Antti Kivilahti 2017-03-15 10:39:22 UTC
Created attachment 61097 [details] [review]
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table.

Exhibit 1:

The MARCXML is lost when the biblio is deleted because the values in biblio_metadata-table from biblioitems-table are never moved to deletedbiblio_metadata-table.
Previously the marcxml was preserved in the oldbiblioitems-table
This is a challenge for gathering statistics
For example item has been ordered, received, and removed from collection during the statistics period
Information in marcxml is needed to accurately categorize the item

Exhibit 2:

[15:22:38] <kivilahtio> also. In Koha there once was a great idea that one biblio-entry can have multiple biblioitems
[15:22:45] <kivilahtio> and the marcxml was tied to the biblioitems-table
[15:23:11] <kivilahtio> this is a nice idea, but nobdy really cared for it and now biblionumber and biblioitemnumber are used interchangeably
[15:23:22] <kivilahtio> except when they get out of sync during a data migration
[15:23:41] <kivilahtio> then things start to work really interestingly
[15:23:47] <kivilahtio> but the point is
[15:23:57] <kivilahtio> biblio_metadata has a biblionumber-column
[15:24:15] <kivilahtio> shouldn't it have a biblioitemnumber-column, just for old time's sake?

In the biblioitems-table we had marcxml. primary keyed with biblioitemnumber.
Now the marcxml is moved to biblio_metadata and the foreign key reference was simplified to biblionumber.
This makes the existing FRBR-bug worse.

Exhibit 3:

Just noticed while doing DB changes, that biblio_metadata is linked to the biblio table instead of the biblioitems-table!

No FRBR there.

Is being fixed.

REPLICATE ISSUE:

Create biblio
Delete biblio
Where is deletedbiblio_metadata-row?

TEST PLAN:

perl misc/devel/update_dbix_class_files.pl

perl t/db_dependent/Biblio_crud.t

perl t/db_dependent/Holds/HoldFulfillmentPolicy.t
 + other tests from Buug 17196
Comment 6 Olli-Antti Kivilahti 2017-03-15 11:07:07 UTC
Created attachment 61098 [details] [review]
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table.

Exhibit 1:

The MARCXML is lost when the biblio is deleted because the values in biblio_metadata-table from biblioitems-table are never moved to deletedbiblio_metadata-table.
Previously the marcxml was preserved in the oldbiblioitems-table
This is a challenge for gathering statistics
For example item has been ordered, received, and removed from collection during the statistics period
Information in marcxml is needed to accurately categorize the item

Exhibit 2:

[15:22:38] <kivilahtio> also. In Koha there once was a great idea that one biblio-entry can have multiple biblioitems
[15:22:45] <kivilahtio> and the marcxml was tied to the biblioitems-table
[15:23:11] <kivilahtio> this is a nice idea, but nobdy really cared for it and now biblionumber and biblioitemnumber are used interchangeably
[15:23:22] <kivilahtio> except when they get out of sync during a data migration
[15:23:41] <kivilahtio> then things start to work really interestingly
[15:23:47] <kivilahtio> but the point is
[15:23:57] <kivilahtio> biblio_metadata has a biblionumber-column
[15:24:15] <kivilahtio> shouldn't it have a biblioitemnumber-column, just for old time's sake?

In the biblioitems-table we had marcxml. primary keyed with biblioitemnumber.
Now the marcxml is moved to biblio_metadata and the foreign key reference was simplified to biblionumber.
This makes the existing FRBR-bug worse.

Exhibit 3:

Just noticed while doing DB changes, that biblio_metadata is linked to the biblio table instead of the biblioitems-table!

No FRBR there.

Is being fixed.

REPLICATE ISSUE:

Create biblio
Delete biblio
Where is deletedbiblio_metadata-row?

TEST PLAN:

perl misc/devel/update_dbix_class_files.pl

perl t/db_dependent/Biblio_crud.t

perl t/db_dependent/Holds/HoldFulfillmentPolicy.t
 + other tests from Buug 17196
Comment 7 Jonathan Druart 2017-03-15 11:46:00 UTC
(In reply to Olli-Antti Kivilahti from comment #1)
> Exhibit 3:
> 
> Just noticed while doing DB changes, that biblio_metadata is linked to the
> biblio table instead of the biblioitems-table!

I think we should keep the link to biblio, both table should be merged.

You are going too far with this patch, we need a quick patch to fix the data lost when deleting a bibliographic record, then we can discuss other points on another bug report. Would you agree with that?
Comment 8 Olli-Antti Kivilahti 2017-03-15 13:05:19 UTC
Created attachment 61101 [details] [review]
Bug 18265 - Followup to Buuug 17196. deletedbiblio_metadata is not used. biblioitemnumber was forgot. FK to wrong table.

Exhibit 1:

The MARCXML is lost when the biblio is deleted because the values in biblio_metadata-table from biblioitems-table are never moved to deletedbiblio_metadata-table.
Previously the marcxml was preserved in the oldbiblioitems-table
This is a challenge for gathering statistics
For example item has been ordered, received, and removed from collection during the statistics period
Information in marcxml is needed to accurately categorize the item

Exhibit 2:

[15:22:38] <kivilahtio> also. In Koha there once was a great idea that one biblio-entry can have multiple biblioitems
[15:22:45] <kivilahtio> and the marcxml was tied to the biblioitems-table
[15:23:11] <kivilahtio> this is a nice idea, but nobdy really cared for it and now biblionumber and biblioitemnumber are used interchangeably
[15:23:22] <kivilahtio> except when they get out of sync during a data migration
[15:23:41] <kivilahtio> then things start to work really interestingly
[15:23:47] <kivilahtio> but the point is
[15:23:57] <kivilahtio> biblio_metadata has a biblionumber-column
[15:24:15] <kivilahtio> shouldn't it have a biblioitemnumber-column, just for old time's sake?

In the biblioitems-table we had marcxml. primary keyed with biblioitemnumber.
Now the marcxml is moved to biblio_metadata and the foreign key reference was simplified to biblionumber.
This makes the existing FRBR-bug worse.

Exhibit 3:

Just noticed while doing DB changes, that biblio_metadata is linked to the biblio table instead of the biblioitems-table!

No FRBR there.

Is being fixed.

REPLICATE ISSUE:

Create biblio
Delete biblio
Where is deletedbiblio_metadata-row?

TEST PLAN:

perl misc/devel/update_dbix_class_files.pl

perl t/db_dependent/Biblio_crud.t

perl t/db_dependent/Holds/HoldFulfillmentPolicy.t
 + other tests from Buug 17196
Comment 9 Olli-Antti Kivilahti 2017-03-15 13:14:48 UTC
(In reply to Jonathan Druart from comment #7)
> (In reply to Olli-Antti Kivilahti from comment #1)
> > Exhibit 3:
> > 
> > Just noticed while doing DB changes, that biblio_metadata is linked to the
> > biblio table instead of the biblioitems-table!
> 
> I think we should keep the link to biblio, both table should be merged.
> 

Do you mean that biblio- and biblioitems-tables should be merged?

Then those need to be separated again when we are moving to FRBR.

This is one fundamentally good architectural design decision in Koha. Don't break it.
Comment 10 Marcel de Rooy 2017-03-15 13:19:03 UTC
(In reply to Olli-Antti Kivilahti from comment #9)
> Do you mean that biblio- and biblioitems-tables should be merged?
> 
> Then those need to be separated again when we are moving to FRBR.
> 
> This is one fundamentally good architectural design decision in Koha. Don't
> break it.

A design decision that has not been implemented, might not be a good one after all..
Comment 11 Jonathan Druart 2017-03-15 13:39:38 UTC
(In reply to Jonathan Druart from comment #7)
> You are going too far with this patch, we need a quick patch to fix the data
> lost when deleting a bibliographic record, then we can discuss other points
> on another bug report. Would you agree with that?

What about this important part?
Comment 12 Olli-Antti Kivilahti 2017-03-16 08:01:30 UTC
(In reply to Jonathan Druart from comment #11)
> (In reply to Jonathan Druart from comment #7)
> > You are going too far with this patch, we need a quick patch to fix the data
> > lost when deleting a bibliographic record,

I have to disagree. You need properly implemented features, not small steps and quick fixes.

> > then we can discuss other points
> > on another bug report. Would you agree with that?
> 
> What about this important part?

I have already spent a significant amount of time (8h+) on this bug.
This preserves existing data structure functionally as it was intended.
Also there is no mention on Bug 17196 about the intent of merging biblio and biblioitems. There is a mention about Koha::MetadataRecord but there is no reason that would require the supposed merge.

I have no interest atm for any discussion about this bug ticket.

You are free to continue doing whatever you please. However I recommend you salvage the test case attached.
Comment 13 Olli-Antti Kivilahti 2017-03-16 10:16:23 UTC
(In reply to Marcel de Rooy from comment #10)
> (In reply to Olli-Antti Kivilahti from comment #9)
> > Do you mean that biblio- and biblioitems-tables should be merged?
> > 
> > Then those need to be separated again when we are moving to FRBR.
> > 
> > This is one fundamentally good architectural design decision in Koha. Don't
> > break it.
> 
> A design decision that has not been implemented, might not be a good one
> after all..

The issues with (MA)chine (R)eadable (C)ard catalogues are aknowledged for decades. Probably even then when the format was being discussed.

Now the official formats for implementing FRBR (RDA -> Bibframe) are at the door and we (Finland) are already migrating to RDA this year. Actually I think we officially did and new records are only available in RDA as we speak.
Looking forward to Bibframe in the near future.

Structuring the DB to model the separation of work from manifestation is fundamentally important to achieve moving Koha to the future.

Removing this distinction will turn us back 30 years.

I have already suffered from the awkwardness caused by this divide (biblio-biblioitems) for many years with Koha, but the problem is not with DB but the internal API.
Better stick to it now, since it is the foothold for pushing us forward.
Naturally the current implementation is not good enough, but it is better than nothing. Otherwise we need to reinvent this wheel.
Comment 14 Jonathan Druart 2017-03-16 17:06:22 UTC
(In reply to Olli-Antti Kivilahti from comment #12)
> (In reply to Jonathan Druart from comment #11)
> > (In reply to Jonathan Druart from comment #7)
> > > You are going too far with this patch, we need a quick patch to fix the data
> > > lost when deleting a bibliographic record,
> 
> I have to disagree. You need properly implemented features, not small steps
> and quick fixes.

Yes and that is something you did not understand yet, but I am sure you will at some point.
That is exactly why you do not manage to get your patches pushed.
Small steps are the way to go if you want your changes to be integrated.
On this bug report you are addressing to many different points, you are just slowing down its move.

I will take care of the moveto deletedbiblio_metadata.

Thanks for the try!
Comment 15 Olli-Antti Kivilahti 2017-03-17 08:01:16 UTC
Created attachment 61199 [details] [review]
Bug 18265 - Followup to Buuug 17196. Fix t/db_dependent/Exporter/Record.t
Comment 16 Jonathan Druart 2017-03-23 20:03:23 UTC
Bug 18284 has been pushed, lowering the severity of this one.

Discussion remains on the move of biblio_metadata.biblionumber to biblioitemnumber.
Comment 17 Marcel de Rooy 2017-03-24 07:44:51 UTC
(In reply to Olli-Antti Kivilahti from comment #13)
> The issues with (MA)chine (R)eadable (C)ard catalogues are aknowledged for
> decades. Probably even then when the format was being discussed.
> 
> Now the official formats for implementing FRBR (RDA -> Bibframe) are at the
> door and we (Finland) are already migrating to RDA this year. Actually I
> think we officially did and new records are only available in RDA as we
> speak.
> Looking forward to Bibframe in the near future.
> 
> Structuring the DB to model the separation of work from manifestation is
> fundamentally important to achieve moving Koha to the future.
> 
> Removing this distinction will turn us back 30 years.
> 
> I have already suffered from the awkwardness caused by this divide
> (biblio-biblioitems) for many years with Koha, but the problem is not with
> DB but the internal API.
> Better stick to it now, since it is the foothold for pushing us forward.
> Naturally the current implementation is not good enough, but it is better
> than nothing. Otherwise we need to reinvent this wheel.

The current implementation of biblio-biblioitems is very poor.
I agree with your argument that we should look to the future, allowing for work, manifestation, etc.
Koha has not been designed with that in mind. Just books-items. So yes, it will need (a lot of) attention.
But biblio-biblioitems is no good foundation.
Comment 18 Katrin Fischer 2023-09-10 16:03:02 UTC
I believe we have been more and more standardizing on biblionumber and this also applies to biblio_metadata. I feel the current schema is correct.

Any vetos to closing this?
Comment 19 David Cook 2023-09-11 04:47:36 UTC
(In reply to Katrin Fischer from comment #18)
> I believe we have been more and more standardizing on biblionumber and this
> also applies to biblio_metadata. I feel the current schema is correct.
> 
> Any vetos to closing this?

I get where Olli-Antti was coming from and overall is probably right in theory but I don't think it was a practical approach.

Looking at the BIBFRAME 2.0 model (https://www.loc.gov/bibframe/docs/bibframe2-model.html) it looks like it could be worthwhile keeping the "biblio", "biblioitems", "items" tables separate as those could evolve into "works", "instances", "items"...

I suppose a MARC expression of the BIBFRAME 2.0 format would merge together all 3 of those with the instance being the central piece of the puzzle. 

But that's not really how Koha works at this point. I think we would need some collective talks and agreements about future directions if we wanted to really adopt the WMI model used by BIBFRAME 2.0. 

--

In other words, I'm not sure if we're ready to close this one yet... but I don't know that I see it moving forward anytime soon either...