Bug 31224

Summary: Koha::Biblio::Metadata->record should use the EmbedItems filter
Product: Koha Reporter: Tomás Cohen Arazi (tcohen) <tomascohen>
Component: Architecture, internals, and plumbingAssignee: Tomás Cohen Arazi (tcohen) <tomascohen>
Status: Pushed to main --- QA Contact: Martin Renvoize (ashimema) <martin.renvoize>
Severity: normal    
Priority: P5 - low CC: andrew, dcook, jonathan.druart, joonas.kylmala, m.de.rooy, martin.renvoize, nick, tomascohen
Version: unspecified   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23247
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31889
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35104
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38286
Change sponsored?: --- Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.11.00
Circulation function:
Bug Depends on: 3150, 23247, 29697, 33083    
Bug Blocks: 38406    
Attachments: [POC] Bug 31224: Add Koha::Biblio->metadata_record
[POC] Bug 31224: Add Koha::Biblio->metadata_record
[POC] Bug 31224: Use metadata_record in opac-MARCdetail.pl
[POC] Bug 31224: Add Koha::Biblio->metadata_record
[POC] Bug 31224: Use metadata_record in opac-MARCdetail.pl
Bug 31224: Add Koha::Biblio->metadata_record
Bug 31224: Use metadata_record in opac-MARCdetail.pl
Bug 31224: (QA follow-up) Fix syntax error
Bug 31224: Convert C4::Record::marcrecord2csv
Bug 31224: Support 'interface' on metadata_record
Bug 31224: Update instances of metadata->record
Bug 31224: Update instances of metadata->record
Bug 31224: Add Koha::Biblio->metadata_record
Bug 31224: Use metadata_record in opac-MARCdetail.pl
Bug 31224: (QA follow-up) Fix syntax error
Bug 31224: Convert C4::Record::marcrecord2csv
Bug 31224: Support 'interface' on metadata_record
Bug 31224: Update instances of metadata->record
Bug 31224: Add Koha::Biblio->metadata_record
Bug 31224: Use metadata_record in opac-MARCdetail.pl
Bug 31224: (QA follow-up) Fix syntax error
Bug 31224: Convert C4::Record::marcrecord2csv
Bug 31224: Support 'interface' on metadata_record
Bug 31224: Update instances of metadata->record
Bug 31224: Add Koha::Biblio->metadata_record
Bug 31224: Use metadata_record in opac-MARCdetail.pl
Bug 31224: Convert C4::Record::marcrecord2csv
Bug 31224: Support 'interface' on metadata_record
Bug 31224: Update instances of metadata->record
Bug 31224: EmbedItems takes an arrayref
Bug 31224: EmbedItems takes an arrayref
Bug 31224: Expand coded values using metadata_record in Server.pm
Bug 31224: Adapt GET /public/biblios/:biblio_id controller
Bug 31224: Add Koha::Biblio->metadata_record
Bug 31224: Use metadata_record in opac-MARCdetail.pl
Bug 31224: Convert C4::Record::marcrecord2csv
Bug 31224: Support 'interface' on metadata_record
Bug 31224: Update instances of metadata->record
Bug 31224: EmbedItems takes an arrayref
Bug 31224: Expand coded values using metadata_record in Server.pm
Bug 31224: Adapt GET /public/biblios/:biblio_id controller
Bug 31224: (follow-up) Update instances of metadata->record
Bug 31224: Move item field removal up to $metadata->record
Bug 31224: Revert to $metadata->record in `marcrecord2csv`

Description Tomás Cohen Arazi (tcohen) 2022-07-22 19:05:13 UTC
Bug 29697 removed GetMarcBiblio in favor of using Koha::Biblio::Metadata->record. YAY!

But it added a redundant Koha::Biblio::Metadata::_embed_items method. It also adds a suspicious 'record' param. I say suspicious because it looks like a trick to have a given MARC::Record object added the items information. Which feels like wrong API design. Specially when we have the Koha::Filter::MARC::EmbedItems processor filter for the task.

Now that things are 'modernized' we really need to revisit this code for a cleanup.
Comment 1 Tomás Cohen Arazi (tcohen) 2022-07-22 20:14:35 UTC
Ok, I submitted to bug 23247 and implementation that doesn't rely on the new embed_items param, so we ponder how coding with RecordProcessor looks like in the controllers.

I will now submit a version that uses RecordProcessor in the method.
Comment 2 Tomás Cohen Arazi (tcohen) 2022-07-22 20:50:58 UTC Comment hidden (obsolete)
Comment 3 Tomás Cohen Arazi (tcohen) 2022-07-22 20:52:32 UTC Comment hidden (obsolete)
Comment 4 Tomás Cohen Arazi (tcohen) 2022-07-22 20:53:22 UTC Comment hidden (obsolete)
Comment 5 Tomás Cohen Arazi (tcohen) 2022-07-22 20:59:55 UTC Comment hidden (obsolete)
Comment 6 Tomás Cohen Arazi (tcohen) 2022-07-22 21:06:36 UTC
Created attachment 138063 [details] [review]
[POC] Bug 31224: Add Koha::Biblio->metadata_record

This patch introduces a higher-level method for retrieving a 'prepared'
record for using in specific contexts. In particular, I only focused on
embedding item information and OPAC view filtering. But we could add a
way to get the record through the ViewPolicy also for staff.

The virtue of this patch is that it explains better than my words why I
think having the *embed_items* and *opac* parameters in
Koha::Biblio::Metadata->record is not ideal. And makes the other
implementation feel like is done at the wrong level.

I know it's been done like that to act as a drop-in replacement fro GetMarcBiblio,
which was a good first step. But we need to revisit how it is used, in general, to come
up with a more refined and useful design. So we do not port the same
design problems from C4::* into Koha::*.

Try to think how it would be used in contexts like opac-MARCdetail.pl or
any other OPAC controller script.

Thanks!
Comment 7 Tomás Cohen Arazi (tcohen) 2022-07-22 21:06:40 UTC
Created attachment 138064 [details] [review]
[POC] Bug 31224: Use metadata_record in opac-MARCdetail.pl
Comment 8 Tomás Cohen Arazi (tcohen) 2022-07-22 21:08:20 UTC
Ok, I added a sample usage.
Comment 9 Jonathan Druart 2022-07-23 06:05:24 UTC
Then, what's next? Please make sure to work on top of bug 27272 please.
Comment 10 Tomás Cohen Arazi (tcohen) 2022-07-25 11:15:29 UTC
(In reply to Jonathan Druart from comment #9)
> Then, what's next? Please make sure to work on top of bug 27272 please.

If you all agree with this proposed implementation, once we fix breakages caused by bug 29697, and bug 27272 pushed, I will implement this.
Comment 11 Martin Renvoize (ashimema) 2022-10-13 07:04:59 UTC
I lost track of this one, sorry.. I agree with the approach here, would you like me to have a look at the other cases Tomas?
Comment 12 Tomás Cohen Arazi (tcohen) 2022-10-18 12:08:51 UTC
(In reply to Martin Renvoize from comment #11)
> I lost track of this one, sorry.. I agree with the approach here, would you
> like me to have a look at the other cases Tomas?

Please do. I can too, after the release.
Comment 13 Martin Renvoize (ashimema) 2022-10-19 09:20:27 UTC
Created attachment 142137 [details] [review]
Bug 31224: Add Koha::Biblio->metadata_record

This patch introduces a higher-level method for retrieving a 'prepared'
record for using in specific contexts. In particular, I only focused on
embedding item information and OPAC view filtering. But we could add a
way to get the record through the ViewPolicy also for staff.

The virtue of this patch is that it explains better than my words why I
think having the *embed_items* and *opac* parameters in
Koha::Biblio::Metadata->record is not ideal. And makes the other
implementation feel like is done at the wrong level.

I know it's been done like that to act as a drop-in replacement fro GetMarcBiblio,
which was a good first step. But we need to revisit how it is used, in general, to come
up with a more refined and useful design. So we do not port the same
design problems from C4::* into Koha::*.

Try to think how it would be used in contexts like opac-MARCdetail.pl or
any other OPAC controller script.

Thanks!
Comment 14 Martin Renvoize (ashimema) 2022-10-19 09:20:32 UTC
Created attachment 142138 [details] [review]
Bug 31224: Use metadata_record in opac-MARCdetail.pl
Comment 15 Martin Renvoize (ashimema) 2022-10-19 09:20:37 UTC
Created attachment 142139 [details] [review]
Bug 31224: (QA follow-up) Fix syntax error
Comment 16 Martin Renvoize (ashimema) 2022-10-19 09:20:41 UTC
Created attachment 142140 [details] [review]
Bug 31224: Convert C4::Record::marcrecord2csv

Update marcrecord2csv to user RecordProcessor directly for embedding
listed items.
Comment 17 Martin Renvoize (ashimema) 2022-10-19 09:20:46 UTC
Created attachment 142141 [details] [review]
Bug 31224: Support 'interface' on metadata_record

We replace the 'opac' boolean with an 'interface' param here to allow
for ViewPolicy filtering depending on 'opac', 'intranet' or 'nothing'
being passed.
Comment 18 Martin Renvoize (ashimema) 2022-10-19 09:20:50 UTC
Created attachment 142142 [details] [review]
Bug 31224: Update instances of metadata->record

We can now call metadata_record directly on the Koha::Biblio object.

This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate.  The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Comment 19 Martin Renvoize (ashimema) 2022-10-19 09:25:12 UTC
OK.. So I've worked through a grep of `metadata->record(` to catch all cases where we were applying filters to the record (well.. all cases that did it directly on fetch and not as a second step) for controllers and modules only at this point.. I will follow-up and do tests next.

After working through I decided to update the new method to accept an 'interface' parameter instead of using 'opac' as a boolean.. this allows for 'opac', 'intranet' and 'undefined' as options giving us a little more control for the ViewPolicy filter use case.

I opted not to port the optional passing of a list of itemnumbers to embed as there's only one use case of it.. so instead I added the RecordProcesser to the controller in that case.

Let me know what you think.. I'm also still somewhat open on the method name.. 'metadata_record' vs 'filtered_record' or 'record_filtered' or something along those lines?
Comment 20 Martin Renvoize (ashimema) 2022-10-19 09:26:09 UTC
Also.. should I work through and try to replace ALL calls to ->metadata->record with ->metadata_record.. and check for superfluous RecordProcessor instantiations now we can do it inline?
Comment 21 Martin Renvoize (ashimema) 2022-10-19 09:38:55 UTC
Created attachment 142143 [details] [review]
Bug 31224: Update instances of metadata->record

We can now call metadata_record directly on the Koha::Biblio object.

This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate.  The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Comment 22 Marcel de Rooy 2022-10-20 12:52:11 UTC
Please fix:

root@master:/usr/share/koha# git grep  EmbedItemsInMarcBiblio
C4/Biblio.pm:        EmbedItemsInMarcBiblio

Exported not existing
Comment 23 Katrin Fischer 2023-06-17 12:35:00 UTC
(In reply to Marcel de Rooy from comment #22)
> Please fix:
> 
> root@master:/usr/share/koha# git grep  EmbedItemsInMarcBiblio
> C4/Biblio.pm:        EmbedItemsInMarcBiblio
> 
> Exported not existing
Comment 24 Tomás Cohen Arazi (tcohen) 2023-08-02 20:16:02 UTC
(In reply to Marcel de Rooy from comment #22)
> Please fix:
> 
> root@master:/usr/share/koha# git grep  EmbedItemsInMarcBiblio
> C4/Biblio.pm:        EmbedItemsInMarcBiblio
> 
> Exported not existing

I don't understand this comment or why this is FQA (because of it?).
This anyway doesn't apply anymore, and the last patch needs to be redone entirely.

Will poke around it.
Comment 25 Martin Renvoize (ashimema) 2024-02-02 14:50:12 UTC
Such a shame things like this just die instead of getting support.
Comment 26 Tomás Cohen Arazi (tcohen) 2024-02-02 14:55:34 UTC
(In reply to Martin Renvoize from comment #20)
> Also.. should I work through and try to replace ALL calls to
> ->metadata->record with ->metadata_record.. and check for superfluous
> RecordProcessor instantiations now we can do it inline?

I'm sorry. Yes to all your suggestions :-D
Comment 27 Martin Renvoize (ashimema) 2024-02-08 16:11:50 UTC
Master moved a chunk.. this needs rebasing on at least bug 33083
Comment 28 Martin Renvoize (ashimema) 2024-02-08 16:53:29 UTC
Created attachment 161933 [details] [review]
Bug 31224: Add Koha::Biblio->metadata_record

This patch introduces a higher-level method for retrieving a 'prepared'
record for using in specific contexts. In particular, I only focused on
embedding item information and OPAC view filtering. But we could add a
way to get the record through the ViewPolicy also for staff.

The virtue of this patch is that it explains better than my words why I
think having the *embed_items* and *opac* parameters in
Koha::Biblio::Metadata->record is not ideal. And makes the other
implementation feel like is done at the wrong level.

I know it's been done like that to act as a drop-in replacement fro GetMarcBiblio,
which was a good first step. But we need to revisit how it is used, in general, to come
up with a more refined and useful design. So we do not port the same
design problems from C4::* into Koha::*.

Try to think how it would be used in contexts like opac-MARCdetail.pl or
any other OPAC controller script.

Thanks!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 29 Martin Renvoize (ashimema) 2024-02-08 16:53:32 UTC
Created attachment 161934 [details] [review]
Bug 31224: Use metadata_record in opac-MARCdetail.pl

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 30 Martin Renvoize (ashimema) 2024-02-08 16:53:35 UTC
Created attachment 161935 [details] [review]
Bug 31224: (QA follow-up) Fix syntax error

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 31 Martin Renvoize (ashimema) 2024-02-08 16:53:37 UTC
Created attachment 161936 [details] [review]
Bug 31224: Convert C4::Record::marcrecord2csv

Update marcrecord2csv to user RecordProcessor directly for embedding
listed items.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 32 Martin Renvoize (ashimema) 2024-02-08 16:53:40 UTC
Created attachment 161937 [details] [review]
Bug 31224: Support 'interface' on metadata_record

We replace the 'opac' boolean with an 'interface' param here to allow
for ViewPolicy filtering depending on 'opac', 'intranet' or 'nothing'
being passed.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 33 Martin Renvoize (ashimema) 2024-02-08 16:53:43 UTC
Created attachment 161938 [details] [review]
Bug 31224: Update instances of metadata->record

We can now call metadata_record directly on the Koha::Biblio object.

This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate.  The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 34 Martin Renvoize (ashimema) 2024-02-08 16:55:51 UTC
Rebased
Comment 35 Martin Renvoize (ashimema) 2024-02-08 17:17:02 UTC
Created attachment 161939 [details] [review]
Bug 31224: Add Koha::Biblio->metadata_record

This patch introduces a higher-level method for retrieving a 'prepared'
record for using in specific contexts. In particular, I only focused on
embedding item information and OPAC view filtering. But we could add a
way to get the record through the ViewPolicy also for staff.

The virtue of this patch is that it explains better than my words why I
think having the *embed_items* and *opac* parameters in
Koha::Biblio::Metadata->record is not ideal. And makes the other
implementation feel like is done at the wrong level.

I know it's been done like that to act as a drop-in replacement fro GetMarcBiblio,
which was a good first step. But we need to revisit how it is used, in general, to come
up with a more refined and useful design. So we do not port the same
design problems from C4::* into Koha::*.

Try to think how it would be used in contexts like opac-MARCdetail.pl or
any other OPAC controller script.

Thanks!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 36 Martin Renvoize (ashimema) 2024-02-08 17:17:05 UTC
Created attachment 161940 [details] [review]
Bug 31224: Use metadata_record in opac-MARCdetail.pl

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 37 Martin Renvoize (ashimema) 2024-02-08 17:17:07 UTC
Created attachment 161941 [details] [review]
Bug 31224: (QA follow-up) Fix syntax error

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 38 Martin Renvoize (ashimema) 2024-02-08 17:17:10 UTC
Created attachment 161942 [details] [review]
Bug 31224: Convert C4::Record::marcrecord2csv

Update marcrecord2csv to user RecordProcessor directly for embedding
listed items.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 39 Martin Renvoize (ashimema) 2024-02-08 17:17:13 UTC
Created attachment 161943 [details] [review]
Bug 31224: Support 'interface' on metadata_record

We replace the 'opac' boolean with an 'interface' param here to allow
for ViewPolicy filtering depending on 'opac', 'intranet' or 'nothing'
being passed.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 40 Martin Renvoize (ashimema) 2024-02-08 17:17:16 UTC
Created attachment 161944 [details] [review]
Bug 31224: Update instances of metadata->record

We can now call metadata_record directly on the Koha::Biblio object.

This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate.  The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 41 Martin Renvoize (ashimema) 2024-02-08 17:21:40 UTC
Right.. we're all rebased and QA scripts are passing.. 

However, not all tests are.. we need to account for the bad xml chars stuff.. whilst we wrap things in evals as part of the rebase.. the errors thrown aren't propagating properly yet so we're failing tests.

I also think we should merge Koha::Biblio->record and Koha::Biblio->metadata_record rather than having both.

Passing the ball to the next dev however.. it's time for me to clock off for the evening.
Comment 42 Tomás Cohen Arazi (tcohen) 2024-02-13 14:02:37 UTC
(In reply to Martin Renvoize from comment #41)
> Right.. we're all rebased and QA scripts are passing.. 
> 
> However, not all tests are.. we need to account for the bad xml chars
> stuff.. whilst we wrap things in evals as part of the rebase.. the errors
> thrown aren't propagating properly yet so we're failing tests.
> 
> I also think we should merge Koha::Biblio->record and
> Koha::Biblio->metadata_record rather than having both.
> 
> Passing the ball to the next dev however.. it's time for me to clock off for
> the evening.

Thanks Martin for resurrecting this. It is a relief to see there's still hope on stuff. I'll finish this once I'm back from this short time off.
Comment 43 Tomás Cohen Arazi (tcohen) 2024-10-28 13:08:09 UTC
Created attachment 173530 [details] [review]
Bug 31224: Add Koha::Biblio->metadata_record

This patch introduces a higher-level method for retrieving a 'prepared'
record for using in specific contexts. In particular, I only focused on
embedding item information and OPAC view filtering. But we could add a
way to get the record through the ViewPolicy also for staff.

The virtue of this patch is that it explains better than my words why I
think having the *embed_items* and *opac* parameters in
Koha::Biblio::Metadata->record is not ideal. And makes the other
implementation feel like is done at the wrong level.

I know it's been done like that to act as a drop-in replacement fro GetMarcBiblio,
which was a good first step. But we need to revisit how it is used, in general, to come
up with a more refined and useful design. So we do not port the same
design problems from C4::* into Koha::*.

Try to think how it would be used in contexts like opac-MARCdetail.pl or
any other OPAC controller script.

Thanks!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 44 Tomás Cohen Arazi (tcohen) 2024-10-28 13:08:12 UTC
Created attachment 173531 [details] [review]
Bug 31224: Use metadata_record in opac-MARCdetail.pl

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 45 Tomás Cohen Arazi (tcohen) 2024-10-28 13:08:15 UTC
Created attachment 173532 [details] [review]
Bug 31224: Convert C4::Record::marcrecord2csv

Update marcrecord2csv to user RecordProcessor directly for embedding
listed items.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 46 Tomás Cohen Arazi (tcohen) 2024-10-28 13:08:18 UTC
Created attachment 173533 [details] [review]
Bug 31224: Support 'interface' on metadata_record

We replace the 'opac' boolean with an 'interface' param here to allow
for ViewPolicy filtering depending on 'opac', 'intranet' or 'nothing'
being passed.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 47 Tomás Cohen Arazi (tcohen) 2024-10-28 13:08:21 UTC
Created attachment 173534 [details] [review]
Bug 31224: Update instances of metadata->record

We can now call metadata_record directly on the Koha::Biblio object.

This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate.  The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 48 Tomás Cohen Arazi (tcohen) 2024-10-28 13:08:24 UTC
Created attachment 173535 [details] [review]
Bug 31224: EmbedItems takes an arrayref

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 49 Tomás Cohen Arazi (tcohen) 2024-10-28 13:13:48 UTC
Ok. Rebased, and found a non-minor glitch with `EmbedItems`.
I think this is mostly solid, I didn't find broken tests (yet). Tested with the advanced editor (which uses /svc/[new_]bib)

I think the patch for `marcrecord2csv` is not correct as we don't have an  'itemnumbers' param being used. I haven't found any use of that method in the code, though.

On a follow-up dev, I would make the `EmbedItems` filter just get a `Koha::Items` resultset.

I think this patchset is fixing a couple leaks too. If this is not gonna move, I might fix them on a separate bug.
Comment 50 Tomás Cohen Arazi (tcohen) 2024-10-28 13:15:56 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #49)
> Tested with the advanced editor (which uses /svc/[new_]bib)

I meant that adding/changing a biblio worked correctly. Both ES and Zebra indexing worked as expected. And all views work as well.
Comment 51 Jonathan Druart 2024-10-28 13:32:06 UTC
Shouldn't we filter by default?

 166         if ( $params->{embed_items} ) {
 167             push @filters, 'EmbedItems';
 168             if ( $params->{interface} && $params->{interface} eq 'opac' ) {
 169                 $options->{items} = $self->items->filter_by_visible_in_opac(

If we don't pass 'opac' we do not filter. If we forget it then it's not filtered either.
Comment 52 Jonathan Druart 2024-10-28 13:52:23 UTC
Shouldn't Koha::OAI::Server::Repository->get_biblio_marcxml call this with expand_coded_fields?

Same for Koha::REST::V1::Biblios->get_public, use metadata_record and remove 'ViewPolicy'?
Comment 53 Tomás Cohen Arazi (tcohen) 2024-10-28 14:44:12 UTC
(In reply to Jonathan Druart from comment #51)
> Shouldn't we filter by default?
> 
>  166         if ( $params->{embed_items} ) {
>  167             push @filters, 'EmbedItems';
>  168             if ( $params->{interface} && $params->{interface} eq 'opac'
> ) {
>  169                 $options->{items} =
> $self->items->filter_by_visible_in_opac(
> 
> If we don't pass 'opac' we do not filter. If we forget it then it's not
> filtered either.

The original implementation falled back to 'opac'. If we want to make sure, we could make the `interface` parameter mandatory and throw an exception if missing. I'd agree with a follow-up implementing it.

(In reply to Jonathan Druart from comment #52)
> Shouldn't Koha::OAI::Server::Repository->get_biblio_marcxml call this with
> expand_coded_fields?

Yes.

> Same for Koha::REST::V1::Biblios->get_public, use metadata_record and remove
> 'ViewPolicy'?

Yes.

---

Overall, I implemented `metadata_record` and provided a sample usage patch. Martin's attempt to use it everywhere might have missed some places. But some might better be handled on their own bugs. As we tried to keep the scope small (i.e. replace `$biblio->metadata->record(*)` with `$biblio->metadata_record(*)`.

I agree with all your remarks, Jonathan. And can work on that if you don't beat me to it.
Comment 54 Tomás Cohen Arazi (tcohen) 2024-10-28 14:51:54 UTC
(In reply to Jonathan Druart from comment #51)
> Shouldn't we filter by default?
> 
>  166         if ( $params->{embed_items} ) {
>  167             push @filters, 'EmbedItems';
>  168             if ( $params->{interface} && $params->{interface} eq 'opac'
> ) {
>  169                 $options->{items} =
> $self->items->filter_by_visible_in_opac(
> 
> If we don't pass 'opac' we do not filter. If we forget it then it's not
> filtered either.

I just remembered: in some context we want to apply the ViewPolicy filter (i.e. OPAC and some staff interface views) but in some others we don't want filtering out things i.e. exporting records, etc. That's why it was built like that.

Maybe we should force the `interface` param and also add a `skip_view_policy` parameter for select use cases (e.g. record indexing).
Comment 55 Tomás Cohen Arazi (tcohen) 2024-10-28 18:46:08 UTC
Created attachment 173588 [details] [review]
Bug 31224: EmbedItems takes an arrayref

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 56 Tomás Cohen Arazi (tcohen) 2024-10-29 00:02:00 UTC
Created attachment 173597 [details] [review]
Bug 31224: Expand coded values using metadata_record in Server.pm

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 57 Tomás Cohen Arazi (tcohen) 2024-10-29 00:02:03 UTC
Created attachment 173598 [details] [review]
Bug 31224: Adapt GET /public/biblios/:biblio_id controller

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 58 Jonathan Druart 2024-10-29 13:06:34 UTC
Created attachment 173636 [details] [review]
Bug 31224: Add Koha::Biblio->metadata_record

This patch introduces a higher-level method for retrieving a 'prepared'
record for using in specific contexts. In particular, I only focused on
embedding item information and OPAC view filtering. But we could add a
way to get the record through the ViewPolicy also for staff.

The virtue of this patch is that it explains better than my words why I
think having the *embed_items* and *opac* parameters in
Koha::Biblio::Metadata->record is not ideal. And makes the other
implementation feel like is done at the wrong level.

I know it's been done like that to act as a drop-in replacement fro GetMarcBiblio,
which was a good first step. But we need to revisit how it is used, in general, to come
up with a more refined and useful design. So we do not port the same
design problems from C4::* into Koha::*.

Try to think how it would be used in contexts like opac-MARCdetail.pl or
any other OPAC controller script.

Thanks!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 59 Jonathan Druart 2024-10-29 13:06:37 UTC
Created attachment 173637 [details] [review]
Bug 31224: Use metadata_record in opac-MARCdetail.pl

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 60 Jonathan Druart 2024-10-29 13:06:41 UTC
Created attachment 173638 [details] [review]
Bug 31224: Convert C4::Record::marcrecord2csv

Update marcrecord2csv to user RecordProcessor directly for embedding
listed items.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 61 Jonathan Druart 2024-10-29 13:06:44 UTC
Created attachment 173639 [details] [review]
Bug 31224: Support 'interface' on metadata_record

We replace the 'opac' boolean with an 'interface' param here to allow
for ViewPolicy filtering depending on 'opac', 'intranet' or 'nothing'
being passed.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 62 Jonathan Druart 2024-10-29 13:06:47 UTC
Created attachment 173640 [details] [review]
Bug 31224: Update instances of metadata->record

We can now call metadata_record directly on the Koha::Biblio object.

This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate.  The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 63 Jonathan Druart 2024-10-29 13:06:50 UTC
Created attachment 173641 [details] [review]
Bug 31224: EmbedItems takes an arrayref

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 64 Jonathan Druart 2024-10-29 13:06:53 UTC
Created attachment 173642 [details] [review]
Bug 31224: Expand coded values using metadata_record in Server.pm

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 65 Jonathan Druart 2024-10-29 13:06:57 UTC
Created attachment 173643 [details] [review]
Bug 31224: Adapt GET /public/biblios/:biblio_id controller

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 66 Jonathan Druart 2024-10-29 13:07:00 UTC
Created attachment 173644 [details] [review]
Bug 31224: (follow-up) Update instances of metadata->record

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 67 Martin Renvoize (ashimema) 2024-10-31 11:35:35 UTC
I'm tempted to call this PQA to keep things moving here.. we've got Tomas, Jonathan and I all onboard with it and though I agree to the comments left, I also agree they make sense as follow-ups on new bugs.
Comment 68 Katrin Fischer 2024-11-07 15:32:42 UTC
Pushed for 24.11!

Well done everyone, thank you!
Comment 69 Katrin Fischer 2024-11-07 16:45:08 UTC
This broke tests, please fix URGENTLY:

git bisect:
[8c5135dadbe8cd6d1df61e46026be8996f97f27c] Bug 31224: Add Koha::Biblio->metadata_record

* t_db_dependent_Record_marcrecord2csv_t
* t_db_dependent_Exporter_Record_t.
Comment 70 Tomás Cohen Arazi (tcohen) 2024-11-07 18:16:15 UTC
(In reply to Katrin Fischer from comment #69)
> This broke tests, please fix URGENTLY:
> 
> git bisect:
> [8c5135dadbe8cd6d1df61e46026be8996f97f27c] Bug 31224: Add
> Koha::Biblio->metadata_record
> 
> * t_db_dependent_Record_marcrecord2csv_t
> * t_db_dependent_Exporter_Record_t.

Ok! I will fix them!
Comment 71 Tomás Cohen Arazi (tcohen) 2024-11-08 00:23:40 UTC
Created attachment 174262 [details] [review]
Bug 31224: Move item field removal up to $metadata->record

This patch addresse the fact Koha needs to strip out stray item field in
MARC records coming from the DB. While this is not ideal, it has worked
so far, limiting the negative effects of bad data to Koha users.

I put a FIXME because I think it deserves to be revisited colectively at
some point. I filed bug 38406 to track the discussion around this.

The `marcmarcrecord2csv.t` tests cover this behavior and I thought
removing it was not in the scope of this bug.

I decided to move the removal one step up in the call chain so it
applies to all calls to `$metadata->record`, not only those that ask to
include items. This is the right thing to do while we keep this
behavior.

To test:
1. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Record/marcrecord2csv.t
=> FAIL: Item information gets extracted from more items than expected
(i.e. the item in the MARC record is not stripped out so 'Withdrawn'
shows 3 times instead of 2).
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests pass!
4. Run:
  k$ prove t/db_dependent/Koha/Biblio/Metadata.t
=> SUCCESS: Tests pass! The behavior change in $metadata->record works
as intended!
5. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 72 Tomás Cohen Arazi (tcohen) 2024-11-08 00:36:54 UTC
Created attachment 174263 [details] [review]
Bug 31224: Revert to $metadata->record in `marcrecord2csv`

This patch reverts the change for marcrecord2csv as
`$biblio->metadata_record` doesn't support a list of itemnumbers, and it
feels we should discuss in another context whether to add it as a
parameter, or make it a Koha::Items iterator. The Koha::Item object is
actually retrieved in marc2csv so there's room for easy improvements
without adding technical debt (i.e. bad method signatures for a single
use case).

This fixes a feature:

1. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Exporter/Record.t
=> FAIL: Test fails because the 'items' are not being filtered on the
passed itemnumbers (i.e. all items are picked for generating the CSV).
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests pass!

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 73 Tomás Cohen Arazi (tcohen) 2024-11-08 00:38:57 UTC
Ok, my patches highlight the problem. The first one fixes some, adding a FIXME for a situation that is explained in the commit message. And then, the second one acknowledges the fact we need to discuss/think a little bit more on how/who will refactor the marc2csv code. Which I think should happen on its own bug.

Gotta go to bed now. Hope others can revisit what I've just submitted and hopefully come up with better solutions :-D
Comment 74 Katrin Fischer 2024-11-08 08:06:10 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #73)
> Ok, my patches highlight the problem. The first one fixes some, adding a
> FIXME for a situation that is explained in the commit message. And then, the
> second one acknowledges the fact we need to discuss/think a little bit more
> on how/who will refactor the marc2csv code. Which I think should happen on
> its own bug.
> 
> Gotta go to bed now. Hope others can revisit what I've just submitted and
> hopefully come up with better solutions :-D

Thanks Tomas! I assumed it was about repeated fields, but didn't think it was for items, very interesting. It sounds like this is the way to go for now, but needs more work/thought in future.
Comment 75 Katrin Fischer 2024-11-08 10:00:51 UTC
Picked 2 follow-up patches ("Move item field..." and "Revert to $metadata->record...") for push to main.