Bug 39296 - Provide a template plugin to return MARC::Record for MARCXML metadata
Summary: Provide a template plugin to return MARC::Record for MARCXML metadata
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Reports (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: David Cook
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-03-11 03:39 UTC by David Cook
Modified: 2025-10-24 03:04 UTC (History)
5 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 39296: Provide a template plugin for getting a MARC::Record from MARCXML (3.11 KB, patch)
2025-03-11 04:26 UTC, David Cook
Details | Diff | Splinter Review
Bug 39296: Provide a template plugin for getting a MARC::Record from MARCXML (3.17 KB, patch)
2025-03-12 01:58 UTC, Phil Ringnalda
Details | Diff | Splinter Review
Bug 39296: [POC] Send objects to the templates (2.40 KB, patch)
2025-10-23 12:22 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2025-03-11 03:39:29 UTC
ExtractValue() is a useful MySQL function but it's limited in what it can do. 

Sometimes, you want to do reporting off a more nuanced understanding of MARC.

I had thought about exposing XML::LibXML::Document and XML::LibXML::XPathContext objects, but they're too powerful/vulnerable to abuse. Instead, I decided that MARC::Record should give all the power we need while not upsetting the status quo or security (after all, we already allow access to MARC::Record objects via $biblio objects in Templates).

In this case, you fetch the whole "metadata" blob from the database in your report, and you transform it using a template using Koha::Template::Plugin::KohaMarcRecord

It lets you perform more advanced reporting using the full MARC record.
Comment 1 David Cook 2025-03-11 04:26:18 UTC
Created attachment 179135 [details] [review]
Bug 39296: Provide a template plugin for getting a MARC::Record from MARCXML

This change takes MARCXML data (e.g. from the biblio_metadata table) and
returns a MARC::Record object which can be used in Reports templates to
allow for more sophisticated reporting using MARCXML data.

Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/tools/letter.pl
2. Add a new notice for Reports with the following content:
[% USE KohaMarcRecord %]
[% SET all_subjects = {} %]
[% FOREACH row IN data %]
    [% SET record= KohaMarcRecord.from_xml(row.metadata) %]
    [% SET fields = record.field('6..') %]
    [% FOREACH field IN fields %]
        [% tmp = field.delete_subfield('code','9') %]
        [% SET subject = field.as_string() %]
        [% IF ( all_subjects.$subject ) %]
            [% all_subjects.$subject = all_subjects.$subject + 1 %]
        [% ELSE %]
            [% all_subjects.$subject = 1 %]
        [% END %]
    [% END %]
[% END %]
<table>
<tr><th>Subject</th><th>Count</th></tr>
[% FOREACH key IN all_subjects.nsort.reverse %]
<tr><td>[% key %]</td><td>[% all_subjects.$key %]</td></tr>
[% END %]
</table>
3. Go to http://localhost:8081/cgi-bin/koha/reports/guided_reports.pl?op=add_form_sql
4. Create a new report with the following SQL:
5. Click "Run with template" and choose the template/notice you've created above
6. Note that you get a list of MARC subjects sorted by the most used to the least used
Comment 2 Phil Ringnalda 2025-03-12 01:56:46 UTC
But if we do that, all my knowledge about how to ExtractValue repeatable fields like 650 as individuals with datafield[@tag=650][1] ... datafield[@tag=650][20] will be useless!
Comment 3 Phil Ringnalda 2025-03-12 01:58:38 UTC
Created attachment 179189 [details] [review]
Bug 39296: Provide a template plugin for getting a MARC::Record from MARCXML

This change takes MARCXML data (e.g. from the biblio_metadata table) and
returns a MARC::Record object which can be used in Reports templates to
allow for more sophisticated reporting using MARCXML data.

Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/tools/letter.pl
2. Add a new notice for Reports with the following content:
[% USE KohaMarcRecord %]
[% SET all_subjects = {} %]
[% FOREACH row IN data %]
    [% SET record= KohaMarcRecord.from_xml(row.metadata) %]
    [% SET fields = record.field('6..') %]
    [% FOREACH field IN fields %]
        [% tmp = field.delete_subfield('code','9') %]
        [% SET subject = field.as_string() %]
        [% IF ( all_subjects.$subject ) %]
            [% all_subjects.$subject = all_subjects.$subject + 1 %]
        [% ELSE %]
            [% all_subjects.$subject = 1 %]
        [% END %]
    [% END %]
[% END %]
<table>
<tr><th>Subject</th><th>Count</th></tr>
[% FOREACH key IN all_subjects.nsort.reverse %]
<tr><td>[% key %]</td><td>[% all_subjects.$key %]</td></tr>
[% END %]
</table>
3. Go to http://localhost:8081/cgi-bin/koha/reports/guided_reports.pl?op=add_form_sql
4. Create a new report with the following SQL:
select metadata from biblio_metadata
5. Click "Run with template" and choose the template/notice you've created above
6. Note that you get a list of MARC subjects sorted by the most used to the least used

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Comment 4 Phil Ringnalda 2025-03-12 01:59:25 UTC
I made a guess at the missing SQL and added it to the test plan.
Comment 5 David Cook 2025-03-12 02:06:26 UTC
(In reply to Phil Ringnalda from comment #4)
> I made a guess at the missing SQL and added it to the test plan.

Thanks, Phil. Not sure what happened there!
Comment 6 David Cook 2025-03-12 02:57:49 UTC
(In reply to Phil Ringnalda from comment #2)
> But if we do that, all my knowledge about how to ExtractValue repeatable
> fields like 650 as individuals with datafield[@tag=650][1] ...
> datafield[@tag=650][20] will be useless!

You delight me, Phil. Haha.
Comment 7 Jonathan Druart 2025-10-23 12:03:09 UTC
Thinking aloud.

We have some magic in the reports already to send items to batch mod tools, if the headers contain "itemnumber".

Could we imagine the same here? If itemnumber or biblionumber in the header then we send the Koha::Objects to generate the notice.

The notice template would be:
[% FOREACH biblio IN biblios %]
    [% SET record = biblio.metadata.record %]

We won't need an additional TT plugin and that will be useful in different other use cases.
Comment 8 Jonathan Druart 2025-10-23 12:22:15 UTC
Created attachment 188347 [details] [review]
Bug 39296: [POC] Send objects to the templates

[% SET all_subjects = {} %]
[% FOREACH d IN data %]
    [% SET record= d.biblio.metadata.record %]
    [% SET fields = record.field('6..') %]
    [% FOREACH field IN fields %]
        [% tmp = field.delete_subfield('code','9') %]
        [% SET subject = field.as_string() %]
        [% IF ( all_subjects.$subject ) %]
            [% all_subjects.$subject = all_subjects.$subject + 1 %]
        [% ELSE %]
            [% all_subjects.$subject = 1 %]
        [% END %]
    [% END %]
[% END %]
<table>
<tr><th>Subject</th><th>Count</th></tr>
[% FOREACH key IN all_subjects.nsort.reverse %]
<tr><td>[% key %]</td><td>[% all_subjects.$key %]</td></tr>
[% END %]
</table>
Comment 9 Jonathan Druart 2025-10-23 12:22:38 UTC
(In reply to Jonathan Druart from comment #8)
> Created attachment 188347 [details] [review] [review]
> Bug 39296: [POC] Send objects to the templates

Far from perfect but this is what I had in mind, for discussion.
Comment 10 David Cook 2025-10-24 02:54:00 UTC
(In reply to Jonathan Druart from comment #7)
> Thinking aloud.
> 
> We have some magic in the reports already to send items to batch mod tools,
> if the headers contain "itemnumber".
> 
> Could we imagine the same here? If itemnumber or biblionumber in the header
> then we send the Koha::Objects to generate the notice.
> 
> The notice template would be:
> [% FOREACH biblio IN biblios %]
>     [% SET record = biblio.metadata.record %]
> 
> We won't need an additional TT plugin and that will be useful in different
> other use cases.

I think I thought a bit about this originally, but I was worried a bit about how methods could be chained together to leak data. 

I'll put my idea in a separate comment.
Comment 12 David Cook 2025-10-24 03:04:41 UTC
(In reply to Jonathan Druart from comment #7)
> We won't need an additional TT plugin and that will be useful in different
> other use cases.

Yeah, I thought it would be interesting to have a solution that was powerful/flexible across a range of use cases, but then I thought about all the ways it could go wrong, so I decided to narrowly scope the template plugin. 

I don't think I could see any security flaws of just introducing MARC::Record via the template plugin. Limited utility overall, but I do use it in production for a few of my libraries, especially for analyzing usage of authorities in bib records.