ExtractValue() is a useful MySQL function but it's limited in what it can do. Sometimes, you want to do reporting off a more nuanced understanding of MARC. I had thought about exposing XML::LibXML::Document and XML::LibXML::XPathContext objects, but they're too powerful/vulnerable to abuse. Instead, I decided that MARC::Record should give all the power we need while not upsetting the status quo or security (after all, we already allow access to MARC::Record objects via $biblio objects in Templates). In this case, you fetch the whole "metadata" blob from the database in your report, and you transform it using a template using Koha::Template::Plugin::KohaMarcRecord It lets you perform more advanced reporting using the full MARC record.
Created attachment 179135 [details] [review] Bug 39296: Provide a template plugin for getting a MARC::Record from MARCXML This change takes MARCXML data (e.g. from the biblio_metadata table) and returns a MARC::Record object which can be used in Reports templates to allow for more sophisticated reporting using MARCXML data. Test plan: 0. Apply the patch and koha-plack --restart kohadev 1. Go to http://localhost:8081/cgi-bin/koha/tools/letter.pl 2. Add a new notice for Reports with the following content: [% USE KohaMarcRecord %] [% SET all_subjects = {} %] [% FOREACH row IN data %] [% SET record= KohaMarcRecord.from_xml(row.metadata) %] [% SET fields = record.field('6..') %] [% FOREACH field IN fields %] [% tmp = field.delete_subfield('code','9') %] [% SET subject = field.as_string() %] [% IF ( all_subjects.$subject ) %] [% all_subjects.$subject = all_subjects.$subject + 1 %] [% ELSE %] [% all_subjects.$subject = 1 %] [% END %] [% END %] [% END %] <table> <tr><th>Subject</th><th>Count</th></tr> [% FOREACH key IN all_subjects.nsort.reverse %] <tr><td>[% key %]</td><td>[% all_subjects.$key %]</td></tr> [% END %] </table> 3. Go to http://localhost:8081/cgi-bin/koha/reports/guided_reports.pl?op=add_form_sql 4. Create a new report with the following SQL: 5. Click "Run with template" and choose the template/notice you've created above 6. Note that you get a list of MARC subjects sorted by the most used to the least used
But if we do that, all my knowledge about how to ExtractValue repeatable fields like 650 as individuals with datafield[@tag=650][1] ... datafield[@tag=650][20] will be useless!
Created attachment 179189 [details] [review] Bug 39296: Provide a template plugin for getting a MARC::Record from MARCXML This change takes MARCXML data (e.g. from the biblio_metadata table) and returns a MARC::Record object which can be used in Reports templates to allow for more sophisticated reporting using MARCXML data. Test plan: 0. Apply the patch and koha-plack --restart kohadev 1. Go to http://localhost:8081/cgi-bin/koha/tools/letter.pl 2. Add a new notice for Reports with the following content: [% USE KohaMarcRecord %] [% SET all_subjects = {} %] [% FOREACH row IN data %] [% SET record= KohaMarcRecord.from_xml(row.metadata) %] [% SET fields = record.field('6..') %] [% FOREACH field IN fields %] [% tmp = field.delete_subfield('code','9') %] [% SET subject = field.as_string() %] [% IF ( all_subjects.$subject ) %] [% all_subjects.$subject = all_subjects.$subject + 1 %] [% ELSE %] [% all_subjects.$subject = 1 %] [% END %] [% END %] [% END %] <table> <tr><th>Subject</th><th>Count</th></tr> [% FOREACH key IN all_subjects.nsort.reverse %] <tr><td>[% key %]</td><td>[% all_subjects.$key %]</td></tr> [% END %] </table> 3. Go to http://localhost:8081/cgi-bin/koha/reports/guided_reports.pl?op=add_form_sql 4. Create a new report with the following SQL: select metadata from biblio_metadata 5. Click "Run with template" and choose the template/notice you've created above 6. Note that you get a list of MARC subjects sorted by the most used to the least used Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
I made a guess at the missing SQL and added it to the test plan.
(In reply to Phil Ringnalda from comment #4) > I made a guess at the missing SQL and added it to the test plan. Thanks, Phil. Not sure what happened there!
(In reply to Phil Ringnalda from comment #2) > But if we do that, all my knowledge about how to ExtractValue repeatable > fields like 650 as individuals with datafield[@tag=650][1] ... > datafield[@tag=650][20] will be useless! You delight me, Phil. Haha.
Thinking aloud. We have some magic in the reports already to send items to batch mod tools, if the headers contain "itemnumber". Could we imagine the same here? If itemnumber or biblionumber in the header then we send the Koha::Objects to generate the notice. The notice template would be: [% FOREACH biblio IN biblios %] [% SET record = biblio.metadata.record %] We won't need an additional TT plugin and that will be useful in different other use cases.
Created attachment 188347 [details] [review] Bug 39296: [POC] Send objects to the templates [% SET all_subjects = {} %] [% FOREACH d IN data %] [% SET record= d.biblio.metadata.record %] [% SET fields = record.field('6..') %] [% FOREACH field IN fields %] [% tmp = field.delete_subfield('code','9') %] [% SET subject = field.as_string() %] [% IF ( all_subjects.$subject ) %] [% all_subjects.$subject = all_subjects.$subject + 1 %] [% ELSE %] [% all_subjects.$subject = 1 %] [% END %] [% END %] [% END %] <table> <tr><th>Subject</th><th>Count</th></tr> [% FOREACH key IN all_subjects.nsort.reverse %] <tr><td>[% key %]</td><td>[% all_subjects.$key %]</td></tr> [% END %] </table>
(In reply to Jonathan Druart from comment #8) > Created attachment 188347 [details] [review] [review] > Bug 39296: [POC] Send objects to the templates Far from perfect but this is what I had in mind, for discussion.
(In reply to Jonathan Druart from comment #7) > Thinking aloud. > > We have some magic in the reports already to send items to batch mod tools, > if the headers contain "itemnumber". > > Could we imagine the same here? If itemnumber or biblionumber in the header > then we send the Koha::Objects to generate the notice. > > The notice template would be: > [% FOREACH biblio IN biblios %] > [% SET record = biblio.metadata.record %] > > We won't need an additional TT plugin and that will be useful in different > other use cases. I think I thought a bit about this originally, but I was worried a bit about how methods could be chained together to leak data. I'll put my idea in a separate comment.
(In reply to Jonathan Druart from comment #7) > We won't need an additional TT plugin and that will be useful in different > other use cases. Yeah, I thought it would be interesting to have a solution that was powerful/flexible across a range of use cases, but then I thought about all the ways it could go wrong, so I decided to narrowly scope the template plugin. I don't think I could see any security flaws of just introducing MARC::Record via the template plugin. Limited utility overall, but I do use it in production for a few of my libraries, especially for analyzing usage of authorities in bib records.