Bug 27859 - MARC export for search results
Summary: MARC export for search results
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Gustafsson
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-04 12:15 UTC by David Gustafsson
Modified: 2024-03-25 00:56 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement enables exporting search results from the staff interface when using Elasticsearch or Open Search. Two formats are available by default: MARC (UTF-8) and MARC XML. The export is run as a background job, with the results available for download from the details page for the completed job. Requires the export_catalog permission. Three new system preferences are used to configure exporting search results: - EnableElasticsearchSearchResultExport: enable the exporting of search results - ElasticsearchSearchResultExportCustomFormats: define custom export formats - ElasticsearchSearchResultExportLimit: limit the number of search results exported
Version(s) released in:


Attachments
Bug 27859: marc search result export (34.10 KB, patch)
2021-03-04 13:06 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (34.00 KB, patch)
2021-03-04 13:09 UTC, David Gustafsson
Details | Diff | Splinter Review
Use patron object instead of userenv (1.96 KB, patch)
2021-09-15 14:14 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (33.89 KB, patch)
2021-09-15 14:20 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (34.60 KB, patch)
2021-12-16 16:04 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.33 KB, patch)
2022-03-07 15:58 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.31 KB, patch)
2022-03-07 16:01 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.32 KB, patch)
2022-04-06 12:22 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.42 KB, patch)
2022-04-06 12:42 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.44 KB, patch)
2022-04-07 12:18 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (48.54 KB, patch)
2023-01-12 18:44 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (48.49 KB, patch)
2023-01-12 18:50 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.22 KB, patch)
2023-01-13 11:26 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.27 KB, patch)
2023-01-13 17:49 UTC, David Nind
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.56 KB, patch)
2023-03-08 13:38 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.98 KB, patch)
2023-03-08 14:20 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.94 KB, patch)
2023-03-08 14:21 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (51.45 KB, patch)
2024-02-22 18:20 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (51.06 KB, patch)
2024-02-22 18:46 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Export staff interface search results when using Elasticsearch (51.36 KB, patch)
2024-03-25 00:52 UTC, David Nind
Details | Diff | Splinter Review
Bug 27859: (follow-up) Update system preference descriptions (4.90 KB, patch)
2024-03-25 00:52 UTC, David Nind
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Gustafsson 2021-03-04 12:15:18 UTC
This patch provides the option of exporting a search result as MARCXML or iso 2709. Ideally the export should be asynchronous but it was much trickier than first thought to fork and detach from Apache so gave that up for now. A link to the export is sent to the current user as an email as a simple way of delivering the response asynchronously (keeping it this way so don't have to refactor when/if the export is made asynchronous).
Comment 1 David Gustafsson 2021-03-04 13:06:48 UTC Comment hidden (obsolete)
Comment 2 David Gustafsson 2021-03-04 13:09:27 UTC Comment hidden (obsolete)
Comment 3 David Gustafsson 2021-03-04 13:14:44 UTC
When writing tests for this patch also discovered and fixed a bug in_array_to_marc($data) where $data is unintentionally modified by replacing the value of hash-ref in that strucure. $data is mutated when it should not which becomes apparent if calling _array_to_marc with the same argument multiple times.
Comment 4 David Gustafsson 2021-09-15 14:14:18 UTC Comment hidden (obsolete)
Comment 5 David Gustafsson 2021-09-15 14:20:27 UTC Comment hidden (obsolete)
Comment 6 David Gustafsson 2021-09-15 14:20:55 UTC
Rebased against master.
Comment 7 David Gustafsson 2021-12-16 16:04:33 UTC Comment hidden (obsolete)
Comment 8 David Gustafsson 2022-03-07 15:58:54 UTC Comment hidden (obsolete)
Comment 9 David Gustafsson 2022-03-07 16:01:36 UTC Comment hidden (obsolete)
Comment 10 Séverine Queune 2022-04-04 15:07:28 UTC
Sorry David, patch needs rebase :/

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 27859: marc search result export
Using index info to reconstruct a base tree...
M	Koha/SearchEngine/Elasticsearch/Search.pm
M	catalogue/search.pl
M	installer/data/mysql/mandatory/sysprefs.sql
M	koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
Auto-merging installer/data/mysql/mandatory/sysprefs.sql
Auto-merging catalogue/search.pl
Auto-merging Koha/SearchEngine/Elasticsearch/Search.pm
CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch/Search.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 27859: marc search result export
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Comment 11 David Gustafsson 2022-04-06 12:22:46 UTC Comment hidden (obsolete)
Comment 12 David Gustafsson 2022-04-06 12:23:22 UTC
Ok! Rebased against master.
Comment 13 David Gustafsson 2022-04-06 12:42:04 UTC Comment hidden (obsolete)
Comment 14 David Gustafsson 2022-04-07 12:18:10 UTC Comment hidden (obsolete)
Comment 15 Katrin Fischer 2022-06-25 15:29:18 UTC
Hi David, 

the QA test tools turn up some issues with this patch, can you please fix?

1) QA test tools

 FAIL	Koha/SearchEngine/Elasticsearch.pm
   FAIL	  spelling
		 prefered ==> preferred

 OK	Koha/SearchEngine/Elasticsearch/Search.pm
 FAIL	catalogue/search.pl
   FAIL	  critic
		# ControlStructures::ProhibitMutatingListFunctions: Got 1 violation(s).  
		# Variables::ProhibitUnusedVariables: Got 1 violation(s).  

 OK	installer/data/mysql/atomicupdate/bug_27859-add_enable_search_result_marc_export_sysprefs.pl
 OK	installer/data/mysql/mandatory/sysprefs.sql
 FAIL	koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
   FAIL	  filters
		wrong_html_filter at line 325 (                                            <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>)
		missing_filter at line 325 (                                            <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>)
		missing_filter at line 360 (                    <div class="dialog message">[% export_message %]</div>)
		missing_filter at line 363 (                    <div class="dialog error">[% export_error %]</div>)

 FAIL	t/db_dependent/Koha/SearchEngine/Elasticsearch.t
   FAIL	  spelling
		 indentical ==> identical
		 indentical ==> identical


2) Translatability

This will not be translatable and should be moved to templates, also the other places where export_links_html is set. HTML should not be generated in .pl files:

+            $export_links_html = $links_count > 1 ?
+            "<p>Some records exceeded the maximum size supported by ISO2709 and was exported as MARCXML instead.</p>" . $export_links_html : $export_links_html;

Also here:
$message = "<p>The export finished successfully:</p>"


3) System preferences

I am a little in doubt about SearchResultMARCExportFromAddress. The from address can not always be set to be a library specific email, especially for installations hosted centrally by a support provider. We usually tend to use the reply-to for individual settings.

But it should for sure be type = email in the .prefs file (activates validation).
Comment 16 David Gustafsson 2023-01-12 18:44:08 UTC Comment hidden (obsolete)
Comment 17 David Gustafsson 2023-01-12 18:46:21 UTC
Thanks for the review! Issues should now be fixed. Now using background job when performing the export so removed the feature for sending results in email.
Comment 18 David Gustafsson 2023-01-12 18:50:06 UTC Comment hidden (obsolete)
Comment 19 David Nind 2023-01-13 03:09:44 UTC
Just one item from testing. 

When I click on the "View detail of the enqueued job" (link is http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I get an error trace page:

Template process failed: file error - background_jobs/search_result_marc_export.inc: not found at /kohadevbox/koha/C4/Templates.pm line 127

    in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127

      124: 
      125:     my $data;
      126:     binmode( STDOUT, ":encoding(UTF-8)" );
      127:     $template->process( $self->filename, $vars, \$data )
      128:       || die "Template process failed: ", $template->error();
      129:     return $data;
      130: }

....

I get the same error if I try to view the job details from Administration > Jobs > Manage jobs
Comment 20 David Gustafsson 2023-01-13 11:26:41 UTC Comment hidden (obsolete)
Comment 21 David Gustafsson 2023-01-13 11:28:41 UTC
(In reply to David Nind from comment #19)
> Just one item from testing. 
> 
> When I click on the "View detail of the enqueued job" (link is
> http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I
> get an error trace page:
> 
> Template process failed: file error -
> background_jobs/search_result_marc_export.inc: not found at
> /kohadevbox/koha/C4/Templates.pm line 127
> 
>     in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127
> 
>       124: 
>       125:     my $data;
>       126:     binmode( STDOUT, ":encoding(UTF-8)" );
>       127:     $template->process( $self->filename, $vars, \$data )
>       128:       || die "Template process failed: ", $template->error();
>       129:     return $data;
>       130: }
> 
> ....
> 
> I get the same error if I try to view the job details from Administration >
> Jobs > Manage jobs

Ok! Forgot to include that file in the commit, have now fixed this.
Comment 22 David Nind 2023-01-13 17:49:53 UTC Comment hidden (obsolete)
Comment 23 David Nind 2023-01-13 17:51:24 UTC
(In reply to David Gustafsson from comment #21)
> Ok! Forgot to include that file in the commit, have now fixed this.

Thanks David! Now signed off.

Tested with ES7.
Comment 24 Katrin Fischer 2023-02-25 00:00:14 UTC
Testing...

1) QA test tools

 FAIL	koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/search_result_marc_export.inc
   FAIL	  filters
		missing_filter at line 14 (                <li>[% format %]: <a href="[% report.export_links.$format | $raw %]">[% report.export_links.$format | html %]</a></li

2) Database update

It would be better to use the new template for system preferences, but the output should include the names of the new preferences.

3) System preference definitions

a) This needs to be 0/1 instead of yes/no:
+            - pref: EnableSearchResultMARCExport
+              type: boolean
+              default: yes
+              choices:
+                  yes: Enable
+                  no: Disable


b) This doesn't read quite right:
"A format have the required properties \"<strong>name</strong>\", \"<strong>fields</strong>\" and an optional \"<strong>multiple</strong>\".<br />"

c) SearchResultMARCExportCustomFormats
It says MARC export... but do we actually export MARC? Maybe a note where to find the filed names would help here. From the name, I had expected MARC fields.

d) ElasticSearch should be Elasticsearch

e) As this appears to be an Elasticserach only feature, all relating prefs should say so. Also you might want to add some links from one pref to the others involved.

I got to stop here now since Elasticsearch doesn't want to cooperate in my koha-testing-docker environment right now :(
Comment 25 David Gustafsson 2023-03-08 13:38:19 UTC Comment hidden (obsolete)
Comment 26 David Gustafsson 2023-03-08 14:20:41 UTC Comment hidden (obsolete)
Comment 27 David Gustafsson 2023-03-08 14:21:43 UTC
Created attachment 147936 [details] [review]
Bug 27859: marc search result export

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the syspref EnableSearchResultMARCExport is enabled
4) Make sure the current user has the tools -> export_catalog permission
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export results" drop down
7) Verify that link(s) with exported data appear when export is
   completed
8) Revoke the permission in 3) and ensure exporting is no longer
   possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 28 David Gustafsson 2023-03-08 14:23:03 UTC
(In reply to Katrin Fischer from comment #24)
> Testing...
> 
> 1) QA test tools
> 
>  FAIL
> koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/
> search_result_marc_export.inc
>    FAIL	  filters
> 		missing_filter at line 14 (                <li>[% format %]: <a href="[%
> report.export_links.$format | $raw %]">[% report.export_links.$format | html
> %]</a></li
> 
> 2) Database update
> 
> It would be better to use the new template for system preferences, but the
> output should include the names of the new preferences.
> 
> 3) System preference definitions
> 
> a) This needs to be 0/1 instead of yes/no:
> +            - pref: EnableSearchResultMARCExport
> +              type: boolean
> +              default: yes
> +              choices:
> +                  yes: Enable
> +                  no: Disable
> 
> 
> b) This doesn't read quite right:
> "A format have the required properties \"<strong>name</strong>\",
> \"<strong>fields</strong>\" and an optional
> \"<strong>multiple</strong>\".<br />"
> 
> c) SearchResultMARCExportCustomFormats
> It says MARC export... but do we actually export MARC? Maybe a note where to
> find the filed names would help here. From the name, I had expected MARC
> fields.
> 
> d) ElasticSearch should be Elasticsearch
> 
> e) As this appears to be an Elasticserach only feature, all relating prefs
> should say so. Also you might want to add some links from one pref to the
> others involved.
> 
> I got to stop here now since Elasticsearch doesn't want to cooperate in my
> koha-testing-docker environment right now :(

Thanks for the feedback! The issues should now be fixed.
Comment 29 David Gustafsson 2023-10-25 12:33:57 UTC
Rebased against master.
Comment 30 Lucas Gass 2023-10-25 17:48:50 UTC
David, unfortunately your tests no longer apply cleanly. ( t/db_dependent/Koha/SearchEngine/Elasticsearch.t ) It looks like a simple fix where you just need to adjust the number of tests. 

QA Tool:
 FAIL	koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
   FAIL	  valid_template
		str/job_progess.inc: not found


I don't see an include in Koha called this. 

Also, don't forget to tidy!
Comment 31 David Gustafsson 2024-02-22 18:20:15 UTC
Created attachment 162342 [details] [review]
Bug 27859: marc search result export

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the syspref EnableSearchResultMARCExport is enabled
4) Make sure the current user has the tools -> export_catalog permission
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export results" drop down
7) Verify that link(s) with exported data appear when export is
   completed
8) Revoke the permission in 3) and ensure exporting is no longer
   possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 32 David Gustafsson 2024-02-22 18:24:19 UTC
The job_progress-files where misspelled in previous version of Koha, that's why the includes now fail and this is now fixed. The qa script complains about POD coverage being greater before for Koha/BackgroundJob/SearchResultExport.pm, but don't really understand why. It's the same as the other background job classes, and a coverage of 1 should be fine?
Comment 33 David Gustafsson 2024-02-22 18:46:39 UTC
Created attachment 162343 [details] [review]
Bug 27859: marc search result export

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the syspref EnableSearchResultMARCExport is enabled
4) Make sure the current user has the tools -> export_catalog permission
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export results" drop down
7) Verify that link(s) with exported data appear when export is
   completed
8) Revoke the permission in 3) and ensure exporting is no longer
   possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 34 David Nind 2024-03-25 00:52:17 UTC
Created attachment 163761 [details] [review]
Bug 27859: Export staff interface search results when using Elasticsearch

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the EnableElasticsearchSearchResultExport system
   preference is enabled
4) Make sure the current user has the export_catalog permission
   (Use all tools > Export bibliographic and holdings data)
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export all results" drop down
7) Verify that link(s) with exported data appear on the
   background job detail page when the export is completed
8) Revoke the permission in step 4 and ensure exporting
   is no longer possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Signed-off-by: David Nind <david@davidnind.com>
Comment 35 David Nind 2024-03-25 00:52:20 UTC
Created attachment 163762 [details] [review]
Bug 27859: (follow-up) Update system preference descriptions

Format and edit the descriptions for the new
EnableElasticsearchSearchResultExport and
ElasticsearchSearchResultExportCustomFormats system
preferences.

Signed-off-by: David Nind <david@davidnind.com>
Comment 36 David Nind 2024-03-25 00:56:30 UTC
I have:
1. Amended the title and test plan for the patch (to update to the new system preference name)
2. Added a follow-up with some edits to the system preference descriptions.

Testing notes using KTD:
1. Start up KTD using Elasticsearch: ktd --es8 up
2. The new system preference EnableElasticsearchSearchResultExport is enabled by default.
3. Test exporting using custom export formats:
   . Test the example custom export formats in the ElasticsearchSearchResultExportCustomFormats system preference description