Bug 27859 - MARC export for search results
Summary: MARC export for search results
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: David Gustafsson
QA Contact:
URL:
Keywords:
Depends on: 39171
Blocks:
  Show dependency treegraph
 
Reported: 2021-03-04 12:15 UTC by David Gustafsson
Modified: 2025-04-23 21:50 UTC (History)
9 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement enables exporting search results from the staff interface when using Elasticsearch or Open Search. Two formats are available by default: MARC (UTF-8) and MARC XML. The export is run as a background job, with the results available for download from the details page for the completed job. Requires the export_catalog permission. Three new system preferences are used to configure exporting search results: - EnableElasticsearchSearchResultExport: enable the exporting of search results - ElasticsearchSearchResultExportCustomFormats: define custom export formats - ElasticsearchSearchResultExportLimit: limit the number of search results exported
Version(s) released in:
Circulation function:


Attachments
Bug 27859: marc search result export (34.10 KB, patch)
2021-03-04 13:06 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (34.00 KB, patch)
2021-03-04 13:09 UTC, David Gustafsson
Details | Diff | Splinter Review
Use patron object instead of userenv (1.96 KB, patch)
2021-09-15 14:14 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (33.89 KB, patch)
2021-09-15 14:20 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (34.60 KB, patch)
2021-12-16 16:04 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.33 KB, patch)
2022-03-07 15:58 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.31 KB, patch)
2022-03-07 16:01 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.32 KB, patch)
2022-04-06 12:22 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.42 KB, patch)
2022-04-06 12:42 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (41.44 KB, patch)
2022-04-07 12:18 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (48.54 KB, patch)
2023-01-12 18:44 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (48.49 KB, patch)
2023-01-12 18:50 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.22 KB, patch)
2023-01-13 11:26 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.27 KB, patch)
2023-01-13 17:49 UTC, David Nind
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.56 KB, patch)
2023-03-08 13:38 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.98 KB, patch)
2023-03-08 14:20 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (50.94 KB, patch)
2023-03-08 14:21 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (51.45 KB, patch)
2024-02-22 18:20 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: marc search result export (51.06 KB, patch)
2024-02-22 18:46 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Export staff interface search results when using Elasticsearch (51.36 KB, patch)
2024-03-25 00:52 UTC, David Nind
Details | Diff | Splinter Review
Bug 27859: (follow-up) Update system preference descriptions (4.90 KB, patch)
2024-03-25 00:52 UTC, David Nind
Details | Diff | Splinter Review
Bug 27859: Export staff interface search results when using Elasticsearch (58.98 KB, patch)
2025-02-17 17:08 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: (follow-up) Update system preference descriptions (4.90 KB, patch)
2025-02-17 17:08 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Export staff interface search results when using Elasticsearch (58.06 KB, patch)
2025-02-19 16:36 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: (follow-up) Update system preference descriptions (4.90 KB, patch)
2025-02-19 16:36 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Export staff interface search results when using Elasticsearch (54.91 KB, patch)
2025-02-19 18:44 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: (follow-up) Update system preference descriptions (4.90 KB, patch)
2025-02-19 18:45 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Fix background job error and js progress callback (2.62 KB, patch)
2025-02-20 12:39 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Don't enable feature by default (2.78 KB, patch)
2025-02-20 12:43 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Fix background job error and js progress callback (2.62 KB, patch)
2025-02-20 12:46 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: Don't enable feature by default (2.78 KB, patch)
2025-02-20 12:46 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: (follow up) Fix background job error and js progress callback (2.63 KB, patch)
2025-02-20 12:57 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: (follow-up) Don't enable feature by default (2.79 KB, patch)
2025-02-20 12:57 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 27859: (follow-up) Correct MARC XML to MARCXML (2.80 KB, patch)
2025-02-20 12:57 UTC, David Gustafsson
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Gustafsson 2021-03-04 12:15:18 UTC
This patch provides the option of exporting a search result as MARCXML or iso 2709. Ideally the export should be asynchronous but it was much trickier than first thought to fork and detach from Apache so gave that up for now. A link to the export is sent to the current user as an email as a simple way of delivering the response asynchronously (keeping it this way so don't have to refactor when/if the export is made asynchronous).
Comment 1 David Gustafsson 2021-03-04 13:06:48 UTC Comment hidden (obsolete)
Comment 2 David Gustafsson 2021-03-04 13:09:27 UTC Comment hidden (obsolete)
Comment 3 David Gustafsson 2021-03-04 13:14:44 UTC
When writing tests for this patch also discovered and fixed a bug in_array_to_marc($data) where $data is unintentionally modified by replacing the value of hash-ref in that strucure. $data is mutated when it should not which becomes apparent if calling _array_to_marc with the same argument multiple times.
Comment 4 David Gustafsson 2021-09-15 14:14:18 UTC Comment hidden (obsolete)
Comment 5 David Gustafsson 2021-09-15 14:20:27 UTC Comment hidden (obsolete)
Comment 6 David Gustafsson 2021-09-15 14:20:55 UTC
Rebased against master.
Comment 7 David Gustafsson 2021-12-16 16:04:33 UTC Comment hidden (obsolete)
Comment 8 David Gustafsson 2022-03-07 15:58:54 UTC Comment hidden (obsolete)
Comment 9 David Gustafsson 2022-03-07 16:01:36 UTC Comment hidden (obsolete)
Comment 10 Séverine Queune 2022-04-04 15:07:28 UTC
Sorry David, patch needs rebase :/

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 27859: marc search result export
Using index info to reconstruct a base tree...
M	Koha/SearchEngine/Elasticsearch/Search.pm
M	catalogue/search.pl
M	installer/data/mysql/mandatory/sysprefs.sql
M	koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
Auto-merging installer/data/mysql/mandatory/sysprefs.sql
Auto-merging catalogue/search.pl
Auto-merging Koha/SearchEngine/Elasticsearch/Search.pm
CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch/Search.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 27859: marc search result export
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Comment 11 David Gustafsson 2022-04-06 12:22:46 UTC Comment hidden (obsolete)
Comment 12 David Gustafsson 2022-04-06 12:23:22 UTC
Ok! Rebased against master.
Comment 13 David Gustafsson 2022-04-06 12:42:04 UTC Comment hidden (obsolete)
Comment 14 David Gustafsson 2022-04-07 12:18:10 UTC Comment hidden (obsolete)
Comment 15 Katrin Fischer 2022-06-25 15:29:18 UTC
Hi David, 

the QA test tools turn up some issues with this patch, can you please fix?

1) QA test tools

 FAIL	Koha/SearchEngine/Elasticsearch.pm
   FAIL	  spelling
		 prefered ==> preferred

 OK	Koha/SearchEngine/Elasticsearch/Search.pm
 FAIL	catalogue/search.pl
   FAIL	  critic
		# ControlStructures::ProhibitMutatingListFunctions: Got 1 violation(s).  
		# Variables::ProhibitUnusedVariables: Got 1 violation(s).  

 OK	installer/data/mysql/atomicupdate/bug_27859-add_enable_search_result_marc_export_sysprefs.pl
 OK	installer/data/mysql/mandatory/sysprefs.sql
 FAIL	koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
   FAIL	  filters
		wrong_html_filter at line 325 (                                            <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>)
		missing_filter at line 325 (                                            <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>)
		missing_filter at line 360 (                    <div class="dialog message">[% export_message %]</div>)
		missing_filter at line 363 (                    <div class="dialog error">[% export_error %]</div>)

 FAIL	t/db_dependent/Koha/SearchEngine/Elasticsearch.t
   FAIL	  spelling
		 indentical ==> identical
		 indentical ==> identical


2) Translatability

This will not be translatable and should be moved to templates, also the other places where export_links_html is set. HTML should not be generated in .pl files:

+            $export_links_html = $links_count > 1 ?
+            "<p>Some records exceeded the maximum size supported by ISO2709 and was exported as MARCXML instead.</p>" . $export_links_html : $export_links_html;

Also here:
$message = "<p>The export finished successfully:</p>"


3) System preferences

I am a little in doubt about SearchResultMARCExportFromAddress. The from address can not always be set to be a library specific email, especially for installations hosted centrally by a support provider. We usually tend to use the reply-to for individual settings.

But it should for sure be type = email in the .prefs file (activates validation).
Comment 16 David Gustafsson 2023-01-12 18:44:08 UTC Comment hidden (obsolete)
Comment 17 David Gustafsson 2023-01-12 18:46:21 UTC
Thanks for the review! Issues should now be fixed. Now using background job when performing the export so removed the feature for sending results in email.
Comment 18 David Gustafsson 2023-01-12 18:50:06 UTC Comment hidden (obsolete)
Comment 19 David Nind 2023-01-13 03:09:44 UTC
Just one item from testing. 

When I click on the "View detail of the enqueued job" (link is http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I get an error trace page:

Template process failed: file error - background_jobs/search_result_marc_export.inc: not found at /kohadevbox/koha/C4/Templates.pm line 127

    in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127

      124: 
      125:     my $data;
      126:     binmode( STDOUT, ":encoding(UTF-8)" );
      127:     $template->process( $self->filename, $vars, \$data )
      128:       || die "Template process failed: ", $template->error();
      129:     return $data;
      130: }

....

I get the same error if I try to view the job details from Administration > Jobs > Manage jobs
Comment 20 David Gustafsson 2023-01-13 11:26:41 UTC Comment hidden (obsolete)
Comment 21 David Gustafsson 2023-01-13 11:28:41 UTC
(In reply to David Nind from comment #19)
> Just one item from testing. 
> 
> When I click on the "View detail of the enqueued job" (link is
> http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I
> get an error trace page:
> 
> Template process failed: file error -
> background_jobs/search_result_marc_export.inc: not found at
> /kohadevbox/koha/C4/Templates.pm line 127
> 
>     in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127
> 
>       124: 
>       125:     my $data;
>       126:     binmode( STDOUT, ":encoding(UTF-8)" );
>       127:     $template->process( $self->filename, $vars, \$data )
>       128:       || die "Template process failed: ", $template->error();
>       129:     return $data;
>       130: }
> 
> ....
> 
> I get the same error if I try to view the job details from Administration >
> Jobs > Manage jobs

Ok! Forgot to include that file in the commit, have now fixed this.
Comment 22 David Nind 2023-01-13 17:49:53 UTC Comment hidden (obsolete)
Comment 23 David Nind 2023-01-13 17:51:24 UTC
(In reply to David Gustafsson from comment #21)
> Ok! Forgot to include that file in the commit, have now fixed this.

Thanks David! Now signed off.

Tested with ES7.
Comment 24 Katrin Fischer 2023-02-25 00:00:14 UTC
Testing...

1) QA test tools

 FAIL	koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/search_result_marc_export.inc
   FAIL	  filters
		missing_filter at line 14 (                <li>[% format %]: <a href="[% report.export_links.$format | $raw %]">[% report.export_links.$format | html %]</a></li

2) Database update

It would be better to use the new template for system preferences, but the output should include the names of the new preferences.

3) System preference definitions

a) This needs to be 0/1 instead of yes/no:
+            - pref: EnableSearchResultMARCExport
+              type: boolean
+              default: yes
+              choices:
+                  yes: Enable
+                  no: Disable


b) This doesn't read quite right:
"A format have the required properties \"<strong>name</strong>\", \"<strong>fields</strong>\" and an optional \"<strong>multiple</strong>\".<br />"

c) SearchResultMARCExportCustomFormats
It says MARC export... but do we actually export MARC? Maybe a note where to find the filed names would help here. From the name, I had expected MARC fields.

d) ElasticSearch should be Elasticsearch

e) As this appears to be an Elasticserach only feature, all relating prefs should say so. Also you might want to add some links from one pref to the others involved.

I got to stop here now since Elasticsearch doesn't want to cooperate in my koha-testing-docker environment right now :(
Comment 25 David Gustafsson 2023-03-08 13:38:19 UTC Comment hidden (obsolete)
Comment 26 David Gustafsson 2023-03-08 14:20:41 UTC Comment hidden (obsolete)
Comment 27 David Gustafsson 2023-03-08 14:21:43 UTC
Created attachment 147936 [details] [review]
Bug 27859: marc search result export

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the syspref EnableSearchResultMARCExport is enabled
4) Make sure the current user has the tools -> export_catalog permission
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export results" drop down
7) Verify that link(s) with exported data appear when export is
   completed
8) Revoke the permission in 3) and ensure exporting is no longer
   possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 28 David Gustafsson 2023-03-08 14:23:03 UTC
(In reply to Katrin Fischer from comment #24)
> Testing...
> 
> 1) QA test tools
> 
>  FAIL
> koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/
> search_result_marc_export.inc
>    FAIL	  filters
> 		missing_filter at line 14 (                <li>[% format %]: <a href="[%
> report.export_links.$format | $raw %]">[% report.export_links.$format | html
> %]</a></li
> 
> 2) Database update
> 
> It would be better to use the new template for system preferences, but the
> output should include the names of the new preferences.
> 
> 3) System preference definitions
> 
> a) This needs to be 0/1 instead of yes/no:
> +            - pref: EnableSearchResultMARCExport
> +              type: boolean
> +              default: yes
> +              choices:
> +                  yes: Enable
> +                  no: Disable
> 
> 
> b) This doesn't read quite right:
> "A format have the required properties \"<strong>name</strong>\",
> \"<strong>fields</strong>\" and an optional
> \"<strong>multiple</strong>\".<br />"
> 
> c) SearchResultMARCExportCustomFormats
> It says MARC export... but do we actually export MARC? Maybe a note where to
> find the filed names would help here. From the name, I had expected MARC
> fields.
> 
> d) ElasticSearch should be Elasticsearch
> 
> e) As this appears to be an Elasticserach only feature, all relating prefs
> should say so. Also you might want to add some links from one pref to the
> others involved.
> 
> I got to stop here now since Elasticsearch doesn't want to cooperate in my
> koha-testing-docker environment right now :(

Thanks for the feedback! The issues should now be fixed.
Comment 29 David Gustafsson 2023-10-25 12:33:57 UTC
Rebased against master.
Comment 30 Lucas Gass (lukeg) 2023-10-25 17:48:50 UTC
David, unfortunately your tests no longer apply cleanly. ( t/db_dependent/Koha/SearchEngine/Elasticsearch.t ) It looks like a simple fix where you just need to adjust the number of tests. 

QA Tool:
 FAIL	koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt
   FAIL	  valid_template
		str/job_progess.inc: not found


I don't see an include in Koha called this. 

Also, don't forget to tidy!
Comment 31 David Gustafsson 2024-02-22 18:20:15 UTC
Created attachment 162342 [details] [review]
Bug 27859: marc search result export

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the syspref EnableSearchResultMARCExport is enabled
4) Make sure the current user has the tools -> export_catalog permission
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export results" drop down
7) Verify that link(s) with exported data appear when export is
   completed
8) Revoke the permission in 3) and ensure exporting is no longer
   possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 32 David Gustafsson 2024-02-22 18:24:19 UTC
The job_progress-files where misspelled in previous version of Koha, that's why the includes now fail and this is now fixed. The qa script complains about POD coverage being greater before for Koha/BackgroundJob/SearchResultExport.pm, but don't really understand why. It's the same as the other background job classes, and a coverage of 1 should be fine?
Comment 33 David Gustafsson 2024-02-22 18:46:39 UTC
Created attachment 162343 [details] [review]
Bug 27859: marc search result export

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the syspref EnableSearchResultMARCExport is enabled
4) Make sure the current user has the tools -> export_catalog permission
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export results" drop down
7) Verify that link(s) with exported data appear when export is
   completed
8) Revoke the permission in 3) and ensure exporting is no longer
   possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 34 David Nind 2024-03-25 00:52:17 UTC
Created attachment 163761 [details] [review]
Bug 27859: Export staff interface search results when using Elasticsearch

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the EnableElasticsearchSearchResultExport system
   preference is enabled
4) Make sure the current user has the export_catalog permission
   (Use all tools > Export bibliographic and holdings data)
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export all results" drop down
7) Verify that link(s) with exported data appear on the
   background job detail page when the export is completed
8) Revoke the permission in step 4 and ensure exporting
   is no longer possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Signed-off-by: David Nind <david@davidnind.com>
Comment 35 David Nind 2024-03-25 00:52:20 UTC
Created attachment 163762 [details] [review]
Bug 27859: (follow-up) Update system preference descriptions

Format and edit the descriptions for the new
EnableElasticsearchSearchResultExport and
ElasticsearchSearchResultExportCustomFormats system
preferences.

Signed-off-by: David Nind <david@davidnind.com>
Comment 36 David Nind 2024-03-25 00:56:30 UTC
I have:
1. Amended the title and test plan for the patch (to update to the new system preference name)
2. Added a follow-up with some edits to the system preference descriptions.

Testing notes using KTD:
1. Start up KTD using Elasticsearch: ktd --es8 up
2. The new system preference EnableElasticsearchSearchResultExport is enabled by default.
3. Test exporting using custom export formats:
   . Test the example custom export formats in the ElasticsearchSearchResultExportCustomFormats system preference description
Comment 37 David Gustafsson 2025-02-17 17:08:30 UTC
Created attachment 178145 [details] [review]
Bug 27859: Export staff interface search results when using Elasticsearch

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the EnableElasticsearchSearchResultExport system
   preference is enabled
4) Make sure the current user has the export_catalog permission
   (Use all tools > Export bibliographic and holdings data)
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export all results" drop down
7) Verify that link(s) with exported data appear on the
   background job detail page when the export is completed
8) Revoke the permission in step 4 and ensure exporting
   is no longer possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 38 David Gustafsson 2025-02-17 17:08:33 UTC
Created attachment 178146 [details] [review]
Bug 27859: (follow-up) Update system preference descriptions

Format and edit the descriptions for the new
EnableElasticsearchSearchResultExport and
ElasticsearchSearchResultExportCustomFormats system
preferences.

Signed-off-by: David Nind <david@davidnind.com>
Comment 39 David Gustafsson 2025-02-17 17:09:49 UTC
Rebased against main.
Comment 40 David Cook 2025-02-17 23:34:34 UTC
What's the use case for this enhancement?

Is the idea that staff do a search rather than a SQL report because it's easier for them to understand?
Comment 41 Stefan Berndtsson 2025-02-18 08:29:59 UTC
> Is the idea that staff do a search rather than a SQL report because it's
> easier for them to understand?

Search is more accessible to the average user (not everyone can create a report). Reports that fetch subfield content from MARC datafields are often VERY slow, where the Elasticsearch index is a lot faster and less demanding on the database and the system.
Comment 42 David Cook 2025-02-18 22:11:44 UTC
(In reply to Stefan Berndtsson from comment #41)
> > Is the idea that staff do a search rather than a SQL report because it's
> > easier for them to understand?
> 
> Search is more accessible to the average user (not everyone can create a
> report). Reports that fetch subfield content from MARC datafields are often
> VERY slow, where the Elasticsearch index is a lot faster and less demanding
> on the database and the system.

Thanks for that, Stefan. I figured that would be the use case, but I wanted to make sure.
Comment 43 David Cook 2025-02-18 22:25:45 UTC
I think that there's a number of other areas where we've been interested in the idea of doing background processing and presenting the user with a link to a download, so this is interesting.

It looks like these files will need to manually cleaned up at the moment? Could a method be added to cleanup_db.pl to automatically clean them up after a certain amount of time?

I do think that librarians would like this feature.
Comment 44 David Cook 2025-02-18 22:30:14 UTC
Comment on attachment 178145 [details] [review]
Bug 27859: Export staff interface search results when using Elasticsearch

Review of attachment 178145 [details] [review]:
-----------------------------------------------------------------

::: installer/data/mysql/atomicupdate/bug_27859-add_enable_search_result_marc_export_sysprefs.pl
@@ +16,5 @@
> +        $dbh->do(q{ INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type) VALUES ('ElasticsearchSearchResultExportLimit', NULL, NULL, 'Search result export limit', 'integer') });
> +        say $out "Added new system preference 'ElasticsearchSearchResultExportLimit'";
> +
> +        $dbh->do(q{ UPDATE systempreferences SET options = 'base64ISO2709|ARRAY' WHERE variable = 'ElasticsearchMARCFormat' });
> +        $dbh->do(q{ UPDATE systempreferences SET value = 'base64ISO2709' WHERE variable = 'ElasticsearchMARCFormat' AND value = 'ISO2709' });

I like the idea of changing ISO2709 to base64ISO2709, as it's annoying to translate between base64ISO2709 and ISO2709 (whereas ARRAY and MARCXML can be matched easily). That said... that seems like a different bug report to me. It'll also conflict with bug 28370 if it gets accepted, so we'll want to be careful there.
Comment 45 David Gustafsson 2025-02-19 16:36:11 UTC
Created attachment 178370 [details] [review]
Bug 27859: Export staff interface search results when using Elasticsearch

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the EnableElasticsearchSearchResultExport system
   preference is enabled
4) Make sure the current user has the export_catalog permission
   (Use all tools > Export bibliographic and holdings data)
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export all results" drop down
7) Verify that link(s) with exported data appear on the
   background job detail page when the export is completed
8) Revoke the permission in step 4 and ensure exporting
   is no longer possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 46 David Gustafsson 2025-02-19 16:36:14 UTC
Created attachment 178371 [details] [review]
Bug 27859: (follow-up) Update system preference descriptions

Format and edit the descriptions for the new
EnableElasticsearchSearchResultExport and
ElasticsearchSearchResultExportCustomFormats system
preferences.

Signed-off-by: David Nind <david@davidnind.com>
Comment 47 David Gustafsson 2025-02-19 17:50:54 UTC
(In reply to David Cook from comment #44)
> Comment on attachment 178145 [details] [review] [review]
> Bug 27859: Export staff interface search results when using Elasticsearch
> 
> Review of attachment 178145 [details] [review] [review]:
> -----------------------------------------------------------------
> 
> :::
> installer/data/mysql/atomicupdate/bug_27859-
> add_enable_search_result_marc_export_sysprefs.pl
> @@ +16,5 @@
> > +        $dbh->do(q{ INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type) VALUES ('ElasticsearchSearchResultExportLimit', NULL, NULL, 'Search result export limit', 'integer') });
> > +        say $out "Added new system preference 'ElasticsearchSearchResultExportLimit'";
> > +
> > +        $dbh->do(q{ UPDATE systempreferences SET options = 'base64ISO2709|ARRAY' WHERE variable = 'ElasticsearchMARCFormat' });
> > +        $dbh->do(q{ UPDATE systempreferences SET value = 'base64ISO2709' WHERE variable = 'ElasticsearchMARCFormat' AND value = 'ISO2709' });
> 
> I like the idea of changing ISO2709 to base64ISO2709, as it's annoying to
> translate between base64ISO2709 and ISO2709 (whereas ARRAY and MARCXML can
> be matched easily). That said... that seems like a different bug report to
> me. It'll also conflict with bug 28370 if it gets accepted, so we'll want to
> be careful there.

Did you link to the wrong bug? Seems it has been pushed and is not related.

I agree it's not ideal to have the format name change as part of this bug, I just hoped I could sneak it in I suppose. Would be pretty simple to break it out I suppose. The syspref value actually was base64ISO2709 in bug 19893 before it was removed since I believe we didn't want to allow the user to select a less performant format, but was reintroduced in bug 22258 as ISO2709 when the ARRAY format was added.
Comment 48 David Gustafsson 2025-02-19 17:54:47 UTC
I might also have added the format name change to this bug as it makes more sense in this context. In the original code the value is never checked, it just defaults to base64ISO2709 if not set to ARRAY, so I guess without the refactoring of the record encoding method the change doesn't make as much sense.
Comment 49 David Gustafsson 2025-02-19 18:44:59 UTC
Created attachment 178376 [details] [review]
Bug 27859: Export staff interface search results when using Elasticsearch

Enable export of staff interface search results in different marc formats.
This feature is only supported when using Elasticsearch.

To test:
1) Apply patch
2) Run installer/data/mysql/updatedatabase.pl
3) Make sure the EnableElasticsearchSearchResultExport system
   preference is enabled
4) Make sure the current user has the export_catalog permission
   (Use all tools > Export bibliographic and holdings data)
5) Perform a search
6) Export the serach result by choosing a format under the
   "Export all results" drop down
7) Verify that link(s) with exported data appear on the
   background job detail page when the export is completed
8) Revoke the permission in step 4 and ensure exporting
   is no longer possible
9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t

Sponsored-by: Gothenburg University Library
Comment 50 David Gustafsson 2025-02-19 18:45:02 UTC
Created attachment 178377 [details] [review]
Bug 27859: (follow-up) Update system preference descriptions

Format and edit the descriptions for the new
EnableElasticsearchSearchResultExport and
ElasticsearchSearchResultExportCustomFormats system
preferences.

Signed-off-by: David Nind <david@davidnind.com>
Comment 51 David Gustafsson 2025-02-19 18:49:08 UTC
Now broken out the format change into bug 39171.
Comment 52 David Cook 2025-02-20 00:36:20 UTC
Failing QA for a couple of things.

1.
New features like EnableElasticsearchSearchResultExport should default to off, but it's currently defaulting to on. 

2. 
In "Export all results", "MARC XML" should be "MARCXML" to be consistent with other interfaces. 

3. 
I'm following the test plan but it keeps saying Failed with this report:

"Search results export for the query " failed with the following errors:"
Comment 53 David Gustafsson 2025-02-20 12:39:40 UTC
Created attachment 178399 [details] [review]
Bug 27859: Fix background job error and js progress callback
Comment 54 David Gustafsson 2025-02-20 12:43:39 UTC
Created attachment 178400 [details] [review]
Bug 27859: Don't enable feature by default
Comment 55 David Gustafsson 2025-02-20 12:46:00 UTC
Created attachment 178401 [details] [review]
Bug 27859: Fix background job error and js progress callback
Comment 56 David Gustafsson 2025-02-20 12:46:04 UTC
Created attachment 178402 [details] [review]
Bug 27859: Don't enable feature by default
Comment 57 David Gustafsson 2025-02-20 12:57:24 UTC
Created attachment 178406 [details] [review]
Bug 27859: (follow up) Fix background job error and js progress callback
Comment 58 David Gustafsson 2025-02-20 12:57:28 UTC
Created attachment 178407 [details] [review]
Bug 27859: (follow-up) Don't enable feature by default
Comment 59 David Gustafsson 2025-02-20 12:57:31 UTC
Created attachment 178408 [details] [review]
Bug 27859: (follow-up) Correct MARC XML to MARCXML
Comment 60 David Gustafsson 2025-02-20 13:11:07 UTC
(In reply to David Cook from comment #52)
> Failing QA for a couple of things.
> 
> 1.
> New features like EnableElasticsearchSearchResultExport should default to
> off, but it's currently defaulting to on. 

Ok, didn't know about that. This has now been fixed.

> 2. 
> In "Export all results", "MARC XML" should be "MARCXML" to be consistent
> with other interfaces. 

Yes, that\s clearly wrong. Fixed there and on the syspref page.

> 3. 
> I'm following the test plan but it keeps saying Failed with this report:
> 
> "Search results export for the query " failed with the following errors:"

My mistake, should have tested it again after the rabase. Had some issues with running scheduled jobs in my environment so that's why, but made sure to fix this to be able to test. The error was caused by my changing that the reture value of search_documents_encode is no longer binary but a perl utf8 string, so mod5_hex fails, so need to encode the data before calculating a hash. There has also been some changes to the previous callback arguments of the updateProgress javascript function, which I also had to be adjusted for.
Comment 61 Roman Dolny 2025-04-23 21:50:28 UTC
What about ElasticsearchSearchResultExportLimit system preference?
I can't confirm it works - set it, but it has no impact for limit number of exported records.

BTW, shouldn't we change "items" to "records" (or "elements"?) in description ("...search result items (Elasticsearch only).")?