Description
David Gustafsson
2021-03-04 12:15:18 UTC
Created attachment 117718 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 117721 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library When writing tests for this patch also discovered and fixed a bug in_array_to_marc($data) where $data is unintentionally modified by replacing the value of hash-ref in that strucure. $data is mutated when it should not which becomes apparent if calling _array_to_marc with the same argument multiple times. Created attachment 124892 [details] [review] Use patron object instead of userenv Created attachment 124893 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Rebased against master. Created attachment 128632 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Make sure the current user has an email under your control set 6) Perform a search 7) Export the serach result by choosing a format under the "Export results" drop down 8) Verify that link(s) with exported data has been mailed to the provided emial 9) Revoke the permission in 3) and make sure exporting is no longer possible 10) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 131442 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Make sure the current user has the tools -> export_catalog permission 6) Make sure the current user has an email under your control set 7) Perform a search 8) Export the serach result by choosing a format under the "Export results" drop down 9) Verify that link(s) with exported data has been mailed to the provided email 10) Revoke the permission in 3) and make sure exporting is no longer possible 11) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 131443 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Make sure the current user has the tools -> export_catalog permission 6) Make sure the current user has an email under your control set 7) Perform a search 8) Export the serach result by choosing a format under the "Export results" drop down 9) Verify that link(s) with exported data has been mailed to the provided email 10) Revoke the permission in 3) and make sure exporting is no longer possible 11) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Sorry David, patch needs rebase :/ Apply? [(y)es, (n)o, (i)nteractive] y Applying: Bug 27859: marc search result export Using index info to reconstruct a base tree... M Koha/SearchEngine/Elasticsearch/Search.pm M catalogue/search.pl M installer/data/mysql/mandatory/sysprefs.sql M koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt Falling back to patching base and 3-way merge... Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt Auto-merging installer/data/mysql/mandatory/sysprefs.sql Auto-merging catalogue/search.pl Auto-merging Koha/SearchEngine/Elasticsearch/Search.pm CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch/Search.pm error: Failed to merge in the changes. Patch failed at 0001 Bug 27859: marc search result export hint: Use 'git am --show-current-patch=diff' to see the failed patch Created attachment 132993 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Make sure the current user has the tools -> export_catalog permission 6) Make sure the current user has an email under your control set 7) Perform a search 8) Export the serach result by choosing a format under the "Export results" drop down 9) Verify that link(s) with exported data has been mailed to the provided email 10) Revoke the permission in 3) and make sure exporting is no longer possible 11) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Ok! Rebased against master. Created attachment 132997 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Add a custom filter in the SearchResultMARCExportCustomFormats syspref, for example by copying one of the example formats 6) Make sure the current user has the tools -> export_catalog permission 7) Make sure the current user has an email under your control set 8) Perform a search 9) Export the search result for each format in the "Export all results" drop down 10) Verify that link(s) with exported data has been mailed to the provided email 11) Revoke the permission in 3) and make sure exporting is no longer possible 12) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 133076 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. The export will be mailed to the configured mail address of the current user. This feature is Elasticsearch only. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the syspref EnableSearchResultMARCExportEmail is enabled 5) Add a custom filter in the SearchResultMARCExportCustomFormats syspref, for example by copying one of the example formats 6) Make sure the current user has the tools -> export_catalog permission 7) Make sure the current user has an email under your control set 8) Perform a search 9) Export the search result for each format in the "Export all results" drop down 10) Verify that link(s) with exported data has been mailed to the provided email 11) Revoke the permission in 3) and make sure exporting is no longer possible 12) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Hi David, the QA test tools turn up some issues with this patch, can you please fix? 1) QA test tools FAIL Koha/SearchEngine/Elasticsearch.pm FAIL spelling prefered ==> preferred OK Koha/SearchEngine/Elasticsearch/Search.pm FAIL catalogue/search.pl FAIL critic # ControlStructures::ProhibitMutatingListFunctions: Got 1 violation(s). # Variables::ProhibitUnusedVariables: Got 1 violation(s). OK installer/data/mysql/atomicupdate/bug_27859-add_enable_search_result_marc_export_sysprefs.pl OK installer/data/mysql/mandatory/sysprefs.sql FAIL koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt FAIL filters wrong_html_filter at line 325 ( <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>) missing_filter at line 325 ( <li><a href="/cgi-bin/koha/catalogue/search.pl?count=[% results_per_page | uri %]&export=1&export_format=[% id %][% PROCESS sort_search_query %]">[% custom_export_formats.$id.name | html %]</a></li>) missing_filter at line 360 ( <div class="dialog message">[% export_message %]</div>) missing_filter at line 363 ( <div class="dialog error">[% export_error %]</div>) FAIL t/db_dependent/Koha/SearchEngine/Elasticsearch.t FAIL spelling indentical ==> identical indentical ==> identical 2) Translatability This will not be translatable and should be moved to templates, also the other places where export_links_html is set. HTML should not be generated in .pl files: + $export_links_html = $links_count > 1 ? + "<p>Some records exceeded the maximum size supported by ISO2709 and was exported as MARCXML instead.</p>" . $export_links_html : $export_links_html; Also here: $message = "<p>The export finished successfully:</p>" 3) System preferences I am a little in doubt about SearchResultMARCExportFromAddress. The from address can not always be set to be a library specific email, especially for installations hosted centrally by a support provider. We usually tend to use the reply-to for individual settings. But it should for sure be type = email in the .prefs file (activates validation). Created attachment 145243 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Thanks for the review! Issues should now be fixed. Now using background job when performing the export so removed the feature for sending results in email. Created attachment 145244 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Just one item from testing. When I click on the "View detail of the enqueued job" (link is http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I get an error trace page: Template process failed: file error - background_jobs/search_result_marc_export.inc: not found at /kohadevbox/koha/C4/Templates.pm line 127 in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127 124: 125: my $data; 126: binmode( STDOUT, ":encoding(UTF-8)" ); 127: $template->process( $self->filename, $vars, \$data ) 128: || die "Template process failed: ", $template->error(); 129: return $data; 130: } .... I get the same error if I try to view the job details from Administration > Jobs > Manage jobs Created attachment 145263 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library (In reply to David Nind from comment #19) > Just one item from testing. > > When I click on the "View detail of the enqueued job" (link is > http://127.0.0.1:8081/cgi-bin/koha/admin/background_jobs.pl?op=view&id=1) I > get an error trace page: > > Template process failed: file error - > background_jobs/search_result_marc_export.inc: not found at > /kohadevbox/koha/C4/Templates.pm line 127 > > in C4::Templates::output at /kohadevbox/koha/C4/Templates.pm line 127 > > 124: > 125: my $data; > 126: binmode( STDOUT, ":encoding(UTF-8)" ); > 127: $template->process( $self->filename, $vars, \$data ) > 128: || die "Template process failed: ", $template->error(); > 129: return $data; > 130: } > > .... > > I get the same error if I try to view the job details from Administration > > Jobs > Manage jobs Ok! Forgot to include that file in the commit, have now fixed this. Created attachment 145300 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> (In reply to David Gustafsson from comment #21) > Ok! Forgot to include that file in the commit, have now fixed this. Thanks David! Now signed off. Tested with ES7. Testing... 1) QA test tools FAIL koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/search_result_marc_export.inc FAIL filters missing_filter at line 14 ( <li>[% format %]: <a href="[% report.export_links.$format | $raw %]">[% report.export_links.$format | html %]</a></li 2) Database update It would be better to use the new template for system preferences, but the output should include the names of the new preferences. 3) System preference definitions a) This needs to be 0/1 instead of yes/no: + - pref: EnableSearchResultMARCExport + type: boolean + default: yes + choices: + yes: Enable + no: Disable b) This doesn't read quite right: "A format have the required properties \"<strong>name</strong>\", \"<strong>fields</strong>\" and an optional \"<strong>multiple</strong>\".<br />" c) SearchResultMARCExportCustomFormats It says MARC export... but do we actually export MARC? Maybe a note where to find the filed names would help here. From the name, I had expected MARC fields. d) ElasticSearch should be Elasticsearch e) As this appears to be an Elasticserach only feature, all relating prefs should say so. Also you might want to add some links from one pref to the others involved. I got to stop here now since Elasticsearch doesn't want to cooperate in my koha-testing-docker environment right now :( Created attachment 147927 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> Created attachment 147935 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> Created attachment 147936 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library (In reply to Katrin Fischer from comment #24) > Testing... > > 1) QA test tools > > FAIL > koha-tmpl/intranet-tmpl/prog/en/includes/background_jobs/ > search_result_marc_export.inc > FAIL filters > missing_filter at line 14 ( <li>[% format %]: <a href="[% > report.export_links.$format | $raw %]">[% report.export_links.$format | html > %]</a></li > > 2) Database update > > It would be better to use the new template for system preferences, but the > output should include the names of the new preferences. > > 3) System preference definitions > > a) This needs to be 0/1 instead of yes/no: > + - pref: EnableSearchResultMARCExport > + type: boolean > + default: yes > + choices: > + yes: Enable > + no: Disable > > > b) This doesn't read quite right: > "A format have the required properties \"<strong>name</strong>\", > \"<strong>fields</strong>\" and an optional > \"<strong>multiple</strong>\".<br />" > > c) SearchResultMARCExportCustomFormats > It says MARC export... but do we actually export MARC? Maybe a note where to > find the filed names would help here. From the name, I had expected MARC > fields. > > d) ElasticSearch should be Elasticsearch > > e) As this appears to be an Elasticserach only feature, all relating prefs > should say so. Also you might want to add some links from one pref to the > others involved. > > I got to stop here now since Elasticsearch doesn't want to cooperate in my > koha-testing-docker environment right now :( Thanks for the feedback! The issues should now be fixed. Rebased against master. David, unfortunately your tests no longer apply cleanly. ( t/db_dependent/Koha/SearchEngine/Elasticsearch.t ) It looks like a simple fix where you just need to adjust the number of tests. QA Tool: FAIL koha-tmpl/intranet-tmpl/prog/en/modules/catalogue/results.tt FAIL valid_template str/job_progess.inc: not found I don't see an include in Koha called this. Also, don't forget to tidy! Created attachment 162342 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library The job_progress-files where misspelled in previous version of Koha, that's why the includes now fail and this is now fixed. The qa script complains about POD coverage being greater before for Koha/BackgroundJob/SearchResultExport.pm, but don't really understand why. It's the same as the other background job classes, and a coverage of 1 should be fine? Created attachment 162343 [details] [review] Bug 27859: marc search result export Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the syspref EnableSearchResultMARCExport is enabled 4) Make sure the current user has the tools -> export_catalog permission 5) Perform a search 6) Export the serach result by choosing a format under the "Export results" drop down 7) Verify that link(s) with exported data appear when export is completed 8) Revoke the permission in 3) and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 163761 [details] [review] Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Signed-off-by: David Nind <david@davidnind.com> Created attachment 163762 [details] [review] Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> I have: 1. Amended the title and test plan for the patch (to update to the new system preference name) 2. Added a follow-up with some edits to the system preference descriptions. Testing notes using KTD: 1. Start up KTD using Elasticsearch: ktd --es8 up 2. The new system preference EnableElasticsearchSearchResultExport is enabled by default. 3. Test exporting using custom export formats: . Test the example custom export formats in the ElasticsearchSearchResultExportCustomFormats system preference description Created attachment 178145 [details] [review] Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 178146 [details] [review] Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> Rebased against main. What's the use case for this enhancement? Is the idea that staff do a search rather than a SQL report because it's easier for them to understand? > Is the idea that staff do a search rather than a SQL report because it's
> easier for them to understand?
Search is more accessible to the average user (not everyone can create a report). Reports that fetch subfield content from MARC datafields are often VERY slow, where the Elasticsearch index is a lot faster and less demanding on the database and the system.
(In reply to Stefan Berndtsson from comment #41) > > Is the idea that staff do a search rather than a SQL report because it's > > easier for them to understand? > > Search is more accessible to the average user (not everyone can create a > report). Reports that fetch subfield content from MARC datafields are often > VERY slow, where the Elasticsearch index is a lot faster and less demanding > on the database and the system. Thanks for that, Stefan. I figured that would be the use case, but I wanted to make sure. I think that there's a number of other areas where we've been interested in the idea of doing background processing and presenting the user with a link to a download, so this is interesting. It looks like these files will need to manually cleaned up at the moment? Could a method be added to cleanup_db.pl to automatically clean them up after a certain amount of time? I do think that librarians would like this feature. Comment on attachment 178145 [details] [review] Bug 27859: Export staff interface search results when using Elasticsearch Review of attachment 178145 [details] [review]: ----------------------------------------------------------------- ::: installer/data/mysql/atomicupdate/bug_27859-add_enable_search_result_marc_export_sysprefs.pl @@ +16,5 @@ > + $dbh->do(q{ INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type) VALUES ('ElasticsearchSearchResultExportLimit', NULL, NULL, 'Search result export limit', 'integer') }); > + say $out "Added new system preference 'ElasticsearchSearchResultExportLimit'"; > + > + $dbh->do(q{ UPDATE systempreferences SET options = 'base64ISO2709|ARRAY' WHERE variable = 'ElasticsearchMARCFormat' }); > + $dbh->do(q{ UPDATE systempreferences SET value = 'base64ISO2709' WHERE variable = 'ElasticsearchMARCFormat' AND value = 'ISO2709' }); I like the idea of changing ISO2709 to base64ISO2709, as it's annoying to translate between base64ISO2709 and ISO2709 (whereas ARRAY and MARCXML can be matched easily). That said... that seems like a different bug report to me. It'll also conflict with bug 28370 if it gets accepted, so we'll want to be careful there. Created attachment 178370 [details] [review] Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 178371 [details] [review] Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> (In reply to David Cook from comment #44) > Comment on attachment 178145 [details] [review] [review] > Bug 27859: Export staff interface search results when using Elasticsearch > > Review of attachment 178145 [details] [review] [review]: > ----------------------------------------------------------------- > > ::: > installer/data/mysql/atomicupdate/bug_27859- > add_enable_search_result_marc_export_sysprefs.pl > @@ +16,5 @@ > > + $dbh->do(q{ INSERT IGNORE INTO systempreferences (variable, value, options, explanation, type) VALUES ('ElasticsearchSearchResultExportLimit', NULL, NULL, 'Search result export limit', 'integer') }); > > + say $out "Added new system preference 'ElasticsearchSearchResultExportLimit'"; > > + > > + $dbh->do(q{ UPDATE systempreferences SET options = 'base64ISO2709|ARRAY' WHERE variable = 'ElasticsearchMARCFormat' }); > > + $dbh->do(q{ UPDATE systempreferences SET value = 'base64ISO2709' WHERE variable = 'ElasticsearchMARCFormat' AND value = 'ISO2709' }); > > I like the idea of changing ISO2709 to base64ISO2709, as it's annoying to > translate between base64ISO2709 and ISO2709 (whereas ARRAY and MARCXML can > be matched easily). That said... that seems like a different bug report to > me. It'll also conflict with bug 28370 if it gets accepted, so we'll want to > be careful there. Did you link to the wrong bug? Seems it has been pushed and is not related. I agree it's not ideal to have the format name change as part of this bug, I just hoped I could sneak it in I suppose. Would be pretty simple to break it out I suppose. The syspref value actually was base64ISO2709 in bug 19893 before it was removed since I believe we didn't want to allow the user to select a less performant format, but was reintroduced in bug 22258 as ISO2709 when the ARRAY format was added. I might also have added the format name change to this bug as it makes more sense in this context. In the original code the value is never checked, it just defaults to base64ISO2709 if not set to ARRAY, so I guess without the refactoring of the record encoding method the change doesn't make as much sense. Created attachment 178376 [details] [review] Bug 27859: Export staff interface search results when using Elasticsearch Enable export of staff interface search results in different marc formats. This feature is only supported when using Elasticsearch. To test: 1) Apply patch 2) Run installer/data/mysql/updatedatabase.pl 3) Make sure the EnableElasticsearchSearchResultExport system preference is enabled 4) Make sure the current user has the export_catalog permission (Use all tools > Export bibliographic and holdings data) 5) Perform a search 6) Export the serach result by choosing a format under the "Export all results" drop down 7) Verify that link(s) with exported data appear on the background job detail page when the export is completed 8) Revoke the permission in step 4 and ensure exporting is no longer possible 9) Run tests in t/db_dependent/Koha/SearchEngine/Elasticsearch.t Sponsored-by: Gothenburg University Library Created attachment 178377 [details] [review] Bug 27859: (follow-up) Update system preference descriptions Format and edit the descriptions for the new EnableElasticsearchSearchResultExport and ElasticsearchSearchResultExportCustomFormats system preferences. Signed-off-by: David Nind <david@davidnind.com> Now broken out the format change into bug 39171. Failing QA for a couple of things. 1. New features like EnableElasticsearchSearchResultExport should default to off, but it's currently defaulting to on. 2. In "Export all results", "MARC XML" should be "MARCXML" to be consistent with other interfaces. 3. I'm following the test plan but it keeps saying Failed with this report: "Search results export for the query " failed with the following errors:" Created attachment 178399 [details] [review] Bug 27859: Fix background job error and js progress callback Created attachment 178400 [details] [review] Bug 27859: Don't enable feature by default Created attachment 178401 [details] [review] Bug 27859: Fix background job error and js progress callback Created attachment 178402 [details] [review] Bug 27859: Don't enable feature by default Created attachment 178406 [details] [review] Bug 27859: (follow up) Fix background job error and js progress callback Created attachment 178407 [details] [review] Bug 27859: (follow-up) Don't enable feature by default Created attachment 178408 [details] [review] Bug 27859: (follow-up) Correct MARC XML to MARCXML (In reply to David Cook from comment #52) > Failing QA for a couple of things. > > 1. > New features like EnableElasticsearchSearchResultExport should default to > off, but it's currently defaulting to on. Ok, didn't know about that. This has now been fixed. > 2. > In "Export all results", "MARC XML" should be "MARCXML" to be consistent > with other interfaces. Yes, that\s clearly wrong. Fixed there and on the syspref page. > 3. > I'm following the test plan but it keeps saying Failed with this report: > > "Search results export for the query " failed with the following errors:" My mistake, should have tested it again after the rabase. Had some issues with running scheduled jobs in my environment so that's why, but made sure to fix this to be able to test. The error was caused by my changing that the reture value of search_documents_encode is no longer binary but a perl utf8 string, so mod5_hex fails, so need to encode the data before calculating a hash. There has also been some changes to the previous callback arguments of the updateProgress javascript function, which I also had to be adjusted for. What about ElasticsearchSearchResultExportLimit system preference? I can't confirm it works - set it, but it has no impact for limit number of exported records. BTW, shouldn't we change "items" to "records" (or "elements"?) in description ("...search result items (Elasticsearch only).")? |