Bugzilla – Attachment 90165 Details for
Bug 22639
Elasticsearch ability export mappings and facets settings
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 22639: Elasticsearch ability to export mappings and facets settings
Bug-22639-Elasticsearch-ability-to-export-mappings.patch (text/plain), 19.60 KB, created by
Michal Denar
on 2019-05-28 18:48:39 UTC
(
hide
)
Description:
Bug 22639: Elasticsearch ability to export mappings and facets settings
Filename:
MIME Type:
Creator:
Michal Denar
Created:
2019-05-28 18:48:39 UTC
Size:
19.60 KB
patch
obsolete
>From c79fd2acf86b7896ee55dd95be874dc2afdb50c6 Mon Sep 17 00:00:00 2001 >From: Axel Amghar <axel.amghar@biblibre.com> >Date: Thu, 23 May 2019 11:48:01 +0200 >Subject: [PATCH] Bug 22639: Elasticsearch ability to export mappings and > facets settings > >With this patch you can download search fields , facets order, biblios and authorities table in search engine configuration > >Test plan: >- apply the patch >- make sure you have "elasticsearch" as value for the syspref "SearchEngine" >- go to search engine configuration in the admin page >- in the search fields part, you have a button "Download ... table" >- download in (.csv), (.tab), (.ods) >- verify that you have the same table in your file than in the search fields table >- make the same test for > : biblios table > : facet order table > : authorities table >- Add some mappings, modify some fields, change the facets order >- Save your modifications >- Then dowload again >- Your new files must be modified > >Signed-off-by: Michal Denar <black23@gmail.com> >--- > admin/searchengine/elasticsearch/mappings.pl | 146 +++++++++++++++++++++ > .../admin/searchengine/elasticsearch/mappings.tt | 126 +++++++++++++++++- > 2 files changed, 267 insertions(+), 5 deletions(-) > >diff --git a/admin/searchengine/elasticsearch/mappings.pl b/admin/searchengine/elasticsearch/mappings.pl >index f91399aa9d..244e0f0ad4 100755 >--- a/admin/searchengine/elasticsearch/mappings.pl >+++ b/admin/searchengine/elasticsearch/mappings.pl >@@ -21,12 +21,16 @@ use Scalar::Util qw(looks_like_number); > use List::Util qw( first ); > use C4::Koha; > use C4::Output; >+use C4::Context; > use C4::Auth; >+use Text::CSV::Encoded; >+use C4::Reports::Guided; > > use Koha::SearchEngine::Elasticsearch; > use Koha::SearchEngine::Elasticsearch::Indexer; > use Koha::SearchMarcMaps; > use Koha::SearchFields; >+use Koha::Util::OpenDocument; > > use Try::Tiny; > >@@ -151,6 +155,138 @@ elsif( $op eq 'reset_confirmed' ) { > elsif( $op eq 'reset_confirm' ) { > $template->param( reset_confirm => 1 ); > } >+elsif ($op eq 'download'){ >+ my $format = $input->param("format"); >+ my $table = $input->param("table"); >+ my ($type,$reportfilename, $csv, $ods_fh, $ods_filepath); >+ >+ if ($format eq "tab"){ >+ $type = 'application/octet-stream'; >+ >+ } elsif ($format eq "csv"){ >+ $type = 'application/csv'; >+ my $delimiter = C4::Context->preference('delimiter') || ','; >+ $delimiter = "\t" if ($delimiter eq 'tabulation'); >+ $csv = Text::CSV::Encoded->new({ encoding_out => 'UTF-8', sep_char => $delimiter}); >+ $csv or die "Text::CSV::Encoded->new({binary => 1}) FAILED: " . Text::CSV::Encoded->error_diag(); >+ >+ } elsif ($format eq "ods"){ >+ $type = 'application/vnd.oasis.opendocument.spreadsheet'; >+ $ods_fh = File::Temp->new( UNLINK => 0 ); >+ $ods_filepath = $ods_fh->filename; >+ } >+ >+ #search_fields var >+ my (@search_field_names, @search_fileds_labels, @search_fileds_types, @search_field_weights); >+ #facet_var >+ my (@display_facet, @label_facet); >+ #biblios and authorities var >+ my (@sort, @facet, @suggestible, @marc_field); >+ >+ my @header_row; >+ if ($table eq "search_fields"){ >+ $reportfilename = "search_fields_table"; >+ @search_field_names = $input->multi_param("search_field_name"); >+ @search_fileds_labels = $input->multi_param("search_field_label"); >+ @search_fileds_types = $input->multi_param("search_field_type"); >+ @search_field_weights = $input->multi_param("search_field_weight"); >+ @header_row = ("Name\t\t\t","Label\t\t\t","Type","Weight"); >+ >+ } elsif ($table eq "facet"){ >+ $reportfilename = "facet_order_table"; >+ @search_field_names = $input->multi_param("search_fields_facet"); >+ @label_facet = $input->multi_param("label_facet"); >+ @display_facet = $input->multi_param("display"); >+ @header_row = ("Search field","Label\t\t","Display"); >+ >+ } else { >+ $reportfilename = $table . "_table"; >+ @search_field_names = $input->multi_param("search_field_names_". $table); >+ @sort = $input->multi_param("sort_". $table); >+ @facet = $input->multi_param("facet_". $table); >+ @suggestible = $input->multi_param("suggestible_". $table); >+ @marc_field = $input->multi_param("marc_field_". $table); >+ @header_row = ("Search field\t","Sortable","Facetable","Suggestible","Mapping"); >+ } >+ >+ my $i = 0; >+ my ($content, $ods_content); >+ >+ if ($format eq 'tab') { >+ $reportfilename .= ".tab"; >+ $content .= join("\t\t", @header_row) . "\n"; >+ $content = Encode::decode('UTF-8', $content); >+ >+ foreach my $search_field_name (@search_field_names){ >+ my @row; >+ if ($table eq "search_fields"){ >+ $search_field_name .= ' ' while (length($search_field_name)< 25); >+ $search_fileds_labels[$i] .= ' ' while (length($search_fileds_labels[$i])< 25); >+ @row = ($search_field_name, $search_fileds_labels[$i], $search_fileds_types[$i], $search_field_weights[$i]); >+ } elsif ($table eq "facet"){ >+ $search_field_name .= ' ' while (length($search_field_name)< 15); >+ $label_facet[$i] .= ' ' while (length($label_facet[$i])< 20); >+ @row = ($search_field_name, $label_facet[$i], $display_facet[$i]); >+ } else { >+ $search_field_name .= ' ' while (length($search_field_name)< 20); >+ $sort[$i] .= ' ' while (length($sort[$i])< 15); >+ $facet[$i] .= ' ' while (length($facet[$i])< 15); >+ $suggestible[$i] .= ' ' while (length($suggestible[$i])< 15); >+ @row = ($search_field_name,$sort[$i],$facet[$i],$suggestible[$i],$marc_field[$i]); >+ } >+ $content .= join("\t\t", @row) . "\n"; >+ $i++; >+ } >+ } elsif( $format eq 'csv' ) { >+ $reportfilename .= ".csv"; >+ if ($csv->combine(@header_row)) { >+ $content .= Encode::decode('UTF-8', $csv->string()) . "\n"; >+ foreach my $search_field_name (@search_field_names){ >+ my @row; >+ if ($table eq "search_fields"){ >+ @row = ($search_field_name,$search_fileds_labels[$i],$search_fileds_types[$i],$search_field_weights[$i]); >+ } elsif ($table eq "facet"){ >+ @row = ($search_field_name, $label_facet[$i], $display_facet[$i]); >+ } else { >+ @row = ($search_field_name,$sort[$i],$facet[$i],$suggestible[$i],$marc_field[$i]); >+ } >+ if ($csv->combine(@row)) { >+ $content .= $csv->string() . "\n"; >+ } >+ $i++; >+ } >+ } >+ } elsif ( $format eq 'ods' ) { >+ $reportfilename .= ".ods"; >+ my @headers = @header_row; >+ push @$ods_content, \@headers; >+ >+ foreach my $search_field_name (@search_field_names){ >+ my @content_row; >+ if ($table eq "search_fields"){ >+ push @content_row, Encode::encode( 'UTF8', $search_field_name ), Encode::encode( 'UTF8', $search_fileds_labels[$i] ) , $search_fileds_types[$i],$search_field_weights[$i]; >+ } elsif ($table eq "facet"){ >+ push @content_row, Encode::encode( 'UTF8', $search_field_name ), Encode::encode( 'UTF8', $label_facet[$i] ) , $display_facet[$i]; >+ } else { >+ push @content_row, Encode::encode( 'UTF8', $search_field_name ), $sort[$i] , $facet[$i], $suggestible[$i], Encode::encode( 'UTF8', $marc_field[$i] ); >+ } >+ push @$ods_content, \@content_row; >+ $i++; >+ } >+ generate_ods($ods_filepath, $ods_content); >+ binmode(STDOUT); >+ open $ods_fh, '<', $ods_filepath; >+ $content .= $_ while <$ods_fh>; >+ unlink $ods_filepath; >+ } >+ >+ print $input->header( >+ -type => $type, >+ -attachment=> $reportfilename >+ ); >+ print $content; >+ exit; >+} > > > my @indexes; >@@ -215,11 +351,21 @@ while ( my $search_field = $search_fields->next ) { > push @all_search_fields, $search_field_unblessed; > } > >+my @extensions = ("csv","tab","ods"); >+my @formats_and_extensions =("Semicolon separated test (.csv)","Tab separated text (.tab)","Open Document Spreadsheet (.ods)"); >+my @formats; >+my $j = 0; >+foreach my $extension (@extensions){ >+ push (@formats , { extension => $extension, format_and_extension =>$formats_and_extensions[$j] }); >+ $j++; >+} >+ > $template->param( > indexes => \@indexes, > all_search_fields => \@all_search_fields, > facetable_fields => \@facetable_fields, > messages => \@messages, >+ formats => \@formats, > ); > > output_html_with_http_headers $input, $cookie, $template->output; >diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt >index 33df1fbd8a..018859f1a6 100644 >--- a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt >+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt >@@ -26,6 +26,18 @@ > return new_line; > } > >+ function create_input(value, name){ >+ $("#save_or_reset").append('<input class="remove" type="hidden" value="' + value + '" name="' + name + '"/>' ); >+ } >+ >+ function create_submit(format, table_name){ >+ $(".remove").remove(); >+ create_input("download","op"); >+ create_input(format,"format"); >+ create_input(table_name,"table"); >+ $("#save_or_reset").submit(); >+ } >+ > $(document).ready(function() { > $("#tabs").tabs(); > $('.delete').click(function() { >@@ -56,12 +68,43 @@ > $("#facet_biblios > table").tableDnD( { > onDragClass: "dragClass", > } ); >+ >+ $("#edit").click(function () { >+ $(".remove").remove(); >+ create_input("edit","op"); >+ $("#save_or_reset").submit(); >+ }); >+ $("#reset_confirm").click(function () { >+ $(".remove").remove(); >+ create_input("reset_confirm","op"); >+ $("#save_or_reset").submit(); >+ }); >+ >+ [% FOREACH format IN formats %] >+ $("#link_[% format.extension | html %]").click( function () { >+ create_submit("[% format.extension | html %]","search_fields"); >+ }); >+ $("#link_[% format.extension | html %]_facet").click( function () { >+ create_submit("[% format.extension | html %]","facet"); >+ }); >+ >+ [% FOREACH index IN indexes %] >+ $("#link_[% index.index_name | html %]_[% format.extension | html %]").click( function () { >+ create_submit("[% format.extension | html %]","[% index.index_name | html %]"); >+ }); >+ [% END %] >+ [% END %] > }); > </script> > <style> > a.add, a.delete { > cursor: pointer; > } >+ >+#inline { >+ display : inline; >+} >+ > </style> > </head> > <body id="admin_searchengine_mappings" class="admin"> >@@ -142,7 +185,7 @@ a.add, a.delete { > </form> > </div> > [% END %] >- <form method="post"> >+ <form method="post" id="save_or_reset" > > <div id="tabs" class="toptabs" style="clear:both"> > <ul> > <li><a href="#search_fields">Search fields</a></li> >@@ -154,6 +197,20 @@ a.add, a.delete { > [% END %] > </ul> > <div id="search_fields"> >+ <div id="inline"> >+ <button class="btn btn-default" type="submit" name="op" value="edit"><i class="fa fa-hdd-o" aria-hidden="true"></i> Save</button> >+ <button class="btn btn-default" type="submit" name="op" value="reset_confirm"><i class="fa fa-refresh" aria-hidden="true"></i> Reset Mappings</button> >+ >+ <div class="btn-group"> >+ <button class="btn btn-default dropdown-toggle" data-toggle="dropdown" class="format"><i class="fa fa-upload"></i> Download Search fields table <span class="caret"></span></button> >+ <ul class="dropdown-menu"> >+ [% FOREACH format IN formats%] >+ <li><a href="#" id="link_[% format.extension | html %]">[% format.format_and_extension | html%]</a></li> >+ [% END %] >+ </ul> >+ </div><br/> >+ </div><br/> >+ > <table class="search_fields"> > <thead> > <tr> >@@ -222,8 +279,43 @@ a.add, a.delete { > </tbody> > </table> > </div> >+ > [% FOREACH index IN indexes %] > <div id="mapping_[% index.index_name | html %]"> >+ <div id="inline"> >+ <button class="btn btn-default" type="submit" name="op" value="edit"><i class="fa fa-hdd-o" aria-hidden="true"></i> Save</button> >+ <button class="btn btn-default" type="submit" name="op" value="reset_confirm"><i class="fa fa-refresh" aria-hidden="true"></i> Reset Mappings</button> >+ >+ <div class="btn-group"> >+ <button class="btn btn-default dropdown-toggle" data-toggle="dropdown" class="format"><i class="fa fa-upload"></i> Download [% index.index_name | html %] table <span class="caret"></span></button> >+ <ul class="dropdown-menu"> >+ [% FOREACH format IN formats%] >+ <li><a href="#" id="link_[% index.index_name | html %]_[% format.extension | html %]">[% format.format_and_extension | html%]</a></li> >+ [% END %] >+ </ul> >+ </div> >+ >+ [% IF index.index_name == "biblios" %] >+ <div class="btn-group"> >+ <button class="btn btn-default dropdown-toggle" data-toggle="dropdown" class="format"><i class="fa fa-upload"></i> Download facet order table <span class="caret"></span></button> >+ <ul class="dropdown-menu"> >+ [% FOREACH format IN formats%] >+ <li><a href="#" id="link_[% format.extension | html %]_facet">[% format.format_and_extension | html%]</a></li> >+ [% END %] >+ </ul> >+ </div> >+ [% END %] >+ <br/> >+ </div><br/> >+ >+ [% FOREACH mappings IN index.mappings %] >+ <input type="hidden" value="[% mappings.search_field_name | html %]" name="search_field_names_[% index.index_name | html %]" /> >+ <input type="hidden" value="[% mappings.sort | html %]" name="sort_[% index.index_name | html %]" /> >+ <input type="hidden" value="[% mappings.facet | html %]" name="facet_[% index.index_name | html %]" /> >+ <input type="hidden" value="[% mappings.suggestible | html %]" name="suggestible_[% index.index_name | html %]" /> >+ <input type="hidden" value="[% mappings.marc_field | html %]" name="marc_field_[% index.index_name | html %]" /> >+ [% END %] >+ > <table class="mappings" data-index_name="[% index.index_name | html %]"> > <thead> > <tr class="nodrag nodrop"> >@@ -357,27 +449,50 @@ a.add, a.delete { > <tr> > <td> > [% f.name | html %] >+ <input type="hidden" value="[% f.name | html %]" name="search_fields_facet" /> > </td> > <td> > [% SWITCH f.name %] > [% CASE 'author' %]Authors >+ <input type="hidden" value="Authors" name="label_facet" /> >+ > [% CASE 'itype' %]Item Types >+ <input type="hidden" value="Item Types" name="label_facet" /> >+ > [% CASE 'location' %]Locations >+ <input type="hidden" value="Locations" name="label_facet" /> >+ > [% CASE 'su-geo' %]Places >+ <input type="hidden" value="Places" name="label_facet" /> >+ > [% CASE 'title-series' %]Series >+ <input type="hidden" value="Series" name="label_facet" /> >+ > [% CASE 'subject' %]Topics >+ <input type="hidden" value="Topics" name="label_facet" /> >+ > [% CASE 'ccode' %]Collections >+ <input type="hidden" value="Collections" name="label_facet" /> >+ > [% CASE 'holdingbranch' %]Holding libraries >+ <input type="hidden" value="Holding libraries" name="label_facet" /> >+ > [% CASE 'homebranch' %]Home libraries >+ <input type="hidden" value="Home libraries" name="label_facet" /> >+ > [% CASE 'ln' %]Language >+ <input type="hidden" value="Language" name="label_facet" /> > [% CASE %][% f | html %] > [% END %] >+ > </td> > <td> > [% IF f.facet_order %] > <input type="checkbox" name="display_facet" value="[% f.name | html %]" checked="checked" /> >+ <input type="hidden" value="1" name="display" /> > [% ELSE %] > <input type="checkbox" name="display_facet" value="[% f.name | html %]" /> >+ <input type="hidden" value="0" name="display" /> > [% END %] > </td> > </tr> >@@ -389,12 +504,13 @@ a.add, a.delete { > </div> > [% END %] > </div> >- <p> >- <button class="btn btn-default" type="submit" name="op" value="edit"><i class="fa fa-hdd-o" aria-hidden="true"></i> Save</button> >- <button class="btn btn-default" type="submit" name="op" value="reset_confirm"><i class="fa fa-refresh" aria-hidden="true"></i> Reset Mappings</button> >- </p> >+ > </form> > >+ <p> >+ <button id ="edit" class="btn btn-default" type="submit" ><i class="fa fa-hdd-o" aria-hidden="true"></i> Save</button> >+ <button id="reset_confirm" class="btn btn-default" type="submit"><i class="fa fa-refresh" aria-hidden="true"></i> Reset Mappings</button> >+ </p> > </main> > </div> <!-- /.col-sm-10.col-sm-push-2 --> > >-- >2.11.0
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=90165">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 22639
:
90022
|
90052
|
90151
|
90165
|
90601
|
90602