Attachment 50996 Details for Bug 14899 – Bug 14899: Add the mapping configuration page in the admin module

[patch] Bug 14899: Add the mapping configuration page in the admin module

Bug-14899-Add-the-mapping-configuration-page-in-th.patch (text/plain), 21.39 KB, created by Jonathan Druart on 2016-04-29 14:15:53 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Jonathan Druart

Created: 2016-04-29 14:15:53 UTC

Size: 21.39 KB

patch

obsolete

>From 271b52885aa1f4269fcb97e189e3c9cb16435d06 Mon Sep 17 00:00:00 2001
>From: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
>Date: Fri, 25 Sep 2015 16:06:55 +0100
>Subject: [PATCH] Bug 14899: Add the mapping configuration page in the admin
> module
>
>This new page (admin/searchengine/elasticsearch/mappings.pl) will permit
>to manage the ES mappings.
>For the biblios and authorities indexes, the different mappings can be
>managed from this single page.
>The interface let you add, remove and update mappings and search fields.
>It's also possible to reorder the mappings, as the order can be important
>in the indexation process. Note that the table can be displayed in a
>different order that the one it was before saving, but the mappings are grouped
>by search field and the order inside the search field is preserved.
>
>Limitations:
>- If something went wrong during the insertion/deletion/modification,
>  the users will loose all these changes.
>
>TODO:
>- Add a specific permission (?)
>- Add some data checks client side (JS)
>- Use checkboxes for facet and suggestible (lazy today...)
>- Understand the difference between the 3 values that sortable can have
>  and improve the value for the options in the select box.
>---
> Koha/Objects.pm                                    |  12 +
> admin/searchengine/elasticsearch/mappings.pl       | 137 ++++++++++
> .../admin/searchengine/elasticsearch/mappings.tt   | 298 +++++++++++++++++++++
> 3 files changed, 447 insertions(+)
> create mode 100755 admin/searchengine/elasticsearch/mappings.pl
> create mode 100644 koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt
>
>diff --git a/Koha/Objects.pm b/Koha/Objects.pm
>index 6c03ff5..390d853 100644
>--- a/Koha/Objects.pm
>+++ b/Koha/Objects.pm
>@@ -190,6 +190,18 @@ sub reset {
>     return $self;
> }
> 
>+=head3 Koha::Objects->delete();
>+
>+Koha::Objects->delete();
>+
>+=cut
>+
>+sub delete {
>+    my ( $self ) = @_;
>+
>+    return $self->_resultset->delete;
>+}
>+
> =head3 Koha::Objects->as_list();
> 
> Koha::Objects->as_list();
>diff --git a/admin/searchengine/elasticsearch/mappings.pl b/admin/searchengine/elasticsearch/mappings.pl
>new file mode 100755
>index 0000000..25b3767
>--- /dev/null
>+++ b/admin/searchengine/elasticsearch/mappings.pl
>@@ -0,0 +1,137 @@
>+#!/usr/bin/perl
>+
>+# This file is part of Koha.
>+#
>+# Koha is free software; you can redistribute it and/or modify it
>+# under the terms of the GNU General Public License as published by
>+# the Free Software Foundation; either version 3 of the License, or
>+# (at your option) any later version.
>+#
>+# Koha is distributed in the hope that it will be useful, but
>+# WITHOUT ANY WARRANTY; without even the implied warranty of
>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>+# GNU General Public License for more details.
>+#
>+# You should have received a copy of the GNU General Public License
>+# along with Koha; if not, see <http://www.gnu.org/licenses>.
>+
>+use Modern::Perl;
>+use CGI;
>+use C4::Koha;
>+use C4::Output;
>+use C4::Auth;
>+
>+use Koha::ElasticSearch;
>+use Koha::SearchMarcMaps;
>+use Koha::SearchFields;
>+
>+my $input = new CGI;
>+my ( $template, $borrowernumber, $cookie ) = get_template_and_user(
>+    {   template_name   => 'admin/searchengine/elasticsearch/mappings.tt',
>+        query           => $input,
>+        type            => 'intranet',
>+        authnotrequired => 0,
>+        flagsrequired   => { superlibrarian => 1 },                          # Create a specific permission?
>+    }
>+);
>+
>+my $index = $input->param('index') || 'biblios';
>+my $op    = $input->param('op')    || 'list';
>+my @messages;
>+
>+my $database = Koha::Database->new();
>+my $schema   = $database->schema;
>+
>+my $marc_type = lc C4::Context->preference('marcflavour');
>+
>+if ( $op eq 'edit' ) {
>+
>+    $schema->storage->txn_begin;
>+
>+    my @field_name = $input->param('search_field_name');
>+    my @field_label = $input->param('search_field_label');
>+    my @field_type = $input->param('search_field_type');
>+
>+    my @index_name          = $input->param('mapping_index_name');
>+    my @search_field_name  = $input->param('mapping_search_field_name');
>+    my @mapping_sort        = $input->param('mapping_sort');
>+    my @mapping_facet       = $input->param('mapping_facet');
>+    my @mapping_suggestible = $input->param('mapping_suggestible');
>+    my @mapping_marc_field  = $input->param('mapping_marc_field');
>+
>+    eval {
>+
>+        for my $i ( 0 .. scalar(@field_name) - 1 ) {
>+            my $field_name = $field_name[$i];
>+            my $field_label = $field_label[$i];
>+            my $field_type = $field_type[$i];
>+            # We cannot use Koha::SearchFields because of the ->type method ...
>+            my $search_field = $schema->resultset('SearchField')->find({ name => $field_name }, { key => 'name' });
>+            $search_field->label($field_label);
>+            $search_field->type($field_type);
>+            $search_field->update;
>+        }
>+
>+        Koha::SearchMarcMaps->search( { marc_type => $marc_type, } )->delete;
>+
>+        for my $i ( 0 .. scalar(@index_name) - 1 ) {
>+            my $index_name          = $index_name[$i];
>+            my $search_field_name  = $search_field_name[$i];
>+            my $mapping_marc_field  = $mapping_marc_field[$i];
>+            my $mapping_facet       = $mapping_facet[$i];
>+            my $mapping_suggestible = $mapping_suggestible[$i];
>+            my $mapping_sort        = $mapping_sort[$i];
>+            $mapping_sort = undef if $mapping_sort eq 'undef';
>+
>+            my $search_field = Koha::SearchFields->find({ name => $search_field_name }, { key => 'name' });
>+            # TODO Check mapping format
>+            my $marc_field = Koha::SearchMarcMaps->find_or_create({ index_name => $index_name, marc_type => $marc_type, marc_field => $mapping_marc_field });
>+            $search_field->add_to_search_marc_maps($marc_field, { facet => $mapping_facet, suggestible => $mapping_suggestible, sort => $mapping_sort } );
>+
>+        }
>+    };
>+    if ($@) {
>+        push @messages, { type => 'error', code => 'error_on_update', message => $@, };
>+        $schema->storage->txn_rollback;
>+    } else {
>+        push @messages, { type => 'message', code => 'success_on_update' };
>+        $schema->storage->txn_commit;
>+    }
>+}
>+
>+my @indexes;
>+
>+for my $index_name (qw| biblios authorities |) {
>+    my $search_fields = $schema->resultset('SearchField')->search(
>+        { 'search_marc_map.index_name' => $index_name, 'search_marc_map.marc_type' => $marc_type, },
>+        {   join => { search_marc_to_fields => 'search_marc_map' },
>+            '+select' => [ 'search_marc_to_fields.facet', 'search_marc_to_fields.suggestible', 'search_marc_to_fields.sort', 'search_marc_map.marc_field' ],
>+            '+as'     => [ 'facet',                       'suggestible',                       'sort',                       'marc_field' ],
>+        }
>+    );
>+
>+    my @mappings;
>+    while ( my $s = $search_fields->next ) {
>+        push @mappings,
>+          { search_field_name  => $s->name,
>+            search_field_label => $s->label,
>+            search_field_type  => $s->type,
>+            marc_field         => $s->get_column('marc_field'),
>+            sort               => $s->get_column('sort') // 'undef', # To avoid warnings "Use of uninitialized value in lc"
>+            suggestible        => $s->get_column('suggestible'),
>+            facet              => $s->get_column('facet'),
>+          };
>+    }
>+
>+    push @indexes, { index_name => $index_name, mappings => \@mappings };
>+}
>+
>+my $search_fields = $schema->resultset('SearchField')->search;
>+my @all_search_fields = $search_fields->search( {}, { order_by => ['name'] } );
>+$template->param(
>+    indexes           => \@indexes,
>+    all_search_fields => \@all_search_fields,
>+    messages          => \@messages,
>+);
>+
>+output_html_with_http_headers $input, $cookie, $template->output;
>diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt
>new file mode 100644
>index 0000000..477f8a1
>--- /dev/null
>+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt
>@@ -0,0 +1,298 @@
>+[% INCLUDE 'doc-head-open.inc' %]
>+<title>Koha &rsaquo; Administration &rsaquo; Elastic Search mappings</title>
>+[% INCLUDE 'doc-head-close.inc' %]
>+<script type="text/javascript" src="[% interface %]/lib/jquery/plugins/jquery.tablednd.js"></script>
>+<script type="text/javascript">
>+    function clean_line( line ) {
>+        $(line).find('input[type="text"]').val("");
>+        $(line).find('select').find('option:first').attr("selected", "selected");
>+    }
>+
>+    function clone_line( line ) {
>+        var new_line = $(line).clone();
>+        $(new_line).removeClass("nodrag nodrop");
>+        $(new_line).find('td:last-child>a').removeClass("add").addClass("delete").html(_("Delete"));
>+        $(new_line).find('[data-id]').each( function() {
>+            $(this).attr({ name: $(this).attr('data-id') }).removeAttr('data-id');
>+        } );
>+        $(new_line).find("select").each( function() {
>+            var attr = $(this).attr('name');
>+            var val = $(line).find('[data-id="' + attr + '"]').val();
>+            $(this).find('option[value="' + val + '"]').attr("selected", "selected");
>+        } );
>+        return new_line;
>+    }
>+
>+    $(document).ready(function() {
>+        $("#tabs").tabs();
>+        $('.delete').click(function() {
>+            $(this).parents('tr').remove();
>+        });
>+
>+        $("table.mappings").tableDnD( {
>+            onDragClass: "dragClass",
>+        } );
>+        $('.add').click(function() {
>+            var table = $(this).closest('table');
>+            var index_name   = $(table).attr('data-index_name');
>+            var line = $(this).closest("tr");
>+            var marc_field = $(line).find('input[data-id="mapping_marc_field"]').val();
>+            if ( marc_field.length > 0 ) {
>+                var new_line = clone_line( line );
>+                new_line.appendTo($('table[data-index_name="'+index_name+'"]>tbody'));
>+                $('.delete').click(function() {
>+                    $(this).parents('tr').remove();
>+                });
>+                clean_line(line);
>+
>+                $(table).tableDnD( {
>+                    onDragClass: "dragClass",
>+                } );
>+            }
>+        });
>+    });
>+</script>
>+<style type="text/css">
>+a.add, a.delete {
>+    cursor: pointer;
>+}
>+</style>
>+</head>
>+<body id="admin_searchengine_mappings" class="admin">
>+[% INCLUDE 'header.inc' %]
>+[% INCLUDE 'cat-search.inc' %]
>+
>+<div id="breadcrumbs"><a href="/cgi-bin/koha/mainpage.pl">Home</a> &rsaquo; <a href="/cgi-bin/koha/admin/admin-home.pl">Administration</a> &rsaquo; Search engine configuration</div>
>+
>+<div id="doc3" class="yui-t1">
>+
>+  <div id="bd">
>+    <div id="yui-main">
>+    <div class="yui-b">
>+
>+    [% FOR m IN messages %]
>+      <div class="dialog [% m.type %]">
>+        [% SWITCH m.code %]
>+        [% CASE 'error_on_update' %]
>+          An error occurred when updateing mappings ([% m.message %]).
>+        [% CASE 'error_on_delete' %]
>+          An error occurred when deleting the existing mappings. Nothing has been changed!
>+          (search field [% m.values.field_name %] with mapping [% m.values.marc_field %].)
>+        [% CASE 'success_on_update' %]
>+          Mapping updated successfully.
>+        [% CASE %]
>+          [% m.code %]
>+        [% END %]
>+      </div>
>+    [% END %]
>+
>+    <h1>Search engine configuration</h1>
>+    <div class="warning">
>+        Warning: Any modification in these configurations will need a total reindexation to be fully taken into account !
>+    </div>
>+    [% IF errors %]
>+        <div class="error">
>+        Errors occurred, Modifications does not apply. Please check following values:
>+          <ul>
>+            [% FOREACH e IN errors %]
>+                <li>
>+                    [% IF ( e.type == "malformed_mapping" ) %]
>+                        The value "[% e.value %]" is not supported for mappings
>+                    [% ELSIF ( e.type == "no_mapping" ) %]
>+                        There is no mapping for the index [% e.value %]
>+                    [% END %]
>+                </li>
>+            [% END %]
>+          </ul>
>+        </div>
>+    [% END %]
>+
>+    <form method="post">
>+        <input type="hidden" name="op" value="edit" />
>+        <div id="tabs" class="toptabs" style="clear:both">
>+            <ul>
>+                <li><a href="#search_fields">Search fields</a></li>
>+                [% FOREACH index IN indexes %]
>+                    [% SWITCH index.index_name %]
>+                        [% CASE 'biblios' %]<li><a href="#mapping_biblios">Biblios</a></li>
>+                        [% CASE 'authorities' %]<li><a href="#mapping_authorities">Authorities</a></li>
>+                    [% END %]
>+                [% END %]
>+            </ul>
>+            <div id="search_fields">
>+              <table class="search_fields">
>+                <thead>
>+                  <tr>
>+                    <th>Name</th>
>+                    <th>Label</th>
>+                    <th>Type</th>
>+                  </tr>
>+                </thead>
>+                <tbody>
>+                  [% FOREACH search_field IN all_search_fields %]
>+                    <tr>
>+                      <td>
>+                        <input type="text" name="search_field_name" value="[% search_field.name %]" />
>+                      </td>
>+                      <td><input type="text" name="search_field_label" value="[% search_field.label %]" />
>+                      <td>
>+                        <select name="search_field_type">
>+                          <option value=""></option>
>+                          [% IF search_field.type == "string" %]
>+                            <option value="string" selected="selected">String</option>
>+                          [% ELSE %]
>+                            <option value="string">String</option>
>+                          [% END %]
>+                          [% IF search_field.type == "date" %]
>+                            <option value="date" selected="selected">Date</option>
>+                          [% ELSE %]
>+                            <option value="date">Date</option>
>+                          [% END %]
>+                          [% IF search_field.type == "number" %]
>+                            <option value="number" selected="selected">Number</option>
>+                          [% ELSE %]
>+                            <option value="number">Number</option>
>+                          [% END %]
>+                          [% IF search_field.type == "boolean" %]
>+                            <option value="boolean" selected="selected">Boolean</option>
>+                          [% ELSE %]
>+                            <option value="boolean">Boolean</option>
>+                          [% END %]
>+                          [% IF search_field.type == "sum" %]
>+                            <option value="sum" selected="selected">Sum</option>
>+                          [% ELSE %]
>+                            <option value="sum">Sum</option>
>+                          [% END %]
>+                        </select>
>+                      </td>
>+                    </tr>
>+                  [% END %]
>+                </tbody>
>+              </table>
>+            </div>
>+            [% FOREACH index IN indexes %]
>+                <div id="mapping_[% index.index_name %]">
>+                    <table class="mappings" data-index_name="[% index.index_name%]">
>+                      <thead>
>+                        <tr class="nodrag nodrop">
>+                          <th>Search field</th>
>+                          <th>Sortable</th>
>+                          <th>Facetable</th>
>+                          <th>Suggestible</th>
>+                          <th>Mapping</th>
>+                          <th></th>
>+                        </tr>
>+                      </thead>
>+                      <tbody>
>+                        [% FOREACH mapping IN index.mappings %]
>+                          <tr>
>+                            <td>
>+                              <input type="hidden" name="mapping_index_name" value="[% index.index_name %]" />
>+                              <input type="hidden" name="mapping_search_field_name" value="[% mapping.search_field_name %]">
>+                              [% mapping.search_field_label %]
>+                            </td>
>+                            <td>
>+                              <select name="mapping_sort">
>+                                [% IF mapping.sort == 'undef' %]
>+                                  <option value="undef" selected="selected">Undef</option>
>+                                [% ELSE %]
>+                                  <option value="undef">Undef</option>
>+                                [% END %]
>+                                [% IF mapping.sort == 0 %]
>+                                  <option value="0" selected="selected">0</option>
>+                                [% ELSE %]
>+                                  <option value="0">0</option>
>+                                [% END %]
>+                                [% IF  mapping.sort == 1 %]
>+                                  <option value="1" selected="selected">1</option>
>+                                [% ELSE %]
>+                                  <option value="1">1</option>
>+                                [% END %]
>+                              </select>
>+                            </td>
>+                            <td>
>+                              <select name="mapping_facet">
>+                                [% IF mapping.facet %]
>+                                  <option value="0">No</option>
>+                                  <option value="1" selected="selected">Yes</option>
>+                                [% ELSE %]
>+                                  <option value="0" selected="selected">No</option>
>+                                  <option value="1">Yes</option>
>+                                [% END %]
>+                              </selected>
>+                            </td>
>+                            <td>
>+                              <select name="mapping_suggestible">
>+                                [% IF mapping.suggestible %]
>+                                  <option value="0">No</option>
>+                                  <option value="1" selected="selected">Yes</option>
>+                                [% ELSE %]
>+                                  <option value="0" selected="selected">No</option>
>+                                  <option value="1">Yes</option>
>+                                [% END %]
>+                              </selected>
>+                            </td>
>+                            <td>
>+                                <input name="mapping_marc_field" type="text" value="[% mapping.marc_field %]" />
>+                            </td>
>+                            <td><a class="delete" style="cursor: pointer;">Delete</a></td>
>+                          </tr>
>+                        [% END %]
>+                      </tbody>
>+                      <tfoot>
>+                        <tr class="nodrag nodrop">
>+                          <td>
>+                            <input data-id="mapping_index_name" type="hidden" value="[% index.index_name %]" />
>+                            <select data-id="mapping_search_field_name">
>+                             [% FOREACH f IN all_search_fields %]
>+                               <option value="[% f.name %]">[% f.name %]</option>
>+                             [% END %]
>+                            /select>
>+                          </td>
>+                          <td>
>+                            <select data-id="mapping_sort">
>+                              <option value="undef">Undef</option>
>+                              <option value="0">0</option>
>+                              <option value="1">1</option>
>+                            </select>
>+                          </td>
>+                          <td>
>+                            <select data-id="mapping_facet">
>+                              [% IF mapping.facet %]
>+                                <option value="0">No</option>
>+                                <option value="1" selected="selected">Yes</option>
>+                              [% ELSE %]
>+                                <option value="0" selected="selected">No</option>
>+                                <option value="1">Yes</option>
>+                              [% END %]
>+                            </selected>
>+                          </td>
>+                          <td>
>+                            <select data-id="mapping_suggestible">
>+                              [% IF mapping.suggestible %]
>+                                <option value="0">No</option>
>+                                <option value="1" selected="selected">Yes</option>
>+                              [% ELSE %]
>+                                <option value="0" selected="selected">No</option>
>+                                <option value="1">Yes</option>
>+                              [% END %]
>+                            </selected>
>+                          </td>
>+                          <td><input data-id="mapping_marc_field" type="text" /></td>
>+                          <td><a class="add">Add</a></td>
>+                        </tr>
>+                      </tfoot>
>+                    </table>
>+                </div>
>+            [% END %]
>+        </div>
>+        <p><input type="submit" value="Save" /></p>
>+    </form>
>+</div>
>+
>+</div>
>+<div class="yui-b">
>+[% INCLUDE 'admin-menu.inc' %]
>+</div>
>+</div>
>+[% INCLUDE 'intranet-bottom.inc' %]
>-- 
>2.7.0

Actions: View | Diff | Splinter Review

Attachments on bug 14899: 42882 | 42883 | 42884 | 42885 | 42886 | 43054 | 43055 | 43056 | 43057 | 43058 | 43099 | 43353 | 43354 | 43355 | 50995 | 50996 | 50997 | 55915 | 55916 | 55917 | 55918 | 55957 | 55958 | 55959 | 55960 | 56182 | 56183 | 56194 | 56195 | 56196 | 56197 | 56198 | 56199