Attachment 78092 Details for Bug 19893 – Bug 19893: Remove serialization format setting

[patch] Bug 19893: Remove serialization format setting

Bug-19893-Remove-serialization-format-setting.patch (text/plain), 4.47 KB, created by Ere Maijala on 2018-08-23 12:36:02 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Ere Maijala

Created: 2018-08-23 12:36:02 UTC

Size: 4.47 KB

patch

obsolete

>From a39995f1ce93a303d139bd67d479d68e74136525 Mon Sep 17 00:00:00 2001
>From: David Gustafsson <david.gustafsson@ub.gu.se>
>Date: Mon, 28 May 2018 16:03:32 +0200
>Subject: [PATCH] Bug 19893: Remove serialization format setting
>
>Default to base64 encoded binary MARC with MARCXML
>fallback if record exceeds maximum size
>
>Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
>---
> Koha/SearchEngine/Elasticsearch.pm                    | 19 +++++++++++++++++--
> Koha/SearchEngine/Elasticsearch/Search.pm             |  7 +++++--
> admin/searchengine/elasticsearch/field_config.yaml    |  6 ++++++
> .../prog/en/modules/admin/preferences/admin.pref      |  8 --------
> 4 files changed, 28 insertions(+), 12 deletions(-)
>
>diff --git a/Koha/SearchEngine/Elasticsearch.pm b/Koha/SearchEngine/Elasticsearch.pm
>index 93d9213..32ba39b 100644
>--- a/Koha/SearchEngine/Elasticsearch.pm
>+++ b/Koha/SearchEngine/Elasticsearch.pm
>@@ -379,11 +379,26 @@ sub marc_records_to_documents {
>         }
>         # TODO: Perhaps should check if $records_document non empty, but really should never be the case
>         $record->encoding('UTF-8');
>-        if ($serialization_format eq 'base64ISO2709') {
>+        my @warnings;
>+        {
>+            # Temporarily intercept all warn signals (MARC::Record carps when record length > 99999)
>+            local $SIG{__WARN__} = sub {
>+                push @warnings, $_[0];
>+            };
>             $record_document->{'marc_data'} = encode_base64(encode('UTF-8', $record->as_usmarc()));
>         }
>-        else {
>+        if (@warnings) {
>+            # Suppress warnings if record length exceeded
>+            unless (substr($record->leader(), 0, 5) eq '99999') {
>+                foreach my $warning (@warnings) {
>+                    carp($warning);
>+                }
>+            }
>             $record_document->{'marc_data'} = $record->as_xml_record($marcflavour);
>+            $record_document->{'marc_format'} = 'MARCXML';
>+        }
>+        else {
>+            $record_document->{'marc_format'} = 'base64ISO2709';
>         }
>         my $id = $record->subfield('999', 'c');
>         push @record_documents, [$id, $record_document];
>diff --git a/Koha/SearchEngine/Elasticsearch/Search.pm b/Koha/SearchEngine/Elasticsearch/Search.pm
>index 3bd27ea..718eb06 100644
>--- a/Koha/SearchEngine/Elasticsearch/Search.pm
>+++ b/Koha/SearchEngine/Elasticsearch/Search.pm
>@@ -369,11 +369,14 @@ sub decode_record_from_result {
>     # Result is passed in as array, will get flattened
>     # and first element will be $result
>     my ( $self, $result ) = @_;
>-    if (C4::Context->preference('ElasticsearchMARCSerializationFormat') eq 'MARCXML') {
>+    if ($result->{marc_format} eq 'base64ISO2709') {
>+        return MARC::Record->new_from_usmarc(decode_base64($result->{marc_data}));
>+    }
>+    elsif ($result->{marc_format} eq 'MARCXML') {
>         return MARC::Record->new_from_xml($result->{marc_data}, 'UTF-8', uc C4::Context->preference('marcflavour'));
>     }
>     else {
>-        return MARC::Record->new_from_usmarc(decode_base64($result->{marc_data}));
>+        die("Missing marc_format field in Elasticsearch result");
>     }
> }
> 
>diff --git a/admin/searchengine/elasticsearch/field_config.yaml b/admin/searchengine/elasticsearch/field_config.yaml
>index 0d60cd8..0baddf3 100644
>--- a/admin/searchengine/elasticsearch/field_config.yaml
>+++ b/admin/searchengine/elasticsearch/field_config.yaml
>@@ -10,6 +10,12 @@ general:
>       type: text
>       analyzer: keyword
>       index: false
>+    marc_format:
>+      store: true
>+      type: text
>+      analyzer: keyword
>+      index: false
>+
> # Search fields
> search:
>   boolean:
>diff --git a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref
>index c393e4b..2e8dce7 100644
>--- a/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref
>+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/admin.pref
>@@ -425,11 +425,3 @@ Administration:
>               choices:
>                 Zebra: Zebra
>                 Elasticsearch: Elasticsearch
>-        -
>-            - "Use"
>-            - pref: ElasticsearchMARCSerializationFormat
>-              default: MARCXML
>-              choices:
>-                MARCXML: MARCXML
>-                base64ISO2709: base64ISO2709
>-            - "as serialization format for MARC records stored in Elasticsearch index. base64ISO2709 is faster and will use less space but have a maximum record length which could cause issues with very large records."
>-- 
>2.7.4

Actions: View | Diff | Splinter Review

Attachments on bug 19893: 70199 | 70200 | 70201 | 70202 | 70230 | 70325 | 73153 | 75109 | 75271 | 75272 | 75353 | 75428 | 75455 | 75588 | 75658 | 75660 | 76612 | 76613 | 76614 | 76629 | 78091 | 78092 | 78093 | 78094 | 78524 | 78561 | 78587 | 78588 | 78589 | 78590 | 78591 | 78592 | 78593 | 78594 | 78690 | 78691 | 78692 | 78693 | 78694 | 78695 | 78696 | 80969 | 80970 | 80971 | 80983 | 81014 | 81015 | 81284 | 81822 | 81901 | 81903 | 81955 | 81957 | 81958 | 81959 | 81960 | 81961 | 81962 | 81963 | 81964 | 81965 | 81966 | 82143 | 82144 | 82145 | 82146 | 82147 | 82148 | 82149 | 82150 | 82151 | 82152 | 82256 | 82257 | 82258 | 82259 | 82260 | 82261 | 82262 | 82263 | 82264 | 82265 | 82266 | 82267