Bug 25669 - ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)
Summary: ElasticSearch 6: [types removal] Specifying types in put mapping requests is ...
Status: Needs documenting
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Kevin Carnes
QA Contact: Martin Renvoize
URL:
Keywords: release-notes-needed
: 27771 (view as bug list)
Depends on:
Blocks: 25439
  Show dependency treegraph
 
Reported: 2020-06-03 19:03 UTC by Victor Grousset/tuxayo
Modified: 2023-05-10 16:14 UTC (History)
16 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00, 22.05.04, 21.11.12, 21.05.18


Attachments
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7) (3.87 KB, patch)
2022-02-08 14:40 UTC, Kevin Carnes
Details | Diff | Splinter Review
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7) (3.86 KB, patch)
2022-02-08 15:19 UTC, Kevin Carnes
Details | Diff | Splinter Review
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7) (14.31 KB, patch)
2022-02-09 08:41 UTC, Kevin Carnes
Details | Diff | Splinter Review
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7) (14.37 KB, patch)
2022-02-18 08:28 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7) (15.24 KB, patch)
2022-02-22 12:39 UTC, Kevin Carnes
Details | Diff | Splinter Review
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7) (15.31 KB, patch)
2022-02-23 14:45 UTC, Kevin Carnes
Details | Diff | Splinter Review
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7) (15.34 KB, patch)
2022-03-08 15:15 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 25669: [alternate] Use include_type_name parameter in ES calls (7.04 KB, patch)
2022-03-28 12:07 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 25669: [alternate] Use include_type_name parameter in ES calls (small fixes) (6.85 KB, patch)
2022-05-25 09:24 UTC, Kevin Carnes
Details | Diff | Splinter Review
Bug 25669: [alternate] Use include_type_name parameter in ES calls (7.10 KB, patch)
2022-06-01 13:12 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 25669: [alternate] Use include_type_name parameter in ES calls (small fixes) (6.91 KB, patch)
2022-06-01 13:12 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 25669: [alternate] Use include_type_name parameter in ES calls (7.16 KB, patch)
2022-06-22 13:51 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 25669: [alternate] Use include_type_name parameter in ES calls (small fixes) (6.97 KB, patch)
2022-06-22 13:51 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 25669: Use include_type_name parameter in ES calls (7.20 KB, patch)
2022-06-23 14:35 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 25669: (follow-up) Minor fixes (6.98 KB, patch)
2022-06-23 14:35 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 25669: [21.11.x] Use include_type_name parameter in ES calls (7.48 KB, patch)
2022-08-20 01:50 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 25669: [21.11.x] (follow-up) Minor fixes (7.23 KB, patch)
2022-08-20 01:51 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Victor Grousset/tuxayo 2020-06-03 19:03:11 UTC
This happens in koha-testing-docker with ES 6.8.8
When indexing:
./misc/search_tools/rebuild_elasticsearch.pl -v -d

Here are both deprecation messages that are outputted and related:


[DEPRECATION] [types removal] Specifying types in put mapping requests is deprecated.
To be compatible with 7.0, the mapping definition should not be nested under the type name, and the parameter include_type_name must be provided and set to false.
- In request: {body => {data => {properties => {abstract => {analyzer => "analyzer_standard",fields => {ci_raw => {normalizer => "icu_folding_normalizer",type => "keyword"},phrase => {analyzer => "analyzer_phrase",search_analyzer => "analyzer_phrase",type => "text"},raw => {normalizer => "nfkc_cf_normalizer",type => "keyword"}},search_analyzer => "analyzer_standard",type => "text"},abstract__sort => {index => "false",numeric => "true",type => "icu_collation_keyword"},acqsource => $VAR1->{body}{data}{properties}{abstract},acqsource__sort => $VAR1->{body}{data}{properties}{abstract__sort},arl => $VAR1->{body}{data}{properties}{abstract},arl__sort => $VAR1->{body}{data}{properties}{abstract__sort},arp => $VAR1->{body}{data}{properties}{abstract},arp__sort => $VAR1->{body}{data}{properties}{abstract_
_sort},author => $VAR1->{body}{data}{properties}{abstract},"author-in-order" => $VAR1->{body}{data}{properties}{abstract},"author-in-order__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"author-name-corporate" => $VAR1->{body}{data}{properties}{abstract},"author-name-corporate__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"author-name-personal" => $VAR1->{body}{data}{properties}{abstract},"author-name-personal__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"author-personal-bibliography" => $VAR1->{body}{data}{properties}{abstract},"author-personal-bibliography__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"author-title" => $VAR1->{body}{data}{properties}{abstract},"author-title__sort" => $VAR1->{body}{data}{properties}{abstract__sort},author__facet => {type => "keyword"},author__sort => $VAR1->{body}{data}{properties}{abstract__sort},author__suggestion => {analyzer => "simple",max_input_length => 100,search_analyzer => "simple",type => "completion"},barcode => $VAR1->{body}{data}{properties}{abstract},barcode__sort => $VAR1->{body}{data}{properties}{abstract__sort},"bgf-number" => $VAR1->{body}{data}{properties}{abstract},"bgf-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"bib-level" => $VAR1->{body}{data}{properties}{abstract},"bib-level__sort" => $VAR1->{body}{data}{properties}{abstract__sort},biblioitemnumber => $VAR1->{body}{data}{properties}{abstract},biblioitemnumber__sort => $VAR1->{body}{data}{properties}{abstract__sort},bio => $VAR1->{body}{data}{properties}{abstract},bio__sort => $VAR
1->{body}{data}{properties}{abstract__sort},"bnb-card-number" => $VAR1->{body}{data}{properties}{abstract},"bnb-card-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},ccode => $VAR1->{body}{data}{properties}{abstract},ccode__facet => $VAR1->{body}{data}{properties}{author__facet},ccode__sort => $VAR1->{body}{data}{properties}{abstract__sort},"classification-source" => $VAR1->{body}{data}{properties}{abstract},"classification-source__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cn-bib-sort" => $V
AR1->{body}{data}{properties}{abstract},"cn-bib-sort__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cn-bib-source" => $VAR1->{body}{data}{properties}{abstract},"cn-bib-source__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cn-class" => $VAR1->{body}{data}{properties}{abstract},"cn-class__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cn-item" => $VAR1->{body}{data}{properties}{abstract},"cn-item__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cn-prefix" => $VAR1->{body}{data}{properties}{abstract},"cn-prefix__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cn-sort" => $VAR1->{body}{data}{properties}{abstract},"cn-sort__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cn-suffix" => $VAR1->{body}{data}{properties}{abstract},"cn-suffix__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"code-geographic" => $VAR1->{body}{data}{properties}{abstract},"code-geographic__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"code-institution" => $VAR1->{body}{data}{properties}{abstract},"code-institution__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"coded-location-qualifier" => $VAR1->{body}{data}{properties}{abstract},"coded-location-qualifier__sort" => $VAR1->{body}{data}{properties}{abstract__sort},coden => $VAR1->{body}{data}{properties}{abstract},coden__sort => $VAR1->{body}{data}{properties}{abstract__sort},"conference-name" => $VAR1->{body}{data}{properties}{abstract},"conference-name__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"control-number" => $VAR1->{body}
{data}{properties}{abstract},"control-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},copydate => $VAR1->{body}{data}{properties}{abstract},copydate__facet => $VAR1->{body}{data}{properties}{author__facet},copydate__sort => $VAR1->{body}{data}{properties}{abstract__sort},copynumber => $VAR1->{body}{data}{properties}{abstract},copynumber__sort => $VAR1->{body}{data}{properties}{abstract__sort},"corporate-name" => $VAR1->{body}{data}{properties}{abstract},"corporate-name__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"cross-reference" => $VAR1->{body}{data}{properties}{abstract},"cross-reference__sort" => $VAR1->{body}{data}{properties}{abstract__sort},ctype => $VAR1->{body}{data}{properties}{abstract},ctype__sort => $VAR1->{body}{data}{properties}{abstract__sort},curriculum => $VAR1->{body}{data}{properties}{abstract},curriculum__sort => $VAR1->{body}{data}{properties}{abstract__sort},damaged => $VAR1->{body}{data}{properties}{abstract},damaged__sort => $VAR1->{body}{data}{properties}{abstract__sort},"date-entered
-on-file" => $VAR1->{body}{data}{properties}{abstract},"date-entered-on-file__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"date-of-acquisition" => $VAR1->{body}{data}{properties}{abstract},"date-of-acquisition__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"date-of-publication" => $VAR1->{body}{data}{properties}{abstract},"date-of-publication__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"date-time-last-modified" => $VAR1->{body}{data}{properties}{abstract},"date-time-last-modified__sort
" => $VAR1->{body}{data}{properties}{abstract__sort},datelastborrowed => $VAR1->{body}{data}{properties}{abstract},datelastborrowed__sort => $VAR1->{body}{data}{properties}{abstract__sort},datelastseen => $VAR1->{body}{data}{properties}{abstract},datelastseen__sort => $VAR1->{body}{data}{properties}{abstract__sort},"dewey-classification" => $VAR1->{body}{data}{properties}{abstract},"dewey-classification__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"dissertation-information" => $VAR1->{body}{data}{properties}{a
bstract},"dissertation-information__sort" => $VAR1->{body}{data}{properties}{abstract__sort},editor => $VAR1->{body}{data}{properties}{abstract},editor__sort => $VAR1->{body}{data}{properties}{abstract__sort},extent => $VAR1->{body}{data}{properties}{abstract},extent__sort => $VAR1->{body}{data}{properties}{abstract__sort},"ff7-00" => $VAR1->{body}{data}{properties}{abstract},"ff7-00__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"ff7-01" => $VAR1->{body}{data}{properties}{abstract},"ff7-01-02" => $VAR1->{body}{
data}{properties}{abstract},"ff7-01-02__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"ff7-01__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"ff7-02" => $VAR1->{body}{data}{properties}{abstract},"ff7-02__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"ff8-23" => $VAR1->{body}{data}{properties}{abstract},"ff8-23__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"ff8-29" => $VAR1->{body}{data}{properties}{abstract},"ff8-29__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"geog
raphic-class" => $VAR1->{body}{data}{properties}{abstract},"geographic-class__sort" => $VAR1->{body}{data}{properties}{abstract__sort},holdingbranch => $VAR1->{body}{data}{properties}{abstract},holdingbranch__facet => $VAR1->{body}{data}{properties}{author__facet},homebranch => $VAR1->{body}{data}{properties}{abstract},homebranch__facet => $VAR1->{body}{data}{properties}{author__facet},"host-item" => $VAR1->{body}{data}{properties}{abstract},"host-item-number" => {null_value => 0,type => "integer"},"host-item-number__sort"
 => $VAR1->{body}{data}{properties}{abstract__sort},"host-item__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"identifier-other" => $VAR1->{body}{data}{properties}{abstract},"identifier-other__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"identifier-publisher-for-music" => $VAR1->{body}{data}{properties}{abstract},"identifier-publisher-for-music__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"identifier-standard" => {analyzer => "analyzer_stdno",fields => {phrase => {analyzer => "analyzer_
phrase",search_analyzer => "analyzer_phrase",type => "text"},raw => {type => "keyword"}},search_analyzer => "analyzer_stdno",type => "text"},"identifier-standard__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"index-term-genre" => $VAR1->{body}{data}{properties}{abstract},"index-term-genre__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"index-term-uncontrolled" => $VAR1->{body}{data}{properties}{abstract},"index-term-uncontrolled__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"indexed-by" =
> $VAR1->{body}{data}{properties}{abstract},"indexed-by__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"interest-age-level" => $VAR1->{body}{data}{properties}{abstract},"interest-age-level__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"interest-grade-level" => $VAR1->{body}{data}{properties}{abstract},"interest-grade-level__sort" => $VAR1->{body}{data}{properties}{abstract__sort},isbn => $VAR1->{body}{data}{properties}{"identifier-standard"},isbn__sort => $VAR1->{body}{data}{properties}{abstract__sor
t},issn => $VAR1->{body}{data}{properties}{"identifier-standard"},issn__sort => $VAR1->{body}{data}{properties}{abstract__sort},issues => $VAR1->{body}{data}{properties}{"host-item-number"},issues__sort => $VAR1->{body}{data}{properties}{abstract__sort},itemnumber => $VAR1->{body}{data}{properties}{"host-item-number"},itemtype => $VAR1->{body}{data}{properties}{abstract},itemtype__sort => $VAR1->{body}{data}{properties}{abstract__sort},itype => $VAR1->{body}{data}{properties}{abstract},itype__facet => $VAR1->{body}{data}{p
roperties}{author__facet},itype__sort => $VAR1->{body}{data}{properties}{abstract__sort},"koha-auth-number" => $VAR1->{body}{data}{properties}{abstract},"koha-auth-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"language-original" => $VAR1->{body}{data}{properties}{abstract},"language-original__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"lc-call-number" => $VAR1->{body}{data}{properties}{abstract},"lc-call-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"lc-card-number" => $
VAR1->{body}{data}{properties}{abstract},"lc-card-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"lexile-number" => $VAR1->{body}{data}{properties}{abstract},"lexile-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},lf => $VAR1->{body}{data}{properties}{abstract},lf__sort => $VAR1->{body}{data}{properties}{abstract__sort},llength => $VAR1->{body}{data}{properties}{abstract},llength__sort => $VAR1->{body}{data}{properties}{abstract__sort},ln => $VAR1->{body}{data}{properties}{abstract},"ln-a
udio" => $VAR1->{body}{data}{properties}{abstract},"ln-audio__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"ln-subtitle" => $VAR1->{body}{data}{properties}{abstract},"ln-subtitle__sort" => $VAR1->{body}{data}{properties}{abstract__sort},ln__facet => $VAR1->{body}{data}{properties}{author__facet},ln__sort => $VAR1->{body}{data}{properties}{abstract__sort},"local-classification" => $VAR1->{body}{data}{properties}{abstract},"local-classification__suggestion" => $VAR1->{body}{data}{properties}{author__suggestion},"
local-number" => $VAR1->{body}{data}{properties}{abstract},location => $VAR1->{body}{data}{properties}{abstract},location__facet => $VAR1->{body}{data}{properties}{author__facet},lost => $VAR1->{body}{data}{properties}{abstract},lost__sort => $VAR1->{body}{data}{properties}{abstract__sort},"map-scale" => $VAR1->{body}{data}{properties}{abstract},"map-scale__sort" => $VAR1->{body}{data}{properties}{abstract__sort},marc_data => {analyzer => "keyword",index => "false",store => "true",type => "text"},marc_data_array => {dynami
c => "true",type => "object"},marc_format => {analyzer => "keyword",index => "false",store => "true",type => "text"},"material-type" => $VAR1->{body}{data}{properties}{abstract},"material-type__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"materials-specified" => $VAR1->{body}{data}{properties}{abstract},"materials-specified__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"microform-generation" => $VAR1->{body}{data}{properties}{abstract},"microform-generation__sort" => $VAR1->{body}{data}{properties}
{abstract__sort},"music-key" => $VAR1->{body}{data}{properties}{abstract},"music-key__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"nal-call-number" => $VAR1->{body}{data}{properties}{abstract},"nal-call-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},name => $VAR1->{body}{data}{properties}{abstract},"name-and-title" => $VAR1->{body}{data}{properties}{abstract},"name-and-title__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"name-geographic" => $VAR1->{body}{data}{properties}{abstract
},"name-geographic__sort" => $VAR1->{body}{data}{properties}{abstract__sort},name__sort => $VAR1->{body}{data}{properties}{abstract__sort},"nlm-call-number" => $VAR1->{body}{data}{properties}{abstract},"not-onloan-count" => $VAR1->{body}{data}{properties}{abstract},"not-onloan-count__sort" => $VAR1->{body}{data}{properties}{abstract__sort},note => $VAR1->{body}{data}{properties}{abstract},note__sort => $VAR1->{body}{data}{properties}{abstract__sort},notforloan => $VAR1->{body}{data}{properties}{"host-item-number"},"number-
db" => $VAR1->{body}{data}{properties}{abstract},"number-db__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"number-govt-pub" => $VAR1->{body}{data}{properties}{abstract},"number-govt-pub__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"number-legal-deposit" => $VAR1->{body}{data}{properties}{abstract},"number-legal-deposit__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"number-local-acquisition" => $VAR1->{body}{data}{properties}{abstract},"number-local-acquisition__sort" => $VAR1->{body}{da
ta}{properties}{abstract__sort},"number-natl-biblio" => $VAR1->{body}{data}{properties}{abstract},"number-natl-biblio__sort" => $VAR1->{body}{data}{properties}{abstract__sort},onloan => {null_value => "false",type => "boolean"},"other-control-number" => $VAR1->{body}{data}{properties}{abstract},"other-control-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"personal-name" => $VAR1->{body}{data}{properties}{abstract},"personal-name__sort" => $VAR1->{body}{data}{properties}{abstract__sort},pl => $VAR1->{body
}{data}{properties}{abstract},pl__sort => $VAR1->{body}{data}{properties}{abstract__sort},price => $VAR1->{body}{data}{properties}{abstract},price__sort => $VAR1->{body}{data}{properties}{abstract__sort},provider => $VAR1->{body}{data}{properties}{abstract},provider__sort => $VAR1->{body}{data}{properties}{abstract__sort},publisher => $VAR1->{body}{data}{properties}{abstract},publisher__facet => $VAR1->{body}{data}{properties}{author__facet},publisher__sort => $VAR1->{body}{data}{properties}{abstract__sort},"reading-grade-
level" => $VAR1->{body}{data}{properties}{abstract},"reading-grade-level__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"record-control-number" => $VAR1->{body}{data}{properties}{abstract},"record-control-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"record-source" => $VAR1->{body}{data}{properties}{abstract},"record-source__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"related-periodical" => $VAR1->{body}{data}{properties}{abstract},"related-periodical__sort" => $VAR1->{body}{dat
a}{properties}{abstract__sort},renewals => $VAR1->{body}{data}{properties}{abstract},renewals__sort => $VAR1->{body}{data}{properties}{abstract__sort},replacementprice => $VAR1->{body}{data}{properties}{abstract},replacementprice__sort => $VAR1->{body}{data}{properties}{abstract__sort},replacementpricedate => $VAR1->{body}{data}{properties}{abstract},replacementpricedate__sort => $VAR1->{body}{data}{properties}{abstract__sort},"report-number" => $VAR1->{body}{data}{properties}{abstract},"report-number__sort" => $VAR1->{bod
y}{data}{properties}{abstract__sort},reserves => $VAR1->{body}{data}{properties}{abstract},reserves__sort => $VAR1->{body}{data}{properties}{abstract__sort},restricted => $VAR1->{body}{data}{properties}{abstract},restricted__sort => $VAR1->{body}{data}{properties}{abstract__sort},rtype => $VAR1->{body}{data}{properties}{abstract},rtype__sort => $VAR1->{body}{data}{properties}{abstract__sort},stack => $VAR1->{body}{data}{properties}{abstract},stack__sort => $VAR1->{body}{data}{properties}{abstract__sort},"stock-number" => $
VAR1->{body}{data}{properties}{abstract},"stock-number__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"su-geo" => $VAR1->{body}{data}{properties}{abstract},"su-geo__facet" => $VAR1->{body}{data}{properties}{author__facet},"su-geo__sort" => $VAR1->{body}{data}{properties}{abstract__sort},subject => $VAR1->{body}{data}{properties}{abstract},"subject-name-personal" => $VAR1->{body}{data}{properties}{abstract},"subject-name-personal__sort" => $VAR1->{body}{data}{properties}{abstract__sort},subject__facet => $VAR1->{
body}{data}{properties}{author__facet},subject__sort => $VAR1->{body}{data}{properties}{abstract__sort},subject__suggestion => $VAR1->{body}{data}{properties}{author__suggestion},suppress => $VAR1->{body}{data}{properties}{onloan},suppress__sort => $VAR1->{body}{data}{properties}{abstract__sort},ta => $VAR1->{body}{data}{properties}{abstract},ta__sort => $VAR1->{body}{data}{properties}{abstract__sort},"thematic-number" => $VAR1->{body}{data}{properties}{abstract},"thematic-number__sort" => $VAR1->{body}{data}{properties}{a
bstract__sort},title => $VAR1->{body}{data}{properties}{abstract},"title-abbreviated" => $VAR1->{body}{data}{properties}{abstract},"title-abbreviated__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-collective" => $VAR1->{body}{data}{properties}{abstract},"title-collective__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-cover" => $VAR1->{body}{data}{properties}{abstract},"title-cover__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-expanded" => $VAR1->{body}{data}{properties
}{abstract},"title-expanded__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-former" => $VAR1->{body}{data}{properties}{abstract},"title-former__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-key" => $VAR1->{body}{data}{properties}{abstract},"title-key__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-later" => $VAR1->{body}{data}{properties}{abstract},"title-later__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-other-variant" => $VAR1->{body}{data}{propertie
s}{abstract},"title-other-variant__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-series" => $VAR1->{body}{data}{properties}{abstract},"title-series__facet" => $VAR1->{body}{data}{properties}{author__facet},"title-series__sort" => $VAR1->{body}{data}{properties}{abstract__sort},"title-uniform" => $VAR1->{body}{data}{properties}{abstract},"title-uniform__sort" => $VAR1->{body}{data}{properties}{abstract__sort},title__sort => $VAR1->{body}{data}{properties}{abstract__sort},title__suggestion => $VAR1->{body}{
data}{properties}{author__suggestion},totalissues => $VAR1->{body}{data}{properties}{abstract},totalissues__sort => $VAR1->{body}{data}{properties}{abstract__sort},"udc-classification" => $VAR1->{body}{data}{properties}{abstract},"udc-classification__sort" => $VAR1->{body}{data}{properties}{abstract__sort},uri => $VAR1->{body}{data}{properties}{abstract},uri__sort => $VAR1->{body}{data}{properties}{abstract__sort},withdrawn => $VAR1->{body}{data}{properties}{abstract},withdrawn__sort => $VAR1->{body}{data}{properties}{abst
ract__sort}}}},ignore => [],method => "PUT",mime_type => "application/json",path => "/koha_kohadev_biblios/_mapping/data",qs => {},serialize => "std"}



[DEPRECATION] [types removal] Specifying types in put mapping requests is deprecated.
To be compatible with 7.0, the mapping definition should not be nested under the type name, and the parameter include_type_name must be provided and set to false.
- In request: {body => {data => {properties => {authtype => {analyzer => "analyzer_standard",fields => {ci_raw => {normalizer => "icu_folding_normalizer",type => "keyword"},phrase => {analyzer => "analyzer_phrase",search_analyzer => "analyzer_phrase",type => "text"},raw => {normalizer => "nfkc_cf_normalizer",type => "keyword"}},search_analyzer => "analyzer_standard",type => "text"},authtype__sort => {index => "false",numeric => "true",type => "icu_collation_keyword"},"corporate-name-see-also-from" => $VAR1->{body}{data}{properties}{authtype},"corporate-name-see-also-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"corporate-name-see-from" => $VAR1->{body}{data}{properties}{authtype},"corporate-name-see-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"descriptive-cata
loging-rules" => $VAR1->{body}{data}{properties}{authtype},"descriptive-cataloging-rules__sort" => $VAR1->{body}{data}{properties}{authtype__sort},heading => $VAR1->{body}{data}{properties}{authtype},"heading-main" => $VAR1->{body}{data}{properties}{authtype},"heading-main__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"heading-use-main-or-added-entry" => $VAR1->{body}{data}{properties}{authtype},"heading-use-main-or-added-entry__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"heading-use-series-added-
entry" => $VAR1->{body}{data}{properties}{authtype},"heading-use-series-added-entry__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"heading-use-subject-added-entry" => $VAR1->{body}{data}{properties}{authtype},"heading-use-subject-added-entry__sort" => $VAR1->{body}{data}{properties}{authtype__sort},heading__sort => $VAR1->{body}{data}{properties}{authtype__sort},"kind-of-record" => $VAR1->{body}{data}{properties}{authtype},"kind-of-record__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"lc-card-number
" => $VAR1->{body}{data}{properties}{authtype},"lc-card-number__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"local-number" => $VAR1->{body}{data}{properties}{authtype},"local-number__sort" => $VAR1->{body}{data}{properties}{authtype__sort},marc_data => {analyzer => "keyword",index => "false",store => "true",type => "text"},marc_data_array => {dynamic => "true",type => "object"},marc_format => {analyzer => "keyword",index => "false",store => "true",type => "text"},match => $VAR1->{body}{data}{properties}{authty
pe},"match-heading" => $VAR1->{body}{data}{properties}{authtype},"match-heading-see-from" => $VAR1->{body}{data}{properties}{authtype},"match-heading-see-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"match-heading__sort" => $VAR1->{body}{data}{properties}{authtype__sort},match__sort => $VAR1->{body}{data}{properties}{authtype__sort},"meeting-name" => $VAR1->{body}{data}{properties}{authtype},"meeting-name-heading" => $VAR1->{body}{data}{properties}{authtype},"meeting-name-heading__sort" => $VAR1->{body}{d
ata}{properties}{authtype__sort},"meeting-name-see-also-from" => $VAR1->{body}{data}{properties}{authtype},"meeting-name-see-also-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"meeting-name-see-from" => $VAR1->{body}{data}{properties}{authtype},"meeting-name-see-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"meeting-name__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"personal-name" => $VAR1->{body}{data}{properties}{authtype},"personal-name-heading" => $VAR1->{body}{data}{prope
rties}{authtype},"personal-name-heading__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"personal-name-see-also-from" => $VAR1->{body}{data}{properties}{authtype},"personal-name-see-also-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"personal-name-see-from" => $VAR1->{body}{data}{properties}{authtype},"personal-name-see-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"personal-name__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"record-source" => $VAR1->{body}{data}{prop
erties}{authtype},"record-source__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"see-also-from" => $VAR1->{body}{data}{properties}{authtype},"see-also-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"see-from" => $VAR1->{body}{data}{properties}{authtype},"see-from__sort" => $VAR1->{body}{data}{properties}{authtype__sort},"subject-heading-thesaurus" => $VAR1->{body}{data}{properties}{authtype},"subject-heading-thesaurus__sort" => $VAR1->{body}{data}{properties}{authtype__sort}}}},ignore => [],metho
d => "PUT",mime_type => "application/json",path => "/koha_kohadev_authorities/_mapping/data",qs => {},serialize => "std"}
Comment 1 Kevin Carnes 2022-02-08 14:40:20 UTC
Created attachment 130288 [details] [review]
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)

Starting with version 7.0 of Elasticsearch type names should not be used and
in version 6.8 there is a warning if you don't use include_type_name when
using put_mapping (requires Search::Elasticsearch@6.8).

To test:
1) Reindex records with koha-elasticsearch or rebuild_elasticsearch.pl
2) Observe any deprecation warnings or errors
3) Apply patch
4) Install Search::Elasticsearch with the same version as Elasticsearch
5) Reindex records with koha-elasticsearch or rebuild_elasticsearch.pl
6) Observe no warning or errors
7) Sign off

Sponsored-by: Lund University Library
Comment 2 Kevin Carnes 2022-02-08 14:48:22 UTC
This is a possible solution. The version checking can be made cleaner or there can be a configuration to specify which version of Elasticsearch is being used. It's also important to have the correct version of Search::Elasticsearch.
Comment 3 Kevin Carnes 2022-02-08 15:19:13 UTC
Created attachment 130293 [details] [review]
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)

Starting with version 7.0 of Elasticsearch type names should not be used and
in version 6.8 there is a warning if you don't use include_type_name when
using put_mapping (requires Search::Elasticsearch@6.8).

To test:
1) Reindex records with koha-elasticsearch or rebuild_elasticsearch.pl
2) Observe any deprecation warnings or errors
3) Apply patch
4) Install Search::Elasticsearch with the same version as Elasticsearch
5) Reindex records with koha-elasticsearch or rebuild_elasticsearch.pl
6) Observe no warning or errors
7) Sign off

Sponsored-by: Lund University Library
Comment 4 Kevin Carnes 2022-02-08 15:35:07 UTC
Should I create a new bug to remove support for multiple types when using mappings in Elasticsearch?
Comment 5 Kevin Carnes 2022-02-09 08:41:42 UTC
Created attachment 130341 [details] [review]
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)

Starting with version 7.0 of Elasticsearch type names should not be used and
in version 6.8 there is a warning if you don't use include_type_name when
using put_mapping (requires Search::Elasticsearch@6.8).

To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) Observe any deprecation warnings or errors about types
3) Apply patch
4) Install Search::Elasticsearch with the same version as Elasticsearch
5) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
6) Observe no warning or errors about types
7) Sign off

Sponsored-by: Lund University Library
Comment 6 Victor Grousset/tuxayo 2022-02-18 08:28:14 UTC
Created attachment 130811 [details] [review]
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)

Starting with version 7.0 of Elasticsearch type names should not be used and
in version 6.8 there is a warning if you don't use include_type_name when
using put_mapping (requires Search::Elasticsearch@6.8).

To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) Observe any deprecation warnings or errors about types
3) Apply patch
4) Install Search::Elasticsearch with the same version as Elasticsearch
5) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
6) Observe no warning or errors about types
7) Sign off

Sponsored-by: Lund University Library
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 7 Victor Grousset/tuxayo 2022-02-18 10:17:58 UTC
Thanks it works :D

(In reply to Kevin Carnes from comment #4)
> Should I create a new bug to remove support for multiple types when using
> mappings in Elasticsearch?

If that's needed for ES7 support or just to have a cleaner ES support then yes. Don't hesitate to open tickets.
Comment 8 Victor Grousset/tuxayo 2022-02-18 14:26:09 UTC
*** Bug 27771 has been marked as a duplicate of this bug. ***
Comment 9 Kevin Carnes 2022-02-22 12:39:17 UTC
Created attachment 131009 [details] [review]
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)

Starting with version 7.0 of Elasticsearch type names should not be used and
in version 6.8 there is a warning if you don't use include_type_name when
using put_mapping (requires Search::Elasticsearch@6.8).

To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) Observe any deprecation warnings or errors about types
3) Apply patch
4) Install Search::Elasticsearch with the same version as Elasticsearch
5) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
6) Observe no warning or errors about types
7) Sign off

Sponsored-by: Lund University Library
Comment 10 Kevin Carnes 2022-02-22 12:44:09 UTC
(In reply to Victor Grousset/tuxayo from comment #7)
> Thanks it works :D
> 
> (In reply to Kevin Carnes from comment #4)
> > Should I create a new bug to remove support for multiple types when using
> > mappings in Elasticsearch?
> 
> If that's needed for ES7 support or just to have a cleaner ES support then
> yes. Don't hesitate to open tickets.

No, it's not needed for ES7, it is just cleaner. It's already included, but can be moved to a new bug.

I also removed version checking and added a temporary elasticsearch_no_type configuration value that can be removed in the future when support for ES6 is dropped
Comment 11 Kevin Carnes 2022-02-23 14:45:54 UTC
Created attachment 131037 [details] [review]
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)

Starting with version 7.0 of Elasticsearch type names should not be used and
in version 6.8 there is a warning if you don't use include_type_name when
using put_mapping (requires Search::Elasticsearch@6.8).

To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) Observe any deprecation warnings or errors about types
3) Apply patch
4) Install Search::Elasticsearch with the same version as Elasticsearch
5) If testing with ES7, enable elasticsearch_no_type in koha-conf.xml
6) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
7) Observe no warning or errors about types
8) Sign off

Sponsored-by: Lund University Library
Comment 12 Julian Maurice 2022-03-08 08:24:49 UTC
(In reply to Kevin Carnes from comment #10)
> I also removed version checking and added a temporary elasticsearch_no_type
> configuration value that can be removed in the future when support for ES6
> is dropped

What was the problem with checking the version ?
Comment 13 Kevin Carnes 2022-03-08 14:08:16 UTC
(In reply to Julian Maurice from comment #12)
> What was the problem with checking the version ?

It sends a request to Elasticsearch to find out the version number every time Koha creates an Elasticsearch client and then parses the version number. This can add up, especially when searching, but perhaps it's not as big an issue for indexing.

It's definitely possible to check the version number when indexing, if people want to avoid adding a configuration, but it seems like others would like to avoid checking for specific versions of Elasticsearch.

I'm happy to hear suggestions on how it should be done.
Comment 14 Julian Maurice 2022-03-08 15:14:57 UTC
This makes sense, thanks.

I don't have any better suggestions, and the patch works as expected so I will sign off.

I tested with ES 6.8.23 and ES 7.17.1.
Two tests are failing with ES 7 in t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t but they should be fixed by bug 28610
Comment 15 Julian Maurice 2022-03-08 15:15:47 UTC
Created attachment 131484 [details] [review]
Bug 25669: ElasticSearch 6: [types removal] Specifying types in put mapping requests is deprecated (incompatible with 7)

Starting with version 7.0 of Elasticsearch type names should not be used and
in version 6.8 there is a warning if you don't use include_type_name when
using put_mapping (requires Search::Elasticsearch@6.8).

To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) Observe any deprecation warnings or errors about types
3) Apply patch
4) Install Search::Elasticsearch with the same version as Elasticsearch
5) If testing with ES7, enable elasticsearch_no_type in koha-conf.xml
6) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
7) Observe no warning or errors about types
8) Sign off

Sponsored-by: Lund University Library
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 16 Nick Clemens 2022-03-25 16:00:10 UTC
I think this should be a system preference, rather than a config switch - it is more easily adjusted

I don't like the idea of needing config switch for a future versions that will want to be on - system preferences can be updated/removed with an automatic update during upgrade - config will need manual intervention
Comment 17 Julian Maurice 2022-03-28 08:23:05 UTC
I don't like to have a switch for that (I'd prefer Koha to detect automatically the ES version) but if we have it, I think it makes sense to have it close to the other ES settings.
For instance, while testing it was easy to switch from one version of ES to another by (de)commenting a block in $KOHA_CONF

I don't understand why we would need to update this setting during a Koha upgrade. ES will not necessarily be upgraded at the same time as Koha. Can you explain a bit more what would be the benefits of having that setting as a syspref ?
Comment 18 Julian Maurice 2022-03-28 09:48:21 UTC
Just tested this with Elasticsearch 8.1.1 (with bug 28610) and it seems to work well :)
Comment 19 Nick Clemens 2022-03-28 12:02:09 UTC
(In reply to Julian Maurice from comment #17)
> I don't like to have a switch for that (I'd prefer Koha to detect
> automatically the ES version) but if we have it, I think it makes sense to
> have it close to the other ES settings.
> For instance, while testing it was easy to switch from one version of ES to
> another by (de)commenting a block in $KOHA_CONF
> 
> I don't understand why we would need to update this setting during a Koha
> upgrade. ES will not necessarily be upgraded at the same time as Koha. Can
> you explain a bit more what would be the benefits of having that setting as
> a syspref ?

I was thinking of the last time we moved required version of ES
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27070

I like the syspref because it can provide more control to the user and is more visible and can include notes about the need, like  ElasticsearchCrossFields. If a conf switch is preferred we should add something to the about page


If it's in the koha-conf - then I think it should be in the elasticsearch stanza:
 <elasticsearch>
     <server>es:9200</server>
     <index_name>koha_kohadev</index_name>
     <no_type>1</no_type>         
 </elasticsearch>

As it will be the standard in the future, i.e. everyone will want no type as ES6 is no longer supported I would rather see the switch the other way:
<include_type_name> to mirror the ES setting and their instructions

Looking deeper while typing this, we can also take advantage of include_type_name parameter to get us through ES7, patch coming
Comment 20 Nick Clemens 2022-03-28 12:07:27 UTC
Created attachment 132321 [details] [review]
Bug 25669: [alternate] Use include_type_name parameter in ES calls

This patch includes the parameter:
include_type_name
on our ES calls

It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.

For ES8 we will need to remove the hardcoded type,

I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support

When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs
Comment 21 Victor Grousset/tuxayo 2022-03-29 08:25:49 UTC
So if for now we only want to support ES 6 & ES 7 the alternate patch can do and no config or sys pref needed?
Comment 22 Nick Clemens 2022-03-29 09:50:54 UTC
(In reply to Victor Grousset/tuxayo from comment #21)
> So if for now we only want to support ES 6 & ES 7 the alternate patch can do
> and no config or sys pref needed?

I think  the goal as discussed with Julian yesterday is:
1 - Apply patch on a Koha using ES6 - indexing/searching continues to work with no reindex
2 - Upgrade ES to 7 - indexing/searching continues to work with no reindex
3 - Reindex Koha records - type 'data' is removed - indexing/searching continues to work
4 - Upgrade ES to 8 - indexing/searching continues to work with no reindex

Step 3 may require a switch, but Julian is looking into it
Comment 23 Victor Grousset/tuxayo 2022-03-29 13:07:40 UTC
I might be mixing stuff up. I though is was about a config switch or syspref to avoid querying the ES version.

Is this still about this? Or is that index not a performance issue to query the version when indexing?

quoting Kevin:

> It sends a request to Elasticsearch to find out the version number every time Koha creates an Elasticsearch client and then parses the version number. This can add up, especially when searching, but perhaps it's not as big an issue for indexing.

> It's definitely possible to check the version number when indexing, if people want to avoid adding a configuration, but it seems like others would like to avoid checking for specific versions of Elasticsearch.
Comment 24 Kevin Carnes 2022-03-30 08:57:00 UTC
(In reply to Victor Grousset/tuxayo from comment #23)
> I might be mixing stuff up. I though is was about a config switch or syspref
> to avoid querying the ES version.
> 
> Is this still about this? Or is that index not a performance issue to query
> the version when indexing?
> 

It was checking the version of Elasticsearch and Search::Elasticsearch to see if it was necessary to include the type.

The alternative solution uses include_type_name to delay removing type names. It will work starting with Elasticsearch 6.7 and Search::Elasticsearch@6.8 and it will also work with Elasticsearch 7, but will give a deprecation warning.

It will work as long as support before 6.7 is not required and support for 6 will be dropped before adding support for version 8. When support for 6 is dropped, Koha shouldn't include_type_name or use type names.

I don't think it's necessary to reindex between upgrades. There's only support for one type name in Elasticsearch and it doesn't seem to cause a problem if the name was data, but there's nothing wrong with reindexing in Elasticsearch 7.
Comment 25 Julian Maurice 2022-03-30 10:17:17 UTC
(In reply to Nick Clemens from comment #22)
> (In reply to Victor Grousset/tuxayo from comment #21)
> > So if for now we only want to support ES 6 & ES 7 the alternate patch can do
> > and no config or sys pref needed?
> 
> I think  the goal as discussed with Julian yesterday is:
> 1 - Apply patch on a Koha using ES6 - indexing/searching continues to work
> with no reindex
> 2 - Upgrade ES to 7 - indexing/searching continues to work with no reindex
> 3 - Reindex Koha records - type 'data' is removed - indexing/searching
> continues to work
> 4 - Upgrade ES to 8 - indexing/searching continues to work with no reindex
> 
> Step 3 may require a switch, but Julian is looking into it

I tested steps 1 and 2 and it works well.
Indexing with ES 7 prints huge warning messages about the types being deprecated, but I think it's not too much of a problem if we have a way to reindex without the types right after the ES upgrade (which is the step 3)

Still unsure about how to do that. We have a few possible solutions:
1) Have a switch in $KOHA_CONF or in a syspref that enable/disable types
2) Have the version of ES in $KOHA_CONF or in a syspref (it might be better to have the version than several "feature switches") 
3) Detect the version of ES at indexing time.

solution 3 might cause performance issues when indexing directly (for instance after a biblio modification). They can be mitigated by either:
a) put the version in memcached or L1 cache, but now there's a cache issue (you'd have to remember to flush the cache when upgrading ES)
b) use bug 27344 to index in a background job

Opinions ?
Comment 26 Victor Grousset/tuxayo 2022-04-07 22:35:47 UTC
Thanks Kevin for the clarifications.

> It will work as long as support before 6.7 is not required

I would say that's not a problem. ES <= 6.7 (even 6.8 actually) has security issues and is upgradable to 6.8.x directly.


(In reply to Julian Maurice from comment #25)
> Still unsure about how to do that. We have a few possible solutions:
> 1) Have a switch in $KOHA_CONF or in a syspref that enable/disable types
> 2) Have the version of ES in $KOHA_CONF or in a syspref (it might be better
> to have the version than several "feature switches") 

If we need a syspref or config switch, we can count on part of instances not knowing that it's needed.

> 3) Detect the version of ES at indexing time.
> 
> solution 3 might cause performance issues when indexing directly (for
> instance after a biblio modification). They can be mitigated by either:
> a) put the version in memcached or L1 cache, but now there's a cache issue
> (you'd have to remember to flush the cache when upgrading ES)

What if we refresh the ES version cache value on each services start?
Surely there isn't the need to upgrade ES on the fly without even restarting the services.
(wait, for installs without plack, do we have the concept of "restarting the services"?)
Or maybe on ES connection init? (I don't know if it's kept alive or opened on each request)

> b) use bug 27344 to index in a background job

It would make it not backportable.
Comment 27 Kevin Carnes 2022-04-11 09:05:19 UTC
(In reply to Victor Grousset/tuxayo from comment #26)
> > It will work as long as support before 6.7 is not required
> 
> I would say that's not a problem. ES <= 6.7 (even 6.8 actually) has security
> issues and is upgradable to 6.8.x directly.
> 

In that case I think it would be possible to avoid checking the ES version. As long as the minimum version of ES is 6.7 and the minimum version of Search::Elasticsearch is 6.8, using include_type_name should work for both ES 6 and 7.

This will give systems with ES 6 time to upgrade to ES 7. When it's decided that support for ES 6 is no longer needed, include_type_name and specifying the type name can be removed from Koha without needing to check the ES version since it's assumed to be ES 7 or ES 8.
Comment 28 Victor Grousset/tuxayo 2022-04-11 18:44:12 UTC
> In that case I think it would be possible to avoid checking the ES version

Even if someone wants to support ES 8, no version check is needed for this? Maybe your last paragraph actually is about this and we can't have ES 6 to 8 support without check?
Comment 29 Kevin Carnes 2022-04-26 08:29:20 UTC
(In reply to Victor Grousset/tuxayo from comment #28)
> > In that case I think it would be possible to avoid checking the ES version
> 
> Even if someone wants to support ES 8, no version check is needed for this?
> Maybe your last paragraph actually is about this and we can't have ES 6 to 8
> support without check?

In order to avoid version checking, support for ES 8 wouldn't be available while there is support for ES 6. So it would be support for ES 6 to 7 and then support for ES 7 to 8.
Comment 30 Victor Grousset/tuxayo 2022-04-30 20:35:30 UTC
For the sake of simplicity and unblocking this, is there any objection going the way of supporting just ES 6 and ES 7 without a version check?
And if someone needs support for the completely non-libre ES 8 they would continue with the more complex solution to have a wide version support. IMHO ES 8 has much less priority so no need to be proactive if it gets us to stall.

I lost track about the patches, is the right one (assuming going with above proposal) this one?: "Bug 25669: [alternate] Use include_type_name parameter in ES calls"
Comment 31 Kevin Carnes 2022-05-17 06:55:45 UTC
No objection, I think Nick's alternate is a good solution. I would recommend removing the data type from the mappings in the test files so that they pass and updating cpanfile to require Search::Elasticsearch@6.80.

(In reply to Victor Grousset/tuxayo from comment #30)
> For the sake of simplicity and unblocking this, is there any objection going
> the way of supporting just ES 6 and ES 7 without a version check?
> And if someone needs support for the completely non-libre ES 8 they would
> continue with the more complex solution to have a wide version support. IMHO
> ES 8 has much less priority so no need to be proactive if it gets us to
> stall.
> 
> I lost track about the patches, is the right one (assuming going with above
> proposal) this one?: "Bug 25669: [alternate] Use include_type_name parameter
> in ES calls"
Comment 32 Kevin Carnes 2022-05-25 09:24:08 UTC
Created attachment 135335 [details] [review]
Bug 25669: [alternate] Use include_type_name parameter in ES calls (small fixes)

Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.
Comment 33 Kevin Carnes 2022-05-25 09:34:24 UTC
To test:
1) Configure Koha to use ES 6.8
2) Make sure that Search::Elasticsearch@6.80 is installed
3) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4) Apply patch
5) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
6) Make sure that all the tests pass
7) Configure Koha to use ES 7
8) Make sure that Search::Elasticsearch has the same version as ES
9) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
10) Apply patch
11) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
12) Make sure that all the tests pass
13) Sign off
Comment 34 Julian Maurice 2022-06-01 13:12:54 UTC
Created attachment 135542 [details] [review]
Bug 25669: [alternate] Use include_type_name parameter in ES calls

This patch includes the parameter:
include_type_name
on our ES calls

It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.

For ES8 we will need to remove the hardcoded type,

I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support

When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 35 Julian Maurice 2022-06-01 13:12:59 UTC
Created attachment 135543 [details] [review]
Bug 25669: [alternate] Use include_type_name parameter in ES calls (small fixes)

Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 36 Julian Maurice 2022-06-01 13:17:37 UTC
- Followed test plan: OK (unit tests do not work with a UNIMARC database, but they do not work better on master either, OK with a MARC21 database)
- Tested upgrading ES 6 to 7: OK - search continues to work without reindexing
- Tested also with latest version of Search::Elasticsearch (7.715): OK
Comment 37 Martin Renvoize 2022-06-22 13:51:33 UTC
Created attachment 136417 [details] [review]
Bug 25669: [alternate] Use include_type_name parameter in ES calls

This patch includes the parameter:
include_type_name
on our ES calls

It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.

For ES8 we will need to remove the hardcoded type,

I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support

When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 38 Martin Renvoize 2022-06-22 13:51:38 UTC
Created attachment 136418 [details] [review]
Bug 25669: [alternate] Use include_type_name parameter in ES calls (small fixes)

Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 39 Martin Renvoize 2022-06-22 13:53:02 UTC
All seems to be working well in tests, QA scripts are happy and the unit tests are passing.

Passing QA.. nice work chaps.
Comment 40 Tomás Cohen Arazi 2022-06-23 14:35:12 UTC
Created attachment 136452 [details] [review]
Bug 25669: Use include_type_name parameter in ES calls

This patch includes the parameter:
include_type_name
on our ES calls

It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.

For ES8 we will need to remove the hardcoded type,

I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support

When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 41 Tomás Cohen Arazi 2022-06-23 14:35:19 UTC
Created attachment 136453 [details] [review]
Bug 25669: (follow-up) Minor fixes

Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 42 Tomás Cohen Arazi 2022-06-23 14:36:21 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 43 Marcel de Rooy 2022-06-24 07:54:17 UTC
(In reply to Tomás Cohen Arazi from comment #42)
> Pushed to master for 22.11.
> 
> Nice work everyone, thanks!

I immediately cannot install master anymore on Debian 10.
Thanks ;)
Comment 44 Victor Grousset/tuxayo 2022-06-26 00:29:29 UTC
What happened?! :o

Something with Search::Elasticsearch 6.80?
Comment 45 Marcel de Rooy 2022-06-27 06:23:52 UTC
(In reply to Victor Grousset/tuxayo from comment #44)
> What happened?! :o
> 
> Something with Search::Elasticsearch 6.80?

Yes. Debian 10 has 6.00
Comment 46 Blou 2022-07-08 18:26:56 UTC
Any followup on this ?

Our master doesn't work, we can't rebuild the mappings anymore.

We get an exception (and the printing of the message fails as well, since args     is empty.  A dumping gets us the variable 'text' at least, that states

Unknown param (include_type_name) in (put_mapping) request.

This is on Ubuntu 20.04, but it would happen everywhere (we use Debian 10 just about everywhere in production).
Comment 47 Mason James 2022-07-15 12:31:15 UTC
(In reply to Blou from comment #46)
> Any followup on this ?
> 
> Our master doesn't work, we can't rebuild the mappings anymore.

hi Blou,
try using the staging repo, it has updated packages

$ cat /etc/apt/sources.list.d/koha.list
deb http://debian.koha-community.org/koha-staging unstable  main
Comment 48 Victor Grousset/tuxayo 2022-07-16 20:41:19 UTC
> try using the staging repo, it has updated packages

Does that mean the problem is solved? Then when can backport, right?
Comment 49 Lucas Gass 2022-07-29 15:30:28 UTC
Backported to 22.05.x for 22.05.04
Comment 50 Arthur Suzuki 2022-08-11 09:03:57 UTC
doesn't apply cleanly on 21.11.x. please provide backport if needed.
Comment 51 Victor Grousset/tuxayo 2022-08-20 01:50:58 UTC
Created attachment 139531 [details] [review]
Bug 25669: [21.11.x] Use include_type_name parameter in ES calls

This patch includes the parameter:
include_type_name
on our ES calls

It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.

For ES8 we will need to remove the hardcoded type,

I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support

When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 150a4b03f337f6a58869c67d3681caa64441d53a)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>

(cherry picked from commit 99757c2d975d224ea191622116e6eb839fff70c1)
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 52 Victor Grousset/tuxayo 2022-08-20 01:51:07 UTC
Created attachment 139532 [details] [review]
Bug 25669: [21.11.x] (follow-up) Minor fixes

Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit e926de5e8f359acf6b8beff541fe13ce2a5ac49e)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>

(cherry picked from commit 22c9ea9737d614a26b06092cda9708e27013e494)
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 53 Victor Grousset/tuxayo 2022-08-20 01:55:19 UTC
(In reply to Arthur Suzuki from comment #50)
> doesn't apply cleanly on 21.11.x. please provide backport if needed.

Here it is, so we can be sure to have it for next release. (Which should therefore finally support ES 7!)
Comment 54 Arthur Suzuki 2022-08-26 08:04:05 UTC
pushed to 21.11.x for 21.11.12
Comment 55 Victor Grousset/tuxayo 2022-08-26 20:34:25 UTC
Backported: Pushed to 21.05.x branch for 21.05.18

About ES 7 support, I asked for the next meeting if we can start advertising it.
Comment 56 Arthur Suzuki 2022-08-30 09:18:00 UTC
thx pushed to 21.11.x for 21.11.12
Comment 57 David Nind 2022-10-06 21:48:55 UTC
Could I get some feedback on the documentation required?

My initial thoughts, as this is a technical change and is not something that I'm in any way familiar with:

1. This change:
   a. release note about what this change does - I have added the release-notes-needed keyword
   b. communication to the main Koha and development mailing lists about Elasticsearch/OpenSearch support for 22.11 and supported releases (or maybe just in the 22.11 release notes)

2. Installing and configuring Elasticsearch/OpenSearch:
   a. I'm not sure what we have currently, and how up-to-date it is. Any pointers to the Wiki and other sources would be appreciated.
   b. Should we have a link from the main Koha installation guide to some instructions on the basics?
   c. What should be in scope? If the Elasticsearch/OpenSearch installation is sufficient, then we should link to that, and only provide any Koha specific things to consider.

3. Guide to using Elasticsearch: this is not something we have now, but something we could potentially work on.

4. Supported version matrix, including OpenSearch options.

5. Material for developers, including the KTD development environment and how to get the various versions working.

For anything more than item 1, I'm proposing to create separate documentation bugs for these:

- Omnibus Elasticsearch/OpenSearch documentation bug

- Specific guides covering:
  . Installation and administration (from a system administration perspective - keeping things running)
  . Using Elasticsearch/OpenSearch: more from a library point of view, including an overview, configuring, options, mappings, etc - the day-to-day things to make sure the search in Koha is working as required.

- Development guide: from a developer perspective and what to do when working on search-related bugs, testing various versions, KTD environment.

Some help would be required with these, as it seems to me to be a quite technical area with some expertise required.

David
Comment 58 Aleisha Amohia 2022-11-07 00:52:09 UTC
This appears to have broken our ES6, we're seeing this error:

Exception 'Koha::Exception' thrown 'Unable to update mappings for index "koha_instance_biblios". Reason was: "request [/koha_instance_biblios/_mapping/data] contains unrecognized parameter: [include_type_name]". Index needs to be recreated and reindexed'
Something went wrong rebuilding indexes 

How do we fix this please? I'm not sure what this change means for us if we're still running ES6.
Comment 59 Kevin Carnes 2022-11-07 09:29:59 UTC
(In reply to Aleisha Amohia from comment #58)
> This appears to have broken our ES6, we're seeing this error:
> 
> Exception 'Koha::Exception' thrown 'Unable to update mappings for index
> "koha_instance_biblios". Reason was: "request
> [/koha_instance_biblios/_mapping/data] contains unrecognized parameter:
> [include_type_name]". Index needs to be recreated and reindexed'
> Something went wrong rebuilding indexes 
> 
> How do we fix this please? I'm not sure what this change means for us if
> we're still running ES6.

Which version of Elasticsearch are you using? I think at least version 6.8 is required to use include_type_name.
Comment 60 Nick Clemens 2022-11-07 12:02:19 UTC
Correct, 6.8 is required for include_type_name:
https://www.elastic.co/blog/moving-from-types-to-typeless-apis-in-elasticsearch-7-0
Comment 61 Victor Grousset/tuxayo 2022-11-27 23:46:15 UTC
Updated doc to reflect this:
https://wiki.koha-community.org/wiki/System_requirements_and_recommendations