Bug 31652

Created attachment 141406 [details] [review]
Bug 31652: Add geo-search

This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.

The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)

Test Plan:

* make sure you're running ElasticSearch 7
  (eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* Set system preference "SaerchEngine" to "Elasticsearch"
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
  misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
  * get the biblionumber (eg 123)
  * curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
  * You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
  * Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
  * curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
  * handcrafted query string:
    * /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
  * HKS3GeoSearch
    * install the plugin and enable it
    * got to OPAC / Advanced Search
    * There is a new input box "Geographic Search" where you can enter lat/long/radius
    * On the search result page a map is shown with pins for each found biblioitem

Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/

This is a combination of 17 commits.

The patch no longer applies 8-(...

git bz apply 31652

Bug 31652 - Add geo-search

141406 - Bug 31652: Add geo-search

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 31652: Add geo-search
.git/rebase-apply/patch:323: trailing whitespace.
# can be single or multiple parameters separated by comma: kw,right-Truncation 
warning: 1 line adds whitespace errors.
Using index info to reconstruct a base tree...
M       Koha/SearchEngine/Elasticsearch.pm
M       Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
M       Koha/SearchEngine/Elasticsearch/Search.pm
M       admin/searchengine/elasticsearch/field_config.yaml
M       admin/searchengine/elasticsearch/mappings.yaml
M       installer/data/mysql/kohastructure.sql
M       koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt
M       opac/opac-search.pl
.git/rebase-apply/patch:323: trailing whitespace.
# can be single or multiple parameters separated by comma: kw,right-Truncation 
warning: 1 line adds whitespace errors.
Falling back to patching base and 3-way merge...
Auto-merging opac/opac-search.pl
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/admin/searchengine/elasticsearch/mappings.tt
Auto-merging installer/data/mysql/kohastructure.sql
CONFLICT (content): Merge conflict in installer/data/mysql/kohastructure.sql
Auto-merging admin/searchengine/elasticsearch/mappings.yaml
Auto-merging admin/searchengine/elasticsearch/field_config.yaml
CONFLICT (content): Merge conflict in admin/searchengine/elasticsearch/field_config.yaml
Auto-merging Koha/SearchEngine/Elasticsearch/Search.pm
Auto-merging Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Auto-merging Koha/SearchEngine/Elasticsearch.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 31652: Add geo-search

Created attachment 153720 [details] [review]
Bug 31652: Add geo-search

This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.

The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)

Test Plan:

* make sure you're running ElasticSearch 7
  (eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* Set system preference "SaerchEngine" to "Elasticsearch"
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
  misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
  * get the biblionumber (eg 123)
  * curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
  * You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
  * Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
  * curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
  * handcrafted query string:
    * /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
  * HKS3GeoSearch
    * install the plugin and enable it
    * got to OPAC / Advanced Search
    * There is a new input box "Geographic Search" where you can enter lat/long/radius
    * On the search result page a map is shown with pins for each found biblioitem

Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/

This is a combination of 17 commits.

I've updated the patch and it should apply now (again) on current master. Please try to signoff again, I'd love to have this in master before Perl Koha Conference (where I'll do a talk on this patch!)

Comment on attachment 153720 [details] [review]
Bug 31652: Add geo-search

Review of attachment 153720 [details] [review]:
-----------------------------------------------------------------

::: Koha/SearchEngine/Elasticsearch/Indexer.pm
@@ +117,4 @@
>      }
>  
>      my $documents = $self->marc_records_to_documents($records);
> +

It looks like this file might not be altered beyond whitespace? Looks like the whitespace change could be removed to simplify the overall patch.

::: Koha/SearchEngine/Elasticsearch/Search.pm
@@ +93,5 @@
>      my $elasticsearch = $self->get_elasticsearch();
> +
> +    # XXX investigate where empty query_string is coming from
> +    delete $query->{query}->{query_string} if
> +      $query->{query}->{query_string} && !%{$query->{query}->{query_string}};

Is this a pre-existing problem or introduced in this patch?

This is/was a pre-existing problem

Created attachment 154388 [details] [review]
Bug 31652: Add geo-search

This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.

The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query.

We added a new SystemPreference `GeoSearchEnabled`. If it is set to
`true`, we show a new advanced search field that allows for input of lat/lng
and distance (radius). On the search result page, a we render a map
using leaflet.js with pins for each found item. Therefor we also added a
new api endpoint that returns lat/lng for each biblio in the selection.

Test Plan:

* make sure you're running ElasticSearch 7
  (eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* Set system preference "SearchEngine" to "Elasticsearch"
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
  misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
  * get the biblionumber (eg 123)
  * curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
  * You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
  * Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
  * curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha:
  * Set system preference "GeoSearchEnabled" to "true"
  * got to OPAC / Advanced Search
  * There is a new input box "Geographic Search" where you can enter lat/long/radius, for example lat=48.2, lng=16.3, distance=100km
  * On the search result page a map is shown with pins for each found biblioitem

Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/

Authors: Mark Hofstetter <mark@hofstetter.at>, Thomas Klausner <domm@plix.at>

The latest patch has some warnings/errors reported from koha-qa.pl. Plus I'm not totally happy with the way the search params are handled. So I guess it will need a bit more work.

(In reply to Thomas Klausner from comment #8)
> The latest patch has some warnings/errors reported from koha-qa.pl. Plus I'm
> not totally happy with the way the search params are handled. So I guess it
> will need a bit more work.

Based on this I am setting this to FQA, to get it out of the 'Needs signoff' queue

Here's some old-ish work on improving ElasticSearch, which could be used/integrated here (as pointed out by Jonathan during PerlKohaCon)

https://github.com/fredericd/Koha/compare/master...newelastic

(In reply to Thomas Klausner from comment #10)
> Here's some old-ish work on improving ElasticSearch, which could be
> used/integrated here (as pointed out by Jonathan during PerlKohaCon)
> 
> https://github.com/fredericd/Koha/compare/master...newelastic

This should go onto a new bug - if we want to change searching at large let's do that, then base this on top of that work

Is is necessary to use external server for static files ?

https://unpkg.com/leaflet@1.9.1/dist/leaflet.css
https://unpkg.com/leaflet@1.9.1/dist/leaflet.js

Can't we embed in sources ?

I don't see a problem in embedding the files if that is "the prefered way of doing things"

Lets see what QA says

Comment on attachment 154388 [details] [review]
Bug 31652: Add geo-search

Review of attachment 154388 [details] [review]:
-----------------------------------------------------------------

::: koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-results.tt
@@ +592,5 @@
>          [% Asset.js("js/openlibrary.js") | $raw %]
>      [% END %]
> +    [% IF ( GeoSearchEnabled ) %]
> +        <link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.1/dist/leaflet.css" integrity="sha256-sA+zWATbFveLLNqWO2gtiw3HL/lh1giY/Inf1BJ0z14=" crossorigin=""/>
> +        <script src="https://unpkg.com/leaflet@1.9.1/dist/leaflet.js" integrity="sha256-NDI0K41gVbWqfkkaHj15IzU7PtMoelkzyKp8TOaFQ3s=" crossorigin=""></script>

As Frido suggested, we don't link out to externally hosted static files. Lots of reasons for that.

Koha already has leaflet.js and leaflet.css although a much older version: 1.0.3+ed36a04

I'd suggest opening a separate issue to upgrade Leaflet and then having that be a dependency of this issue.

I think we should definitely not link to external files but keep shipping our own. I believe updating the ones we ship would be a good first step here and hopefully not too hard. I think maybe they are used for Mana?

git@github.com:HKS3/koha-branches.git

it's in the branch 

topic-es-2312

for those who want to see it, please send me your github id

I now created a public git branch 

https://github.com/HKS3/koha-devel/tree/topic-es-2312

working on a cypress test

Created attachment 164666 [details] [review]
Bug 31652: Add geo-search

This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.

The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)

Test Plan:

* make sure you're running ElasticSearch 7
  (eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* Set system preference "SaerchEngine" to "Elasticsearch"
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
  misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
  * get the biblionumber (eg 123)
  * curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
  * You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
  * Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
  * curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
  * handcrafted query string:
    * /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
  * HKS3GeoSearch
    * install the plugin and enable it
    * got to OPAC / Advanced Search
    * There is a new input box "Geographic Search" where you can enter lat/long/radius
    * On the search result page a map is shown with pins for each found biblioitem

Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/

Created attachment 164689 [details] [review]
Bug 31652: Add geo-search

This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.

The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)

Test Plan:

* make sure you're running ElasticSearch 7
  (eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* Set system preference "SaerchEngine" to "Elasticsearch"
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
  misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
  * get the biblionumber (eg 123)
  * curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
  * You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
  * Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
  * curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
  * handcrafted query string:
    * /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
  * HKS3GeoSearch
    * install the plugin and enable it
    * got to OPAC / Advanced Search
    * There is a new input box "Geographic Search" where you can enter lat/long/radius
    * On the search result page a map is shown with pins for each found biblioitem

Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Created attachment 164690 [details] [review]
Bug 31652: (follow-up) Fix update and remove fixed fixme

Update test count as well

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Worked with Mark and Martin to move this forward - this version simply adds the ability to do a geo-search - it would need work for an interface to make the searching exposed, but I think this will provide a useful base. Passing QA

Please check:

1) Are the changes to name / label in kohastructure intended?

   `id` int(11) NOT NULL AUTO_INCREMENT,
-  `name` varchar(255) NOT NULL COMMENT 'the name of the field as it will be stored in the search engine',
-  `label` varchar(255) NOT NULL COMMENT 'the human readable name of the field, for display',
-  `type` enum('','string','date','number','boolean','sum','isbn','stdno','year','callnumber') NOT NULL COMMENT 'what type of data this holds, relevant when storing it in the search engine',
+  `name` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'the name of the field as it will be stored in the search engine',
+  `label` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'the human readable name of the field, for display',
+  `type` enum('','string','date','number','boolean','sum','isbn','stdno','year','callnumber','geo_point') COLLATE utf8mb4_unicode_ci NOT NULL COMMENT 'what type of data this holds, relevant when storing it in the search engine',

2) System preference GeoSearchEnabled

I believe this should be removed and due to rewrites? There is no other mention of this pref.

+++ b/koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/searching.pref
@@ -99,6 +99,13 @@ Searching:
             - "List of search fields (separated by | or ,) that should not be autotruncated by Elasticsearch even if <a href='/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=QueryAutoTruncate'>QueryAutoTruncate</a> is set to 'Yes':"
             - pref: ESPreventAutoTruncate
               class: long
+        -
+            - pref: GeoSearchEnabled
+              type: boolean
+              choices:
+                  1: Enable
+                  0: Disable
+            - 'GeoSearch via Elasticsearch'
     Search form:

3) Capitalization

Usually I just fix these sneakily, but so we don't forget with the other points:

+                                              <option value="geo_point" selected="selected">Geo Point</option>

We only capitalize the first word, so this should read: Geo Point (I know Call Number in the same spot is wrong as well, but no need to add more)


Please fix!

Created attachment 165707 [details] [review]
Bug 31652: Add geo-search

This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.

The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)

Test Plan:

* make sure you're running ElasticSearch 7
  (eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* Set system preference "SaerchEngine" to "Elasticsearch"
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
  misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
  * get the biblionumber (eg 123)
  * curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
  * You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
  * Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
  * curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
  * handcrafted query string:
    * /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
  * HKS3GeoSearch
    * install the plugin and enable it
    * got to OPAC / Advanced Search
    * There is a new input box "Geographic Search" where you can enter lat/long/radius
    * On the search result page a map is shown with pins for each found biblioitem

Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Additional finetuning:

- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23

Created attachment 165708 [details] [review]
Bug 31652: Add geo-search

This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.

The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)

Test Plan:

* make sure you're running ElasticSearch 7
  (eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
  misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
  * get the biblionumber (eg 123)
  * curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
  * You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
  * Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
  * curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
  * handcrafted query string:
    * /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
  * HKS3GeoSearch
    * install the plugin and enable it
    * got to OPAC / Advanced Search
    * There is a new input box "Geographic Search" where you can enter lat/long/radius
    * On the search result page a map is shown with pins for each found biblioitem

Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Additional finetuning:

- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23

Please switch back to PQA when you are done!

I assume you mean status = "Passed QA"? I've now set it to this :-)

I thought I would have a go at testing, even though it has passed QA.

Unfortunately, the patches don't apply - sha1/fake ancestor error:

git bz apply 31652

Bug 31652 - Add geo-search

164690 - Bug 31652: (follow-up) Fix update and remove fixed fixme
165708 - Bug 31652: Add geo-search

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 31652: (follow-up) Fix update and remove fixed fixme
error: sha1 information is lacking or useless (Koha/SearchEngine/Elasticsearch/Search.pm).
error: could not build fake ancestor
Patch failed at 0001 Bug 31652: (follow-up) Fix update and remove fixed fixme

I think the patch sequence is wrong now. I was going to do 2 git bz apply -i here to fix on applying.

(In reply to Katrin Fischer from comment #29)
> I think the patch sequence is wrong now. I was going to do 2 git bz apply -i
> here to fix on applying.

I tried that, but it didn't work for me 8-)

When creating the current patch, I did a rebase on the fresh main and squashed the three commits (original, "(follow-up)" and my fixes according to Katrins comment) into one commit.

So there should only be one patch:
  Bug 31652: Add geo-search (14.16 KB, patch)
  2024-04-27 16:58 UTC, Thomas Klausner 


Not sure how/if I can obsolte this one:
  Bug 31652: (follow-up) Fix update and remove fixed fixme (2.72 KB, patch)
  2024-04-11 13:19 UTC, Nick Clemens 

Or if it is possible (or how to do it) to tell git bz to only use my rebased/squashed patch

(In reply to Thomas Klausner from comment #31)
> When creating the current patch, I did a rebase on the fresh main and
> squashed the three commits (original, "(follow-up)" and my fixes according
> to Katrins comment) into one commit.
> 
> So there should only be one patch:
>   Bug 31652: Add geo-search (14.16 KB, patch)
>   2024-04-27 16:58 UTC, Thomas Klausner 
> 
> 
> Not sure how/if I can obsolte this one:
>   Bug 31652: (follow-up) Fix update and remove fixed fixme (2.72 KB, patch)
>   2024-04-11 13:19 UTC, Nick Clemens 
> 
> Or if it is possible (or how to do it) to tell git bz to only use my
> rebased/squashed patch

You can obsolete the patch from Bugzilla:
1. Under the attachments section, click the 'Details' link beside the patch to obsolete.
2. In the title/header for the patch, click 'Edit details' again.
3. Click the tick box beside 'obsolete'.
4. Add a comment.
5. Click 'Submit'.

I obsolted the old patch, now git bz works again:

  kohadev-koha@kohadevbox:koha(another_31652_try)$ git bz apply 31652

  Bug 31652 - Add geo-search

  165708 - Bug 31652: Add geo-search

  Apply? [(y)es, (n)o, (i)nteractive] y
  Applying: Bug 31652: Add geo-search


Please try again

Hi Thomas.

I can confirm the bug now applies!

I had a go at testing (not that this is required, as it has already passed QA). Things I noted:
- database update required after applying the patch
- also needed to reset the Elasticsearch mappings
- I didn't get any search results in the OPAC
- wasn't sure how to install the HKS3GeoSearch plugin 

Here are my testing notes (using KTD):
1. Start up KTD with Elasticsearch: ktd --es7 up
2. Apply the patch
3. Update the database: updatedatabase
4. Restart everything
5. Edit the books framework and make 034$s and 034$t visible in the editor
6. Add example coordinates for a couple of records (I added lat=48.216 and lon=16.395 to records 262 and 126)
7. Reset the Elasticsearch mappings (Administration > Catalog > Search engine configuration (Elasticsearch) > Reset mappings)
8. Rebuild the search index: koha-elasticsearch --rebuild -d -b -a kohadev
9. Check if indexing working: curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
- The result I got:

curl http://es:9200/koha_kohadev_biblios/_doc/262?pretty | grep -A5 geolocation
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1078    "geolocation" : [ 0      0      0 --:--:-- --:--:-- --:--:--     0
3      {
         "lon" : "16.395",
         "lat" : "48.216"
1      }
0    ],
0 10783    0     0  2164k      0 --:--:-- --:--:-- --:--:-- 2632k

10. Query Elasticsearch directly:

- Result:
curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
{
....

A lot of information returned - basically the details for the two records that I added the coordinates to.
...
}

11. OPAC query:

http://127.0.0.1:8080/cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
- Puts details in the search field: lat:48.25 lng:18.35 distance:100km
- Results displayed:

No results found!
No results found for that in catalog. 

12. HKS3GeoSearch Koha Plugin:
- No release available. Does this mean I have to clone the repository into the plugins folder? (I tried cloning the repository into /var/lib/koha/kohadev/plugins, but nothing appears in the list of plugins.)

David

Pushed for 24.05!

Well done everyone, thank you!

Not backported to 23.11.x

I've tried to add some release notes - please correct/extend as you see fit.

I've made some edits to the release notes.

I was able to install the plugin, having the kpz available made that much easier.

Results from searching:
1. I used the latitude and longitude from the test plan and added these to some records.
2. The curl queries return the results as in the test plan (the lat and lon values are in the Elasticsearch details).
3. If I search in the OPAC using the example query, the map appears, but it says no search results found.
4. If I search for a word in the records, the map appears with search results, but there are no pins showing on the map.

Is there something else I need to do?

Here is what I did (using KTD on the latest main, tested using ktd --es7 up and jktd --es8 up):
1. Installed the plugin, enabled, and restarted everything.
2. Made 034$s and 034$t visible in the editor for the BKS framework.
3. Added some latitude and longitude values to two records (I used the values from the test plan - lat=48.216, lon=16.395).
4. Checked that the mappings exist under Search engine configuration (Elasticsearch) for geolocation_lat and geolocation_lon. They did, so I didn't reset the mappings (I think that is right).
5. I reindexed, even though I'm not sure that this is required (koha-elasticsearch --rebuild -d -b -a kohadev).
6. OPAC keyword search - searched with a keyword that returns the records I added values to: 
   - Result ==> Map appears (without pins), with the search results including the records with the latitude and longitude values
7. Direct query (from the test plan): <opac-url>:8080/cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
  . Result: Map appears, but no results
8. If I change the 100km to 1000km in the query, then the results are listed and there is a pin for one of the records (but not the other).
9. Something else I noted: These details appear in the search box - lat:48.25 lng:18.35 distance:1000km. If I click the search button again, the URL is changed so that it has %3A instead of a +, and there are no results (<opac-url>:8080/cgi-bin/koha/opac-search.pl?idx=&q=lat%3A48.25+lng%3A18.35+distance%3A1000km&weight_search=1).

Apart from a very basic knowledge about latitude and longitude, I'm guessing the query is saying give me any records that are X km from the latitude and longitude.

Are you able to provide some latitude and longitude values that could be added to some records, and the queries to use to display some pins on the map? (For queries, either the values to enter in the advanced search form field, or the direct query.)

This would make it easy for people to try it out with something that works. (And it would be great to demonstrate all your hard work!)

Just noting this presentation, will be of great help when updating the documentation (even though some things have changed since it was shared): https://domm.plix.at/talks/2023_helsinki_koha_geosearch/

hi,

I added 

https://github.com/HKS3/HKS3GeoSearch/blob/dev/TO_TEST.md

https://github.com/HKS3/HKS3GeoSearch/tree/dev/scripts

https://github.com/HKS3/HKS3GeoSearch/blob/dev/scripts/insert_geo_data.pl

which adds 3 geo data point to 3 books (hardcoded for the dev install)

so if now do a search around 

http://kohadev.mydnsname.org:8080/cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48+lng:16+distance:50km&do=Search

3 books should be found

we are working on the a point and click interface to search

Thanks Mark, that really helps!

I added the example co-ordinates to the example records, and used the query - the pins were nicely displayed in the OPAC.

Feel free to cc me in on any future bugs, as I'm happy to help with testing and sign off.

David

Created attachment 167223 [details]
MARC21 cartographic bib record with coordinates

Bib record 446 for a nautical chart, from our catalog, keys.bywatersolutions.com

Created attachment 167224 [details]
Another MARC21 catalog record with coordinates

Bib record 447 for a nautical chart with coordinates, from our catalog, keys.bywatersolutions.com

Created attachment 167225 [details]
A third MARC21 bib record with coordinates

Bib record no.41753 for a nautical chart, with coordinates, from our catalog, keys.bywatersolutions.com

Hi!  I've uploaded three fully cataloged cartographic records with correctly entered coordinates for nautical charts from our catalog, in case they will help!  And for extra fun, they link to onlin images of the charts in case anyone would like to take a look.:)

I am super excited about this development--I've been taking the time to enter coordinates into cartographic records for years and years and years with hopes that they'd be useful in the future!

Cheerio!
h2, A Real Cataloger:)

This bug seems to change things mostly in the "backend" of Koha, paving the way for further changes. But right now, nothing much to change in the manual as far as I could see. I added the new mappings, but that's it.

(In reply to Caroline Cyr La Rose from comment #46)
> This bug seems to change things mostly in the "backend" of Koha, paving the
> way for further changes. But right now, nothing much to change in the manual
> as far as I could see. I added the new mappings, but that's it.

That's the case (see comments 37 to 40 for how I tested, after it was pushed). It is usable now using the plugin mentioned in the release notes.