Bug 18829

Summary: Elasticsearch - Add ability to view the ES indexed record
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens (kidclamp) <nick>
Status: Needs documenting --- QA Contact:
Severity: enhancement    
Priority: P5 - low CC: alex.arnaud, axel.amghar, axel.amghar, ere.maijala, fridolin.somers, jonathan.druart, katrin.fischer, kyle, marjorie.barry-vila, matt.blenkinsop, severine.queune
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17382
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00
Attachments: Bug 18829 - Elasticsearch - Add ability to view the ES indexed record
Bug 18829: Elasticsearch - Add ability to view the ES indexed record
Bug 18829: Elasticsearch - Add ability to view the ES indexed record
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: Elasticsearch - Add ability to view the ES indexed record
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: (QA follow-up) Fix punctuation, capitalization and page title
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: (QA follow-up) Fix punctuation, capitalization and page title
Bug 18829: (follow-up) Return 404 when record cannot be loaded
Bug 18829: (follow-up) Don't messup record encoding
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record
Bug 18829: (QA follow-up) Fix punctuation, capitalization and page title
Bug 18829: (follow-up) Return 404 when record cannot be loaded
Bug 18829: (follow-up) Don't messup record encoding
Bug 18829: (QA follow-up) I18N, test character and breadcrumbs
Bug 18829: (QA follow-up) Replace forbidden indirect syntax

Description Nick Clemens (kidclamp) 2017-06-20 14:54:40 UTC
It would be nice for the end user (or developer) to have a way to view the record as it exists in the ES index - it will help in adjusting mappings, understanding results, and identifying is a record is not being index
Comment 1 Nick Clemens (kidclamp) 2017-06-20 14:58:50 UTC Comment hidden (obsolete)
Comment 2 axel 2019-04-29 09:29:13 UTC Comment hidden (obsolete)
Comment 3 axel 2019-04-29 09:36:44 UTC Comment hidden (obsolete)
Comment 4 axel Amghar 2019-05-02 14:55:13 UTC Comment hidden (obsolete)
Comment 5 axel Amghar 2019-05-02 15:05:04 UTC
Created attachment 89249 [details] [review]
Bug 18829: Elasticsearch - Add ability to view the ES indexed record

This is groundwork - we need to make the display prettier, but for now
it does at least dump the record into the template

We probably also need a swtich to enable/disable this - or just move it
into the left hand tabs?
Comment 6 axel Amghar 2019-05-02 15:05:08 UTC
Created attachment 89250 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElsaticSearch" as value
- Search whatever you want in Search the catalog and select a notice
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
Comment 7 Fridolin Somers 2019-05-28 14:26:53 UTC
Oups the ES connexion is hardcoded :

+                $.getJSON("http://localhost:9200/koha_kohadev_biblios/data/", biblionumber, function( data ){

You must get server+port and index name from C4::Context.
Comment 8 Nick Clemens (kidclamp) 2019-06-06 13:28:17 UTC
Created attachment 90370 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElasticSearch" as value
- Search whatever you want in Search the catalog and select a record
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
Comment 9 Nick Clemens (kidclamp) 2019-06-06 13:30:55 UTC
Squashed as axel rewrote my patch :-)

Some issues:
1 - Missing filter from QA tool - this is tricky too - if I add </script> to the title I can break the view - we need to make sure the JOSN can't break
2 - We should order the view, at least for _source - this could be later enhancement, but would make it much easier to read
Comment 10 Nick Clemens (kidclamp) 2020-08-12 13:12:42 UTC
Created attachment 108103 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElasticSearch" as value
- Search whatever you want in Search the catalog and select a record
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
- note fields are alphabetized
- from the command line delete the es record:
    curl -XDELETE es:9200/koha_kohadev_biblios/data/5
- click the preview link, it syas record not found
- check the logs - you see the error on missing record
Comment 11 Nick Clemens (kidclamp) 2020-08-12 13:17:26 UTC
Created attachment 108104 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElasticSearch" as value
- Search whatever you want in Search the catalog and select a record
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
- note fields are alphabetized
- from the command line delete the es record:
    curl -XDELETE es:9200/koha_kohadev_biblios/data/5
- click the preview link, it syas record not found
- check the logs - you see the error on missing record
Comment 12 Fridolin Somers 2020-08-13 15:08:30 UTC
Nice work, I like it.
Comment 13 Fridolin Somers 2020-08-13 15:10:46 UTC
Created attachment 108191 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElasticSearch" as value
- Search whatever you want in Search the catalog and select a record
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
- note fields are alphabetized
- from the command line delete the es record:
    curl -XDELETE es:9200/koha_kohadev_biblios/data/5
- click the preview link, it syas record not found
- check the logs - you see the error on missing record

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Comment 14 Katrin Fischer 2020-08-22 23:39:13 UTC
Created attachment 108944 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElasticSearch" as value
- Search whatever you want in Search the catalog and select a record
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
- note fields are alphabetized
- from the command line delete the es record:
    curl -XDELETE es:9200/koha_kohadev_biblios/data/5
- click the preview link, it syas record not found
- check the logs - you see the error on missing record

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 15 Katrin Fischer 2020-08-22 23:39:18 UTC
Created attachment 108945 [details] [review]
Bug 18829: (QA follow-up) Fix punctuation, capitalization and page title

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 16 Katrin Fischer 2020-08-22 23:39:37 UTC
I think this is a nice little enhancement as is. 

I'd love to see codemirror JSON highlighting used on the modal in the future.
Comment 17 Jonathan Druart 2020-08-26 13:18:15 UTC
1. 
+catch{
+    @es_fields = ("Error fetching record: see logs for details");
+    warn $_;
+};

This is not the way to deal with error, you should return an error, here certainly a 404.

2. Incorrect unicode characters handling.
Add '❤' to the title then click the link, you get a JS error: "URIError: malformed URI sequence" and the modal is empty.
Comment 18 Nick Clemens (kidclamp) 2023-03-27 12:10:17 UTC
Created attachment 148757 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElasticSearch" as value
- Search whatever you want in Search the catalog and select a record
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
- note fields are alphabetized
- from the command line delete the es record:
    curl -XDELETE es:9200/koha_kohadev_biblios/data/5
- click the preview link, it syas record not found
- check the logs - you see the error on missing record

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 19 Nick Clemens (kidclamp) 2023-03-27 12:10:20 UTC
Created attachment 148758 [details] [review]
Bug 18829: (QA follow-up) Fix punctuation, capitalization and page title

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 20 Nick Clemens (kidclamp) 2023-03-27 12:10:23 UTC
Created attachment 148759 [details] [review]
Bug 18829: (follow-up) Return 404 when record cannot be loaded
Comment 21 Nick Clemens (kidclamp) 2023-03-27 12:10:26 UTC
Created attachment 148760 [details] [review]
Bug 18829: (follow-up) Don't messup record encoding
Comment 22 Katrin Fischer 2023-04-10 17:08:04 UTC
Created attachment 149400 [details] [review]
Bug 18829: Elasticsearch - Put a better display for the view of ES indexed record

To test:
- apply the patch
- go to global sysPref
- make sure that SearchEngine have "ElasticSearch" as value
- Search whatever you want in Search the catalog and select a record
- Search the link (Ctrl f) : "Elasticsearch Record :" , click on the link
- make sure that the pop-up open and you should see the elasticsearch result in JSON (the pop-up look the same as MARC preview)
- note fields are alphabetized
- from the command line delete the es record:
    curl -XDELETE es:9200/koha_kohadev_biblios/data/5
- click the preview link, it syas record not found
- check the logs - you see the error on missing record

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 23 Katrin Fischer 2023-04-10 17:08:07 UTC
Created attachment 149401 [details] [review]
Bug 18829: (QA follow-up) Fix punctuation, capitalization and page title

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 24 Katrin Fischer 2023-04-10 17:08:11 UTC
Created attachment 149402 [details] [review]
Bug 18829: (follow-up) Return 404 when record cannot be loaded
Comment 25 Katrin Fischer 2023-04-10 17:08:15 UTC
Created attachment 149403 [details] [review]
Bug 18829: (follow-up) Don't messup record encoding

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 26 Katrin Fischer 2023-04-10 17:08:18 UTC
Created attachment 149404 [details] [review]
Bug 18829: (QA follow-up) I18N, test character and breadcrumbs

* Removes a 'heart' character from the file, probably added for testing
* Corrects breadbrumbs to make QA test tool happy
* Makes JS message translatable (test by manipulating the link to an invalid record number in DOM)

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 27 Tomás Cohen Arazi 2023-04-14 18:39:08 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 29 Tomás Cohen Arazi 2023-04-17 11:26:39 UTC
Created attachment 149745 [details] [review]
Bug 18829: (QA follow-up) Replace forbidden indirect syntax

This pattern is forbidden and was overlooked by QA.

To test:
1. Run:
   $ ktd --shell
  k$ prove t/00-testcritic.t
=> FAIL: Tests fail with 'Subroutine "new" called using indirect
syntax...'
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests pass
4. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 30 Matt Blenkinsop 2023-04-18 12:21:10 UTC
Enhancement - not backporting to 22.11.x

Nice work everyone!