Bug 30745 - ES : Search never return result using after-date and/or before-date in labels/label-item-search.pl
Summary: ES : Search never return result using after-date and/or before-date in labels...
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low normal with 4 votes (vote)
Assignee: Emily-Rose Francoeur
QA Contact: Testopia
URL:
Keywords:
: 28987 30908 (view as bug list)
Depends on:
Blocks:
 
Reported: 2022-05-13 08:12 UTC by Alex Arnaud
Modified: 2024-03-26 18:29 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 30745: Use query builder and search_compat in label-item-search (3.24 KB, patch)
2022-05-13 08:15 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 30745: handle ES ranges for date of acquisition (4.02 KB, patch)
2022-05-13 08:15 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 30745: Use query builder and search_compat in label-item-search (3.31 KB, patch)
2022-06-07 15:11 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 30745: handle ES ranges for date of acquisition (4.08 KB, patch)
2022-06-07 15:11 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 30745: Use query builder and search_compat in label-item-search (3.42 KB, patch)
2022-06-12 11:06 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 30745: handle ES ranges for date of acquisition (4.19 KB, patch)
2022-06-12 11:06 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 30745: Use query builder and search_compat in label-item-search (2.93 KB, patch)
2023-10-16 13:41 UTC, Emily-Rose Francoeur
Details | Diff | Splinter Review
Bug 30745: Use query builder and search_compat in label-item-search (2.93 KB, patch)
2023-10-16 14:15 UTC, Emily-Rose Francoeur
Details | Diff | Splinter Review
Bug 30745: handle ES ranges for date of acquisition (4.20 KB, patch)
2023-10-16 14:15 UTC, Emily-Rose Francoeur
Details | Diff | Splinter Review
Bug 30745: Display pagination for the results page (6.64 KB, patch)
2023-10-26 15:34 UTC, Emily-Rose Francoeur
Details | Diff | Splinter Review
Bug 30745 : Fix undefined value error while using zebra (1.62 KB, patch)
2024-03-06 20:15 UTC, Phan Tung Bui
Details | Diff | Splinter Review
Bug 30745 : Fix pagination error (5.36 KB, patch)
2024-03-26 18:29 UTC, Phan Tung Bui
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Arnaud 2022-05-13 08:12:55 UTC
With elasticsearch as search engine, using the "after date" or "before date" form elements in labels/label-item-search.pl never return any results.

However, searching a specific date of acquisition in the index "Acquisition date" works.

Bug 16355 report a similar problem, but there is already a fix and it seems to concern only MARC21.

Patch is coming.
Comment 1 Alex Arnaud 2022-05-13 08:15:23 UTC
Created attachment 134952 [details] [review]
Bug 30745: Use query builder and search_compat in label-item-search

Test plan:
  - apply this patch,
  - make some searches in labels/label-item-search.pl using
    the 3 form input (index, after date, before date),
  - check it works like before
Comment 2 Alex Arnaud 2022-05-13 08:15:30 UTC
Created attachment 134953 [details] [review]
Bug 30745: handle ES ranges for date of acquisition

Test plan:
  - Enable elasticsearch
  - git to labels/label-item-search.pl
  - search using after date only
    => check you get items with date-of-acquisition greater than after
date
  - search using before date only
    => check you get items with date-of-acquisition less than before
date
  - search using after and before date
    => check you get items with date-of-acquisition between after and
before
  - Combine this searches with a specified index
  - enable zebra and repeat these tests
Comment 3 Shi Yao Wang 2022-05-24 21:30:29 UTC
For Bug 16355, the patch doesn't apply and the last comment is from 2018.

I tested the patch with elasticsearch and it works for me, but I do not have a functioning zebra to test with. Do I signoff nonetheless?
Comment 4 Andrew Fuerste-Henry 2022-06-07 13:56:15 UTC
*** Bug 30908 has been marked as a duplicate of this bug. ***
Comment 5 Andrew Fuerste-Henry 2022-06-07 15:11:33 UTC
Created attachment 135784 [details] [review]
Bug 30745: Use query builder and search_compat in label-item-search

Test plan:
  - apply this patch,
  - make some searches in labels/label-item-search.pl using
    the 3 form input (index, after date, before date),
  - check it works like before

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 6 Andrew Fuerste-Henry 2022-06-07 15:11:37 UTC
Created attachment 135785 [details] [review]
Bug 30745: handle ES ranges for date of acquisition

Test plan:
  - Enable elasticsearch
  - git to labels/label-item-search.pl
  - search using after date only
    => check you get items with date-of-acquisition greater than after
date
  - search using before date only
    => check you get items with date-of-acquisition less than before
date
  - search using after and before date
    => check you get items with date-of-acquisition between after and
before
  - Combine this searches with a specified index
  - enable zebra and repeat these tests

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 7 Katrin Fischer 2022-06-12 10:51:37 UTC
Alex, please don't forget to assign the bug :)
Comment 8 Katrin Fischer 2022-06-12 11:06:16 UTC
Created attachment 135966 [details] [review]
Bug 30745: Use query builder and search_compat in label-item-search

Test plan:
  - apply this patch,
  - make some searches in labels/label-item-search.pl using
    the 3 form input (index, after date, before date),
  - check it works like before

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

https://bugs.koha-community.org/show_bug.cgi?id=30845
Comment 9 Katrin Fischer 2022-06-12 11:06:21 UTC
Created attachment 135967 [details] [review]
Bug 30745: handle ES ranges for date of acquisition

Test plan:
  - Enable elasticsearch
  - git to labels/label-item-search.pl
  - search using after date only
    => check you get items with date-of-acquisition greater than after
date
  - search using before date only
    => check you get items with date-of-acquisition less than before
date
  - search using after and before date
    => check you get items with date-of-acquisition between after and
before
  - Combine this searches with a specified index
  - enable zebra and repeat these tests

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

https://bugs.koha-community.org/show_bug.cgi?id=30845
Comment 10 Katrin Fischer 2022-06-12 11:07:29 UTC
This tests well, so happy to sign-off, but we are missing unit tests for the changes to Querybuilder.pm. Let us know if you need help with this.
Comment 11 Katrin Fischer 2022-07-16 10:33:08 UTC
*** Bug 28987 has been marked as a duplicate of this bug. ***
Comment 12 Fridolin Somers 2023-07-21 00:15:43 UTC
Patches need rebase
Comment 13 Emily-Rose Francoeur 2023-10-16 13:41:48 UTC
Created attachment 157192 [details] [review]
Bug 30745: Use query builder and search_compat in label-item-search

Test plan:
  - apply this patch,
  - make some searches in labels/label-item-search.pl using
    the 3 form input (index, after date, before date),
  - check it works like before

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

https://bugs.koha-community.org/show_bug.cgi?id=30845
Comment 14 Emily-Rose Francoeur 2023-10-16 14:15:00 UTC
Created attachment 157194 [details] [review]
Bug 30745: Use query builder and search_compat in label-item-search

Test plan:
  - apply this patch,
  - make some searches in labels/label-item-search.pl using
    the 3 form input (index, after date, before date),
  - check it works like before

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

https://bugs.koha-community.org/show_bug.cgi?id=30845
Comment 15 Emily-Rose Francoeur 2023-10-16 14:15:03 UTC
Created attachment 157195 [details] [review]
Bug 30745: handle ES ranges for date of acquisition

Test plan:
  - Enable elasticsearch
  - git to labels/label-item-search.pl
  - search using after date only
    => check you get items with date-of-acquisition greater than after
date
  - search using before date only
    => check you get items with date-of-acquisition less than before
date
  - search using after and before date
    => check you get items with date-of-acquisition between after and
before
  - Combine this searches with a specified index
  - enable zebra and repeat these tests

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

https://bugs.koha-community.org/show_bug.cgi?id=30845
Comment 16 Emily-Rose Francoeur 2023-10-16 14:18:52 UTC
The patch has been rebased.
Comment 17 Emily-Rose Francoeur 2023-10-26 15:34:12 UTC
Created attachment 157934 [details] [review]
Bug 30745: Display pagination for the results page

I have fixed the pagination issue when the search returns multiple results.

Test plan:
    - Enable elasticsearch
    - git to labels/label-item-search.pl
    - search using after date only
            => check you get items with date-of-acquisition greater than after
date
    - search using before date only
            => check you get items with date-of-acquisition less than before
date
    - search using after and before date
            => check you get items with date-of-acquisition between after and
before
    - Combine this searches with a specified index
    - enable zebra and repeat these tests
pagination tests
    - Perform a search that returns more than 20 notices
    - Navigate through different result pages using page numbers, the "Next" button, and the "Previous" button
	    => On each page, check that "Results X through X of X" displays the correct information
Comment 18 Marie-Luce Laflamme 2024-01-04 15:24:18 UTC
Is there a kind volunteer who could spare some time to test this rebased patch?
Your feedback is much appreciated.
Cheers!
Comment 19 David Nind 2024-01-07 14:23:46 UTC
I was about to sign off, but then came across this Zebra only error - I get this error after the patches are applied (and a reindex):

   Error: Can't use an undefined value as an ARRAY reference at /kohadevbox/koha/labels/label-item-search.pl line 88
   ....

1. Generated by putting a letter in the input box for these index searches (without any dates in the filters): publication date, standard number, ISBN, ISSN, call number, acquisition date

2. I could also generate this error with a number in a text field, for example: Author index, 2011 (but with 2000 no error is generated)

3. With Elasticsearch, no results are displayed and the form just refreshes (both before and after the patches, and the same with Zebra before the patches)

Other notes
===========

1. I couldn't work out how or where cgi-bin/koha/labels/label-item-search.pl is used/accessed from - something to do with acquisitions?

2. I've changed the assignee.

3. I didn't test with a UNIMARC installation.

Testing notes (using koha-testing-docker)
=========================================

Setup
~~~~~

To test with Elasticsearch or OpenSearch - start up KTD with Elasticsearch or OpenSearch: ktd --es8 up OR ktd --os2 up (I didn't test with Open Search, just Zebra and Elasticsearch)

Go to this page in the staff interface: http://127.0.0.1:8081/cgi-bin/koha/labels/label-item-search.pl

Searching using Zebra (after KTD is started with --es8):
- Change the SearchEngine system preference to Zebra
- Reload http://127.0.0.1:8081/cgi-bin/koha/labels/label-item-search.pl
- Reindex: koha-rebuild-zebra -d -f -v kohadev
- Repeat all the example searches - you should get the same results

Demonstrating the issue and that the patches fix it
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Notes: In the sample data, the acquisition date is recorded in 952$d - Date acquired. 
       For many items, 09/04/2014 is used.
       For some items, these dates are used 05/08/2014, 05/07/2014, 10/15/2014, 12/04/2014, 1/05/2015.
       Many items don't have a date recorded.
       If there is no value in the index field, it is ignored.
       If an index is selected and a value is entered, the results should be filtered by any values entered in 'Added on or after date' and 'Added on or before date' fields.

Before the patches: index searches work, but when used with 'Added on or after date' and 'Added on or before date' options, the search results ignore them. However, entering a date for the 'Acquisition date' index does work.

A title index search:
- Title index: perl
  => Expected result: 11
  => Actual result: 11

A combined search with dates:
- Title index: perl
- Added on or after date: 09/04/2014
  => Expected result: 10 (1 perl record has no items)
  => Actual result: No results

A combined search with dates:
- Title index: perl
- Added on or before date: 09/04/2014
  => Expected result: 10 (1 perl record has no items)
  => Actual result: No results

Acquisition date index search:
- Index: select 'Acquisition date' from the dropdown list
- Value to enter: 09/04/2014
  => Expected result: 352 results
  => Actual result: 352 result

Acquisition date index search:
- Index: select 'Acquisition date' from the dropdown list
- Value to enter: 09/04/2014
- Added on or after date: 09/04/2014
  OR
- Added on or before date: 09/04/2014
  OR
  Use both or none
  => Expected result: 352 results
  => Actual result: 352 results

Search using 'Added on or after date' (after date) only:
- Added on or after date: 09/04/2014
  => Expected result: should get 408 records
  => Actual result: 0 results

Search using 'Added on or before date' (before date) only:
- Added on or after date: 09/04/2014
  => Expected result: should get 354 records
  => Actual result: 0 results

Publication date index search using 2011 (AFTER patches are applied - doesn't work as expected BEFORE the patches are applied):
- Results if no before or after filter added: 8 (as expected)
- Results if Added on or after date: 09/04/2014 - 8 (as expected)
- Results if Added on or before date: 09/04/2014 - 7 (as expected)
- Results for a date range - 7 (as expected):
  . Added on or after date: 09/04/2014
  . Added on or before date: 09/05/2014
- Results for a date range - 8 (as expected):
  . Added on or after date: 09/04/2014
  . Added on or before date: 10/15/2014
- Results for a date range - 1 (as expected):
  . Added on or after date: 09/05/2014
  . Added on or before date: 10/15/2014
(One record and all its items has acquisition dates of 10/15/2014 - Blackstone's statutes on company law 2014-2015; the rest have 09/04/2014 acquisition dates)

Search using just a date range (and no index) (AFTER patches are applied - doesn't work as expected BEFORE the patches are applied):
- Added on or after date: 09/04/2014
  => should get 408 results
- Added on or before date: 09/03/2014
  => should get 3 results
- Added on or before date: today's date
  => should get 410 results
    (note: 408 + 3 doesn't equal 410 but it is correct, as one record has items with different acquisition dates: The C programming language has items with different acquisition dates - 1 x 09/04/2014 and 3 x 05/08/2014)
- Added on or after date: 09/05/2014
  => should get 56 results
- Added on or after date: 12/04/2014
  => should get 40 results
- Added on or after date: 09/05/2014
- Added on or before date: 12/03/2014
  => should get 16 records


Pagination of search results
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Perform a search that returns more than 20 results -  for example Title index, a:

2. Navigate through different result page navigation using page numbers, the "Next" button, and the "Previous" button
  => On each page, check that "Results X through X of X" displays the correct information
     - 20 results are displayed for each page
     - the total number of results should not change
Comment 20 Phan Tung Bui 2024-03-06 20:15:48 UTC
Created attachment 162855 [details] [review]
Bug 30745 : Fix undefined value error while using zebra

Plan test : 
- Apply this batch
- Set Zebra as searchEngine
- Put a letter in the input box for these index searches (without any dates in the filters): publication date, standard number, ISBN, ISSN, call number, acquisition date
- The page's supposed to refresh itself (so that it has the same behavior as elasticSearch)
Comment 21 David Nind 2024-03-25 04:50:01 UTC
Thanks for fixing the error identified in the previous testing. This no longer occurs (using Zebra and putting a letter in number index fields such as ISBN, Publication date, and so on). 

However, for:

1. Search results: the pagination is no longer working - it just goes back to the search form if you click on next or any of the numbers. There are some console errors on the search results page (using the web developer tools)

2. The select all and clear all links no longer work
Comment 22 David Nind 2024-03-25 05:51:45 UTC
Why the pagination and links aren't working working may be because of the CSRF changes in bug 34478 and bug 36192 (but I don't know enough to know if that is the cause).
Comment 23 Phan Tung Bui 2024-03-26 18:29:35 UTC
Created attachment 163955 [details] [review]
Bug 30745 : Fix pagination error

1. I've fixed the pagination option
2. "Select all" and "clear all" buttons seem all functional for me. Could you double check it with my new test plan?

Test Plan 
- Activate Zebra as search engine
- Cataloging ->  Label Creator 
- Click on "New" then "Label Batch"
- Click on "Add item(s)"
- Choose a search option (For exemple i put "a" for "Title" so that we can obtain a result that has multiple pages)
- Click on "Search"
- Scroll all the pages using page number or "Previous" and "Next" button to test the pagination option
- Click on "Select all" or "Clear all" to test the functionality of both buttons then click on "Add checked", the barcode of the seleccted items will be added to the "Add by barcode(s) or itemnumbers(s)" case