Bug 40966 - 'whole_record' and 'weighted_fields' not passed around
Summary: 'whole_record' and 'weighted_fields' not passed around
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low major
Assignee: Tomás Cohen Arazi (tcohen)
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 40968
  Show dependency treegraph
 
Reported: 2025-10-07 14:26 UTC by Tomás Cohen Arazi (tcohen)
Modified: 2025-12-27 10:25 UTC (History)
3 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around (2.14 KB, patch)
2025-10-07 14:42 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Regression tests (2.68 KB, patch)
2025-10-07 14:52 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around (2.38 KB, patch)
2025-10-07 14:52 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Regression tests (2.86 KB, patch)
2025-10-15 21:24 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around (9.45 KB, patch)
2025-10-15 21:24 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Regression tests (2.87 KB, patch)
2025-12-26 20:45 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around (13.62 KB, patch)
2025-12-26 20:45 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around (13.66 KB, patch)
2025-12-26 21:00 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 40966: Regression tests (2.90 KB, patch)
2025-12-27 10:10 UTC, David Nind
Details | Diff | Splinter Review
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around (13.69 KB, patch)
2025-12-27 10:10 UTC, David Nind
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Tomás Cohen Arazi (tcohen) 2025-10-07 14:26:40 UTC
To reproduce:

1. Set ElasticsearchMARCFormat="Searchable array"
2. Reindex all the things:
   $ ktd --shell
  k$ koha-elasticsearch --rebuild -d -v kohadev
3. Go to advanced search, using the 'More options' option.
4. Check 'Search entire MARC record' and 'Apply weights to search'
5. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
6. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
Comment 1 Tomás Cohen Arazi (tcohen) 2025-10-07 14:42:30 UTC Comment hidden (obsolete)
Comment 2 Tomás Cohen Arazi (tcohen) 2025-10-07 14:52:01 UTC
Created attachment 187531 [details] [review]
Bug 40966: Regression tests

Tests verify that whole_record and weighted_fields parameters are
correctly included in the query_cgi string returned by build_query_compat(),
ensuring these parameters are preserved in facet links and pagination.

To test:
1. Apply this patch
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: The values are not being passed around
Comment 3 Tomás Cohen Arazi (tcohen) 2025-10-07 14:52:04 UTC
Created attachment 187532 [details] [review]
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around

The ES query builder correctly handles this parameters (passed from
search.pl and opac-search.pl) when it builds the ES query. But the
returned 'query_cgi' that is used later for keeping the search results
query context for building things like facets, just doesn't pass this
values around, leading to a loss of them on the generated links.

This patch makes sure they are passed around.

To test:

1. Apply the regression tests
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail! These params are not passed around!
3. Set ElasticsearchMARCFormat="Searchable array"
4. Reindex all the things:
  k$ koha-elasticsearch --rebuild -d -v kohadev
5. Go to advanced search, using the 'More options' option.
6. Check 'Search entire MARC record' and 'Apply weights to search'
7. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
8. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
9. Apply this patch
10. Restart all
  k$ restart_all
11. Refresh the search results
12. Repeat 6
=> SUCCESS: Links have the needed parameters!
13. Repeat 2
=> SUCCESS: Tests pass! the parameters are passed around!
14. Sign off :-D
Comment 4 Nick Clemens (kidclamp) 2025-10-15 15:19:58 UTC
This fixes it for the 'on' case - but if I uncheck "Apply field weights to search" - do a search, then click a facet, the weights are applied again

The issue is from:
my $weight_search    = $params->{'advsearch'} ? $params->{'weight_search'} || 0 : 1;
Comment 5 Tomás Cohen Arazi (tcohen) 2025-10-15 21:24:35 UTC
Created attachment 187945 [details] [review]
Bug 40966: Regression tests

Tests verify that whole_record and weighted_fields parameters are
correctly included in the query_cgi string returned by build_query_compat(),
ensuring these parameters are preserved in facet links and pagination.

To test:
1. Apply this patch
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: The values are not being passed around
Comment 6 Tomás Cohen Arazi (tcohen) 2025-10-15 21:24:39 UTC
Created attachment 187946 [details] [review]
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around

The ES query builder correctly handles this parameters (passed from
search.pl and opac-search.pl) when it builds the ES query. But the
returned 'query_cgi' that is used later for keeping the search results
query context for building things like facets, just doesn't pass this
values around, leading to a loss of them on the generated links.

This patch makes sure they are passed around.

To test:

1. Apply the regression tests
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail! These params are not passed around!
3. Set ElasticsearchMARCFormat="Searchable array"
4. Reindex all the things:
  k$ koha-elasticsearch --rebuild -d -v kohadev
5. Go to advanced search, using the 'More options' option.
6. Check 'Search entire MARC record' and 'Apply weights to search'
7. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
8. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
9. Apply this patch
10. Restart all
  k$ restart_all
11. Refresh the search results
12. Repeat 6
=> SUCCESS: Links have the needed parameters!
13. Repeat 2
=> SUCCESS: Tests pass! the parameters are passed around!
14. Sign off :-D
Comment 7 David Nind 2025-10-25 20:24:33 UTC
For step 7, I'm assuming you put 'FULL TEXT' in the first keyword field.

I'm getting this for search results, instead of results and facets (normal advanced searches work fine):

  No results found
  You did not specify any search criteria.

  Error: Unable to perform your search. Please try again.
Comment 8 Tomás Cohen Arazi (tcohen) 2025-10-25 20:25:39 UTC
(In reply to David Nind from comment #7)
> For step 7, I'm assuming you put 'FULL TEXT' in the first keyword field.
> 
> I'm getting this for search results, instead of results and facets (normal
> advanced searches work fine):
> 
>   No results found
>   You did not specify any search criteria.
> 
>   Error: Unable to perform your search. Please try again.

I chose phrase search instead of keyword.
Comment 9 David Nind 2025-10-25 21:49:05 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #8)
> (In reply to David Nind from comment #7)
> > For step 7, I'm assuming you put 'FULL TEXT' in the first keyword field.
> > 
> > I'm getting this for search results, instead of results and facets (normal
> > advanced searches work fine):
> > 
> >   No results found
> >   You did not specify any search criteria.
> > 
> >   Error: Unable to perform your search. Please try again.
> 
> I chose phrase search instead of keyword.

I chose 'Keyword as phrase' - without 'Search entire MARC record' I get a normal no results page, with it ticked I get the same error as above.

With both patches applied, the tests now pass but I get the same error.

(I'm using ktd --es7 up, as I haven't been able to get es8 to work for a while on KTD.)
Comment 10 David Nind 2025-12-26 02:23:54 UTC
I had another go at testing (using ES7), but couldn't get any search results (got an error) when checking "Search entire MARC record".

Testing notes, from steps 5-7:
1. Search > Advanced search
2. More options
3. Changed keyword for the first 'Search for' to 'Keyword as phrase'
4. Search term entered: agrovoc
5. Checked "Search entire MARC record" ("Apply field weights to search" was already checked")
5. I get:

     No results found
     You did not specify any search criteria.

     Error: Unable to perform your search. Please try again.

6. The URL includes `whole_record=on` and `weight_search=on`, but not `weighted_fields=on`. The full URL is:

/cgi-bin/koha/catalogue/search.pl?advsearch=1&idx=kw%2Cphr&q=agrovoc&whole_record=on&weight_search=on&sort_by=relevance

7. Note: I do get search results when only "Apply field weights to search" s checked, with 'Subject as phrase' and 'Keyword as phrase'. I get the error if I have "Search entire MARC record" checked as well.
8. If I apply the second patch and repeat the above steps, I still get the same error as in step 5, but the URL is now:

/cgi-bin/koha/catalogue/search.pl?advsearch=1&idx=kw%2Cphr&q=agrovoc&whole_record=on&weight_search_submitted=1&weight_search=on&sort_by=relevance

9. With the second patch, the tests pass.
10. All of the above was in the staff interface. For the OPAC, I don't have a "Search entire MARC record" option to check for the advanced search.
Comment 11 Tomás Cohen Arazi (tcohen) 2025-12-26 20:45:55 UTC
Created attachment 190740 [details] [review]
Bug 40966: Regression tests

Tests verify that whole_record and weighted_fields parameters are
correctly included in the query_cgi string returned by build_query_compat(),
ensuring these parameters are preserved in facet links and pagination.

To test:
1. Apply this patch
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: The values are not being passed around
Comment 12 Tomás Cohen Arazi (tcohen) 2025-12-26 20:45:57 UTC
Created attachment 190741 [details] [review]
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around

The ES query builder correctly handles this parameters (passed from
search.pl and opac-search.pl) when it builds the ES query. But the
returned 'query_cgi' that is used later for keeping the search results
query context for building things like facets, just doesn't pass this
values around, leading to a loss of them on the generated links.

This patch makes sure they are passed around.

To test:

1. Apply the regression tests
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail! These params are not passed around!
3. Set ElasticsearchMARCFormat="Searchable array"
4. Reindex all the things:
  k$ koha-elasticsearch --rebuild -d -v kohadev
5. Go to advanced search, using the 'More options' option.
6. Check 'Search entire MARC record' and 'Apply weights to search'
7. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
8. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
9. Apply this patch
10. Restart all
  k$ restart_all
11. Refresh the search results
12. Repeat 6
=> SUCCESS: Links have the needed parameters!
13. Repeat 2
=> SUCCESS: Tests pass! the parameters are passed around!
14. Sign off :-D
Comment 13 Tomás Cohen Arazi (tcohen) 2025-12-26 21:00:07 UTC
Created attachment 190742 [details] [review]
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around

The ES query builder correctly handles this parameters (passed from
search.pl and opac-search.pl) when it builds the ES query. But the
returned 'query_cgi' that is used later for keeping the search results
query context for building things like facets, just doesn't pass this
values around, leading to a loss of them on the generated links.

This patch makes sure they are passed around.

To test:

1. Apply the regression tests
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail! These params are not passed around!
3. Set ElasticsearchMARCFormat="Searchable array"
4. Reindex all the things:
  k$ koha-elasticsearch --rebuild -d -v kohadev
5. Go to advanced search, using the 'More options' option.
6. Check 'Search entire MARC record' and 'Apply weights to search'
7. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
8. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
9. Apply this patch
10. Restart all
  k$ restart_all
11. Refresh the search results
12. Repeat 6
=> SUCCESS: Links have the needed parameters!
13. Repeat 2
=> SUCCESS: Tests pass! the parameters are passed around!
14. Sign off :-D
Comment 14 Tomás Cohen Arazi (tcohen) 2025-12-26 21:02:59 UTC
Hi, David. Thanks for taking the time! I've just started a fresh KTD like this [1]:

```shell
cd ~/git/koha-testing-docker
ktd --proxy --search-engine es9 up -d
```

It is using Elasticsearch 9 (tried with 8 too). With the fixed patches I submitted. things work as expected both in staff and OPAC on both ES versions. I tested searching 'agrovoc' on the 'More options' (in both interfaces) with and without 'Apply field weights...' and 'Search entire MARC...' (the latter only applies to the staff interface). Chosen options are passed around on the facets (this was working) and now also on the 'Edit this search' and 'Return to the last advanced search' respectively.

Somehow a part of the dev got lost I think. I introduced 'weight_search_submitted' to keep track of the user unchecking 'weight_search' but the template was not taking it into account. Sorry for that. Probably a bad commit.


[1] You don't need to use `--proxy` for it to work.
Comment 15 David Nind 2025-12-26 23:52:01 UTC
Thanks Tomás for looking at this again!

I'll have to leave re-tetsing until I've got a working environment with Elasticsearch ES8 or ES9 - hopefully someone who as working environment can test!

Notes:
- I've tried to retest, but I can't get KTD to work with either ES8 or ES9, only ES7.
- I get similar errors to comment #10 step 5.
- I think it is obviously something to do with my local environment (will attempt to setup KTD on my Windows desktop).
- I also tried on the sandboxes (Open Fifth and BibLibre - errors when doing a full reindex after switching to Elasticserch; ByWater Solutions - "You are not from an authorized IP.").
Comment 16 David Nind 2025-12-27 10:10:56 UTC
Created attachment 190745 [details] [review]
Bug 40966: Regression tests

Tests verify that whole_record and weighted_fields parameters are
correctly included in the query_cgi string returned by build_query_compat(),
ensuring these parameters are preserved in facet links and pagination.

To test:
1. Apply this patch
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: The values are not being passed around

Signed-off-by: David Nind <david@davidnind.com>
Comment 17 David Nind 2025-12-27 10:10:59 UTC
Created attachment 190746 [details] [review]
Bug 40966: Make sure 'whole_record' and 'weighted_fields' are passed around

The ES query builder correctly handles this parameters (passed from
search.pl and opac-search.pl) when it builds the ES query. But the
returned 'query_cgi' that is used later for keeping the search results
query context for building things like facets, just doesn't pass this
values around, leading to a loss of them on the generated links.

This patch makes sure they are passed around.

To test:

1. Apply the regression tests
2. Run:
   $ ktd --shell
  k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail! These params are not passed around!
3. Set ElasticsearchMARCFormat="Searchable array"
4. Reindex all the things:
  k$ koha-elasticsearch --rebuild -d -v kohadev
5. Go to advanced search, using the 'More options' option.
6. Check 'Search entire MARC record' and 'Apply weights to search'
7. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
8. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
9. Apply this patch
10. Restart all
  k$ restart_all
11. Refresh the search results
12. Repeat 6
=> SUCCESS: Links have the needed parameters!
13. Repeat 2
=> SUCCESS: Tests pass! the parameters are passed around!
14. Sign off :-D

Signed-off-by: David Nind <david@davidnind.com>
Comment 18 David Nind 2025-12-27 10:25:26 UTC
Got ES8 and ES9 working with KTD on a Windows desktop. Things worked as per the test plan.

I used 'world' for the 'Keyword as phrase' search term.