Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. (the atomicupdate sql-file is a bit obfuscated since had to be able to run in multiple time without error because of our local deploy-script, when moving this all procedure related code can be removed, only the "ALTER TABLE.."-lines are relevant.)
Created attachment 74245 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) No reindex is really required, but still, reindex authorities and biblios to verify that this still works. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Created attachment 74249 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) No reindex is really required, but still, reindex authorities and biblios to verify that this still works. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Created attachment 74301 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) No reindex is really required, but still, reindex authorities and biblios to verify that this still works. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Added a *.perl update script instead of the custom SQL-script.
Created attachment 74373 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) No reindex is really required, but still, reindex authorities and biblios to verify that this still works. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Had forgotten to take new "search" field into account for mappings, and also fixed where had not removed all of the _all-field mapping (which would have caused indexing to crash).
Created attachment 74418 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) No reindex is really required, but still, reindex authorities and biblios to verify that this still works. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Discovered search where missing in '+as', and '+select' clause, which causes indexing to crash, will fix.
Created attachment 74419 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) No reindex is really required, but still, reindex authorities and biblios to verify that this still works. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Searchable parameter on search fields (staff/opac) is nice. But, is searchable parameter on mapping useful? We can just remove the mapping, right? This patch could be rebased on bug 18316, that brings boost for simple search.
Yes, I think the best option would be to try getting bug_18316 merged, and then refactor this upon that. I completely missed bug_18316 before starting to work on this. I tried applying bug_18316 but was unable to do so because of some git-error, think there might be something wonky with the current patches.
And about the searchable parameter, I think we are going to need this in the future so included it as did not see a major downside of doing so. The problem this might solve is that we would want to have exclusive "sort"-field mappings, not included in the main field. The idea is to allow mappings to be written on the form "035a|035b|035c", and this would mean that the first field that has a value gets mapped, the rest skipped. We have disabled sorting since it's very unpredictable for multivalued fields, and this was seen as a possible future solution.
Hi David, if you write bug 18316 instead of bug_18316, bugzilla will automagically link it. :)
I a bit lost because of last comments and allusion about merging it with bug 18316 : should this one be tested ?
I think this patch in it's current state needs to be refactored with regard to the changes introduces in Bug 18316 once has been pushed to master. I will do this, but will probably wait until it has been pushed (which it seems close to be).
I have started working on this now since 18316 has been merged. Will update beginning of next week.
Created attachment 82503 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
This is still work in progress. Have not adjusted tests for the refactored code for example. There are parts of this patch regarding fixes for authorities search that could be a separate patch, but since removing '_all' also requires adjusting this patch such a patch would probably cause merge conflict with this one. So it could be simple to bundle theses changes as one patch.
*but since removing '_all' also requires adjusting this _part_ such a patch would probably cause merge conflict with this one. So it could be simple to bundle theses changes as one patch.
I will write a summary of the changes with motivations behind each one as this can be difficult to understand by just looking at the code. I'm also a little bit confused my self with regards to the correct behaviour of build_auth_query_compat for example. And would probably have to setup a Zebra instance to find out exactly how the different options are supposed to work.
Created attachment 82544 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Here is a more verbose explaction of most of the changes: - Remove syspref QueryWeightFields as this seems not to be used anyway. - Replace hard coded 'biblios' and 'authorities' in mappings.pl with constants. - Add _search_fields method in QueryParser for returning weighted/unweighted search fields for OPAC/STaffClient (as optional subfield, used primarily for authorities search). Refactor authorities search. This part is a a litte bit iffy: What mainly confuses me is the motivation behind the differentiation between is/=, and "exact". With regards to elastic that operator only seems to be used in C4/Matcher.pm. It also occurs in Koha/QueryParser/Driver/PQF.pm and Koha/REST/Plugin/Query.pm but i don't believe Elastic is used as backend for those. What I can gather there where actually no difference between is/= and "exact" before my changes as is/= performed a "term" query on "<field>.phrase" and "exact" performed a "match_phrase" on the same field, which is pointless since the "phrase" subfield is not tokenized (used keyword analyser). So I simplified this and use the same query for both is/= and "exact". As "exact" queries seems to be used for matching, I also thought it would be more fitting to perform a case insensitive match instead of also removing punctuation (which could result in unexpected matches). I also noticed that the Authorities type drop-down in the authorities search was not respected, so this is not included in the query if selected. Also noted there is an $and_or option to build_authorities_query_compat that is not used, this should probably be addressed, but not fixed in this patch. Tidy up the index/fields configuration a little bit: - Rename all occurrences of misspelled "analyser" to "analyzer". - Rename "my_normalizer" to more descriptive "nfkc_cf_normalizer". - Rename "normalizer_keyword" to more descriptive "icu_folding_normalizer". - Rename subfield "lc_raw" to "ci_raw" (case insensitive). I don't think this subfield actually was case insensitive before since "my_normalizer" was used wich is only performing UTF normalization, not case folding. - Don't use "phrase" subfield for sorting, this is incorrect and will result in strange results since byte order of characters is used. There is also no need for a subfield for the sort field. I removed the subfield and changed type to "icu_collation_keyword", which will attempt to sort in order with the least conflicts between languages. Ideally there should be a syspref for setting collation language to sort by. This is a pretty trivial fix so might open a new issue for that. As a result of this "phrase" is almost not used at all, and could probably be removed with phrase queries on the search field instead in a later stage (which would decrease index size considerably).
Created attachment 83016 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Rebase against master (but currently untested).
Created attachment 83019 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Fixed incorrectly named is_mapped_biblios to mapped_biblios in mappings.tt.
David, one request here would be to include the opac/staff columns to the mapping yaml file as well - changes that can be lost on a mapping reload are harder to track and keep in place
Yes, agree. I had forgotten about that. Should be easy to fix. Might be able to do so Friday or next week.
Just rebased this, but need to test some more and make sure tests are not broken. Should have that done by tomorrow.
Created attachment 87358 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Not ready for signoff though. One test fails, I know the reason but it's not trivial to fix (association need to be used in test data and don't know if that is possible with TestBuilder).
Created attachment 87411 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Created attachment 87412 [details] [review] Bug 20589: fix QueryBuilder tests
Created attachment 87415 [details] [review] Bug 20589: Exclude boolean fields from search fields
Created attachment 87416 [details] [review] Bug 20589: Remove QueryWeightFields syspref since no longer in use
Hello David, First: i like this work :). The idea of targeting only searchable fields instead of systematically fall back on the entire record make results more consistent (even with boosting). however, here are some observations: When searching on a specific field (i.e "Autrement" as publisher), we keep all others searchable fields in the target list: "query_string": { "query": "(publisher:autrement)", "fields": [ "LC-card-number", "Local-number", "publisher", + all searchable fields ] } That has no matter on the results here since the following query returns the same results: "query_string": { "query": "(publisher:autrement)", } But keeping field list seems to be useless. A better thing would be to use only fields list instead of having the target in query. Like that: "query_string": { "query": "(autrement)", "fields": [ publisher ] } This also returns the same result (for me) and allows user to make more complete searches such like "autrement AND jean" If i request this: "query_string": { "query": "(publisher:autrement OR jean)" "fields": [ "LC-card-number", "Local-number", "publisher", + all searchable ], } I get many irrelevant results (more than 600 in my database that are not related with what i typed) Or this one: "query_string": { "query": "(autrement OR jean)" "fields": [ "publisher" ], } Returns only records containing "Autrement" or "jean" in publisher. Seems more correct to me. Sorry for the long comment, waiting for your feedback
@Alex Arnaud Thanks for the comments. Regarding the query "(publisher:autrement OR jean)" I think it should be written as "publisher:(autrement OR jean)" to get the results you want. Also, surrounding parentheses can be left out of all the queries. They are only needed if you need to group sub-expressions. I don't think that Koha should try to reduce the field-list by parsing the query, this greatly increases complexity without any real benefit.
Created attachment 92454 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Created attachment 92455 [details] [review] Bug 20589: fix QueryBuilder tests
Created attachment 92456 [details] [review] Bug 20589: Exclude boolean fields from search fields
Created attachment 92457 [details] [review] Bug 20589: Add option for searching entire record if record stored as array
Rebased and tested, this works excellently, I would love to move this forward, but I see a few issues in the patches: 1 - You removed QueryWeightedFields, but Zebra uses this, I obsoleted that patch 2 - There are changes to the authority searches here, things are improved, but a bit too strict, they don't relate to this bug, can you move those to a new report? 3 - Schema changes are mixed into first patch, they should be their own so RM can generate separately 4 - I did a minor rebase on top of 22258 and added a followup to allow searchig the entire record if using marc array Let me know if this makes sense or if I went too far anywhere
Created attachment 93238 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Created attachment 93239 [details] [review] Bug 20589: fix QueryBuilder tests
Created attachment 93240 [details] [review] Bug 20589: Exclude boolean fields from search fields
Created attachment 93241 [details] [review] Bug 20589: Add option for searching entire record if record stored as array
(In reply to Nick Clemens from comment #42) > Rebased and tested, this works excellently, I would love to move this > forward, but I see a few issues in the patches: > 1 - You removed QueryWeightedFields, but Zebra uses this, I obsoleted that > patch > 2 - There are changes to the authority searches here, things are improved, > but a bit too strict, they don't relate to this bug, can you move those to a > new report? > 3 - Schema changes are mixed into first patch, they should be their own so > RM can generate separately > 4 - I did a minor rebase on top of 22258 and added a followup to allow > searchig the entire record if using marc array > > Let me know if this makes sense or if I went too far anywhere Thanks for the feedback! Rebased aagainst master since did not apply again. 1) Ok, I obviously missed that used by Zebra, thanks! 2) I gave this a lot of thought, but on the other hand not sure got it right. It would be possible to break out some changes introduced into separate issues, but they would all depend on each other and would be harder to maintain. So it's bundled together for practical reasons and time constraints. I might have time later on to split things up, but unfortunately not at the moment. 3) Forgot to fix this, but will do that. 4) Great, thanks :)
Created attachment 93247 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library
Created attachment 93248 [details] [review] Bug 20589: fix QueryBuilder tests
Created attachment 93249 [details] [review] Bug 20589: Exclude boolean fields from search fields
Created attachment 93250 [details] [review] Bug 20589: Add option for searching entire record if record stored as array
Created attachment 93251 [details] [review] Bug 20589: DO NOT PUSH Schema changes
Created attachment 93348 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93349 [details] [review] Bug 20589: fix QueryBuilder tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93350 [details] [review] Bug 20589: Exclude boolean fields from search fields Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93351 [details] [review] Bug 20589: Add option for searching entire record if record stored as array Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93352 [details] [review] Bug 20589: DO NOT PUSH Schema changes Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93353 [details] [review] Bug 20589: Remove expanded_facet variable and fix tests Thi is a rebase from 14419 to remove an unused variable Also update tests to expect the new expected result Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Added minor followup Everything seems to work as expected. The authority searching changes seem to be an improvement: previously an 'exact' search matched any term, now it must match the entire field. Punctuation could be excluded in the future, but exact match should eb exact so I think good starts with searches and sorting are improved
Created attachment 93379 [details] [review] Bug 20589: Add field boosting and use query_string fields parameter Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field. How to test: 1) Reindex authorities and biblios. 2) Search biblios and try to verify that this works as expected. 3) Search authorities and try to verify that this works as expected. 4) Go to "Search engine configuration" 5) Change some "Boost", "Staff client", and "OPAC" settings and save. 6) Verify that those settings where saved accordingly. 7) Click the "Biblios" or "Authorities" tab and change one or more "Searchable" settings 8) Verfiy that those settings where saved accordingly. 9) Try to verify that these settings has taken effect by peforming some biblios and/or authorities searches. Sponsorded-by: Gothenburg Univesity Library Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Created attachment 93380 [details] [review] Bug 20589: fix QueryBuilder tests Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Created attachment 93381 [details] [review] Bug 20589: Exclude boolean fields from search fields Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Created attachment 93382 [details] [review] Bug 20589: Add option for searching entire record if record stored as array Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Created attachment 93383 [details] [review] Bug 20589: DO NOT PUSH Schema changes Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Created attachment 93384 [details] [review] Bug 20589: Remove expanded_facet variable and fix tests Thi is a rebase from 14419 to remove an unused variable Also update tests to expect the new expected result Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
- Proposed features works - QA tools ok, - code ok, - work with ES 5.x and 6.x - also fix a relevance sorting issue. Great! Passed QA
Nice work! Pushed to master for 19.11.00
Thanks all, great enhancement
Wonderful, thanks everyone!
Hmm, I think it's this one that causing a fair few breaks in tests on master right now.. I've narrowed down some of the issues.. for example I don't think we need the Koha::SearchFields->weighted_fields tests at all any more (you've removed the method entirely in your patch but missed this test). But, there are some more I'm less confident about.. can you guys take a look please?
Created attachment 93590 [details] [review] Bug 20589: (RM follow-up) Fix Koha::SearchField.t This patch removes the unit test for the weighted_fields method which this bug removed.
Created attachment 93624 [details] [review] Bug 20589: (follow-up) Fix tests
(In reply to Martin Renvoize from comment #70) > Hmm, I think it's this one that causing a fair few breaks in tests on master > right now.. > > I've narrowed down some of the issues.. for example I don't think we need > the Koha::SearchFields->weighted_fields tests at all any more (you've > removed the method entirely in your patch but missed this test). > > But, there are some more I'm less confident about.. can you guys take a look > please? I attached a patch, I think the mappings test is probably not the best, but I think we should delete/replace it on its own bug, this gets things running again for now
Thanks for the fast followup Nick :) Pushed