Hi, There is a problem with date range search in advanced search with ES. Search "2001-2010" will give a list of results with 2001-2010 entered in marc field like 260$c2001-2010 or 264$2001-2010 Records with just 2001 in marc field will not be displayed in search results. Maybe it's just a bad configuration in searchengine/elasticsearch/mappings.pl? Regards, Marjorie
That sounds like it's not using the controlled date but 260/264.
I have the same issue. Does it work somewhere? This seems to be a major regression.
It's not really a mapping problem (though there may be additional mapping issues), but a problem in the Elasticsearch query parser not handling st-year query type.
Created attachment 88063 [details] [review] Bug 22524: Fix publication date search with Elasticsearch Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing. Test plan: 1. Reset mappings and reindex biblios. 2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass. 3. Try that all of the following year range type work in publication date search in advanced search: yyyy yyyy-yyyy -yyyy yyyy- Note that bug 22413 is needed for paging etc. to work properly.
Created attachment 88758 [details] [review] Bug 22524: Fix publication date search with Elasticsearch Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing. Test plan: 1. Reset mappings and reindex biblios. 2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass. 3. Try that all of the following year range type work in publication date search in advanced search: yyyy yyyy-yyyy -yyyy yyyy-
Hi Ere, I'm not sure why, but search with range don't work for me. Test passed. If I try 1989-2001 I get "No results match your search with limit(s): 'copydate:1989-2001'. " I chcecked all twoce but same results. Mike
Created attachment 88998 [details] [review] Bug 22524: Fix publication date and year search with Elasticsearch Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing. Test plan: 1. Reset mappings and reindex biblios. 2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass. 3. Try that all of the following year range type work in publication date search and year limit in advanced search: yyyy yyyy-yyyy -yyyy yyyy-
Mike, I didn't notice the year limit and only fixed the publication date in the normal search terms. Now both should work.
Bug 19575 also renamed : date-time-last-modified => date/time-last-modified Can we fix in this bug ?
Thanks for pointing that out. I'll add to the patch here.
Created attachment 88999 [details] [review] Bug 22524: Fix publication date and year search with Elasticsearch Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing. Test plan: 1. Reset mappings and reindex biblios. 2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass. 3. Try that all of the following year range type work in publication date search and year limit in advanced search: yyyy yyyy-yyyy -yyyy yyyy-
Created attachment 89000 [details] [review] Bug 22524: Fix date/time-last-modified search with Elasticsearch Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping.
Hi Ere, I get error: Applying: Bug 22524: Fix publication date search with Elasticsearch Applying: Bug 22524: Fix publication date and year search with Elasticsearch Using index info to reconstruct a base tree... M Koha/SearchEngine/Elasticsearch/QueryBuilder.pm M admin/searchengine/elasticsearch/mappings.yaml M t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Falling back to patching base and 3-way merge... Auto-merging t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t CONFLICT (content): Merge conflict in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Auto-merging Koha/SearchEngine/Elasticsearch/QueryBuilder.pm CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch/QueryBuilder.pm error: Failed to merge in the changes. Can you rebase it on the top of master. Thank you.
I thikn the first patch was not obsoleted - can you try again?
Hi Katrin, still bad: pplying: Bug 22524: Fix publication date and year search with Elasticsearch Using index info to reconstruct a base tree... M Koha/SearchEngine/Elasticsearch/QueryBuilder.pm M admin/searchengine/elasticsearch/mappings.yaml M t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Falling back to patching base and 3-way merge... Auto-merging t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t CONFLICT (content): Merge conflict in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Auto-merging Koha/SearchEngine/Elasticsearch/QueryBuilder.pm CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch/QueryBuilder.pm error: Failed to merge in the changes.
Katrin, thanks for fixing my mess. The patch applies fine for me.
Created attachment 89152 [details] [review] Bug 22524: Fix publication date and year search with Elasticsearch Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing. Test plan: 1. Reset mappings and reindex biblios. 2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass. 3. Try that all of the following year range type work in publication date search and year limit in advanced search: yyyy yyyy-yyyy -yyyy yyyy- Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Created attachment 89153 [details] [review] Bug 22524: Fix date/time-last-modified search with Elasticsearch Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping. Signed-off-by: Liz Rea <wizzyrea@gmail.com>
I am sorry, but this doesn't work for me. With the patches applied, I find all records for my search for 2012-, ifen records from the 1950s... :( I have: - reset mappings - run updatedatabase - reset mappings Maybe I am missing something, please double check the test plan.
It looks like we use both the 008 and the 260 here; I have a record that has 2006 in the 008, but 1995 in the 260 - that record turns up in the search even though I might not initially expect it to - maybe this is what you are seeing? Should we not consider both the 008 and the 260/264?
Hrm, no that's not it because I just found a record in the results after checking more than 30 correct ones that doesn't have a date in the range in any of the supported fields. So yeah, something is messed up here. Liz
(In reply to Liz Rea from comment #21) > Hrm, no that's not it because I just found a record in the results after > checking more than 30 correct ones that doesn't have a date in the range in > any of the supported fields. So yeah, something is messed up here. > > Liz I think 260/264 is too messy in my opinion for doing numeric seraches. In the past we only used 008 as it has a normalized form of the date. I think that would make sense to me here too.
I think I agree, pub date searches should use the 008, but non-marc libraries coming to Koha might not know how to use the 008 and that... kind of frightens me.
(In reply to Liz Rea from comment #23) > I think I agree, pub date searches should use the 008, but non-marc > libraries coming to Koha might not know how to use the 008 and that... kind > of frightens me. But they might also not add useful data to 260/264...
Created attachment 91562 [details] [review] Bug 22524: Fix publication date and year search with Elasticsearch Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing. Test plan: 1. Reset mappings and reindex biblios. 2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass. 3. Try that all of the following year range type work in publication date search and year limit in advanced search: yyyy yyyy-yyyy -yyyy yyyy- Signed-off-by: Liz Rea <wizzyrea@gmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 91563 [details] [review] Bug 22524: Fix date/time-last-modified search with Elasticsearch Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping. Signed-off-by: Liz Rea <wizzyrea@gmail.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
I think since mappings can be adjusted by libraries we can skirt the issue of which fields should be included - we are finding most libraries want to tweak the mappings rather than use the default file. Needs will vary by library type too - including extra out of the box to show options/functions would be my opinion so this should be good as is. I couldn't find records that didn't fit the searches in my returns. Can you retest Liz/Katrin and share sample records? If we don't have examples of incorrect records I would be happy for my SO to be a QA stamp. The mapping fix here is important so should be split if this one gets stuck
(In reply to Nick Clemens from comment #27) > I think since mappings can be adjusted by libraries we can skirt the issue > of which fields should be included - we are finding most libraries want to > tweak the mappings rather than use the default file. Needs will vary by > library type too - including extra out of the box to show options/functions > would be my opinion so this should be good as is. > > I couldn't find records that didn't fit the searches in my returns. Can you > retest Liz/Katrin and share sample records? > > If we don't have examples of incorrect records I would be happy for my SO to > be a QA stamp. > > The mapping fix here is important so should be split if this one gets stuck Ah, the 260/008 confusion seems to stem from bug 22801 - I think that should depend on this one Publication date in Adv search dropdown will use pubdate The 'publication date range' on opac and 'Year' limit on staff both use copydate in ES right now
Moving to PQA to let RM decide
Nice work! Pushed to master for 19.11.00
Pushed to 19.05.x for 19.05.03
backported to 18.11.x for 18.11.10
*** Bug 20492 has been marked as a duplicate of this bug. ***