Created attachment 182559 [details] Publication Date: Newest to Oldest / Oldest to Newest not indexing properly In Koha version 24.11, sorting search results by publication year (newest to oldest/oldest to newest) in the OPAC does not work as expected, even though the correct data exists in MARC field `264$c`. The issue appears to be related to how the publication year is indexed or handled during OPAC search result sorting. ### Steps to Reproduce: 1. Enter bibliographic records with valid publication years in MARC field `264$c`. 2. Rebuild Zebra/Elasticsearch index. 3. Perform a general search in the OPAC. 4. Attempt to sort by "Publication Date: Newest to Oldest and oldest to newest". #### Expected Result: - Results should be sorted by publication year descending (e.g., 2024, 2023, 2022). #### Actual Result: - Sorting appears incorrect or inconsistent. - Records do not appear in chronological order by publication year. ### Additional Information: - I have verified that the data is present and sortable via direct SQL query using XPath and regex extraction. - Indexing configuration seems to map `264$c` to `publication_date`, but sorting behavior does not reflect this. - Rebuilding the index did not resolve the issue. - This affects user experience in the public OPAC where accurate sorting by publication date is critical. ### Suggested Fix / Enhancement: 1. Ensure that `264$c` is properly mapped in the Search Engine Configuration for use in `publication_date`. 2. Confirm that the value is normalized and indexed as a numeric field (e.g., only the 4-digit year) for proper sorting. 3. Consider adding documentation or warnings if certain MARC subfields are known to cause issues with sorting/indexing. ```If anyone has experienced this issue before or found a solution, please share your insights!```
I'd argue that it should use the normalized date in 008 and it might do that already. Have you verified you have 008 (Date 1) in your data set correctly? 264 is repeatable, selecting the right one for sorting is not trivial.
(In reply to Katrin Fischer from comment #1) > I'd argue that it should use the normalized date in 008 and it might do that > already. Have you verified you have 008 (Date 1) in your data set correctly? > 264 is repeatable, selecting the right one for sorting is not trivial. Thank you for your response, Katrin! You're absolutely right that the 008 field (Date 1) is a standard place for publication year in MARC records. However, in our case, we rarely use the 008 field because most of our records are either locally created or imported without full fixed fields. We primarily rely on variable fields like `260$c` for the publication year. In our workflow, we use `260$c` as the main source for publication year. this is how many libraries handle manually entered records where 008 may not always be filled out completely. We noticed that while the data in `260$c` is properly stored and can be extracted/sorted via direct SQL query, the OPAC does not sort these results correctly by publication year. This suggests there may be an issue with how the OPAC indexing layer (Zebra/Elasticsearch) handles sorting based on variable MARC fields like `260$c`. Is there a way to ensure that Koha's OPAC search layer reliably uses `260$c` for sorting by publication year? Any guidance or insights from your experience would be greatly appreciated! thank you.
The date range search on advanced search also uses 008. You might want to add the missing ones to your data. Sorting by 008 is the expected behavior.
(In reply to Katrin Fischer from comment #3) > The date range search on advanced search also uses 008. You might want to > add the missing ones to your data. Sorting by 008 is the expected behavior. Thank you for the clarification, Katrin! I understand that using the 008 field for sorting and filtering is the expected behavior in Koha, especially since itβs also used by other features like the date range search in advanced search. We have tried manually updating some of our records to ensure the 008 field contains valid publication years. While a few of them now sort correctly, but most still do not, and the issue persists across many records. From my observation: The first 20 pages of results usually sort correctly by publication year, but starting from around page 20 onward (depending on the total number of results), sorting breaks down and becomes inconsistent. This makes me think that the issue might not only be about which field is used for sorting, but also how the index handles large result sets or how the data is stored/normalized in the index. Could it be that: - Not all 008 values are being properly extracted or normalized during indexing? - Or there's an issue with pagination and how the index returns sorted results beyond a certain threshold? Any insight into what could cause this kind of behavior would be greatly appreciated. Thank you again for your time and support.
OK, that might be a different issue. Are you using Zebra or Elasticsearch?
(In reply to Katrin Fischer from comment #5) > OK, that might be a different issue. Are you using Zebra or Elasticsearch? Zebra
Zebra has a limit on how many records are sorted. It's for performance reasons I guess. It's configured in one of the Zebra config files I believe here: https://git.koha-community.org/Koha-community/Koha/src/commit/d7af2cacff2692c803579117953f3cd13c94c878/etc/zebradb/zebra-biblios-dom.cfg sortmax:1000 Maybe check what you have configured and if it works for smaller result sets.
(In reply to Katrin Fischer from comment #7) > Zebra has a limit on how many records are sorted. It's for performance > reasons I guess. It's configured in one of the Zebra config files I believe > here: > https://git.koha-community.org/Koha-community/Koha/src/commit/ > d7af2cacff2692c803579117953f3cd13c94c878/etc/zebradb/zebra-biblios-dom.cfg > > sortmax:1000 > > Maybe check what you have configured and if it works for smaller result sets. Thank you for the detailed information, Katrin. I will take some time to review our Zebra configuration, particularly the sortmax setting in the zebra-biblios-dom.cfg file. I'll test whether adjusting this parameter affects the sorting behavior for publication years in the OPAC. I'll keep this thread open and will update you with my findings once I've completed the necessary checks. Best regards, Shahrum