|
Description
Jonathan Druart
2015-09-25 14:38:52 UTC
Created attachment 42882 [details] [review] Bug 14899: Add Koha::SearchField[s] and Koha::SearchMarcMap[s] classes Created attachment 42883 [details] [review] Bug 14899: Add tableDND JS lib Created attachment 42884 [details] [review] Bug 14899: DB changes This patch applies several changes to the DB structure introduced by the previous ES patch set: 1/ Add a search_field.label column => It will be easier to understand what the search_field does. Indeed, some are not user friendly: acqdate, an, pl, ff8-23, ln, etc. TODO later: Update the labels with correct values (at the moment label=name) 2/ Update the foreign key search_marc_to_field.search_marc_map_id with on delete cascase. This will permit to automatically remove the entries in search_marc_to_field when they are deleted in search_marc_map. 3/ Remove the index_name_2 unique constraint on the search_marc_map table. I don't understand how this could be useful, it was defined like (index_name, marc_type, marc_field), so it means a mapping cannot be defined twice with different values for facet, suggestible and sort. This limitation does not seem to make sense. Robin, please correct me if I am wrong :) Created attachment 42885 [details] [review] Bug 14899: Add the mapping configuration page in the admin module This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: - If something went wrong during the insertion/deletion/modification, the users will loose all these changes. - Using the interface the user can defined several labels for a same field. The script will retrieve the first change and apply it, next ones will be avoided. TODO: - Add a specific permission (?) - Add some data checks client side (JS) - Use checkboxes for facet and suggestible (lazy today...) - Understand the difference between the 3 values that sortable can have and improve the value for the options in the select box. Created attachment 42886 [details] [review] Bug 14899: Add a link to the new page in the admin To apply on top of the ES work, see the catalyst/elastic_search branch. http://127.0.0.1:8080/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl An error has occurred! Error 500 In Koha this typically means that the Koha team is working on new features Wait while system maintenance is being done or email the Koha administrator. Use top menu bar to navigate to another part of Koha. (In reply to Peter Zhao from comment #7) > http://127.0.0.1:8080/cgi-bin/koha/admin/searchengine/elasticsearch/mappings. > pl > > > An error has occurred! > Error 500 > > In Koha this typically means that the Koha team is working on new > features > Wait while system maintenance is being done or email the Koha > administrator. > Use top menu bar to navigate to another part of Koha. What is the error in the log file? How did you test? To test: $ git remote -v # should at least return catalyst git://git.catalyst.net.nz/koha.git (fetch) catalyst git://git.catalyst.net.nz/koha.git (push) $ git fetch catalyst $ git checkout -b bug_14899 catalyst/elastic_search $ git bz apply 14899 $ perl installer/data/mysql/updatedatabase.pl Created attachment 43054 [details] [review] Bug 14899: Add Koha::SearchField[s] and Koha::SearchMarcMap[s] classes Created attachment 43055 [details] [review] Bug 14899: Add tableDND JS lib Created attachment 43056 [details] [review] Bug 14899: DB changes This patch applies several changes to the DB structure introduced by the previous ES patch set: 1/ Add a search_field.label column => It will be easier to understand what the search_field does. Indeed, some are not user friendly: acqdate, an, pl, ff8-23, ln, etc. TODO later: Update the labels with correct values (at the moment label=name) 2/ Update the foreign key search_marc_to_field.search_marc_map_id with on delete cascase. This will permit to automatically remove the entries in search_marc_to_field when they are deleted in search_marc_map. 3/ Remove the index_name_2 unique constraint on the search_marc_map table. I don't understand how this could be useful, it was defined like (index_name, marc_type, marc_field), so it means a mapping cannot be defined twice with different values for facet, suggestible and sort. This limitation does not seem to make sense. Robin, please correct me if I am wrong :) Created attachment 43057 [details] [review] Bug 14899: Add the mapping configuration page in the admin module This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: - If something went wrong during the insertion/deletion/modification, the users will loose all these changes. - Using the interface the user can defined several labels for a same field. The script will retrieve the first change and apply it, next ones will be avoided. TODO: - Add a specific permission (?) - Add some data checks client side (JS) - Use checkboxes for facet and suggestible (lazy today...) - Understand the difference between the 3 values that sortable can have and improve the value for the options in the select box. Created attachment 43058 [details] [review] Bug 14899: Add a link to the new page in the admin Patches updated, they now apply against the up-to-date catalyst/elastic_search branch. (In reply to Jonathan Druart from comment #11) > 3/ Remove the index_name_2 unique constraint on the search_marc_map table. > I don't understand how this could be useful, it was defined like (index_name, > marc_type, marc_field), so it means a mapping cannot be defined twice with > different values for facet, suggestible and sort. > This limitation does not seem to make sense. > Robin, please correct me if I am wrong :) You are not wrong. I just wrote a big thing about what I was intending, realised I was wrong, so deleted it :) I think I was confusing index_name and field name, or something. I'll merge this in as part of the ES branch like you suggest, as it'll solve some issues. Though, it'd be best to just put DB changes in the .sql file for now and let that load them in from there, rather than creating a whole set of transactions to manage. Created attachment 43099 [details] [review] Bug 14899: schema changes to make the database better Created attachment 43353 [details] [review] Bug 14899: Add tableDND JS lib Created attachment 43354 [details] [review] Bug 14899: Add the mapping configuration page in the admin module This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings and search fields. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: - If something went wrong during the insertion/deletion/modification, the users will loose all these changes. TODO: - Add a specific permission (?) - Add some data checks client side (JS) - Use checkboxes for facet and suggestible (lazy today...) - Understand the difference between the 3 values that sortable can have and improve the value for the options in the select box. Created attachment 43355 [details] [review] Bug 14899: Add a link to the new page in the admin .git/rebase-apply/patch:305: trailing whitespace.
warning: 1 line adds whitespace errors.
.git/rebase-apply/patch:305: trailing whitespace.
warning: 1 line applied after fixing whitespace errors.
error: Failed to merge in the changes. at ./getter.pl line 196.
Created attachment 50995 [details] [review] Bug 14899: Add tableDND JS lib Created attachment 50996 [details] [review] Bug 14899: Add the mapping configuration page in the admin module This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings and search fields. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: - If something went wrong during the insertion/deletion/modification, the users will loose all these changes. TODO: - Add a specific permission (?) - Add some data checks client side (JS) - Use checkboxes for facet and suggestible (lazy today...) - Understand the difference between the 3 values that sortable can have and improve the value for the options in the select box. Created attachment 50997 [details] [review] Bug 14899: Add a link to the new page in the admin Created attachment 55915 [details] [review] Bug 14899: Add tableDND JS lib Created attachment 55916 [details] [review] Bug 14899: Add the mapping configuration page in the admin module This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings and search fields. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: - If something went wrong during the insertion/deletion/modification, the users will loose all these changes. TODO: - Add a specific permission (?) - Add some data checks client side (JS) - Use checkboxes for facet and suggestible (lazy today...) - Understand the difference between the 3 values that sortable can have and improve the value for the options in the select box. Created attachment 55917 [details] [review] Bug 14899: Add a link to the new page in the admin Created attachment 55918 [details] [review] Bug 14899: Add a reset option For developpers only at the moment, if you want to reset your mappings, you can hit: admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 It seems to work, adding/deleting/updating values affects only the current marcflavour, adding a field to an index and performing a reindex results in good search results. It would be nice to remain on the current tab when saving Bad data here can break indexing ( save date-entered-on-file with '650a/4-8' ) so permissions are definitely needed Created attachment 55957 [details] [review] Bug 14899: Add tableDND JS lib Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Created attachment 55958 [details] [review] Bug 14899: Add the mapping configuration page in the admin module This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings and search fields. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: - If something went wrong during the insertion/deletion/modification, the users will loose all these changes. TODO: - Add a specific permission (?) - Add some data checks client side (JS) - Use checkboxes for facet and suggestible (lazy today...) - Understand the difference between the 3 values that sortable can have and improve the value for the options in the select box. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Created attachment 55959 [details] [review] Bug 14899: Add a link to the new page in the admin Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Created attachment 55960 [details] [review] Bug 14899: Add a reset option For developpers only at the moment, if you want to reset your mappings, you can hit: admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Created attachment 56182 [details] [review] Bug 14899: Add tableDND JS lib Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 56183 [details] [review] Bug 14899: Add tableDND JS lib Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 56194 [details] [review] Bug 14899: Add tableDND JS lib Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 56195 [details] [review] Bug 14899: Add tableDND JS lib Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 56196 [details] [review] Bug 14899: Add the mapping configuration page in the admin module This new page (admin/searchengine/elasticsearch/mappings.pl) will permit to manage the ES mappings. For the biblios and authorities indexes, the different mappings can be managed from this single page. The interface let you add, remove and update mappings and search fields. It's also possible to reorder the mappings, as the order can be important in the indexation process. Note that the table can be displayed in a different order that the one it was before saving, but the mappings are grouped by search field and the order inside the search field is preserved. Limitations: - If something went wrong during the insertion/deletion/modification, the users will loose all these changes. TODO: - Add a specific permission (?) - Add some data checks client side (JS) - Use checkboxes for facet and suggestible (lazy today...) - Understand the difference between the 3 values that sortable can have and improve the value for the options in the select box. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 56197 [details] [review] Bug 14899: Add a link to the new page in the admin Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 56198 [details] [review] Bug 14899: Add a reset option For developpers only at the moment, if you want to reset your mappings, you can hit: admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 56199 [details] [review] Bug 14899 - Convert links to buttons, add icons Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Pushed to Master - Should be in the November 2016 release. Thanks! Pushed to 16.05.x, for 16.05.06 release |