http://git.catalyst.net.nz/gw?p=koha.git;a=shortlog;h=refs/heads/elasticsearch_browser Currently the required info is being stored alongside the records saved in Elasticsearch. Now I just need to get stuff in place to read it out... Created attachment 41804 [details] [review] Bug 14567: backend code for browse interface This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. Created attachment 41805 [details] [review] Bug 14567: frontend template and JS for browse interface Created attachment 41806 [details] [review] Bug 14567: recompile CSS with less This is working. It's currently visible at http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-browse.pl though it may not stay there for too long. The patches attached are intended to be applied on top of the elasticsearch branch, they won't work without it. (In reply to Robin Sheat from comment #5) > This is working. It's currently visible at > http://elasticsearch.koha.catalystdemo.net.nz/cgi-bin/koha/opac-browse.pl > though it may not stay there for too long. > > The patches attached are intended to be applied on top of the elasticsearch > branch, they won't work without it. I've put a more permanent version at http://browse.koha.catalystdemo.net.nz/cgi-bin/koha/opac-browse.pl It's sharing a database (though not an ES index) with the elasticsearch demo, though I've cherrypicked the schema changes over to prevent things getting too weird. I tried it out here: http://browse.koha.catalystdemo.net.nz It seems nice, but it doesn't seem like a true browse to me--what I look for in a browse is to enter a search term, and have the results returned to be what is exactly before my searched term in the given index, then where the searched term is/would be, then what is exactly after the searched term in the index. So there would be no need for "exact, fuzzy, really fuzzy," because when I browse an index, I want the results to be what is before and after that term in the given index. That said, when I searched "exact" for "Author": Rowling I get: Rowling, J. K. Rowling, J.K. This is nice, but, again, it doesn't seem like a "browse" to me--these are just search results. When I click the names, I get the exact same search results as I do via Advanced Search, Author Phrase: http://browse.koha.catalystdemo.net.nz/cgi-bin/koha/opac-search.pl?idx=au%2Cphr&q=rowling&op=and&idx=kw&op=and&idx=kw&do=Search&sort_by=relevance&limit= If I Browse via "Fuzzy" with: Rowling I get these results: Rollins, Henry, Rowland, Sarah-Jane. Rowlands, David. Rowlands, Gena. Rowling, J. K. Rowling, J.K. Which does show me some results which should be in the author index above Rowling, but I get no results that should be in the author index after Rowling. "Really fuzzy" is the same. I would like to see the searched for term in the middle of hits that occur before and after it in the index, e.g. something like this (an author "Browse" result, with made up examples below Rowling, J.K.): Rollins, Henry, Rowland, Sarah-Jane. Rowlands, David. Rowlands, Gena. Rowling, J. K. Rowling, J.K. Rowlins, Fred. Rozalind, Sue. Rualla, Max. Ruscha, Ed. Ruther, Martin. Quahog, Amir. Quaker Alliance for Peace. I hope this is helpful! --h2 (In reply to Heather from comment #7) > I tried it out here: http://browse.koha.catalystdemo.net.nz > It seems nice, but it doesn't seem like a true browse to me--what I look for > in a browse is to enter a search term, and have the results returned to be > what is exactly before my searched term in the given index, then where the > searched term is/would be, then what is exactly after the searched term in > the index. So there would be no need for "exact, fuzzy, really fuzzy," > because when I browse an index, I want the results to be what is before and > after that term in the given index. There are technical reasons why this is particularly hard, essentially that pulling the index terms out in order is not something that can readily be done without patching Elasticsearch itself, which I'm considering out of scope of this project :) However, when I thought about it with that in mind, I got to wondering what the point of a complete alphabetical index really is to a library user. There's no semantic relationship between a word that starts with R and a word that starts with S, why would it be meaningful to put one beside the other. A subject of "Symbols" is not closely related to a subject of "Tarantulas", just because they're alphabetically close. However, an author name of "Stephen" is (sorta) semantically related to an author name of "Steven", if the user hasn't quite remembered the whole name or the spelling. > That said, when I searched "exact" for "Author": Rowling > I get: > Rowling, J. K. > Rowling, J.K. This is a GIGO situation with the data that is the result of it not being terribly cleanly catalogued in the first place. I tend to prefer working with data like this as it is better at highlighting odd edge cases. I've been thinking about ways to correct for cases like this (where it's just a punctuation or whitespace difference), but it presents some issues relating to how it'd be done reliably with no false positives. So I've postponed further thinking about it for now. In an ideal world, everyone will be using authorities anyway, and this won't happen :) > This is nice, but, again, it doesn't seem like a "browse" to me--these are > just search results. When I click the names, I get the exact same search > results as I do via Advanced Search, Author Phrase: It is doing pretty much the same thing as a search, yes. Essentially, it's goal is to be an easier way to look for some specific things, possibly when you're a bit unsure about the details. I think that subjects will benefit from this the most actually, as scrolling through "Birds -- Kakapo", "Birds -- Kea", "Birds -- Kiwi" is something that has real use. Additionally, a search for "Kiwi" will also find "Birds -- Kiwi" (though this behaviour isn't visible in the data that's currently on the test site due to the way it's catalogued.) > If I Browse via "Fuzzy" with: Rowling > I get these results: > Rollins, Henry, > Rowland, Sarah-Jane. > Rowlands, David. > Rowlands, Gena. > Rowling, J. K. > Rowling, J.K. > > Which does show me some results which should be in the author index above > Rowling, but I get no results that should be in the author index after > Rowling. "Really fuzzy" is the same. I'll double check this, but it's most likely that in this database, there's just nothing further that matches. Actually, in this case, fuzzy and extra fuzzy will be doing the same thing as there's a max edit distance limit for the fuzziness, and I think it will have been hit. I'm probably going to remove the "extra fuzzy" option anyway, it risks being more confusing than anything. > I would like to see the searched for term in the middle of hits that occur > before and after it in the index, e.g. something like this (an author > "Browse" result, with made up examples below Rowling, J.K.): But why? What does an alphabetically adjacent author tell you about that set of authors, and what does it provide that a library user would find useful? (Technical reasons for why this is hard aside.) > I hope this is helpful! --h2 Thanks for checking it out! (In reply to Robin Sheat from comment #8) > (In reply to Heather from comment #7) > > I tried it out here: http://browse.koha.catalystdemo.net.nz > > It seems nice, but it doesn't seem like a true browse to me--what I look for > > in a browse is to enter a search term, and have the results returned to be > > what is exactly before my searched term in the given index, then where the > > searched term is/would be, then what is exactly after the searched term in > > the index. So there would be no need for "exact, fuzzy, really fuzzy," > > because when I browse an index, I want the results to be what is before and > > after that term in the given index. > > There are technical reasons why this is particularly hard, essentially that > pulling the index terms out in order is not something that can readily be > done without patching Elasticsearch itself, which I'm considering out of > scope of this project :) > > However, when I thought about it with that in mind, I got to wondering what > the point of a complete alphabetical index really is to a library user. > There's no semantic relationship between a word that starts with R and a > word that starts with S, why would it be meaningful to put one beside the > other. A subject of "Symbols" is not closely related to a subject of > "Tarantulas", just because they're alphabetically close. However, an author > name of "Stephen" is (sorta) semantically related to an author name of > "Steven", if the user hasn't quite remembered the whole name or the spelling. > > > That said, when I searched "exact" for "Author": Rowling > > I get: > > Rowling, J. K. > > Rowling, J.K. > > This is a GIGO situation with the data that is the result of it not being > terribly cleanly catalogued in the first place. I tend to prefer working > with data like this as it is better at highlighting odd edge cases. > > I've been thinking about ways to correct for cases like this (where it's > just a punctuation or whitespace difference), but it presents some issues > relating to how it'd be done reliably with no false positives. So I've > postponed further thinking about it for now. In an ideal world, everyone > will be using authorities anyway, and this won't happen :) > > > This is nice, but, again, it doesn't seem like a "browse" to me--these are > > just search results. When I click the names, I get the exact same search > > results as I do via Advanced Search, Author Phrase: > > It is doing pretty much the same thing as a search, yes. Essentially, it's > goal is to be an easier way to look for some specific things, possibly when > you're a bit unsure about the details. I think that subjects will benefit > from this the most actually, as scrolling through > "Birds -- Kakapo", "Birds -- Kea", "Birds -- Kiwi" is something that has > real use. Additionally, a search for "Kiwi" will also find "Birds -- Kiwi" > (though this behaviour isn't visible in the data that's currently on the > test site due to the way it's catalogued.) > > > If I Browse via "Fuzzy" with: Rowling > > I get these results: > > Rollins, Henry, > > Rowland, Sarah-Jane. > > Rowlands, David. > > Rowlands, Gena. > > Rowling, J. K. > > Rowling, J.K. > > > > Which does show me some results which should be in the author index above > > Rowling, but I get no results that should be in the author index after > > Rowling. "Really fuzzy" is the same. > > I'll double check this, but it's most likely that in this database, there's > just nothing further that matches. Actually, in this case, fuzzy and extra > fuzzy will be doing the same thing as there's a max edit distance limit for > the fuzziness, and I think it will have been hit. I'm probably going to > remove the "extra fuzzy" option anyway, it risks being more confusing than > anything. > > > I would like to see the searched for term in the middle of hits that occur > > before and after it in the index, e.g. something like this (an author > > "Browse" result, with made up examples below Rowling, J.K.): > > But why? What does an alphabetically adjacent author tell you about that set > of authors, and what does it provide that a library user would find useful? > (Technical reasons for why this is hard aside.) > > > I hope this is helpful! --h2 > > Thanks for checking it out! Hi, Robin! One of the main purposes of a complete alphabetical browse for the library user is exactly what you mentioned: when the user is unsure of the spelling, but has most of it, e.g., "Stephens" or "Stevens," or something like "Ralphs," or "Rolphs." And, with subjects, the user will receive a list with all subdivisions listed & will see those, rather than just search results that don't show the subject terms. Yes--GIGO is an issue in the above browse, and ideally all access points are under authority control, but the browse is useful for libraries that have order records that are not yet cataloged, and who only do their authority control/cataloging upon receipt of the item. The above issue in the spacing of "Rowling, J. K." could occur with order records vs. fully cataloged & received items. The alphabetic browse is also invaluable for the catalogers on the staff side doing authority control--it's the easiest way to spot heading variations. In the Koha public opac, a basic search always gives you a list of search results that never shows subject terms--the facets on the left show *some* subject terms but, e.g., if you do a basic keyword search in your demo catalog for "Birds," the subject facets on the left have subdivisions broken out & not associated with the main heading: Animals (2) Birds (6) Maoritanga. (1) Mystery and suspense stories. (1) Mythology, Maori. (1) Natural history (2) New Zealand (3) New Zealand. (3) Penguins. (1) Pictorial works. (1) Show more (It's the same in our catalog, i.e. this search only gives you facets on the left that are main headings, not headings *with* their subdivisions: http://keys.bywatersolutions.com/cgi-bin/koha/opac-search.pl?q=ships) So an alphabetical browse that does return subject terms with their subdivision, like in your example, would be useful to a user, e.g.: Birds--Kakapo. Birds--Kea. Birds--Kuwait. Birds--Pictorial works. Birds--Qatar. (BTW, the Authority Search in the demo catalog has a problem--do an authority search with default options & the term "Birds" & you get: Software error: Can't call method "each" on unblessed reference at /opt/kohaclones/browse/Koha/SearchEngine/Elasticsearch/Search.pm line 265. For help, please send mail to the webmaster ([no address given]), giving this error message and the time and date of the error.") Do a similar authority search in our catalog for ships: http://keys.bywatersolutions.com/cgi-bin/koha/opac-authorities-home.pl?op=do_search&type=opac&operator=contains&value=ships&marclist=any&and_or=and&orderby=HeadingAsc And you get close to a subject heading browse result--the only problem is that it returns only main headings, not an alphabetic list of the terms *with subdivisions* as used in the records, which would be very useful to the user. I bring this up because, IMO, as it is, authority control/authority search is not an alternative to a true alphabetical catalog browse for the public opac user, of subject terms with subdivisions as used in the catalog. I hope this is helpful! Best, --h2 I was thinking this morning, maybe a 'starts with' search option could help to bring the result list a little closer to an alphabetic index. Could this be an option? I have asked some of the coworkers to give the search a test, but some of the results are quite confusing. Example: Exact, Author: Miller The first result shown is "Millard, Annie". I didn't expect that for the exact search option, but only with fuzzy activated. Another issue could be related to ISBD punctuation in the fields. Often there are 2 entries which what seems to be the same author, only different by a . at the end: Example: Exact, Author: Smith Smith, Alexander Gordon. Smith, Alexander Gordon Smith, Jim Smith, Jim. Would be good if those could be merged for display and search. (In reply to Katrin Fischer from comment #10) > I was thinking this morning, maybe a 'starts with' search option could help > to bring the result list a little closer to an alphabetic index. Could this > be an option? It sounds to me like it could be an option, if the results were an alphabetic list of terms from the searched index, rather than a list of records. > I have asked some of the coworkers to give the search a test, but some of > the results are quite confusing. > > Example: Exact, Author: Miller > The first result shown is "Millard, Annie". > I didn't expect that for the exact search option, but only with fuzzy > activated. It sounds like your "exact" is kind of "fuzzy!" > Another issue could be related to ISBD punctuation in the fields. Often > there are 2 entries which what seems to be the same author, only different > by a . at the end: > > Example: Exact, Author: Smith > Smith, Alexander Gordon. > Smith, Alexander Gordon > Smith, Jim > Smith, Jim. > > Would be good if those could be merged for display and search. Could this also be an issue of variations in the headings' $q fields, which Koha tends to not display in search results, and when used there's no punctuation between the end of the $a and the beginning of the $q subfields? In your example, above, the headings could be for different people, and so should not be merged, e.g., could be: 100 1_ Smith, Alexander Gordon. 100 1_ Smith, Alexander Gordon $q (Alexander Gordon Frank) 100 1_ Smith, Jim $q (James Allen) 100 1_ Smith, Jim. (The headings' variations are most likely the result of GIGO, & so proper authority control would collate them.) --h2 (In reply to Heather from comment #9) > One of the main purposes of a complete alphabetical browse for the library > user is exactly what you mentioned: when the user is unsure of the spelling, > but has most of it, e.g., "Stephens" or "Stevens," or something like > "Ralphs," or "Rolphs." Right, and that is handled here by allowing fuzzier searching. Having everything there doesn't seem like it'd make that particularly easy, as things might be separated quite widely visually even if there's only a one character difference. Y'know what'd be interesting, incorporating "sounds like" searching to it. I'll leave that for another time though. I also shudder to think how that'd be internationalised :) > The alphabetic browse is also invaluable for the catalogers on the staff > side doing authority control--it's the easiest way to spot heading > variations. Certainly, but I'm consciously not focussing on that side of things, or stuff will get too big too fast :) > (BTW, the Authority Search in the demo catalog has a problem--do an > authority search with default options & the term "Birds" & you get: > Software error: > Can't call method "each" on unblessed reference at > /opt/kohaclones/browse/Koha/SearchEngine/Elasticsearch/Search.pm line 265. > For help, please send mail to the webmaster ([no address given]), giving > this error message and the time and date of the error.") That doesn't surprise me just because this code is new and largely unstressed, though I'll look into it, it shouldn't be happening. > Do a similar authority search in our catalog for ships: > http://keys.bywatersolutions.com/cgi-bin/koha/opac-authorities-home. > pl?op=do_search&type=opac&operator=contains&value=ships&marclist=any&and_or=a > nd&orderby=HeadingAsc > > And you get close to a subject heading browse result--the only problem is > that it returns only main headings, not an alphabetic list of the terms > *with subdivisions* as used in the records, which would be very useful to > the user. I bring this up because, IMO, as it is, authority > control/authority search is not an alternative to a true alphabetical > catalog browse for the public opac user, of subject terms with subdivisions > as used in the catalog. Just FYI, this isn't using the authority database, it's using the actual search indices. I may swap out the database some time soon with one that has a more "properly" catalogued catalogue, to see how that looks. (In reply to Katrin Fischer from comment #10) > I was thinking this morning, maybe a 'starts with' search option could help > to bring the result list a little closer to an alphabetic index. Could this > be an option? This is exactly what is there right now, or I misunderstand you :) > Example: Exact, Author: Miller > The first result shown is "Millard, Annie". > I didn't expect that for the exact search option, but only with fuzzy > activated. Hmm. That must be a bug. That shouldn't be happening. > Another issue could be related to ISBD punctuation in the fields. Often > there are 2 entries which what seems to be the same author, only different > by a . at the end: > > Example: Exact, Author: Smith > Smith, Alexander Gordon. > Smith, Alexander Gordon > Smith, Jim > Smith, Jim. > > Would be good if those could be merged for display and search. I've been thinking about this. It's a bit problematic, due to way the MARC->ES conversion works. However, I want to figure out how to tap into this for some other things too, if I can attach bits of Perl to certain points in the conversion, it might be doable. We're still going to get other weird results though, e.g. Rowling, J. K. Rowling, J.K. fixing that in a way that doesn't cause false positives, or make the results inaccurate, may be hard. > Just FYI, this isn't using the authority database, it's using the actual > search indices. Oh, right! I'm sorry, I think I wasn't very clear--I was trying to say that there's no way right now for a user to ever see the subject headings with subdivisions as used in the records. Maybe this new browse feature, when browsing subjects, could return an alphabetic list of subject headings *with* subdivisions from the records. I hope this is clearer! > I may swap out the database some time soon with one that has a more > "properly" catalogued catalogue, to see how that looks. Our catalog is not the cleanest (I haven't done all our authority work & headings clean up), but it's pretty good--if there's any way our records, a subset of our records, or even we can help with this, we'd be very, very happy to help. Our catalog is here, http://keys.bywatersolutions.com/ supported by ByWater, so perhaps if y'all working on this bug contact Brendan at ByWater, there's a way that we can help test with our catalog? Or perhaps even ByWater could make our records or a subset of our records available so y'all have a more "cataloged catalog" for testing? --h2 (In reply to Heather from comment #13) > Oh, right! I'm sorry, I think I wasn't very clear--I was trying to say that > there's no way right now for a user to ever see the subject headings with > subdivisions as used in the records. Maybe this new browse feature, when > browsing subjects, could return an alphabetic list of subject headings > *with* subdivisions from the records. I hope this is clearer! Ah, this will do that. You just can't see it because this catalogue doesn't handle subjects like that. > Our catalog is not the cleanest (I haven't done all our authority work & > headings clean up), but it's pretty good--if there's any way our records, a > subset of our records, or even we can help with this, we'd be very, very > happy to help. Our catalog is here, http://keys.bywatersolutions.com/ > supported by ByWater, so perhaps if y'all working on this bug contact > Brendan at ByWater, there's a way that we can help test with our catalog? > Or perhaps even ByWater could make our records or a subset of our records > available so y'all have a more "cataloged catalog" for testing? More test data is good, I've pinged Brendan to have a look into it. Rebased this on top of the current ES code. Created attachment 56253 [details] [review] Bug 14567: backend code for browse interface This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. Created attachment 56254 [details] [review] Bug 14567: frontend template and JS for browse interface Created attachment 56255 [details] [review] Bug 14567: recompile CSS with less Created attachment 56256 [details] [review] Bug 14567 - (Rebase followups) Dirty rebase, I need to do some cleanup but this should work on current master Created attachment 56262 [details] [review] Bug 14567: backend code for browse interface This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. Created attachment 56263 [details] [review] Bug 14567: frontend template and JS for browse interface Created attachment 56264 [details] [review] Bug 14567: recompile CSS with less Created attachment 56270 [details] [review] [SIGNED-OFF] Bug 14567: backend code for browse interface This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Created attachment 56271 [details] [review] [SIGNED-OFF] Bug 14567: frontend template and JS for browse interface Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Created attachment 56272 [details] [review] [SIGNED-OFF] Bug 14567: recompile CSS with less Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Created attachment 56273 [details] [review] [SIGNED-OFF] Bug 14567 (QA followup) Fix typos from tidying Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> This does not work at all, please retest. Created attachment 57141 [details] [review] Bug 14567 - Fix paths (rebase from 17372) If we get a test instance set up locally is there a way to see signed off bugs? Thanks. Yes, I would suggest checking out kohadevbox: https://github.com/digibib/kohadevbox Once you have a developer instance setup you can test any patches, signed off or needing signoff :-) (In reply to J Schmidt from comment #30) > If we get a test instance set up locally is there a way to see signed off > bugs? Thanks. Scalar value @_[0] better written as $_[0] at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/Search.pm line 126. Scalar value @_[0] better written as $_[0] at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/Search.pm line 128. Can't use string ("(d)") as an ARRAY ref while "strict refs" in use at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 108. Please do a *complete* pass on this patchset before sending it back to the QA queue: - Remove debug statements - Add a test plan: - Provide full test coverage - The opac-browse template looks wrong (does not follow the usual pattern, is it expected?) - And finally, test Created attachment 61660 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Squashed and rebased - moving some code to a dependency bug Created attachment 64477 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 3 - Test searching for author, title, and subject 4 - Verify that results are returned in expected order 5 - Experiment with fuzziness 6 - Click any result and verify specific titles are correct 7 - Click through title to record and verify it is the correct record Created attachment 64478 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 3 - Test searching for author, title, and subject 4 - Verify that results are returned in expected order 5 - Experiment with fuzziness 6 - Click any result and verify specific titles are correct 7 - Click through title to record and verify it is the correct record Created attachment 64779 [details] [review] [SIGNED-OFF] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 3 - Test searching for author, title, and subject 4 - Verify that results are returned in expected order 5 - Experiment with fuzziness 6 - Click any result and verify specific titles are correct 7 - Click through title to record and verify it is the correct record Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Created attachment 69614 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 3 - Test searching for author, title, and subject 4 - Verify that results are returned in expected order 5 - Experiment with fuzziness 6 - Click any result and verify specific titles are correct 7 - Click through title to record and verify it is the correct record Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Comment on attachment 69614 [details] [review] Bug 14567: Add a browse interface to the OPAC Review of attachment 69614 [details] [review]: ----------------------------------------------------------------- It's quite basic still, but that makes it easy to test and it's really fast on my system. Some mostly small things that should be easy to check/fix: 1) There is currently no way to access the page from the OPAC if you don't know the address. Should we not add it to the top navigation? Maybe with an id to make it easy to hide? 2) Will long lists be paged? I couldn't get it to page on my system but wonder how it will look for a really long list. ::: Koha/SearchEngine/Elasticsearch/Browse.pm @@ +178,5 @@ > +=head1 AUTHOR > + > +=over 4 > + > +=item Robin Sheat C<< <robin@catalyst.net.nz> >> 5) Was the C intentional? ::: koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-browse.tt @@ +83,5 @@ > + </div><!-- / .container-fluid --> > +</div><!-- / .main --> > +[% INCLUDE 'opac-bottom.inc' %] > +[% BLOCK jsinclude %] > +<script type="text/javascript" src="[% interface %]/[% theme %]/js/browse.js"> 3) FAIL koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-browse.tt FAIL forbidden patterns forbidden pattern: Include [% KOHA_VERSION %] in js paths (bug 12904) - May be false positive (line 87) Not a false one here I think! ::: opac/opac-browse.pl @@ +91,5 @@ > + ); > + print to_json( \@output ); > +} > + > +# This is a temporary, MARC21-only thing that will grab titles, and authors 4) Still MARC21 only? Created attachment 83127 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 3 - Test searching for author, title, and subject 4 - Verify that results are returned in expected order 5 - Experiment with fuzziness 6 - Click any result and verify specific titles are correct 7 - Click through title to record and verify it is the correct record Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Created attachment 83128 [details] [review] Bug 14567: Rebase adjustments and QA fixes (In reply to Katrin Fischer from comment #38) > 1) There is currently no way to access the page from the OPAC if you don't > know the address. Should we not add it to the top navigation? Maybe with an > id to make it easy to hide? Yes, can add a followup once we are ready :-) > 2) Will long lists be paged? I couldn't get it to page on my system but > wonder how it will look for a really long list. Not yet, future enhancement I think > 5) Was the C intentional? Assuming not and removing > 3) FAIL koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-browse.tt > FAIL forbidden patterns > forbidden pattern: Include [% KOHA_VERSION %] in js paths (bug 12904) - > May be false positive (line 87) Not showing anymore > 4) Still MARC21 only? Not that I can tell Updated to use scss, you will need to run: yarn build --view=opac It would be great if we could also add a feature to browse by call number. (In reply to Jessie Zairo from comment #42) > It would be great if we could also add a feature to browse by call number. Hi Jesse, please file a separate bug for this depending on this one. Thx! Hi Nick, coming back to this: QA scripts pass now (good!) Am I right that this is ES only? But: 1) There is an issue with the CSS files (please fix first of all): $ node_modules/.bin/gulp build --view=opac [14:17:24] Using gulpfile ~/kohaclone/gulpfile.js [14:17:24] Starting 'build'... Error in plugin 'sass' Message: koha-tmpl/opac-tmpl/bootstrap/css/src/opac.scss Error: Invalid CSS after "... solid #E3E3E3;": expected "}", was ".border-radius-all(" on line 3281 of koha-tmpl/opac-tmpl/bootstrap/css/src/opac.scss >> border: 1px solid #E3E3E3; -----------------------------------^ 2) Please add the link to the navigation (formerly 1). I feel it would make sense to have this depending on a system preference as the other additional features in the navigation bar do. It should also note that this is for Elasticsearch only. But beware: We already got OpacBrowser and OpacBrowseReults... (probably OpacBrowser should be renamed OpacBrowseAuthorities...). 3) opac-browse.pl doesn't use the standard use CGI qw ( -utf8 ); 4) Follow-up bug or something to resolve before pushing? +# This is the temporary entrance point to the API. Once bug #13799 is settled, +# it should be ported to using that. Created attachment 85401 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 3 - Test searching for author, title, and subject 4 - Verify that results are returned in expected order 5 - Experiment with fuzziness 6 - Click any result and verify specific titles are correct 7 - Click through title to record and verify it is the correct record Signed-off-by: Sebastian Hierl <s.hierl@aarome.org> Created attachment 85402 [details] [review] Bug 14567: Rebase adjustments and QA fixes Created attachment 85403 [details] [review] Bug 14567: (follow-up) Add fixes and make syspref controlled To test: 1 - Apply patches 2 - Run updatedatabase 3 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl 4 - Should see "Feature not enabled" 5 - Activate the pref 6 - Reload, page should now load and be in the nav at top of page 7 - Switch to zebra - page not enabled again, link gone (In reply to Katrin Fischer from comment #44) > Hi Nick, coming back to this: > > QA scripts pass now (good!) > > Am I right that this is ES only? Yes > 1) There is an issue with the CSS files (please fix first of all): Fixed > 2) Please add the link to the navigation Okay, please open a new bug for making sane names :-) > 3) opac-browse.pl doesn't use the standard use CGI qw ( -utf8 ); Fixed > 4) Follow-up bug or something to resolve before pushing? Follow-up bug I believe - would be a bigger change to route through the API Created attachment 85887 [details] [review] Bug 14567: (QA follow-up) Fix html structure Created attachment 85888 [details] [review] Bug 14567: (QA follow-up) Remove use of not needed modules I have only one concern here: You can't say from system preference place/descipriton it is only Opac related! I would to see this preference in OPAC category, or at least update descirption? Marking failed QA means I would like to have an answer here... Created attachment 86453 [details] [review] Bug 14567: (follow-up) Rename pref and move to Opac section +# This is the temporary entrance point to the API. Once bug #13799 is settled, +# it should be ported to using that. +my $api = $query->param('api'); Should we file a separate bug for this? Sorry, can't continue here - having issus with getting Elasticsearch running: http://paste.koha-community.org/2623 (kohadevbox) Ere, could you maybe have a look at this? So, we now have two competing efforts to provide sort of an index browse: this and bug 22592. This allows for fuzziness, which I suppose could be useful, but bug 22592 aims to fix the existing index scan in the staff interface. I'd be fine with using this method (if it can be made to work properly) instead of the facet prefix method in the other bug, but I think it should be wired to the index scan functionality in staff UI too. There are also a couple of issues that need a bit of further attention: - "Browse interface" is a poor link text. Since the actual breadcrumb uses "Browse search", why not use that in the menu too? - I'm not quite following the logic of how the actual fuzziness is determined, and it's not obvious in the UI that what the user chooses may not be what they get. At the very least there should be comments in the code describing why it does what it does. - Looks like a longer term may get truncated causing the search for records to fail. An example subject term I encountered is "A history of geographical discovery in the sevente" - I think the UI needs to be polished a bit more. Currently you may get a long list where the whole page scrolls, and clicking on any of the results further down in the list won't have any apparent effect. Maybe make the browse list a scrollable box or something? Created attachment 98509 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 3 - Test searching for author, title, and subject 4 - Verify that results are returned in expected order 5 - Experiment with fuzziness 6 - Click any result and verify specific titles are correct 7 - Click through title to record and verify it is the correct record (In reply to Katrin Fischer from comment #53) > +# This is the temporary entrance point to the API. Once bug #13799 is > settled, > +# it should be ported to using that. > +my $api = $query->param('api'); > > Should we file a separate bug for this? Yes, eventually, API endpoint for the search engine is going to be a larger task - comment amended to explain what this variable currently does >- "Browse interface" is a poor link text. Since the actual breadcrumb uses "Browse search", why not use that in the menu too? Updated >- I'm not quite following the logic of how the actual fuzziness is determined, >and it's not obvious in the UI that what the user chooses may not be what they >get. At the very least there should be comments in the code describing why it >does what it does. There are comments in the Browse.pm - it uses # of substitutions, ES limits to 2 >- Looks like a longer term may get truncated causing the search for records to >fail. An example subject term I encountered is "A history of geographical >discovery in the sevente" Fixed in the mappings we cna set a max - raised to 100 >- I think the UI needs to be polished a bit more. Did you run "yarn build --view=opac"? The list is scrollable I don't think this has to be the same as the scan from 22592 - I think allowing both would allow for comparison to decide if one does work better. On test DB they are comparable at least. Created attachment 99060 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Have ES running and some records in it 2 - Enable the new system preference: OpacBrowseSearch 3 - Rebuild the OPAC CSS (see https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client) 4 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url for your site) 5 - Test searching for author, title, and subject 6 - Verify that results are returned in expected order 7 - Experiment with fuzziness 8 - Click any result and verify specific titles are correct 9 - Click through title to record and verify it is the correct record Signed-off-by: David Nind <david@davidnind.com> False positive ;) FAIL spelling Sheat ==> Sheath, sheet, cheat Testing further... Notes: - Pref appears on Search/OpenUrl - I think move up by one: Features - Test plan and commit message needs a little update: * Run yarn build --view opac * Mention OpacBrowseSearch system preference ... ? - Capitalization: Really Fuzzy As the QA queue is really full right now, I will leave the follow-ups to the devs right now. But they should be quick and I will come back asap! Improvement ideas (separate bugs): - Show number of records for each list entry - It would be cool if you could move up and down list entries with the arrow keys, making it quick move between entries and see linked records Created attachment 99292 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Apply patches 2 - Update database (updatedatabase on kohadevbox) 3 - Compile the CSS https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client yarn build --view=opac on kohadevbox 4 - Enable the new syspref OpacBrowseSearch 5 - Have ES running and some records in it SearchEngine syspref set to Elasticsearch 6 - Browse to opac home, click 'Browse search' link for your site) 7 - Test searching for author, title, and subject 8 - Verify that results are returned in expected order 9 - Experiment with fuzziness https://www.elastic.co/guide/en/elasticsearch/reference/5.6/common-options.html#fuzziness Options are: exact (0 edits), fuzzy (1 edit), very fuzzy (2 edits) 10 - Click any result and verify specific titles are correct 11 - Click through title to record and verify it is the correct record 12 - Test that disabling pref removes the link on the opac home Signed-off-by: David Nind <david@davidnind.com> Created attachment 99298 [details] [review] Bug 14567: Add a browse interface to the OPAC This is an interface for quick and efficient browsing through records. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that match that. You can then scroll through these and select the one you're after. Selecting it provides a list of records that match that particular search. To Test: 1 - Apply patches 2 - Update database (updatedatabase on kohadevbox) 3 - Compile the CSS https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client yarn build --view=opac on kohadevbox 4 - Enable the new syspref OpacBrowseSearch 5 - Have ES running and some records in it SearchEngine syspref set to Elasticsearch 6 - Browse to opac home, click 'Browse search' link for your site) 7 - Test searching for author, title, and subject 8 - Verify that results are returned in expected order 9 - Experiment with fuzziness https://www.elastic.co/guide/en/elasticsearch/reference/5.6/common-options.html#fuzziness Options are: exact (0 edits), fuzzy (1 edit), very fuzzy (2 edits) 10 - Click any result and verify specific titles are correct 11 - Click through title to record and verify it is the correct record 12 - Test that disabling pref removes the link on the opac home Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de> Who sponsored this one Nick? Nice work everyone! Pushed to master for 20.05 enhancement not backported to 19.11.x |
Created attachment 41096 [details] Very early mockup of UI This bug is to track development of a browse interface for the OPAC. See attached mockup. The idea here is that people start typing in the browser box, and as they type, the list below filters according to what they type. They can choose what field they want to search, it'll all update pretty much straight away. The fuzziness factor determines how similar things will have to be, from exact, to missing a character being OK, to transpositions and so forth being accounted for. Exact tuning probably will be done with use. As they type, authors (or subjects, or titles, etc.) will populate in the lower box, effectively as though it were a prefix search. When selected (like the [ ] in the mockup indicates), the relevant results show in the right-hand panel.