Bug 14567

Summary: Add an elasticsearch driven browse interface to the OPAC
Product: Koha Reporter: Robin Sheat <robin>
Component: Searching - ElasticsearchAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact:
Severity: new feature    
Priority: P5 - low CC: 1joynelson, ere.maijala, heather_hernandez, jonathan.druart, josef.moravec, jzairo, katrin.fischer, marjorie.barry-vila, martin.renvoize, mkstephens, nick, ztajoli
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=22592
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29439
Change sponsored?: Sponsored Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
This is an interface for quick and efficient browsing through records with Elasticsearch. It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to enter the prefix of an author, title, or subject and it'll give you a list of the options that begin with the text you entered. You can then scroll forward through these and select the one you're after. Selecting a result provides a list of records that match that particular search. **New system preference**: `OpacBrowseSearch` defaults to disabled.
Version(s) released in:
20.05.00
Bug Depends on: 12478    
Bug Blocks: 24691, 24692    
Attachments: Very early mockup of UI
Bug 14567: backend code for browse interface
Bug 14567: frontend template and JS for browse interface
Bug 14567: recompile CSS with less
Bug 14567: backend code for browse interface
Bug 14567: frontend template and JS for browse interface
Bug 14567: recompile CSS with less
Bug 14567 - (Rebase followups)
Bug 14567: backend code for browse interface
Bug 14567: frontend template and JS for browse interface
Bug 14567: recompile CSS with less
[SIGNED-OFF] Bug 14567: backend code for browse interface
[SIGNED-OFF] Bug 14567: frontend template and JS for browse interface
[SIGNED-OFF] Bug 14567: recompile CSS with less
[SIGNED-OFF] Bug 14567 (QA followup) Fix typos from tidying
Bug 14567 - Fix paths (rebase from 17372)
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Add a browse interface to the OPAC
[SIGNED-OFF] Bug 14567: Add a browse interface to the OPAC
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Rebase adjustments and QA fixes
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Rebase adjustments and QA fixes
Bug 14567: (follow-up) Add fixes and make syspref controlled
Bug 14567: (QA follow-up) Fix html structure
Bug 14567: (QA follow-up) Remove use of not needed modules
Bug 14567: (follow-up) Rename pref and move to Opac section
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Add a browse interface to the OPAC
Bug 14567: Add a browse interface to the OPAC

Description Robin Sheat 2015-07-20 04:49:55 UTC
Created attachment 41096 [details]
Very early mockup of UI

This bug is to track development of a browse interface for the OPAC.

See attached mockup.

The idea here is that people start typing in the browser box, and as they type, the list below filters according to what they type. They can choose what field they want to search, it'll all update pretty much straight away. The fuzziness factor determines how similar things will have to be, from exact, to missing a character being OK, to transpositions and so forth being accounted for. Exact tuning probably will be done with use.

As they type, authors (or subjects, or titles, etc.) will populate in the lower box, effectively as though it were a prefix search. When selected (like the [ ] in the mockup indicates), the relevant results show in the right-hand panel.
Comment 1 Robin Sheat 2015-07-27 06:22:52 UTC Comment hidden (obsolete)
Comment 2 Robin Sheat 2015-08-24 05:28:39 UTC Comment hidden (obsolete)
Comment 3 Robin Sheat 2015-08-24 05:28:42 UTC Comment hidden (obsolete)
Comment 4 Robin Sheat 2015-08-24 05:28:45 UTC Comment hidden (obsolete)
Comment 5 Robin Sheat 2015-08-24 05:33:25 UTC Comment hidden (obsolete)
Comment 6 Robin Sheat 2015-08-26 06:37:50 UTC Comment hidden (obsolete)
Comment 7 Heather 2015-08-28 21:26:27 UTC
I tried it out here: http://browse.koha.catalystdemo.net.nz
It seems nice, but it doesn't seem like a true browse to me--what I look for in a browse is to enter a search term, and have the results returned to be what is exactly before my searched term in the given index, then where the searched term is/would be, then what is exactly after the searched term in the index.  So there would be no need for "exact, fuzzy, really fuzzy," because when I browse an index, I want the results to be what is before and after that term in the given index.

That said, when I searched "exact" for "Author": Rowling
I get:
Rowling, J. K.
Rowling, J.K.

This is nice, but, again, it doesn't seem like a "browse" to me--these are just search results.  When I click the names, I get the exact same search results as I do via Advanced Search, Author Phrase:
http://browse.koha.catalystdemo.net.nz/cgi-bin/koha/opac-search.pl?idx=au%2Cphr&q=rowling&op=and&idx=kw&op=and&idx=kw&do=Search&sort_by=relevance&limit=

If I Browse via "Fuzzy" with: Rowling
I get these results:
Rollins, Henry,
Rowland, Sarah-Jane.
Rowlands, David.
Rowlands, Gena.
Rowling, J. K.
Rowling, J.K.

Which does show me some results which should be in the author index above Rowling, but I get no results that should be in the author index after Rowling.  "Really fuzzy" is the same.

I would like to see the searched for term in the middle of hits that occur before and after it in the index, e.g. something like this (an author "Browse" result, with made up examples below Rowling, J.K.):
Rollins, Henry,
Rowland, Sarah-Jane.
Rowlands, David.
Rowlands, Gena.
Rowling, J. K.
Rowling, J.K.
Rowlins, Fred.
Rozalind, Sue.
Rualla, Max.
Ruscha, Ed.
Ruther, Martin.
Quahog, Amir.
Quaker Alliance for Peace.

I hope this is helpful!  --h2
Comment 8 Robin Sheat 2015-08-31 02:43:24 UTC
(In reply to Heather from comment #7)
> I tried it out here: http://browse.koha.catalystdemo.net.nz
> It seems nice, but it doesn't seem like a true browse to me--what I look for
> in a browse is to enter a search term, and have the results returned to be
> what is exactly before my searched term in the given index, then where the
> searched term is/would be, then what is exactly after the searched term in
> the index.  So there would be no need for "exact, fuzzy, really fuzzy,"
> because when I browse an index, I want the results to be what is before and
> after that term in the given index.

There are technical reasons why this is particularly hard, essentially that pulling the index terms out in order is not something that can readily be done without patching Elasticsearch itself, which I'm considering out of scope of this project :)

However, when I thought about it with that in mind, I got to wondering what the point of a complete alphabetical index really is to a library user. There's no semantic relationship between a word that starts with R and a word that starts with S, why would it be meaningful to put one beside the other. A subject of "Symbols" is not closely related to a subject of "Tarantulas", just because they're alphabetically close. However, an author name of "Stephen" is (sorta) semantically related to an author name of "Steven", if the user hasn't quite remembered the whole name or the spelling.

> That said, when I searched "exact" for "Author": Rowling
> I get:
> Rowling, J. K.
> Rowling, J.K.

This is a GIGO situation with the data that is the result of it not being terribly cleanly catalogued in the first place. I tend to prefer working with data like this as it is better at highlighting odd edge cases.

I've been thinking about ways to correct for cases like this (where it's just a punctuation or whitespace difference), but it presents some issues relating to how it'd be done reliably with no false positives. So I've postponed further thinking about it for now. In an ideal world, everyone will be using authorities anyway, and this won't happen :)

> This is nice, but, again, it doesn't seem like a "browse" to me--these are
> just search results.  When I click the names, I get the exact same search
> results as I do via Advanced Search, Author Phrase:

It is doing pretty much the same thing as a search, yes. Essentially, it's goal is to be an easier way to look for some specific things, possibly when you're a bit unsure about the details. I think that subjects will benefit from this the most actually, as scrolling through
"Birds -- Kakapo", "Birds -- Kea", "Birds -- Kiwi" is something that has real use. Additionally, a search for "Kiwi" will also find "Birds -- Kiwi" (though this behaviour isn't visible in the data that's currently on the test site due to the way it's catalogued.)

> If I Browse via "Fuzzy" with: Rowling
> I get these results:
> Rollins, Henry,
> Rowland, Sarah-Jane.
> Rowlands, David.
> Rowlands, Gena.
> Rowling, J. K.
> Rowling, J.K.
> 
> Which does show me some results which should be in the author index above
> Rowling, but I get no results that should be in the author index after
> Rowling.  "Really fuzzy" is the same.

I'll double check this, but it's most likely that in this database, there's just nothing further that matches. Actually, in this case, fuzzy and extra fuzzy will be doing the same thing as there's a max edit distance limit for the fuzziness, and I think it will have been hit. I'm probably going to remove the "extra fuzzy" option anyway, it risks being more confusing than anything.

> I would like to see the searched for term in the middle of hits that occur
> before and after it in the index, e.g. something like this (an author
> "Browse" result, with made up examples below Rowling, J.K.):

But why? What does an alphabetically adjacent author tell you about that set of authors, and what does it provide that a library user would find useful? (Technical reasons for why this is hard aside.)

> I hope this is helpful!  --h2

Thanks for checking it out!
Comment 9 Heather 2015-08-31 15:19:23 UTC
(In reply to Robin Sheat from comment #8)
> (In reply to Heather from comment #7)
> > I tried it out here: http://browse.koha.catalystdemo.net.nz
> > It seems nice, but it doesn't seem like a true browse to me--what I look for
> > in a browse is to enter a search term, and have the results returned to be
> > what is exactly before my searched term in the given index, then where the
> > searched term is/would be, then what is exactly after the searched term in
> > the index.  So there would be no need for "exact, fuzzy, really fuzzy,"
> > because when I browse an index, I want the results to be what is before and
> > after that term in the given index.
> 
> There are technical reasons why this is particularly hard, essentially that
> pulling the index terms out in order is not something that can readily be
> done without patching Elasticsearch itself, which I'm considering out of
> scope of this project :)
> 
> However, when I thought about it with that in mind, I got to wondering what
> the point of a complete alphabetical index really is to a library user.
> There's no semantic relationship between a word that starts with R and a
> word that starts with S, why would it be meaningful to put one beside the
> other. A subject of "Symbols" is not closely related to a subject of
> "Tarantulas", just because they're alphabetically close. However, an author
> name of "Stephen" is (sorta) semantically related to an author name of
> "Steven", if the user hasn't quite remembered the whole name or the spelling.
> 
> > That said, when I searched "exact" for "Author": Rowling
> > I get:
> > Rowling, J. K.
> > Rowling, J.K.
> 
> This is a GIGO situation with the data that is the result of it not being
> terribly cleanly catalogued in the first place. I tend to prefer working
> with data like this as it is better at highlighting odd edge cases.
> 
> I've been thinking about ways to correct for cases like this (where it's
> just a punctuation or whitespace difference), but it presents some issues
> relating to how it'd be done reliably with no false positives. So I've
> postponed further thinking about it for now. In an ideal world, everyone
> will be using authorities anyway, and this won't happen :)
> 
> > This is nice, but, again, it doesn't seem like a "browse" to me--these are
> > just search results.  When I click the names, I get the exact same search
> > results as I do via Advanced Search, Author Phrase:
> 
> It is doing pretty much the same thing as a search, yes. Essentially, it's
> goal is to be an easier way to look for some specific things, possibly when
> you're a bit unsure about the details. I think that subjects will benefit
> from this the most actually, as scrolling through
> "Birds -- Kakapo", "Birds -- Kea", "Birds -- Kiwi" is something that has
> real use. Additionally, a search for "Kiwi" will also find "Birds -- Kiwi"
> (though this behaviour isn't visible in the data that's currently on the
> test site due to the way it's catalogued.)
> 
> > If I Browse via "Fuzzy" with: Rowling
> > I get these results:
> > Rollins, Henry,
> > Rowland, Sarah-Jane.
> > Rowlands, David.
> > Rowlands, Gena.
> > Rowling, J. K.
> > Rowling, J.K.
> > 
> > Which does show me some results which should be in the author index above
> > Rowling, but I get no results that should be in the author index after
> > Rowling.  "Really fuzzy" is the same.
> 
> I'll double check this, but it's most likely that in this database, there's
> just nothing further that matches. Actually, in this case, fuzzy and extra
> fuzzy will be doing the same thing as there's a max edit distance limit for
> the fuzziness, and I think it will have been hit. I'm probably going to
> remove the "extra fuzzy" option anyway, it risks being more confusing than
> anything.
> 
> > I would like to see the searched for term in the middle of hits that occur
> > before and after it in the index, e.g. something like this (an author
> > "Browse" result, with made up examples below Rowling, J.K.):
> 
> But why? What does an alphabetically adjacent author tell you about that set
> of authors, and what does it provide that a library user would find useful?
> (Technical reasons for why this is hard aside.)
> 
> > I hope this is helpful!  --h2
> 
> Thanks for checking it out!

Hi, Robin!

One of the main purposes of a complete alphabetical browse for the library user is exactly what you mentioned: when the user is unsure of the spelling, but has most of it, e.g., "Stephens" or "Stevens," or something like "Ralphs," or "Rolphs."  And, with subjects, the user will receive a list with all subdivisions listed & will see those, rather than just search results that don't show the subject terms.

Yes--GIGO is an issue in the above browse, and ideally all access points are under authority control, but the browse is useful for libraries that have order records that are not yet cataloged, and who only do their authority control/cataloging upon receipt of the item.  The above issue in the spacing of "Rowling, J. K." could occur with order records vs. fully cataloged & received items.

The alphabetic browse is also invaluable for the catalogers on the staff side doing authority control--it's the easiest way to spot heading variations.

In the Koha public opac, a basic search always gives you a list of search results that never shows subject terms--the facets on the left show *some* subject terms but, e.g., if you do a basic keyword search in your demo catalog for "Birds," the subject facets on the left have subdivisions broken out & not associated with the main heading:
    Animals (2)
    Birds (6)
    Maoritanga. (1)
    Mystery and suspense stories. (1)
    Mythology, Maori. (1)
    Natural history (2)
    New Zealand (3)
    New Zealand. (3)
    Penguins. (1)
    Pictorial works. (1)
    Show more
(It's the same in our catalog, i.e. this search only gives you facets on the left that are main headings, not headings *with* their subdivisions: http://keys.bywatersolutions.com/cgi-bin/koha/opac-search.pl?q=ships)

So an alphabetical browse that does return subject terms with their subdivision, like in your example, would be useful to a user, e.g.:
Birds--Kakapo.
Birds--Kea.
Birds--Kuwait.
Birds--Pictorial works.
Birds--Qatar.

(BTW, the Authority Search in the demo catalog has a problem--do an authority search with default options & the term "Birds" & you get:
Software error:
Can't call method "each" on unblessed reference at /opt/kohaclones/browse/Koha/SearchEngine/Elasticsearch/Search.pm line 265.
For help, please send mail to the webmaster ([no address given]), giving this error message and the time and date of the error.")

Do a similar authority search in our catalog for ships:
http://keys.bywatersolutions.com/cgi-bin/koha/opac-authorities-home.pl?op=do_search&type=opac&operator=contains&value=ships&marclist=any&and_or=and&orderby=HeadingAsc

And you get close to a subject heading browse result--the only problem is that it returns only main headings, not an alphabetic list of the terms *with subdivisions* as used in the records, which would be very useful to the user.  I bring this up because, IMO, as it is, authority control/authority search is not an alternative to a true alphabetical catalog browse for the public opac user, of subject terms with subdivisions as used in the catalog.

I hope this is helpful!  Best,
--h2
Comment 10 Katrin Fischer 2015-08-31 15:34:20 UTC
I was thinking this morning, maybe a 'starts with' search option could help to bring the result list a little closer to an alphabetic index. Could this be an option?

I have asked some of the coworkers to give the search a test, but some of the results are quite confusing. 

Example: Exact, Author: Miller
The first result shown is "Millard, Annie". 
I didn't expect that for the exact search option, but only with fuzzy activated.

Another issue could be related to ISBD punctuation in the fields. Often there are 2 entries which what seems to be the same author, only different by a . at the end:

Example: Exact, Author: Smith
Smith, Alexander Gordon.
Smith, Alexander Gordon
Smith, Jim
Smith, Jim.

Would be good if those could be merged for display and search.
Comment 11 Heather 2015-08-31 16:05:53 UTC
(In reply to Katrin Fischer from comment #10)
> I was thinking this morning, maybe a 'starts with' search option could help
> to bring the result list a little closer to an alphabetic index. Could this
> be an option?

It sounds to me like it could be an option, if the results were an alphabetic list of terms from the searched index, rather than a list of records.

> I have asked some of the coworkers to give the search a test, but some of
> the results are quite confusing. 
> 
> Example: Exact, Author: Miller
> The first result shown is "Millard, Annie". 
> I didn't expect that for the exact search option, but only with fuzzy
> activated.

It sounds like your "exact" is kind of "fuzzy!"

> Another issue could be related to ISBD punctuation in the fields. Often
> there are 2 entries which what seems to be the same author, only different
> by a . at the end:
> 
> Example: Exact, Author: Smith
> Smith, Alexander Gordon.
> Smith, Alexander Gordon
> Smith, Jim
> Smith, Jim.
> 
> Would be good if those could be merged for display and search.

Could this also be an issue of variations in the headings' $q fields, which Koha tends to not display in search results, and when used there's no punctuation between the end of the $a and the beginning of the $q subfields?  In your example, above, the headings could be for different people, and so should not be merged, e.g., could be:
100 1_ Smith, Alexander Gordon.
100 1_ Smith, Alexander Gordon $q (Alexander Gordon Frank)
100 1_ Smith, Jim $q (James Allen)
100 1_ Smith, Jim.

(The headings' variations are most likely the result of GIGO, & so proper authority control would collate them.)
--h2
Comment 12 Robin Sheat 2015-09-01 01:23:55 UTC
(In reply to Heather from comment #9)
> One of the main purposes of a complete alphabetical browse for the library
> user is exactly what you mentioned: when the user is unsure of the spelling,
> but has most of it, e.g., "Stephens" or "Stevens," or something like
> "Ralphs," or "Rolphs."  

Right, and that is handled here by allowing fuzzier searching. Having everything there doesn't seem like it'd make that particularly easy, as things might be separated quite widely visually even if there's only a one character difference.

Y'know what'd be interesting, incorporating "sounds like" searching to it. I'll leave that for another time though. I also shudder to think how that'd be internationalised :)

> The alphabetic browse is also invaluable for the catalogers on the staff
> side doing authority control--it's the easiest way to spot heading
> variations.

Certainly, but I'm consciously not focussing on that side of things, or stuff will get too big too fast :)

> (BTW, the Authority Search in the demo catalog has a problem--do an
> authority search with default options & the term "Birds" & you get:
> Software error:
> Can't call method "each" on unblessed reference at
> /opt/kohaclones/browse/Koha/SearchEngine/Elasticsearch/Search.pm line 265.
> For help, please send mail to the webmaster ([no address given]), giving
> this error message and the time and date of the error.")

That doesn't surprise me just because this code is new and largely unstressed, though I'll look into it, it shouldn't be happening. 

> Do a similar authority search in our catalog for ships:
> http://keys.bywatersolutions.com/cgi-bin/koha/opac-authorities-home.
> pl?op=do_search&type=opac&operator=contains&value=ships&marclist=any&and_or=a
> nd&orderby=HeadingAsc
> 
> And you get close to a subject heading browse result--the only problem is
> that it returns only main headings, not an alphabetic list of the terms
> *with subdivisions* as used in the records, which would be very useful to
> the user.  I bring this up because, IMO, as it is, authority
> control/authority search is not an alternative to a true alphabetical
> catalog browse for the public opac user, of subject terms with subdivisions
> as used in the catalog.

Just FYI, this isn't using the authority database, it's using the actual search indices.

I may swap out the database some time soon with one that has a more "properly" catalogued catalogue, to see how that looks.

(In reply to Katrin Fischer from comment #10)
> I was thinking this morning, maybe a 'starts with' search option could help
> to bring the result list a little closer to an alphabetic index. Could this
> be an option?

This is exactly what is there right now, or I misunderstand you :)

> Example: Exact, Author: Miller
> The first result shown is "Millard, Annie". 
> I didn't expect that for the exact search option, but only with fuzzy
> activated.

Hmm. That must be a bug. That shouldn't be happening.

> Another issue could be related to ISBD punctuation in the fields. Often
> there are 2 entries which what seems to be the same author, only different
> by a . at the end:
> 
> Example: Exact, Author: Smith
> Smith, Alexander Gordon.
> Smith, Alexander Gordon
> Smith, Jim
> Smith, Jim.
> 
> Would be good if those could be merged for display and search.

I've been thinking about this. It's a bit problematic, due to way the MARC->ES conversion works. However, I want to figure out how to tap into this for some other things too, if I can attach bits of Perl to certain points in the conversion, it might be doable.

We're still going to get other weird results though, e.g.

Rowling, J. K.
Rowling, J.K.

fixing that in a way that doesn't cause false positives, or make the results inaccurate, may be hard.
Comment 13 Heather 2015-09-01 15:44:04 UTC
> Just FYI, this isn't using the authority database, it's using the actual
> search indices.

Oh, right!  I'm sorry, I think I wasn't very clear--I was trying to say that there's no way right now for a user to ever see the subject headings with subdivisions as used in the records.  Maybe this new browse feature, when browsing subjects, could return an alphabetic list of subject headings *with* subdivisions from the records.  I hope this is clearer!
 
> I may swap out the database some time soon with one that has a more
> "properly" catalogued catalogue, to see how that looks.

Our catalog is not the cleanest (I haven't done all our authority work & headings clean up), but it's pretty good--if there's any way our records, a subset of our records, or even we can help with this, we'd be very, very happy to help.  Our catalog is here, http://keys.bywatersolutions.com/ supported by ByWater, so perhaps if y'all working on this bug contact Brendan at ByWater, there's a way that we can help test with our catalog?  Or perhaps even ByWater could make our records or a subset of our records available so y'all have a more "cataloged catalog" for testing?

--h2
Comment 14 Robin Sheat 2015-09-02 03:07:34 UTC
(In reply to Heather from comment #13)
> Oh, right!  I'm sorry, I think I wasn't very clear--I was trying to say that
> there's no way right now for a user to ever see the subject headings with
> subdivisions as used in the records.  Maybe this new browse feature, when
> browsing subjects, could return an alphabetic list of subject headings
> *with* subdivisions from the records.  I hope this is clearer!

Ah, this will do that. You just can't see it because this catalogue doesn't handle subjects like that.

> Our catalog is not the cleanest (I haven't done all our authority work &
> headings clean up), but it's pretty good--if there's any way our records, a
> subset of our records, or even we can help with this, we'd be very, very
> happy to help.  Our catalog is here, http://keys.bywatersolutions.com/
> supported by ByWater, so perhaps if y'all working on this bug contact
> Brendan at ByWater, there's a way that we can help test with our catalog? 
> Or perhaps even ByWater could make our records or a subset of our records
> available so y'all have a more "cataloged catalog" for testing?

More test data is good, I've pinged Brendan to have a look into it.
Comment 15 Robin Sheat 2015-10-14 01:54:14 UTC Comment hidden (obsolete)
Comment 16 Nick Clemens 2016-10-12 11:50:29 UTC Comment hidden (obsolete)
Comment 17 Nick Clemens 2016-10-12 11:50:36 UTC Comment hidden (obsolete)
Comment 18 Nick Clemens 2016-10-12 11:50:43 UTC Comment hidden (obsolete)
Comment 19 Nick Clemens 2016-10-12 11:50:50 UTC Comment hidden (obsolete)
Comment 20 Nick Clemens 2016-10-12 11:51:36 UTC Comment hidden (obsolete)
Comment 21 Nick Clemens 2016-10-12 12:26:28 UTC Comment hidden (obsolete)
Comment 22 Nick Clemens 2016-10-12 12:26:37 UTC Comment hidden (obsolete)
Comment 23 Nick Clemens 2016-10-12 12:26:47 UTC Comment hidden (obsolete)
Comment 24 Nick Clemens 2016-10-12 12:57:00 UTC Comment hidden (obsolete)
Comment 25 Nick Clemens 2016-10-12 12:57:08 UTC Comment hidden (obsolete)
Comment 26 Nick Clemens 2016-10-12 12:57:19 UTC Comment hidden (obsolete)
Comment 27 Nick Clemens 2016-10-12 12:57:29 UTC Comment hidden (obsolete)
Comment 28 Jonathan Druart 2016-10-25 08:49:37 UTC Comment hidden (obsolete)
Comment 29 Nick Clemens 2016-11-03 19:48:54 UTC Comment hidden (obsolete)
Comment 30 J Schmidt 2016-11-03 19:56:59 UTC Comment hidden (obsolete)
Comment 31 Nick Clemens 2016-11-03 20:00:27 UTC Comment hidden (obsolete)
Comment 32 Jonathan Druart 2016-11-23 10:16:54 UTC
Scalar value @_[0] better written as $_[0] at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/Search.pm line 126.
Scalar value @_[0] better written as $_[0] at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/Search.pm line 128.
Can't use string ("(d)") as an ARRAY ref while "strict refs" in use at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 108.


Please do a *complete* pass on this patchset before sending it back to the QA queue:
- Remove debug statements
- Add a test plan:
- Provide full test coverage
- The opac-browse template looks wrong (does not follow the usual pattern, is it expected?)
- And finally, test
Comment 33 Nick Clemens 2017-03-28 14:05:52 UTC Comment hidden (obsolete)
Comment 34 Nick Clemens 2017-06-20 21:22:06 UTC Comment hidden (obsolete)
Comment 35 Nick Clemens 2017-06-20 21:26:56 UTC
Created attachment 64478 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
1 - Have ES running and some records in it
2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url
for your site)
3 - Test searching for author, title, and subject
4 - Verify that results are returned in expected order
5 - Experiment with fuzziness
6 - Click any result and verify specific titles are correct
7 - Click through title to record and verify it is the correct record
Comment 36 Nick Clemens 2017-07-03 16:08:49 UTC
Created attachment 64779 [details] [review]
[SIGNED-OFF] Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
1 - Have ES running and some records in it
2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url
for your site)
3 - Test searching for author, title, and subject
4 - Verify that results are returned in expected order
5 - Experiment with fuzziness
6 - Click any result and verify specific titles are correct
7 - Click through title to record and verify it is the correct record

Signed-off-by: Sebastian Hierl <s.hierl@aarome.org>
Comment 37 Nick Clemens 2017-12-07 14:57:28 UTC
Created attachment 69614 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
1 - Have ES running and some records in it
2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url
for your site)
3 - Test searching for author, title, and subject
4 - Verify that results are returned in expected order
5 - Experiment with fuzziness
6 - Click any result and verify specific titles are correct
7 - Click through title to record and verify it is the correct record

Signed-off-by: Sebastian Hierl <s.hierl@aarome.org>
Comment 38 Katrin Fischer 2018-04-09 06:03:13 UTC
Comment on attachment 69614 [details] [review]
Bug 14567: Add a browse interface to the OPAC

Review of attachment 69614 [details] [review]:
-----------------------------------------------------------------

It's quite basic still, but that makes it easy to test and it's really fast on my system.

Some mostly small things that should be easy to check/fix:

1) There is currently no way to access the page from the OPAC if you don't know the address. Should we not add it to the top navigation? Maybe with an id to make it easy to hide?
2) Will long lists be paged? I couldn't get it to page on my system but wonder how it will look for a really long list.

::: Koha/SearchEngine/Elasticsearch/Browse.pm
@@ +178,5 @@
> +=head1 AUTHOR
> +
> +=over 4
> +
> +=item Robin Sheat C<< <robin@catalyst.net.nz> >>

5) Was the C intentional?

::: koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-browse.tt
@@ +83,5 @@
> +    </div><!-- / .container-fluid -->
> +</div><!-- / .main -->
> +[% INCLUDE 'opac-bottom.inc' %]
> +[% BLOCK jsinclude %]
> +<script type="text/javascript" src="[% interface %]/[% theme %]/js/browse.js">

3) FAIL	koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-browse.tt
   FAIL	  forbidden patterns
		forbidden pattern: Include [% KOHA_VERSION %] in js paths (bug 12904) - May be false positive (line 87)

Not a false one here I think!

::: opac/opac-browse.pl
@@ +91,5 @@
> +    );
> +    print to_json( \@output );
> +}
> +
> +# This is a temporary, MARC21-only thing that will grab titles, and authors

4) Still MARC21 only?
Comment 39 Nick Clemens 2018-12-12 14:49:01 UTC
Created attachment 83127 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
1 - Have ES running and some records in it
2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url
for your site)
3 - Test searching for author, title, and subject
4 - Verify that results are returned in expected order
5 - Experiment with fuzziness
6 - Click any result and verify specific titles are correct
7 - Click through title to record and verify it is the correct record

Signed-off-by: Sebastian Hierl <s.hierl@aarome.org>
Comment 40 Nick Clemens 2018-12-12 14:49:07 UTC
Created attachment 83128 [details] [review]
Bug 14567: Rebase adjustments and QA fixes
Comment 41 Nick Clemens 2018-12-12 14:51:06 UTC
(In reply to Katrin Fischer from comment #38)
> 1) There is currently no way to access the page from the OPAC if you don't
> know the address. Should we not add it to the top navigation? Maybe with an
> id to make it easy to hide?
Yes, can add a followup once we are ready :-)

> 2) Will long lists be paged? I couldn't get it to page on my system but
> wonder how it will look for a really long list.
Not yet, future enhancement I think

> 5) Was the C intentional?
Assuming not and removing

> 3) FAIL	koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-browse.tt
>    FAIL	  forbidden patterns
> 		forbidden pattern: Include [% KOHA_VERSION %] in js paths (bug 12904) -
> May be false positive (line 87)

Not showing anymore

> 4) Still MARC21 only?
Not that I can tell


Updated to use scss, you will need to run:
yarn build --view=opac
Comment 42 Jessie Zairo 2018-12-20 15:46:14 UTC
It would be great if we could also add a feature to browse by call number.
Comment 43 Katrin Fischer 2019-02-02 14:15:31 UTC
(In reply to Jessie Zairo from comment #42)
> It would be great if we could also add a feature to browse by call number.

Hi Jesse, please file a separate bug for this depending on this one. Thx!
Comment 44 Katrin Fischer 2019-02-02 14:30:59 UTC
Hi Nick, coming back to this:

QA scripts pass now (good!)

Am I right that this is ES only?

But:

1) There is an issue with the CSS files (please fix first of all):

$ node_modules/.bin/gulp build --view=opac
[14:17:24] Using gulpfile ~/kohaclone/gulpfile.js
[14:17:24] Starting 'build'...
Error in plugin 'sass'
Message:
    koha-tmpl/opac-tmpl/bootstrap/css/src/opac.scss
Error: Invalid CSS after "... solid #E3E3E3;": expected "}", was ".border-radius-all("
        on line 3281 of koha-tmpl/opac-tmpl/bootstrap/css/src/opac.scss
>>         border: 1px solid #E3E3E3;
   -----------------------------------^

2) Please add the link to the navigation (formerly 1). I feel it would make sense to have this depending on a system preference as the other additional features in the navigation bar do. It should also note that this is for Elasticsearch only. But beware: We already got OpacBrowser and OpacBrowseReults... (probably OpacBrowser should be renamed OpacBrowseAuthorities...).

3) opac-browse.pl doesn't use the standard use CGI qw ( -utf8 );

4) Follow-up bug or something to resolve before pushing?

+# This is the temporary entrance point to the API. Once bug #13799 is settled,
+# it should be ported to using that.
Comment 45 Nick Clemens 2019-02-20 18:22:22 UTC
Created attachment 85401 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
1 - Have ES running and some records in it
2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url
for your site)
3 - Test searching for author, title, and subject
4 - Verify that results are returned in expected order
5 - Experiment with fuzziness
6 - Click any result and verify specific titles are correct
7 - Click through title to record and verify it is the correct record

Signed-off-by: Sebastian Hierl <s.hierl@aarome.org>
Comment 46 Nick Clemens 2019-02-20 18:22:26 UTC
Created attachment 85402 [details] [review]
Bug 14567: Rebase adjustments and QA fixes
Comment 47 Nick Clemens 2019-02-20 18:22:29 UTC
Created attachment 85403 [details] [review]
Bug 14567: (follow-up) Add fixes and make syspref controlled

To test:
1 - Apply patches
2 - Run updatedatabase
3 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl
4 - Should see "Feature not enabled"
5 - Activate the pref
6 - Reload, page should now load and be in the nav at top of page
7 - Switch to zebra - page not enabled again, link gone
Comment 48 Nick Clemens 2019-02-20 18:23:57 UTC
(In reply to Katrin Fischer from comment #44)
> Hi Nick, coming back to this:
> 
> QA scripts pass now (good!)
> 
> Am I right that this is ES only?
Yes
> 1) There is an issue with the CSS files (please fix first of all):
Fixed
> 2) Please add the link to the navigation 
Okay, please open a new bug for making sane names :-)
> 3) opac-browse.pl doesn't use the standard use CGI qw ( -utf8 );
Fixed
> 4) Follow-up bug or something to resolve before pushing?
Follow-up bug I believe - would be a bigger change to route through the API
Comment 49 Josef Moravec 2019-03-01 08:18:48 UTC
Created attachment 85887 [details] [review]
Bug 14567: (QA follow-up) Fix html structure
Comment 50 Josef Moravec 2019-03-01 08:18:52 UTC
Created attachment 85888 [details] [review]
Bug 14567: (QA follow-up) Remove use of not needed modules
Comment 51 Josef Moravec 2019-03-01 08:21:26 UTC
I have only one concern here: You can't say from system preference place/descipriton it is only Opac related! I would to see this preference in OPAC category, or at least update descirption?

Marking failed QA means I would like to have an answer here...
Comment 52 Nick Clemens 2019-03-11 13:49:06 UTC
Created attachment 86453 [details] [review]
Bug 14567: (follow-up) Rename pref and move to Opac section
Comment 53 Katrin Fischer 2019-04-09 11:45:23 UTC
+# This is the temporary entrance point to the API. Once bug #13799 is settled,
+# it should be ported to using that.
+my $api = $query->param('api');

Should we file a separate bug for this?
Comment 54 Katrin Fischer 2019-04-09 11:52:05 UTC
Sorry, can't continue here - having issus with getting Elasticsearch running: 
http://paste.koha-community.org/2623 (kohadevbox)
Comment 55 Katrin Fischer 2019-04-09 11:56:01 UTC
Ere, could you maybe have a look at this?
Comment 56 Ere Maijala 2019-04-15 12:16:05 UTC
So, we now have two competing efforts to provide sort of an index browse: this and bug 22592. This allows for fuzziness, which I suppose could be useful, but bug 22592 aims to fix the existing index scan in the staff interface. I'd be fine with using this method (if it can be made to work properly) instead of the facet prefix method in the other bug, but I think it should be wired to the index scan functionality in staff UI too.

There are also a couple of issues that need a bit of further attention:

- "Browse interface" is a poor link text. Since the actual breadcrumb uses "Browse search", why not use that in the menu too?

- I'm not quite following the logic of how the actual fuzziness is determined, and it's not obvious in the UI that what the user chooses may not be what they get. At the very least there should be comments in the code describing why it does what it does.

- Looks like a longer term may get truncated causing the search for records to fail. An example subject term I encountered is "A history of geographical discovery in the sevente"

- I think the UI needs to be polished a bit more. Currently you may get a long list where the whole page scrolls, and clicking on any of the results further down in the list won't have any apparent effect. Maybe make the browse list a scrollable box or something?
Comment 57 Nick Clemens 2020-02-05 18:58:17 UTC
Created attachment 98509 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
1 - Have ES running and some records in it
2 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url
for your site)
3 - Test searching for author, title, and subject
4 - Verify that results are returned in expected order
5 - Experiment with fuzziness
6 - Click any result and verify specific titles are correct
7 - Click through title to record and verify it is the correct record
Comment 58 Nick Clemens 2020-02-05 19:04:07 UTC
(In reply to Katrin Fischer from comment #53)
> +# This is the temporary entrance point to the API. Once bug #13799 is
> settled,
> +# it should be ported to using that.
> +my $api = $query->param('api');
> 
> Should we file a separate bug for this?

Yes, eventually, API endpoint for the search engine is going to be a larger task - comment amended to explain what this variable currently does 

>- "Browse interface" is a poor link text. Since the actual breadcrumb uses "Browse search", why not use that in the menu too?

Updated

>- I'm not quite following the logic of how the actual fuzziness is determined,
>and it's not obvious in the UI that what the user chooses may not be what they
>get. At the very least there should be comments in the code describing why it
>does what it does.

There are comments in the Browse.pm - it uses # of substitutions, ES limits to 2

>- Looks like a longer term may get truncated causing the search for records to >fail. An example subject term I encountered is "A history of geographical
>discovery in the sevente"

Fixed in the mappings we cna set a max - raised to 100

>- I think the UI needs to be polished a bit more.

Did you run "yarn build --view=opac"?

The list is scrollable


I don't think this has to be the same as the scan from 22592 - I think allowing both would allow for comparison to decide if one does work better. On test DB they are comparable at least.
Comment 59 David Nind 2020-02-15 10:32:37 UTC
Created attachment 99060 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
1 - Have ES running and some records in it
2 - Enable the new system preference: OpacBrowseSearch
3 - Rebuild the OPAC CSS (see https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client)
4 - Browse to http://localhost:8080/cgi-bin/koha/opac-browse.pl (or url
    for your site)
5 - Test searching for author, title, and subject
6 - Verify that results are returned in expected order
7 - Experiment with fuzziness
8 - Click any result and verify specific titles are correct
9 - Click through title to record and verify it is the correct record

Signed-off-by: David Nind <david@davidnind.com>
Comment 60 Katrin Fischer 2020-02-19 22:22:32 UTC
False positive ;)
   FAIL	  spelling
		 Sheat  ==> Sheath, sheet, cheat

Testing further...
Comment 61 Katrin Fischer 2020-02-19 22:37:55 UTC
Notes:
- Pref appears on Search/OpenUrl - I think move up by one: Features
- Test plan and commit message needs a little update:

* Run yarn build --view opac
* Mention OpacBrowseSearch system preference
... ?

- Capitalization: Really Fuzzy

As the QA queue is really full right now, I will leave the follow-ups to the devs right now. But they should be quick and I will come back asap!


Improvement ideas (separate bugs):
- Show number of records for each list entry
- It would be cool if you could move up and down list entries with the arrow keys, making it quick move between entries and see linked records
Comment 62 Nick Clemens 2020-02-20 03:01:55 UTC
Created attachment 99292 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
 1 - Apply patches
 2 - Update database (updatedatabase on kohadevbox)
 3 - Compile the CSS
   https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client
   yarn build --view=opac on kohadevbox
 4 - Enable the new syspref OpacBrowseSearch
 5 - Have ES running and some records in it
     SearchEngine syspref set to Elasticsearch
 6 - Browse to opac home, click 'Browse search' link
for your site)
 7 - Test searching for author, title, and subject
 8 - Verify that results are returned in expected order
 9 - Experiment with fuzziness
     https://www.elastic.co/guide/en/elasticsearch/reference/5.6/common-options.html#fuzziness
     Options are: exact (0 edits), fuzzy (1 edit), very fuzzy (2 edits)
10 - Click any result and verify specific titles are correct
11 - Click through title to record and verify it is the correct record
12 - Test that disabling pref removes the link on the opac home

Signed-off-by: David Nind <david@davidnind.com>
Comment 63 Katrin Fischer 2020-02-20 07:10:21 UTC
Created attachment 99298 [details] [review]
Bug 14567: Add a browse interface to the OPAC

This is an interface for quick and efficient browsing through records.

It presents a page at /cgi-bin/koha/opac-browse.pl that allows you to
enter the prefix of an author, title, or subject and it'll give you a
list of the options that match that. You can then scroll through these
and select the one you're after. Selecting it provides a list of records
that match that particular search.

To Test:
 1 - Apply patches
 2 - Update database (updatedatabase on kohadevbox)
 3 - Compile the CSS
   https://wiki.koha-community.org/wiki/Working_with_SCSS_in_the_OPAC_and_staff_client
   yarn build --view=opac on kohadevbox
 4 - Enable the new syspref OpacBrowseSearch
 5 - Have ES running and some records in it
     SearchEngine syspref set to Elasticsearch
 6 - Browse to opac home, click 'Browse search' link
for your site)
 7 - Test searching for author, title, and subject
 8 - Verify that results are returned in expected order
 9 - Experiment with fuzziness
     https://www.elastic.co/guide/en/elasticsearch/reference/5.6/common-options.html#fuzziness
     Options are: exact (0 edits), fuzzy (1 edit), very fuzzy (2 edits)
10 - Click any result and verify specific titles are correct
11 - Click through title to record and verify it is the correct record
12 - Test that disabling pref removes the link on the opac home

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 64 Martin Renvoize 2020-02-20 09:08:57 UTC
Who sponsored this one Nick?
Comment 65 Martin Renvoize 2020-02-20 09:11:01 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 66 Joy Nelson 2020-03-13 00:50:16 UTC
enhancement not backported to 19.11.x