Bug 10891

Summary: Make facets customisable
Product: Koha Reporter: Mathieu Saby <mathsabypro>
Component: SearchingAssignee: Galen Charlton <gmcharlt>
Status: RESOLVED DUPLICATE QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P3 CC: d.gavio, dcook, f.demians, jdemuth, katrin.fischer, marjorie.barry-vila, mtompset, nengard, p.thrasher, tomascohen, veron, vfernandes, weinheimer.jim.l
Version: master   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10892
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8360
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: config page design

Description Mathieu Saby 2013-09-16 07:41:41 UTC
Currently, facets are hard coded in Search.pm and Koha.pm.
This bug will provide a way for each library to customize facets.

Here is my plan : 

1. move facets configuration into a YAML block of data, still stored in Koha.pm. No functional changes at that step

2. store configuration in a YAML file and create a page for allowing librarians to change 
- the fields used for each Koha standard facets
- the sorting order of results inside each facet (ie. alphabetic a-z or z-a, nbr of occurencies 1-100 100-1)
- the displaying orders of facets blocks (ie. Authors before Libraires etc)
- the visibility of each facets (ie. ability to hide "Title" or "Author" facet)

3. define new standard Koha facets (that could be masked by users if they don't like them)
- precise subjects and author facets (geographical subjects, etc.)
- date of publication
- language

4. allow librarians to define custom facets

5. allow facets on coded fieds (0XX in MARC21 and 0XX / 1XX in Unimarc)

6. allow opac facets to be different from staff interface facets

That's all folks!

For the moment, I have only written the 1st patch ;-)
Maybe the best would be to make several bugs an attach them to this one?
Comment 1 Frédéric Demians 2013-09-16 08:01:33 UTC
Great initiative!

Few remarks:

Your step 2 will raise issues. Modifying a text file from a Koha web page is a security risk. A YAML string in a syspref is preferable. 

As you know, defining facets requires not only modifying facets data structure in Search.pm, but also defining indexes in Zebra, both info coordinated. Since altering Zebra indexes from Web UI isn't possible at all, I'm not sure there is so much interest in allowing users to partially defines facets from Koha Web admin module.

But your plan step 3 may be a solution. You could pre-define facets, without omitting Zebra configuration, as much as possible, and add a way to enable/disable them. For very local configuration, you can add a local facets YAML string, which would complete the standard definition.
Comment 2 Mathieu Saby 2013-09-16 08:16:09 UTC
Hi Frederic
I was thinking of a syspref first. It would be easier for me to do (2 sysprefs in fact, one for unimarc, the other for marc21/normarc)
But I thought it was dangerous to provide a YAML block directly in sysprefs, because if the user makes a mistake, he can break everything easily. And it won't be very user friendly...

So I thought I could use a model the page made by Biblibre for configuring SolR facets. But I am not sure of that.

For the "full customisation", as you say, it won't work if Zebra is not configure in the same way. Maybe we could just add a BIG message in RED to warn people? But I am not at step 4 yet!

Mathieu
Comment 3 Mathieu Saby 2013-09-16 08:32:15 UTC
I forgot one little improvement : 
Currently, the link "show more" under each facet relaunch an other search. That's not good. We should get all facets, and only hide/show them with CSS!


Mathieu
Comment 4 Mathieu Saby 2013-09-16 10:27:04 UTC
This bug will be an "umbrella" bug for discussing the way of making this enhancement.
A 1st patch will be attached to bug 10892

By the w

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10893
Comment 5 Mathieu Saby 2013-09-16 10:28:12 UTC
oups...

By the way, I found an error in current configuration (for location facet in UNIMARC). Patch attached to new bug 10893


Mathieu
Comment 6 David Cook 2013-09-17 02:12:11 UTC
You'll also want to include the Leader so that a Record Type facet can be included  (although record type would also probably need to take into account the 006/007/008 so that is easier said than done).

I really like the idea of making facets better. I wish you luck!
Comment 7 Mathieu Saby 2013-09-17 07:24:52 UTC
yes, good idea for the leader ;-)

Mathieu
Comment 8 Mathieu Saby 2013-09-17 16:09:31 UTC
1st patch attached on Bug 10892

Mathieu
Comment 9 Mathieu Saby 2013-09-21 08:18:05 UTC
In fact, creating a new sql table for facets is maybe a more logical solution than using YAML.

M. Saby
Comment 10 Frédéric Demians 2013-09-21 08:30:14 UTC
> In fact, creating a new sql table for facets is maybe a more logical
> solution than using YAML.

Why creating a new lilliputian table, just to store a data structure,
hierarchical, which map cleanly into a serialization format like YAML,
JSON, or whatever? The question, concerning the facet data structure is
more:

- where to store it? file, a syspref (locked), a table
- how to create a UI to edit it?
- how to use it in order to modify search faceting? and so automatically
  modifying Zebra configuration
Comment 11 Mathieu Saby 2013-09-21 08:38:55 UTC
> - automatically modifying Zebra configuration

In your dreams ;-)
Out of the scope of this bug, but that would be a terrific idea!

If you like the facet idea, could you take a look to bug 10892 and say me if this first step is correct?

Mathieu
Comment 12 Mathieu Saby 2013-09-21 08:48:25 UTC
(By the way, for making zebra configurable from staff interface, the more simple would be everyone use DOM indexing. Less work to do...
But as you know, it is not yet possible for UNIMARC. I have made some patches which were signed off last week. I hope they will soon pass QA and be pushed, but some work remains to do for indexing 1XX coded fields)

Mathieu
Comment 13 Mathieu Saby 2013-09-25 21:24:52 UTC
After discussion on koha-devel I'm trying to put the configuration in JSON string, stored in a new syspref, which will be editable only through a new page in staff admin.

The JSON will look like that :

[
  {
    "code": "branches",
    "label": "Libraries",
    "index": {"UNIMARC":"branch","MARC21":"branch"},
    "mandatory": 1,
    "INTRAorder": 1,
    "OPACorder": 1,
    "mapping": {"UNIMARC":["995b"],"MARC21":["952a"]},
    "expandafter": null,
    "sort": null
  },
  {
    "code": "location",
    "label": "Libraries",
    "index": {"UNIMARC":"location","MARC21":"location"},
    "mandatory": 1,
    "INTRAorder": 2,
    "OPACorder": 2,
    "mapping": {"UNIMARC":["995e"],"MARC21":["952c"]},
    "expandafter": null,
    "sort": null
  },
  {
    "code": "language",
    "label": "Language of document",
    "index": {"UNIMARC":"ln","MARC21":"ln"},
    "mandatory": 1,
    "INTRAorder": 3,
    "OPACorder": 3,
    "mapping": {"UNIMARC":["101a"],"MARC21":["041a"]},
    "expandafter": null,
    "sort": null
  },
  {
    "code": "languageoforiginal",
    "label": "Language of original",
    "index": {"UNIMARC":"ln","MARC21":"ln"},
    "mandatory": 1,
    "INTRAorder": 4,
    "OPACorder": 4,
    "mapping": {"UNIMARC":["101c"],"MARC21":["041h"]},
    "expandafter": null,
    "sort": null
  }, 
  {
    "code": "typeofmaterial",
    "label": "Type of material",
    "index": {"UNIMARC":"Material-type"},
    "mandatory": 1,
    "INTRAorder": 5,
    "OPACorder": 5,
    "mapping": {"UNIMARC":["200b"]},
    "expandafter": null,
    "sort": null
  }
]

- code is a unique code for each facet
- label is the name displayed in english templates (of course, templates will ne to be edited to take into account the new facets)
- index is an object with the name of zebra index in unimarc and marc21 conf files. If only unimarc index is stored, the facet will be editable only if marcflavor = unimarc. The same for Marc21
Most often, it is the same index, but not in every cases
- mandatory = 1 means the facet is a standard Koha facet defined in Koha source code, not a completely user defined facet; mandatory facets can be display or not, and configured. but they can't be suppressed from the syspref.
For the moment, the syspref will only contains mandatory syspref. But it will be easy to allow librarians to define their own facets
- INTRAOrder and OPACOrder : order of display of this facet in intranet and OPAC. If 0 or null, the facet is not displayed. The order can be different on OPAC and intranet
- mapping : the fields and subfields in UNIMARC and MARC21. Syntax : 606abx. digits of label could be replaced by . : 6.. match 600-699
- expandafter : This field will give the abiliy to override the limit of 6 values displayed by default. If null, only show the 6 first values in each sysypref, and collapse the rest. Otherwise, show the XXX first values
- sort : This field will give the abiliy to override the default sorting (alphabetic). If null, use the defaut sorting. Otherwise, define a custom sorting according to some codes (not yet defined. for ex : alpha_desc, alpha_asc, occurences_desc, occurences_asc)


M. Saby
Comment 14 David Cook 2013-09-25 23:55:37 UTC
(In reply to mathieu saby from comment #13)
> After discussion on koha-devel I'm trying to put the configuration in JSON
> string, stored in a new syspref, which will be editable only through a new
> page in staff admin.
> 
> The JSON will look like that :
> 
> [
>   {
>     "code": "branches",
>     "label": "Libraries",
>     "index": {"UNIMARC":"branch","MARC21":"branch"},
>     "mandatory": 1,
>     "INTRAorder": 1,
>     "OPACorder": 1,
>     "mapping": {"UNIMARC":["995b"],"MARC21":["952a"]},
>     "expandafter": null,
>     "sort": null
>   },
>   {
>     "code": "location",
>     "label": "Libraries",
>     "index": {"UNIMARC":"location","MARC21":"location"},
>     "mandatory": 1,
>     "INTRAorder": 2,
>     "OPACorder": 2,
>     "mapping": {"UNIMARC":["995e"],"MARC21":["952c"]},
>     "expandafter": null,
>     "sort": null
>   },
>   {
>     "code": "language",
>     "label": "Language of document",
>     "index": {"UNIMARC":"ln","MARC21":"ln"},
>     "mandatory": 1,
>     "INTRAorder": 3,
>     "OPACorder": 3,
>     "mapping": {"UNIMARC":["101a"],"MARC21":["041a"]},
>     "expandafter": null,
>     "sort": null
>   },
>   {
>     "code": "languageoforiginal",
>     "label": "Language of original",
>     "index": {"UNIMARC":"ln","MARC21":"ln"},
>     "mandatory": 1,
>     "INTRAorder": 4,
>     "OPACorder": 4,
>     "mapping": {"UNIMARC":["101c"],"MARC21":["041h"]},
>     "expandafter": null,
>     "sort": null
>   }, 
>   {
>     "code": "typeofmaterial",
>     "label": "Type of material",
>     "index": {"UNIMARC":"Material-type"},
>     "mandatory": 1,
>     "INTRAorder": 5,
>     "OPACorder": 5,
>     "mapping": {"UNIMARC":["200b"]},
>     "expandafter": null,
>     "sort": null
>   }
> ]
> 
> - code is a unique code for each facet
> - label is the name displayed in english templates (of course, templates
> will ne to be edited to take into account the new facets)
> - index is an object with the name of zebra index in unimarc and marc21 conf
> files. If only unimarc index is stored, the facet will be editable only if
> marcflavor = unimarc. The same for Marc21
> Most often, it is the same index, but not in every cases
> - mandatory = 1 means the facet is a standard Koha facet defined in Koha
> source code, not a completely user defined facet; mandatory facets can be
> display or not, and configured. but they can't be suppressed from the
> syspref.
> For the moment, the syspref will only contains mandatory syspref. But it
> will be easy to allow librarians to define their own facets
> - INTRAOrder and OPACOrder : order of display of this facet in intranet and
> OPAC. If 0 or null, the facet is not displayed. The order can be different
> on OPAC and intranet
> - mapping : the fields and subfields in UNIMARC and MARC21. Syntax : 606abx.
> digits of label could be replaced by . : 6.. match 600-699
> - expandafter : This field will give the abiliy to override the limit of 6
> values displayed by default. If null, only show the 6 first values in each
> sysypref, and collapse the rest. Otherwise, show the XXX first values
> - sort : This field will give the abiliy to override the default sorting
> (alphabetic). If null, use the defaut sorting. Otherwise, define a custom
> sorting according to some codes (not yet defined. for ex : alpha_desc,
> alpha_asc, occurences_desc, occurences_asc)
> 
> 
> M. Saby

Looking good, Mathieu.

However, how might control fields or the leader factor into the mapping object? 

Admittedly, I think that your patch is for the present rather than the future, so that's just a question and not a criticism :).
Comment 15 Mathieu Saby 2013-09-26 07:51:02 UTC
Currently, coded fields are not supported.
So they probably won't be in the first version of the patch.
The perl code that uses "mappings" array must be improved later for that. And we could add something like mapping:["UNIMARC":["101a0","101a2","101a3-4"] for map positions 0, 2, 3-4 of 101a.

I add an other idea : give the choice to "group" some facets.
For that I can add a value "hierarchicallabel" in each facets
For ex, for facet "Author (people)":

"label": "Author (people)"
"hierarchiclabel": ["Author","(people)"]

And add a syspref for choosing which kind of label is to be used.
So, with hierarchiclabel, you could have facets like that on OPAC or intranet:

*Location*
- Branches
...
- Shelving location
...
*Authors
- People
...
- Organizations
...


Would you like it?

Mathieu
Comment 16 Katrin Fischer 2013-09-26 09:19:18 UTC
Just a reminder that controlfields in MARC21 have no subfields :)
101a3-4 can work for UNIMARC, but the syntax is maybe not ideal for MARC21. I think that can be solved later tho.
Comment 17 Mathieu Saby 2013-09-26 09:23:04 UTC
Oh yes, I know...
That's why DOM indexing was broken with Unimarc. See 
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8252

Mathieu
Comment 18 Mathieu Saby 2013-09-26 21:54:51 UTC
Created attachment 21505 [details]
config page design

I've almost finished the config page. But for the moment it is parsing an harcoded json string, not yet in a syspref. And these new facets are not used by opac and staff search.
Here is how it will look.

Mathieu
Comment 19 Katrin Fischer 2013-09-26 22:14:28 UTC
Comment on attachment 21505 [details]
config page design

It looks quite nice!
Comment 20 Frédéric Demians 2013-09-27 06:44:50 UTC
Sweet!

In a brave new worlds, it would be great to split facets definition and
facet layout in OPAC/intra. So:

  - Section 1: a table with your current columns, except both columns
    Order
  - Section 2: a ordered list of OPAC facets
  - Section 3: a ordered list of intra facets

This way you could avoid showing the facet display order, which isn't
that much user-friendly. You could use jQuery UI 'sortable' widget. For
example: 

  http://jqueryui.com/sortable/#empty-lists
Comment 21 Katrin Fischer 2013-09-27 06:49:05 UTC
I think Jared has done a drag and drop for the sorting of the search result plugins. Maybe this could be helpful if you want to do the drag and drop. But it might be an issue with accessibility.
Comment 22 Mathieu Saby 2013-09-27 12:01:01 UTC
(In reply to Frédéric Demians from comment #20)
> Sweet!
> 
> In a brave new worlds, it would be great to split facets definition and
> facet layout in OPAC/intra. So:
> 
>   - Section 1: a table with your current columns, except both columns
>     Order
>   - Section 2: a ordered list of OPAC facets
>   - Section 3: a ordered list of intra facets

Good idea, in 3 tabs.

> 
> This way you could avoid showing the facet display order, which isn't
> that much user-friendly. You could use jQuery UI 'sortable' widget. For
> example: 
> 
>   http://jqueryui.com/sortable/#empty-lists


I'll look at it. What I need is something to re-sort the table in real time when sbdy click on the link for changing the order.

Mathieu
Comment 23 Mathieu Saby 2013-11-11 12:00:55 UTC
I will wait to see some patch attached to bug 11232. It could make a big conflict with this one.

Mathieu
Comment 24 Mathieu Saby 2013-11-23 12:57:55 UTC
I won't have time to work on that, and anyway I suppose it is better to wait for the gobal refactoring of facets.

Mathieu
Comment 25 Mark Tompsett 2013-12-03 00:16:05 UTC
(In reply to mathieu saby from comment #2)
> Hi Frederic
> I was thinking of a syspref first. It would be easier for me to do (2
> sysprefs in fact, one for unimarc, the other for marc21/normarc)

A Koha system can't be both unimarc and marc21/normarc at the same time. Why would you need two?
Comment 26 Mathieu Saby 2013-12-03 08:13:31 UTC
(In reply to M. Tompsett from comment #25)
> (In reply to mathieu saby from comment #2)
> > Hi Frederic
> > I was thinking of a syspref first. It would be easier for me to do (2
> > sysprefs in fact, one for unimarc, the other for marc21/normarc)
> 
> A Koha system can't be both unimarc and marc21/normarc at the same time. Why
> would you need two?

Some facets can have sense in a unimarc system, other in a marc21 system (some coded fields only exists in unimarc, some new rda fields only exists in marc21 etc...)
But you are right, I suppose we could have unique syspref with different content, depending on marcflavor choosen when you install Koha.

Anyway, with the global rewriting of facets, I don't know if the idea of storing facets in a syspref will still be interesting.

Mathieu
Comment 27 Vitor Fernandes 2014-08-04 11:11:16 UTC
*** Bug 12712 has been marked as a duplicate of this bug. ***
Comment 28 Mark Tompsett 2014-08-04 11:27:55 UTC
Has progress on this been made?
Comment 29 Mathieu Saby 2014-08-05 15:57:25 UTC
Hi
I stopped working on that when bug 11232 was created, because I feared a big conflict. But nothing is still to test on that bug :(
We could also wait to see the work on Elasticsearch.
Anyway, I'm not working actively on Koha anymore, so I won't propose a patch for bug 10891, but we can keep the bug open, because I still think it would be a nice enhancement.

Mathieu
Comment 30 Tomás Cohen Arazi 2014-12-02 22:45:19 UTC
(In reply to mathieu saby from comment #29)
> Hi
> I stopped working on that when bug 11232 was created, because I feared a big
> conflict. But nothing is still to test on that bug :(
> We could also wait to see the work on Elasticsearch.
> Anyway, I'm not working actively on Koha anymore, so I won't propose a patch
> for bug 10891, but we can keep the bug open, because I still think it would
> be a nice enhancement.
> 
> Mathieu

Mathieu, don't you want to come back? :-D
Comment 31 Mathieu Saby 2014-12-04 20:10:24 UTC
(In reply to Tomás Cohen Arazi from comment #30)
> (In reply to mathieu saby from comment #29)
> > Hi
> > I stopped working on that when bug 11232 was created, because I feared a big
> > conflict. But nothing is still to test on that bug :(
> > We could also wait to see the work on Elasticsearch.
> > Anyway, I'm not working actively on Koha anymore, so I won't propose a patch
> > for bug 10891, but we can keep the bug open, because I still think it would
> > be a nice enhancement.
> > 
> > Mathieu
> 
> Mathieu, don't you want to come back? :-D

Sorry, i won't have time for that. But please keep the bug open if somebody wants to add a patch ;-) I still think it would be a nice improvement

Mathieu
Comment 32 Katrin Fischer 2016-01-18 22:35:27 UTC

*** This bug has been marked as a duplicate of bug 15544 ***