Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles
Summary: Datatables in acqusitions do not ignore "stopwords" in titles
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Acquisitions (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Kyle M Hall
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-11-05 16:31 UTC by Kyle M Hall
Modified: 2016-12-05 21:25 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 11203 - Datatables on parcel.pl do not ignore "stopwords" in titles (1.74 KB, patch)
2013-11-05 16:32 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles (3.56 KB, patch)
2013-11-08 13:48 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles (3.68 KB, patch)
2016-03-04 11:09 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles (4.41 KB, patch)
2016-04-19 12:19 UTC, Kyle M Hall
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles (4.51 KB, patch)
2016-04-19 13:39 UTC, Owen Leonard
Details | Diff | Splinter Review
[PASSED QA] Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles (4.52 KB, patch)
2016-04-20 20:04 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Kyle M Hall 2013-11-05 16:31:09 UTC
Sorting by the "summary" column does not produce the preferred results. Title beginning with 'a', 'an', or 'the' are sorted using those articles.

Test plan:
1) Place an order for 2 items with the titles "Alpha" and "The Alpha", along with some other records with titles starting with something between 'a' and 't', and 't' and 'z'
2) Sort the "pending orders" table, note the incorrect sorting
3) Receive all the items
4) Sort the "already received" table, note the incorrect sorting
5) Apply the patch
6) Repeat steps 1-4, note the corrected sorting
Comment 1 Kyle M Hall 2013-11-05 16:32:00 UTC Comment hidden (obsolete)
Comment 2 Kyle M Hall 2013-11-08 13:48:00 UTC Comment hidden (obsolete)
Comment 3 Chris Cormack 2013-12-28 04:42:03 UTC
How does this work for other languages, die der das etc? Or is it english only?
Comment 4 Katrin Fischer 2013-12-28 07:55:12 UTC
Hm, there is http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=koha-tmpl/intranet-tmpl/prog/en/includes/datatables.inc;h=703d1e3e9637aad3923e2ab2fbe557c2b55869db;hb=e81513199cba41a254820347442719a6fd50ec0b which includes a line with the articles that is translatable. I am not sure if that is what gets used with 'anti-the', but it might. Even better than a translatable line would be making this a system preference, so you can change it anytime without having to update the po files.
Comment 5 David Cook 2014-01-06 23:57:22 UTC
(In reply to Katrin Fischer from comment #4)
> Hm, there is
> http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=koha-tmpl/intranet-
> tmpl/prog/en/includes/datatables.inc;
> h=703d1e3e9637aad3923e2ab2fbe557c2b55869db;
> hb=e81513199cba41a254820347442719a6fd50ec0b which includes a line with the
> articles that is translatable. I am not sure if that is what gets used with
> 'anti-the', but it might. Even better than a translatable line would be
> making this a system preference, so you can change it anytime without having
> to update the po files.

Yep, CONFIG_EXCLUDE_ARTICLES_FROM_SORT is the variable that provides the articles.

You can see this in "git\koha-tmpl\intranet-tmpl\prog\en\js\datatables.js"

It's a modified version of this Datatable plugin:
http://datatables.net/plug-ins/sorting#anti_the

A system preference might be an idea, although I wonder how that would look? It seems to me that you would need to express quite a few different key->value relationships. 

Plus, you would need to have translators providing the values for that system preference, at least initially. Would translators be up for that?
Comment 6 David Cook 2014-01-07 00:04:16 UTC
Applying: Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/acqui/parcel.tt
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/acqui/parcel.tt
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/acqui/basket.tt
Failed to merge in the changes.
Patch failed at 0001 Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles
Comment 7 Katrin Fischer 2014-01-07 05:58:35 UTC
Hm, not sure about "key-value-relationships" - wouldn't it be just a word list? You don't want to sort differently when switching templates, so I think the actual langauge of a word is not needed.
Comment 8 David Cook 2014-01-07 22:58:55 UTC
(In reply to Katrin Fischer from comment #7)
> Hm, not sure about "key-value-relationships" - wouldn't it be just a word
> list? You don't want to sort differently when switching templates, so I
> think the actual langauge of a word is not needed.

Mmm, good point. I'm trying to think of examples where an article in one language might be a non-article in another language...

'The' is an English article, but 'Thé' is a French noun that we wouldn't want to ignore. 

I suppose the regex might differentiate letters with diacritics...

'A' is the French preposition 'A'/'à' (often when capitalized, the article doesn't have a diacritic). 

Tricky.

Here is a more comprehensive list of articles in multiple languages...

http://library.princeton.edu/departments/tsd/katmandu/catcopy/article.html

--

That said, an English install might have records in multiple languages and you would probably want to sort all of them without articles.

I don't know if there is a way of offering a 100% consistent sort across languages though. Of course, using the articles of the selected language isn't very consistent either, so I'm tempted to say that the system preference is the best bet.

I suppose the system preference could make it easier to deal with problems. If you just have English, you could just use English articles. If you just have French, you could just use French, etc. You might even be able to use a few articles from a few languages.

I think French + English would have problems though (because of 'A' and maybe 'The').

German + English also looks like it would have problems. "Die" is a pretty common English noun. "Den" maybe less so, but still. 

Of course, this is all just from a list. I'd be interested to hear from more native speakers.

--

Other problematic words I see:

'as' => Portuguese/Gallegan||Galician
'bat' => Basque
'am' => Gaelic
'den' => Danish/German/Norwegian/Swedish
'die' => Afrikaans/German/Yiddish
'et' => Danish/Norwegian (maybe...)
'he' => Hawaiian
'hen' => Greek
'hi' => Icelandic
'i' => Italian
'in' => Friesian
'it' => Friesian
'nina' => Tagalog
'os' => Portugeuse (both for English OS and the French noun Os)
'to' => Greek (Need a native speaker for this one. I thought 'to' was the Greek pronoun for the English 'it')
'ton' => Greek
Comment 9 David Cook 2014-01-07 23:06:13 UTC
'Des' in German is an article, but that's a partitive article in French ('some'/'any' in English) that we probably wouldn't want to ignore.

I suppose the ideal system would know the language of the string and ignore the requisite articles, but...in acquisitions we don't necessarily even have a record so it's tough to know the language.

Although...I seem to recall seeing a "locale" box somewhere in 3.15 recently...maybe that's something that we should use more of around Koha?

I suppose that has the potential to be quite useful or quite messy/difficult though...
Comment 10 Katrin Fischer 2014-01-08 08:50:35 UTC
I guess you are right, there are some overlaps. A list of words won't ever be perfect, but I think perfect is not really achievable here without going totally overboard :) I think a list of words in a pref would give more flexibility, allow the feature to be deactivated totally or adjusted to each libraries collection.
Comment 11 David Cook 2014-01-09 00:21:09 UTC
(In reply to Katrin Fischer from comment #10)
> I guess you are right, there are some overlaps. A list of words won't ever
> be perfect, but I think perfect is not really achievable here without going
> totally overboard :) I think a list of words in a pref would give more
> flexibility, allow the feature to be deactivated totally or adjusted to each
> libraries collection.

Agreed :)
Comment 12 Mark Tompsett 2015-04-04 21:17:35 UTC
Is this left as "Patch doesn't apply" because someone thinks anti-the isn't a good multilingual solution? It's easy enough to rescue if you know which columns need tweaking in parcel.tt, but I'm curious why the delay in saving this.
Comment 13 Jonathan Druart 2016-03-04 11:09:16 UTC Comment hidden (obsolete)
Comment 14 Owen Leonard 2016-03-17 13:53:19 UTC
I didn't get far into this before running into Bug 16089, but I did find that sorting on acqui/basket.pl wasn't correct. I'm curious why you use

{ "sType": "anti-the", "aTargets": [ 0 ] }

...instead of...

{ "sType": "anti-the", "aTargets": [ "anti-the" ] }


...with a class on the <th>: <th class="anti-the">

I find that much simpler to work with because you don't have to account for columns being conditionally included or not.
Comment 15 Kyle M Hall 2016-04-19 12:18:28 UTC
(In reply to Owen Leonard from comment #14)
> I didn't get far into this before running into Bug 16089, but I did find
> that sorting on acqui/basket.pl wasn't correct. I'm curious why you use
> 
> { "sType": "anti-the", "aTargets": [ 0 ] }
> 
> ...instead of...
> 
> { "sType": "anti-the", "aTargets": [ "anti-the" ] }
> 
> 
> ...with a class on the <th>: <th class="anti-the">
> 
> I find that much simpler to work with because you don't have to account for
> columns being conditionally included or not.

Only because the existant aTargets used indexes, but I think your suggestion is a better idea!
Comment 16 Kyle M Hall 2016-04-19 12:19:13 UTC Comment hidden (obsolete)
Comment 17 Owen Leonard 2016-04-19 13:39:27 UTC Comment hidden (obsolete)
Comment 18 Katrin Fischer 2016-04-20 20:04:36 UTC
Created attachment 50482 [details] [review]
[PASSED QA] Bug 11203 - Datatables in acqusitions do not ignore "stopwords" in titles

Sorting by the "summary" column does not produce the preferred results.
Title beginning with 'a', 'an', or 'the' are sorted using those
articles.

Test plan:
1) Place an order for 2 items with the titles "Alpha" and "The Alpha",
   along with some other records with titles starting with something
   between 'a' and 't', and 't' and 'z'
2) Sort the "pending orders" table, note the incorrect sorting
3) Receive all the items
4) Sort the "already received" table, note the incorrect sorting
5) Apply the patch
6) Repeat steps 1-4, note the corrected sorting
7) Check acqui/basket.pl for correct sorting
8) Check acqui/invoice.pl for correct sorting

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 19 Kyle M Hall 2016-04-29 11:51:42 UTC
Pushed to master for the March 2016 release!
Comment 20 Julian Maurice 2016-05-16 10:02:22 UTC
Patch pushed to 3.22.x, will be in 3.22.7