Bug 7445 - Clicking on a tag gives "Language ... does not exist"
Summary: Clicking on a tag gives "Language ... does not exist"
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: 3.6
Hardware: All All
: P5 - low normal (vote)
Assignee: Magnus Enger
QA Contact: Ian Walls
Depends on:
Reported: 2012-01-13 17:10 UTC by Magnus Enger
Modified: 2013-12-05 20:01 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:

Bug 7445 - Clicking on a tag gives "Language ... does not exist" (1.76 KB, patch)
2012-04-02 13:39 UTC, Magnus Enger
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 7445 - Clicking on a tag gives "Language ... does not exist" (1.84 KB, patch)
2012-04-04 21:05 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Enger 2012-01-13 17:10:20 UTC
To reproduce: 

- Install another language alongside English
- Make sure this language is chosen in the OPAC
- Click on a tag, either in a result list or on the "Tag cloud" page
- The message "Language 'nb-no' does not exist" is displayed (or some other language code)

When English is the chosen language, things work as expected. 

I have tested this on current master and 3.6.2, and the problem is the same in both places. 

It can be seen in master here: http://head.bibkat.no/cgi-bin/koha/opac-tags.pl (Remember to choose Norwegian as language) See also attached screenshot.
Comment 1 Magnus Enger 2012-01-13 17:12:16 UTC
Checking a little more...

This gives the error: 

This works: 
Comment 2 Magnus Enger 2012-01-17 17:40:40 UTC
The problem seems to stem from Lingua::Stem::Snowball and specifically the call to it on line 741 of C4/Search.pm - in the _build_stemmed_operand function: 

    my $stemmer = Lingua::Stem::Snowball->new( lang => $lang,
                                               encoding => "UTF-8" );

For Norwegian, $lang holds "nb-NO". Replacing that temporarily with a literal value of "no" solves the problem. 

The documentation (e.g. http://search.cpan.org/dist/Lingua-Stem-Snowball/lib/Lingua/Stem/Snowball.pm) for Lingua::Stem::Snowball->new says this about the lang parameter: 

"lang: An ISO code taken from the table of supported languages, above." 

The table referenced only lists two-letter language codes, specifically these: da, nl, en, fi, fr, de, hu, it, no, pt, ro, ru, es, sv, tr. 

The syspref QueryStemming influences the observed behviour. When the chosen language in the OPAC is Norwegian (nb-NO), this works regardless of what QueryStemming is set to: 


Clicking on a tag in the result list gives this URL: 


When QueryStemming = "Try" this fails with the "Language ... does not exist" message. 

When QueryStemming = "Don't try" a result list is shown, as expected. 

Still confused...
Comment 3 Magnus Enger 2012-01-17 20:34:56 UTC
In 3.4.6 it works for Swedish, regardless of what QueryStemming is set to, but not for Norwegian. There's a difference between 3.4.6 and current master:

In 3.4.6 $lang is taken from C4::Templates::getlanguagecookie: 

389 sub getlanguagecookie {
390     my ($query) = @_;
391     my $lang;
392     if ($query->cookie('KohaOpacLanguage')){
393         $lang = $query->cookie('KohaOpacLanguage') ;
394     }else{
395         $lang = $ENV{HTTP_ACCEPT_LANGUAGE};
397     }
398     $lang = substr($lang, 0, 2);
400     return $lang;
401 }

In current master it comes from C4::Templates::getlanguage: 

311 sub getlanguage {
312     my ($query, $interface) = @_;
314     # Select a language based on cookie, syspref available languages & browser
315     my $is_intranet = $interface eq 'intranet';
316     my @languages = split(",", C4::Context->preference(
317         $is_intranet ? 'language' : 'opaclanguages'));
319     my $lang;
321     # cookie
322     if ( $query->cookie('KohaOpacLanguage') ) {
323         $lang = $query->cookie('KohaOpacLanguage');
324         $lang =~ s/[^a-zA-Z_-]*//; # sanitize cookie
325     }
328     unless ($lang) {
329         my $http_accept_language = $ENV{ HTTP_ACCEPT_LANGUAGE };
330         $lang = accept_language( $http_accept_language,
331             getTranslatedLanguages($interface,'prog') );
332     }
334     # Ignore a lang not selected in sysprefs
335     $lang = undef  unless first { $_ eq $lang } @languages;
337     # Fall back to English if necessary
338     $lang = 'en' unless $lang;
340     return $lang;
341 }

In 3.4.6 "sv-SE" is reduced to "sv" on line 398 and this is a valid language for Lingua::Stem::Snowball. "nb-NO" is reduced to "nb", which is not valid in the eyes of Lingua::Stem::Snowball. C4::Templates::getlanguage returns the unreduced strings, neither of which is valid for Lingua::Stem::Snowball.
Comment 4 Albert Oller 2012-01-17 21:00:12 UTC
Hi Magnus,

we are working on similar problems.  One site has es-ES language pack installed, 6.02.  All of the English tags work, but when switching to Spanish, the tags won't work if there is only a single item returned.

Language 'es-es' does not exist 

At another site, with only English, none of the tags works:
Language 'en-us,en;q=0.5' does not exist

Turning off Stemming, does allow the tags to work.

Best regards,
Comment 5 Magnus Enger 2012-01-17 21:14:22 UTC
Looks to me like there are two problems here:

> Error:
> Language 'es-es' does not exist 
> At another site, with only English, none of the tags works:
> Error:
> Language 'en-us,en;q=0.5' does not exist

1. These "long" language codes need to be reduced to two-letter codes that Lingua::Stem::Snowball will accept. 

2. Why does the error from the stemmer only show up for opac-search.pl?tag=code&q=code and not for opac-search.pl?q=code ?
Comment 6 Fridolin Somers 2012-01-27 09:28:32 UTC
(In reply to comment #5)
> 1. These "long" language codes need to be reduced to two-letter codes that
> Lingua::Stem::Snowball will accept. 

I confirm Lingua::Stem::Snowball needs a 2-characters lang.

It can be simply corrected in Search.pm : 
my $stemmer = Lingua::Stem::Snowball->new( lang => lc(substr($lang, 0, 2)), encoding => "UTF-8" );

See : http://annocpan.org/~CREAMYG/Lingua-Stem-Snowball-0.952/lib/Lingua/Stem/Snowball.pm

(Sorry, I don't have time to create a patch).
Comment 7 Magnus Enger 2012-01-27 09:35:37 UTC
(In reply to comment #6)
> my $stemmer = Lingua::Stem::Snowball->new( lang => lc(substr($lang, 0, 2)),

If only it were that simple... ;-) 

For the two variants of Norwegian, this would result in
nb-NO -> nb
nn-NO -> nn
but neither of these are valid for Lingua::Stem::Snowball, which expects Norwegian to be coded as "no". 

I have done a quick and dirty hack for my installations which looks like this: 

 $lang = substr($lang, 0, 2);
 if ($lang eq 'nb' || $lang eq 'nn') {
     $lang = 'no';

but that is probably not the way to go...
Comment 8 Marc Véron 2012-02-21 15:13:53 UTC
The solution of Fridolyn Somers works on my server for German and French, and I expect it to work for Italian as well.

I think the most pragmatic solution would be to implement the code of Magnus Enger, because Norwegian seems to be the only exception for the list of languages supported by Snowball, see:

 Language   | ISO code 
 Danish     | da       
 Dutch      | nl       
 English    | en       
 Finnish    | fi       
 French     | fr       
 German     | de       
 Hungarian  | hu       
 Italian    | it       
 Norwegian  | no       
 Portuguese | pt       
 Romanian   | ro       
 Russian    | ru       
 Spanish    | es       
 Swedish    | sv       
 Turkish    | tr
Comment 9 Magnus Enger 2012-04-02 13:39:41 UTC Comment hidden (obsolete)
Comment 10 Katrin Fischer 2012-04-04 21:05:43 UTC
Created attachment 8888 [details] [review]
[SIGNED-OFF] Bug 7445 - Clicking on a tag gives "Language ... does not exist"

To test:
- Syspref QueryStemming = Try
- Install Norwegian bokmål:
  cd misc/translator/
  perl translate install nb-NO
- Go to Home › Administration › System Preferences > I18N/L10N
  and enable "Norsk bokmål(nb-NO)" for opaclanguages as well as
  setting opaclanguagesdisplay = Allow
- Make sure you have selected "Norsk bokmål" as the active language
  in the OPAC
- Find a record that has a tag (which does not contain any digits)
- Click on the tag and see that you get the error in the title of
  this bug
- Apply the patch
- Click on the tag again and the error should be gone

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Easy to test with a great test plan. Works nicely.
Comment 11 Ian Walls 2012-04-06 13:20:18 UTC
This patch just does a substring call to give Snowball what it needs, and adds handling for the only exception we currently know about.  If we start finding more exceptions, we may want to revisit our methodology, but for now, I think having 1 special case is okay.

Passed QA.
Comment 12 Paul Poulain 2012-04-06 15:50:12 UTC
QA comment: this code should, in theory, not be accepted because it's a workaround using something hardcoded. But Marc Veron, in comment 8 seems to prove that's really the only exception, and it's more a problem with Lingua::Stem than a problem in Koha itself, so OK to push

... patch pushed
Comment 13 Jared Camins-Esakov 2012-05-09 16:59:59 UTC
This bug will be included in the Koha 3.6.5 release.