Hi
I did not look yet, and I will, as it could be usefull, but I have considering a more important more work for management of document languages in Koha for years. The current way of dealing with it is just horrible! 3 SQL tables, no way to update them easily, no way to translate them automaticaly...

It's out of the scope of your bug, but if you are interested in language management, you will probably have an opinion. Could you tell me what you think of that?

A enh could be : 
- store somewhere (a table? a yaml file?) the complete list of iso 639-2 codes, used in UNIMARC, with the "native name" of this language
- store somewhere the complete list of MARC21 codes used in MARC21 (mostly the same as iso 639-2, but not always), with the "native name" of this language
- include in the 2 lists "obsolete" codes for the 2 lists, because when a code is marked as obsolete by the agency responsible for a list (ISO or LoC), it does not mean automaticaly that the records using it are updated in every library...
Maybe store the 2 lists (iso 639 and marc codes) in the same table or the same yaml file, using for a key the iso 639-2(or iso 639-3 if needed) code (we must check if there are some languages existing in one list and not in the other...)
These 2 lists won't be editable by users

- map each of these code with the english name of the list, and store the resulting file (xml?) somewhere in koha templates where it could be translated
- make some PO files for translating this list

The PO files could be make automaticaly (need some work...) from data freely available on Lexvo.org website (GLP licence).

effects for users on OPAC : 
- in language search dropdown list, show for "name depending on OPAC locale language - native name". Sort the list alphabeticaly according to the name on OPAC locale language. Ex : if OPAC is in french : "allemand - deutsch" ; in english : "german - deutshc"
- when OPAC is switch in some language, the name of the language used in the record would be translated. Ex : for a UNIMARC record with 101$a = "ger", display "german" if OPAC is in english, "allemand" if it in french...
- we could also build facets on document language.

administration on staff interface:
- show the complete list of avalaible language on a page
- let the staff check a box before the name of the languages they want to be in the dropdown list on avanced search page.

M. Saby

Patch applied cleanly, go forth and signoff

I agree that the current way seems complicated, but it does more than just control the list on the advanced search page. It also controls the BIDI display, by telling the templates when a language is LTR or RTL. We have to see where else it is used before changing anything here.

Oh, the tables also control the hierarchies for the language chooser and multible languages en-nz, en-US, etc.

(In reply to Katrin Fischer from comment #4)
> I agree that the current way seems complicated, but it does more than just
> control the list on the advanced search page. It also controls the BIDI
> display, by telling the templates when a language is LTR or RTL. We have to
> see where else it is used before changing anything here.

That's not a problem : direction of display could be retreived automaticaly.
It depends on the kind of "script" used by the language
http://en.wikipedia.org/wiki/Right-to-left
And you can get the type of script of each language on several pages.

For ex : http://www.lexvo.org/page/iso639-3/ara use arabic (and syriac) script (lvont:usesScript lexvo:script/Arab), whitch is written RTL.

But of course, this would be a big-work-enh.
So, the patch of Blou deserves attention ;-)

Mathieu

(In reply to Katrin Fischer from comment #5)
> Oh, the tables also control the hierarchies for the language chooser and
> multible languages en-nz, en-US, etc.

I thinks that's not good.
The display of OPAC should be deconnected from the language of the records. That's a completely different point.
You can have only iso 639-2 codes in records, while you can have more precise codes in PO (fr-fr, fr-ca etc)

Mathieu

I think that's why we have 3 tables :)

(In reply to Katrin Fischer from comment #8)
> I think that's why we have 3 tables :)

Mmmmm I understand now!

Not saying we should keep them - just that rewriting will need some thought put into it. And I think this patch would be a nice enh.

(In reply to Katrin Fischer from comment #10)
> Not saying we should keep them - just that rewriting will need some thought
> put into it. And I think this patch would be a nice enh.

Well, am I glad I made sure to not impact anything else.  I was first wondering if "just removing the language entries from a table would do it for the libraries", but then I saw the JOIN on the tables involved and decided not to touch it.  

If the language features have reached maturity, then it might be a good time to start thinking about refactoring it safely, now that all the pieces can be considered.  But the solution doesn't necessarily involve touching the tables if it is all nicely hidden behind a single and clear API.  And as they say : "if ain't broke, don't mess it" :)

Btw, only the installer and the advanced search used the getAllLanguages function.

Thanks for the discussion, that's very good source of information to me.

Blou

(In reply to Blou from comment #11)
> (In reply to Katrin Fischer from comment #10)
> > Not saying we should keep them - just that rewriting will need some thought
> > put into it. And I think this patch would be a nice enh.
> 
> Well, am I glad I made sure to not impact anything else.  I was first
> wondering if "just removing the language entries from a table would do it
> for the libraries", but then I saw the JOIN on the tables involved and
> decided not to touch it.  
> 
> If the language features have reached maturity, then it might be a good time
> to start thinking about refactoring it safely, now that all the pieces can
> be considered.  But the solution doesn't necessarily involve touching the
> tables if it is all nicely hidden behind a single and clear API.  And as
> they say : "if ain't broke, don't mess it" :)


Maybe we could fill the tables with all the 639-2 codes, and use your syspref not for overriding the table, but for picking up in it the codes we want to display in the dropdown list (and the order of the codes in the syspref would alter the order of the list)?
If so, the syspref could have a defaut value the list of the codes currently displayed.
That assume that all records are using iso 639-2, while technically it is not the case for MARC21 records. But Koha is currently not making distinction between iso 639-2 and MARC21 codes, so it could be a future enh.

Mathieu

I forgot to mention that in France we use an authorized value "LANG" with all the iso 639-2 codes for "translating" the UNIMARC 101a field.
This is not really good, because if we swich our OPAC in english or spanish, the codes are still translated in the french name of the language.

I don't know how it is done in MARC21. But for me one of the goal of a future enh could make the use of this authorized value obsolete.

Mathieu

Does it obsolete your first patch? If so, you can change the status of the 1st patch (Details > Edit details > obsolete)

M. Saby

Comment on attachment 21764 [details] [review]
Bugs 10986 - Limit the list of languages in advanced search

This new patch is exactly the same as the previous one.  It should not have been attached, I apologize for the confusion.  I was trying to get my git-bz to work.

Thanks. What do you think of my comment 12?

Mathieu

(In reply to mathieu saby from comment #17)
> Thanks. What do you think of my comment 12?
> 
> Mathieu

Technically, it does that.  The tableS contain all the language information, and the preference determines which one is used by the drop down.  The default is EMPTY.  I find that cleaner and easier to maintain long term.  If nothing is in the preference, the whole list is returned.

I must point out I don't know the difference between Marc21 and iso codes.  I detailed it as using iso 639-2 just because some variable in the code was refering to it.  The fact is, it is using whichever code we are currently building our "language hash" with.

(In reply to Blou from comment #18)
> (In reply to mathieu saby from comment #17)
> > Thanks. What do you think of my comment 12?
> > 
> > Mathieu
> 
> Technically, it does that.  The tableS contain all the language information,
> and the preference determines which one is used by the drop down.  The
> default is EMPTY.  I find that cleaner and easier to maintain long term.  If
> nothing is in the preference, the whole list is returned.
> 
> I must point out I don't know the difference between Marc21 and iso codes. 
> I detailed it as using iso 639-2 just because some variable in the code was
> refering to it.  The fact is, it is using whichever code we are currently
> building our "language hash" with.


I am not an expert, but the differences seems to be slight, and mainly between the names of the languages, not the codes themselves.
http://www.loc.gov/marc/languages/introduction.pdf

"ISO 639-2 (Codes for the representation of names of languages-- Part 2: alpha-3 code) was based on the MARC Code List for Languages and published in 1998. In the 22 cases where the ISO 639-2 list has two alternative codes, the bibliographic code is the same as the MARC code. Language names in ISO 639-2
are not necessarily the same as those in MARC, particularly because of the practice of correlating the MARC language names with those used in
Library of Congress Subject Headings. The MARC list includes references
for unused forms of language names, while the ISO list has in some cases included alternative name forms,but many are lacking, since this practice of supplying alternate forms has only recently been implemented.
In addition the MARC documentation includes a list of individual languages under collective codes or language groups, while the ISO list only includes the group codes themselves.The Library of Congress is maintenance agency for both lists, and the two are kept compatible in terms of code additions and deletion"


M. Saby

(In reply to mathieu saby from comment #19)
> (In reply to Blou from comment #18)
> > (In reply to mathieu saby from comment #17)
> > > Thanks. What do you think of my comment 12?
> > > 
> > > Mathieu
> > 
> > Technically, it does that.  The tableS contain all the language information,
> > and the preference determines which one is used by the drop down.  The
> > default is EMPTY.  I find that cleaner and easier to maintain long term.  If
> > nothing is in the preference, the whole list is returned.
> > 
OK for me.
But I as soon as your patch is pushed, I think I or other french ppl will add to the SQL the iso codes missing for french libraries (french regional languages, old french, middle french, etc.). And other people will do the same for other codes I suppose. So the best would be to add all iso codes in one operation to avoid mistakes. But this could be done after your patch is pushed.
So I think every library will soon need to use your syspref if they don't want to have 400 codes in the dropdown menu ;-) But that's not an issue if the syspref is working well.
Just a question : if the syspref is ita|fre or fre|ita, does it change the order of display of the languages in the dropdown menu?

Mathieu

> Just a question : if the syspref is ita|fre or fre|ita, does it change the
> order of display of the languages in the dropdown menu?

From what I remember, the syspref is just a filter on the languages list, it doesn't affect the order

As you said, the order is not altered.
It works well with "," or  "|" separator.
I tried to put a lot of codes in the syspref to see if there is a limit. I hope not, but I don't think so.
I sign off.

I my opinion further enh could be :
- add the list of all iso codes in database
- make the order dependant of the order of codes in syspref, or of the alphabetcial order of languages in the langugage of OPAC (currenty the sorting order is based on the english name)
- make 2 sysprefs (one for OPAC and one for Staff)
- display the "native" name after the translated one (ex : "italien (italiano)"). It would improve UX, because the main public of a french or english library is not italian speaking people.

Mathieu

QA comment:
The patch works as expected. The syspref makes sense for me.
Some QA issues, I will provide a QA followup.
Marked as Passed QA.

Created attachment 22774 [details] [review]
Bugs 10986 - Limit the list of languages in advanced search

In Advanced Search, the list of available language is long and will only get longer.  For a library offering books in 2-3 languages, that is offering too much info to the user (most of the small libraries we deal with only offer documents in two languages).

Code changes:
Languages.pm: Extract getAllLanguages to make a more customizable getLanguages (have getAllLanguage call it, so rest of codebase is oblivious to the change).  Build array returned based on system pref if corresponding argument is set.
search.pl and opac-search.pl: call getLanguages instead of getAllLanguages.

TESTING
0) All language codes are iso 639-2 (three characters)
1) in OPAC, Advanced search, open Language box, acknowledge 30+ items.
2) in Intranet, go to system preferences AdvancedSearchLanguages, enter "ita|eng"
3) back in OPAC, refresh screen, acknowledge only Italian and English are listed.
4) in Intranet, click Search then click "More options" to make the Language box appear.  Acknowledge limited options.
5) Regression Test: Back to the preference, empty the field then save.  Go back to the OPAC and Intranet search, refresh the page, then the Language drop-box will now contain 30+ items.

Signed-off-by: Mathieu Saby <mathieu.saby@univ-rennes2.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Created attachment 22777 [details] [review]
Bug 10986: QA followup

This patch fixes some qa issues:

 FAIL   C4/Languages.pm
   FAIL   critic
        # Variables::ProhibitConditionalDeclarations: Got 1 violation(s).
 FAIL   installer/data/mysql/updatedatabase.pl
   FAIL   forbidden patterns
        forbidden pattern: merge marker (<<<<<<<) (line 7157)

 FAIL   installer/data/mysql/sysprefs.sql
   FAIL   sysprefs_order
        Not blocker: Sysprefs AdvancedSearchLanguages is bad placed (see bug 10610)

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

(forgot to remove the coma at the end of the syspref.sql file)

Pushed to master, along with follow-ups, including ones adding unit tests.

Thanks, Phillippe!