Bug 30150 - Dutch option in language limits is garbled
Summary: Dutch option in language limits is garbled
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: 21.11
Hardware: All All
: P5 - low enhancement
Assignee: Owen Leonard
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-02-21 23:42 UTC by Eric Phetteplace
Modified: 2022-06-27 06:26 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
screenshot of bug (144.78 KB, image/png)
2022-02-21 23:42 UTC, Eric Phetteplace
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Eric Phetteplace 2022-02-21 23:42:11 UTC
Created attachment 130998 [details]
screenshot of bug

To recreate:

- add Dutch "dut" to the AdvancedSearchLanguages option
- go to the advanced search page /cgi-bin/koha/opac-search.pl
- click open the Language select menu 
- note that the text for Dutch is "nedərlɑns (Dutch)"

It should render as "nedərlɑns (Dutch)" but instead the numerical character references for the fourth and seventh characters are rendered literally, as if they were escaped.

I haven't looked into this further so I don't know _why_ this is happening but Koha's OPAC uses charset=utf-8 so I think we should be able to simply use the actual characters without any references. Perhaps it's a problem with the encoding of strings in the database? But when I run a report to pull all the rows from the `language_descriptions` table, the word "nedərlɑns" renders correctly.
Comment 1 Katrin Fischer 2022-02-26 18:54:54 UTC
Hi Eric, I see "Nederlands (Dutch)".

What was your query to the language_descriptions table?

For me it looks like this:

MariaDB [koha_kohadev]> select * from language_descriptions where subtag = "nl";
+--------+----------+------+-----------------+-----+
| subtag | type     | lang | description     | id  |
+--------+----------+------+-----------------+-----+
| nl     | language | de   | Niederländisch  | 148 |
| nl     | language | en   | Dutch           | 146 |
| nl     | language | fr   | Néerlandais     | 147 |
| nl     | language | nl   | Nederlands      | 145 |
| NL     | region   | en   | Netherlands     | 319 |
| NL     | region   | nl   | Nederland       | 320 |
+--------+----------+------+-----------------+-----+
Comment 2 Eric Phetteplace 2022-02-28 17:25:48 UTC
I just used a report `select * FROM language_descriptions` and found the Dutch entries further down. If I run your query I get this:

subtag	type	lang	description	id
nl	language	de	Niederländisch	138
nl	language	en	Dutch	26
nl	language	fr	Néerlandais	27
nl	language	nl	nedərlɑns	25
NL	region	en	Netherlands	299
NL	region	nl	Nederland	300

Should we change the "nedərlɑns" description in the database to "Nederlands" then? It displays right in the reports results, the only problem is on the advanced search page. I see the same results you've shown on koha-testing-docker so perhaps this was changed at some point?
Comment 3 Katrin Fischer 2022-02-28 18:30:18 UTC
(In reply to Eric Phetteplace from comment #2)
> I just used a report `select * FROM language_descriptions` and found the
> Dutch entries further down. If I run your query I get this:
> 
> subtag	type	lang	description	id
> nl	language	de	Niederländisch	138
> nl	language	en	Dutch	26
> nl	language	fr	Néerlandais	27
> nl	language	nl	nedərlɑns	25
> NL	region	en	Netherlands	299
> NL	region	nl	Nederland	300
> 
> Should we change the "nedərlɑns" description in the database to "Nederlands"
> then? It displays right in the reports results, the only problem is on the
> advanced search page. I see the same results you've shown on
> koha-testing-docker so perhaps this was changed at some point?

I am not sure if we can trust the report here... what you see here are nedərlɑns HTML-entities that would look correct in the browser as it renders them (unless they are escaped, as it probably happens on the advanced search. Could you try and download the CSV export to compare?

About the right naming I am not sure, but adding Marcel for maybe some help.
Comment 4 Eric Phetteplace 2022-02-28 18:49:57 UTC
You're correct, it's stored in the database with HTML character references, but renders as you would want on the reports page. This is the CSV export's content from our live Koha instance (21.05.09.000):

subtag,type,lang,description,id
nl,language,de,Niederländisch,138
nl,language,en,Dutch,26
nl,language,fr,Néerlandais,27
nl,language,nl,nedərlɑns,25
NL,region,en,Netherlands,299
NL,region,nl,Nederland,300

So the question is why the character references aren't interpreted in the <select> menu on the advanced search page. All I could think to check is the encoding, but I see that both pages have "<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />" so I'm stumped.
Comment 5 Katrin Fischer 2022-02-28 21:05:58 UTC
I believe the reason is that we are using the | html filter in the select - which is as it should be and escapes any code that might be there. The problem is in the database, as you said earlier we should just store the name in utf-8 and without the entities. Do you have access to your database to fix this with SQL?

We might still want to look into why it was not updated when it was for others - it seems ok on newer installations, but that could take a while.
Comment 6 Eric Phetteplace 2022-02-28 21:09:34 UTC
(In reply to Katrin Fischer from comment #5)
> I believe the reason is that we are using the | html filter in the select -
> which is as it should be and escapes any code that might be there. The
> problem is in the database, as you said earlier we should just store the
> name in utf-8 and without the entities. Do you have access to your database
> to fix this with SQL?
> 
> We might still want to look into why it was not updated when it was for
> others - it seems ok on newer installations, but that could take a while.

Our instance is hosted by ByWater but I'll reach out to them and link to this bug report.
Comment 7 Marcel de Rooy 2022-03-01 12:12:40 UTC
(In reply to Eric Phetteplace from comment #2)
> I just used a report `select * FROM language_descriptions` and found the
> Dutch entries further down. If I run your query I get this:
> 
> subtag	type	lang	description	id
> nl	language	de	Niederländisch	138
> nl	language	en	Dutch	26
> nl	language	fr	Néerlandais	27
> nl	language	nl	nedərlɑns	25
> NL	region	en	Netherlands	299
> NL	region	nl	Nederland	300
> 
> Should we change the "nedərlɑns" description in the database to "Nederlands"
> then? 

Yes, Nederlands is the correct spelling. Change it. The first description seems to be a fonetic one. But apart from replacing it, I would say that there is some encoding issue too.

I already have this entry:
| nl     | language | nl   | Nederlands      |  25 |
Comment 8 Katrin Fischer 2022-06-25 16:26:22 UTC
Should we create a just-in-case database update here in case there are more old installations with the wrong description?
Comment 9 Marcel de Rooy 2022-06-27 06:26:54 UTC
(In reply to Katrin Fischer from comment #8)
> Should we create a just-in-case database update here in case there are more
> old installations with the wrong description?

Good plan