Bug 23675

Summary:	UseACQFrameworkForBiblioRecords default framework is missing LDR breaking encoding
Product:	Koha	Reporter:	Caroline Cyr La Rose <caroline.cyr-la-rose>
Component:	Acquisitions	Assignee:	Jonathan Druart <jonathan.druart>
Status:	CLOSED FIXED	QA Contact:	Testopia <testopia>
Severity:	normal
Priority:	P5 - low	CC:	andrew, bin.wen, fridolin.somers, jonathan.druart, martin.renvoize, mik, tomascohen, victor
Version:	Main
Hardware:	All
OS:	All
Change sponsored?:	---	Patch complexity:	Trivial patch
Documentation contact:		Documentation submission:
Text to go in the release notes:		Version(s) released in:	21.05.00,20.11.04,20.05.10
Circulation function:
Bug Depends on:	19289
Bug Blocks:
Attachments:	Bug 23765 - After TranslateNotices is set to 'Don't allow', email settings still show multiple languages Bug 23675: Add leader and default value for ACQ Bug 23675: Add leader and default value for ACQ Bug 23675: Add leader and default value for ACQ

Description Caroline Cyr La Rose 2019-09-25 17:25:24 UTC

When using UseACQFrameworkForBiblioRecords, there is no field 000 and that is the field that tells Koha which character encoding to use. So diacritics and other non-standard characters appear all mangled.

To reproduce :
1- Go to Administration > Global system preferences and activate UseACQFrameworkForBiblioRecords
2- Go to Acquisitions and add to basket from new (empty) record
3- Enter a title with diacritics, for example Pépé le petit chat
4- From the basket, click on the title

The diacritics in the title are not correct.

If you edit the record from there, and correct the diacritics, it should be okay since there is a field 000 and the 9th position is a.

If you modify the ACQ framework to add field 000 with the default value of "     nam a22     7a 4500" and redo the steps above it should work and the diacritics should be OK.

Furthermore, the plugin for filling out the leader does not work in the order form. I don't know if this is something we would want. But if we have to show the leader in the form, I think it would be best if we could change it using the plugin.

Comment 1 Katrin Fischer 2019-09-30 14:31:05 UTC

This is caused by the LDR missing from the default framwork for ACQ.

I think we should probably have the system add a sane default for records created through acq that are missing the LDR - it contains the encoding information and Koha is unhappy without it.

Comment 2 Bin Wen 2019-10-08 15:22:54 UTC Comment hidden (obsolete)

Created attachment 93884 [details] [review]
Bug 23765 - After TranslateNotices is set to 'Don't allow', email settings still show multiple languages

To test:
1. Enable multi-languages
2. Set the preference 'TranslateNotices' on  'Allow'
3. Go to: tools==>Notices & slips==>Edit, make sure it has multilingual email templates.
4. Set the preference TranslateNotices on 'Don't allow'.
5. Go to: tools==>Notices & slips==>Edit ,the template still shows all the languages.
6. Apply the patch.
7. Repeat the steps 4 and 5
8. Success. It only shows the default template when TranslateNotices is 'Dont allow'.

Comment 3 Bin Wen 2019-10-08 15:37:33 UTC

Sorry, wrong patch push.

Comment 4 Michael Kuhn 2021-02-04 14:51:30 UTC

I have tested the original case with 18.11.03, 19.11.05 and 20.11.00. I have used framework ACQ as downloadable at https://wiki.koha-community.org/wiki/MARC_frameworks

It does not help just adding MARC 000, or adding MARC 000 and setting a default value of "     nam a22     7a 4500".

In all cases special characters are mutilated, for example Unicode strings "äu" (octal 303 244 165) and "ög" (octal 303 266 147) become octal 165 314 203 and 147 314 262.

Comment 5 Katrin Fischer 2021-02-04 15:27:21 UTC

(In reply to Michael Kuhn from comment #4)
> I have tested the original case with 18.11.03, 19.11.05 and 20.11.00. I have
> used framework ACQ as downloadable at
> https://wiki.koha-community.org/wiki/MARC_frameworks
> 
> It does not help just adding MARC 000, or adding MARC 000 and setting a
> default value of "     nam a22     7a 4500".
> 
> In all cases special characters are mutilated, for example Unicode strings
> "äu" (octal 303 244 165) and "ög" (octal 303 266 147) become octal 165 314
> 203 and 147 314 262.

I just tsted in our 20.05 and with a framework with 000 set up with the default and 245 with umlauts. The record displays correctly after saving.
I have set 000 to mandatory and displaying in the editor. Does visibility make a difference?

Comment 6 Michael Kuhn 2021-02-04 16:11:03 UTC

All right! In framework ACQ (as downloadable from https://wiki.koha-community.org/wiki/MARC_frameworks ) I have now

* added MARC 000
* set that field as Mandatory
* added the default value "     nam a22     7a 4500"
* made it visible for OPAC, Intranet and Editor

Using these settings I was able to create an order with special characters in MARC 245$a that still show correctly when opened in the catalog. Thank you very much!

Comment 7 Katrin Fischer 2021-02-04 16:52:17 UTC

Hi Michael, glad it worked!

We should still find a way to make this work a bit nicer, but at least the feature can be used now :)

Comment 8 Jonathan Druart 2021-02-08 10:56:39 UTC

Created attachment 116492 [details] [review]
Bug 23675: Add leader and default value for ACQ

When UseACQFrameworkForBiblioRecords is set, the ACQ framework will be
use when ordering.
But the leader is not defined for this framework and then we will face
encoding issues (I haven't go far investigating this as the suggested
solution worked correctly).

The idea is to define a 000 and default to '     nam a22     7a 4500'
(which is what returns the marc21_leader.pl plugin) for ACQ.

Test plan:
0. Apply this patch and reset_all
1. Create a new basket and place a new order
2. Note that the leader is now displayed on top of the other ACQ
subfields
3. Fill in 000 with "❤️", "ö" or other UTF-8 characters you like
4. Save
5. Notice that 245$a in biblio_metadata is correctly filled with the
title you entered

Comment 9 Katrin Fischer 2021-02-08 20:06:16 UTC

Created attachment 116520 [details] [review]
Bug 23675: Add leader and default value for ACQ

When UseACQFrameworkForBiblioRecords is set, the ACQ framework will be
use when ordering.
But the leader is not defined for this framework and then we will face
encoding issues (I haven't go far investigating this as the suggested
solution worked correctly).

The idea is to define a 000 and default to '     nam a22     7a 4500'
(which is what returns the marc21_leader.pl plugin) for ACQ.

Test plan:
0. Apply this patch and reset_all
1. Create a new basket and place a new order
2. Note that the leader is now displayed on top of the other ACQ
subfields
3. Fill in 245 with "❤️", "ö" or other UTF-8 characters you like
4. Save
5. Notice that 245$a in biblio_metadata is correctly filled with the
title you entered

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Comment 10 Katrin Fischer 2021-02-08 20:10:37 UTC

This is a good first step, but it's quite ugly and hard to edit in the form. We don't have the helper plugins there.

It would be nice if it still worked when hidden (the visibility settings don't appear to work at all in acq). Or if we could add a default one automatically if none is set in the form to avoid the problems with the encoding. 

This might be a generally useful fix as the encoding problem has appeared a few times in difference contexts.

Comment 11 Martin Renvoize (ashimema) 2021-02-22 16:11:20 UTC

Created attachment 117164 [details] [review]
Bug 23675: Add leader and default value for ACQ

When UseACQFrameworkForBiblioRecords is set, the ACQ framework will be
use when ordering.
But the leader is not defined for this framework and then we will face
encoding issues (I haven't go far investigating this as the suggested
solution worked correctly).

The idea is to define a 000 and default to '     nam a22     7a 4500'
(which is what returns the marc21_leader.pl plugin) for ACQ.

Test plan:
0. Apply this patch and reset_all
1. Create a new basket and place a new order
2. Note that the leader is now displayed on top of the other ACQ
subfields
3. Fill in 245 with "❤️", "ö" or other UTF-8 characters you like
4. Save
5. Notice that 245$a in biblio_metadata is correctly filled with the
title you entered

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Comment 12 Martin Renvoize (ashimema) 2021-02-22 16:11:41 UTC

Works as expected and makes sense.

Passing QA

Comment 13 Jonathan Druart 2021-02-23 12:20:59 UTC

Pushed to master for 21.05, thanks to everybody involved!

Comment 14 Fridolin Somers 2021-02-25 09:38:11 UTC

Pushed to 20.11.x for 20.11.04

Comment 15 Andrew Fuerste-Henry 2021-03-05 14:00:55 UTC

Pushed to 20.05.x for 20.05.10

Comment 16 Victor Grousset/tuxayo 2021-03-18 23:49:10 UTC

Not backported to oldoldstable (19.11.x). Feel free to ask if it's needed.