Bug 23675 - UseACQFrameworkForBiblioRecords default framework is missing LDR breaking encoding
Summary: UseACQFrameworkForBiblioRecords default framework is missing LDR breaking enc...
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Acquisitions (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Jonathan Druart
QA Contact: Testopia
URL:
Keywords:
Depends on: 19289
Blocks:
  Show dependency treegraph
 
Reported: 2019-09-25 17:25 UTC by Caroline Cyr La Rose
Modified: 2021-03-18 23:49 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Text to go in the release notes:
Version(s) released in:
21.05.00,20.11.04,20.05.10


Attachments
Bug 23765 - After TranslateNotices is set to 'Don't allow', email settings still show multiple languages (1.92 KB, patch)
2019-10-08 15:22 UTC, Bin Wen
Details | Diff | Splinter Review
Bug 23675: Add leader and default value for ACQ (2.98 KB, patch)
2021-02-08 10:56 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 23675: Add leader and default value for ACQ (3.05 KB, patch)
2021-02-08 20:06 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 23675: Add leader and default value for ACQ (3.10 KB, patch)
2021-02-22 16:11 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Caroline Cyr La Rose 2019-09-25 17:25:24 UTC
When using UseACQFrameworkForBiblioRecords, there is no field 000 and that is the field that tells Koha which character encoding to use. So diacritics and other non-standard characters appear all mangled.

To reproduce :
1- Go to Administration > Global system preferences and activate UseACQFrameworkForBiblioRecords
2- Go to Acquisitions and add to basket from new (empty) record
3- Enter a title with diacritics, for example Pépé le petit chat
4- From the basket, click on the title

The diacritics in the title are not correct.

If you edit the record from there, and correct the diacritics, it should be okay since there is a field 000 and the 9th position is a.

If you modify the ACQ framework to add field 000 with the default value of "     nam a22     7a 4500" and redo the steps above it should work and the diacritics should be OK.

Furthermore, the plugin for filling out the leader does not work in the order form. I don't know if this is something we would want. But if we have to show the leader in the form, I think it would be best if we could change it using the plugin.
Comment 1 Katrin Fischer 2019-09-30 14:31:05 UTC
This is caused by the LDR missing from the default framwork for ACQ.

I think we should probably have the system add a sane default for records created through acq that are missing the LDR - it contains the encoding information and Koha is unhappy without it.
Comment 2 Bin Wen 2019-10-08 15:22:54 UTC Comment hidden (obsolete)
Comment 3 Bin Wen 2019-10-08 15:37:33 UTC
Sorry, wrong patch push.
Comment 4 Michael Kuhn 2021-02-04 14:51:30 UTC
I have tested the original case with 18.11.03, 19.11.05 and 20.11.00. I have used framework ACQ as downloadable at https://wiki.koha-community.org/wiki/MARC_frameworks

It does not help just adding MARC 000, or adding MARC 000 and setting a default value of "     nam a22     7a 4500".

In all cases special characters are mutilated, for example Unicode strings "äu" (octal 303 244 165) and "ög" (octal 303 266 147) become octal 165 314 203 and 147 314 262.
Comment 5 Katrin Fischer 2021-02-04 15:27:21 UTC
(In reply to Michael Kuhn from comment #4)
> I have tested the original case with 18.11.03, 19.11.05 and 20.11.00. I have
> used framework ACQ as downloadable at
> https://wiki.koha-community.org/wiki/MARC_frameworks
> 
> It does not help just adding MARC 000, or adding MARC 000 and setting a
> default value of "     nam a22     7a 4500".
> 
> In all cases special characters are mutilated, for example Unicode strings
> "äu" (octal 303 244 165) and "ög" (octal 303 266 147) become octal 165 314
> 203 and 147 314 262.

I just tsted in our 20.05 and with a framework with 000 set up with the default and 245 with umlauts. The record displays correctly after saving.
I have set 000 to mandatory and displaying in the editor. Does visibility make a difference?
Comment 6 Michael Kuhn 2021-02-04 16:11:03 UTC
All right! In framework ACQ (as downloadable from https://wiki.koha-community.org/wiki/MARC_frameworks ) I have now

* added MARC 000
* set that field as Mandatory
* added the default value "     nam a22     7a 4500"
* made it visible for OPAC, Intranet and Editor

Using these settings I was able to create an order with special characters in MARC 245$a that still show correctly when opened in the catalog. Thank you very much!
Comment 7 Katrin Fischer 2021-02-04 16:52:17 UTC
Hi Michael, glad it worked!

We should still find a way to make this work a bit nicer, but at least the feature can be used now :)
Comment 8 Jonathan Druart 2021-02-08 10:56:39 UTC
Created attachment 116492 [details] [review]
Bug 23675: Add leader and default value for ACQ

When UseACQFrameworkForBiblioRecords is set, the ACQ framework will be
use when ordering.
But the leader is not defined for this framework and then we will face
encoding issues (I haven't go far investigating this as the suggested
solution worked correctly).

The idea is to define a 000 and default to '     nam a22     7a 4500'
(which is what returns the marc21_leader.pl plugin) for ACQ.

Test plan:
0. Apply this patch and reset_all
1. Create a new basket and place a new order
2. Note that the leader is now displayed on top of the other ACQ
subfields
3. Fill in 000 with "❤️", "ö" or other UTF-8 characters you like
4. Save
5. Notice that 245$a in biblio_metadata is correctly filled with the
title you entered
Comment 9 Katrin Fischer 2021-02-08 20:06:16 UTC
Created attachment 116520 [details] [review]
Bug 23675: Add leader and default value for ACQ

When UseACQFrameworkForBiblioRecords is set, the ACQ framework will be
use when ordering.
But the leader is not defined for this framework and then we will face
encoding issues (I haven't go far investigating this as the suggested
solution worked correctly).

The idea is to define a 000 and default to '     nam a22     7a 4500'
(which is what returns the marc21_leader.pl plugin) for ACQ.

Test plan:
0. Apply this patch and reset_all
1. Create a new basket and place a new order
2. Note that the leader is now displayed on top of the other ACQ
subfields
3. Fill in 245 with "❤️", "ö" or other UTF-8 characters you like
4. Save
5. Notice that 245$a in biblio_metadata is correctly filled with the
title you entered

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 10 Katrin Fischer 2021-02-08 20:10:37 UTC
This is a good first step, but it's quite ugly and hard to edit in the form. We don't have the helper plugins there.

It would be nice if it still worked when hidden (the visibility settings don't appear to work at all in acq). Or if we could add a default one automatically if none is set in the form to avoid the problems with the encoding. 

This might be a generally useful fix as the encoding problem has appeared a few times in difference contexts.
Comment 11 Martin Renvoize 2021-02-22 16:11:20 UTC
Created attachment 117164 [details] [review]
Bug 23675: Add leader and default value for ACQ

When UseACQFrameworkForBiblioRecords is set, the ACQ framework will be
use when ordering.
But the leader is not defined for this framework and then we will face
encoding issues (I haven't go far investigating this as the suggested
solution worked correctly).

The idea is to define a 000 and default to '     nam a22     7a 4500'
(which is what returns the marc21_leader.pl plugin) for ACQ.

Test plan:
0. Apply this patch and reset_all
1. Create a new basket and place a new order
2. Note that the leader is now displayed on top of the other ACQ
subfields
3. Fill in 245 with "❤️", "ö" or other UTF-8 characters you like
4. Save
5. Notice that 245$a in biblio_metadata is correctly filled with the
title you entered

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 12 Martin Renvoize 2021-02-22 16:11:41 UTC
Works as expected and makes sense.

Passing QA
Comment 13 Jonathan Druart 2021-02-23 12:20:59 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 14 Fridolin Somers 2021-02-25 09:38:11 UTC
Pushed to 20.11.x for 20.11.04
Comment 15 Andrew Fuerste-Henry 2021-03-05 14:00:55 UTC
Pushed to 20.05.x for 20.05.10
Comment 16 Victor Grousset/tuxayo 2021-03-18 23:49:10 UTC
Not backported to oldoldstable (19.11.x). Feel free to ask if it's needed.