Bug 30373

Summary: Rewrite UNIMARC installer data to YAML
Product: Koha Reporter: Bernardo Gonzalez Kriegel <bgkriegel>
Component: I18N/L10NAssignee: Bernardo Gonzalez Kriegel <bgkriegel>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: arthur.suzuki, f.demians, fridolin.somers, john, jonathan.druart, kyle, mathsabypro, sonia.bouis, thibault.keromnes
Version: Main   
Hardware: All   
OS: All   
URL: https://gitlab.com/bgkriegel/Koha/-/commits/bz30373
Change sponsored?: --- Patch complexity: Medium patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.05.00
Circulation function:
Bug Depends on:    
Bug Blocks: 32722, 34663, 37924, 27619, 30477, 36576    
Attachments: Rewrite unimarc_framework_DEFAULT to YAML
Bug 30373: Add UNIMARC FA (fast add) framework
Bug 30373: Rewrite authorities_normal_unimarc to YAML
Bug 30373: Enable translation of UNIMARC frameworks
Sample UNIMARC install po file with fr-FR translations
Bug 30373: Hide items.dateaccessioned from the editor
Bug 30373: Show 325$a
Bug 30373: Add TYPEDOC authorised_values_category
Bug 30373: Fix visibility, authorised values and descriptions
Bug 30373: Fix unimarc authorised values
Bug 30373: Fix xt/find-license-problems.t
Bug 30373: (followup) Country codes auth values
Bug 30373: (followup) Language codes auth values
Bug 30373: (followup) UNIMARC relator codes auth values
Bug 30373: (followup) Optional SUGGEST auth values
Bug 30373: (followup) Enable 'qualif' authorised_value
Bug 30373: (followup) Add BNF z3950 server

Description Bernardo Gonzalez Kriegel 2022-03-28 03:47:46 UTC
A complete rewrite (and update) of UNIMARC installation files from SQL to YAML is needed to allow the translation of its frameworks.

Both bibliographic and authorities frameworks have been updated up to 2019.

References:
* Bibliographic: https://www.ifla.org/g/unimarc-rg/unimarc-bibliographic-3rd-edition-with-updates/
* Authorities: https://www.ifla.org/g/unimarc-rg/unimarc-authorities-3rd-edition-with-updates/

Some patches are compressed.
Comment 1 Bernardo Gonzalez Kriegel 2022-03-28 05:06:21 UTC
Created attachment 132279 [details]
Rewrite unimarc_framework_DEFAULT to YAML

YAML version of unimarc_framework_DEFAULT + ACQ framework

To test:
0. Apply this patch, recreate database and restart_all
1. Follow web installer, selecting UNIMARC, finish
2. Default UNIMARC framework must load without problem
3. In staff interface check existence of Default and ACQ frameworks

ACQ is buit in the thame way as marc21, by copome fields/subfields.
Comment 2 Bernardo Gonzalez Kriegel 2022-03-28 05:08:06 UTC
Created attachment 132280 [details] [review]
Bug 30373: Add UNIMARC FA (fast add) framework

This patch creates optional folder and a sample
fast add framework

To test:
0. On top of first patch, repeat procedure to do an UNIMARC install
1. Verify existence of FA framework
Comment 3 Bernardo Gonzalez Kriegel 2022-03-28 05:10:22 UTC
Created attachment 132281 [details]
Bug 30373: Rewrite authorities_normal_unimarc to YAML

YAML version of authorities_normal_unimarc + sample auth types
Includes

This files works like it's marc21 counterpart:
* Defines an authority default framework
* Creates sample auth types by copy

Auth type codes and select fields/subfields are based on
french files in fr-FR install directory.

To test:
1. Apply this patch
2. Repeat UNIMARC install procedure as previous patches
3. Verify authorities_normal_unimarc.yml is loaded without problem
4. In the staff interface verify all auth types are present
Comment 4 Bernardo Gonzalez Kriegel 2022-03-28 05:11:18 UTC
Created attachment 132282 [details] [review]
Bug 30373: Enable translation of UNIMARC frameworks

This patch adds a new translation file, *-installer-UNIMARC.po

To test:
1. On top of all previous patches
2. Apply this patch
3. Create (or update) some language
Ej.
    misc/translation/translate create xx-YY

    verify new file misc/translation/po/xx-YY-installer-UNIMARC.po
4. Install new language
    misc/translation/translate install xx-YY

    verify new dirs:
        installer/data/mysql/xx-YY/marcflavour/unimarc/{mandatory,optional}

5. Repeat install procedure selecting xx-YY language and UNIMARC
   verify all frameworks are present
Comment 5 Bernardo Gonzalez Kriegel 2022-03-28 05:46:24 UTC
to the UNIMARC users:
This patch set may still need some work, but it is usable.
All frameworks have been updated carefully (up to 2019), but there may be bugs, that can be fixed.

For authorities, the fr-FR case was used as a basis, in particular for the types of authorities and the definition of which subfields are mandatory. But in this case, fr-FR, the number of mandatory fields differs a lot between the default and the different authority types. That may need to be reviewed.

I hope you can try it and check if it is useful.

As an alternative, I offer to base the files exclusively on the current fr-FR files, but with English legends and valid subfields, and leave the update for another opportunity.

In any case, it is very easy to generate an almost complete translation for fr-FR and it-IT, recovering in the latter case one of the languages for which a directory previously existed, and for which a bug is open (Bug 29538)
Comment 6 Jonathan Druart 2022-03-28 07:43:49 UTC
*** Bug 29538 has been marked as a duplicate of this bug. ***
Comment 7 Arthur Suzuki 2022-03-30 15:26:09 UTC
I applyied all patches on this repository : https://git.biblibre.com/biblibre/kohac/src/branch/bz30373

This way the bug can be installed and tested on a sandbox.
Best,
Arthur
Comment 8 Thibault Keromnès 2022-04-01 12:29:54 UTC
Tested with Arthur's git on a sandbox. 
First tests (comments #1, #2 and #3) worked well (we have Default, ACQ, FA and Authorities frameworks)

Test on comment #4 failed, we couldn't install a new language. 
"misc/translation/translate create xx-YY" didn't create "misc/translation/po/xx-YY-installer-UNIMARC.po"

On the default framework itself it looks really good, we just think it'd be better not to collapse the following fields : 
001 
010$d
020 
021  
106 
3XX 
4XX 
5XX 
8XX 
9XX
We imported all authorities types, it's great, thanks a lot.
Comment 9 Bernardo Gonzalez Kriegel 2022-04-01 20:19:38 UTC
(In reply to Thibault Keromnès from comment #8)
> Tested with Arthur's git on a sandbox. 
> First tests (comments #1, #2 and #3) worked well (we have Default, ACQ, FA
> and Authorities frameworks)
> 
> Test on comment #4 failed, we couldn't install a new language. 
> "misc/translation/translate create xx-YY" didn't create
> "misc/translation/po/xx-YY-installer-UNIMARC.po"

It's strange, new po file creation depends on the last patch, and that is present in the biblibre repo.
I just tried and I have not found problems.
I'll attach a sample xx-YY-installer-UNIMARC.po with french translations.
 
> On the default framework itself it looks really good, we just think it'd be
> better not to collapse the following fields : 
> 001 
> 010$d
> 020 
> 021  
> 106 
> 3XX 
> 4XX 
> 5XX 
> 8XX 
> 9XX
> We imported all authorities types, it's great, thanks a lot.

About visibility of fields/subfields, I'll gladly follow any advice, as in the case of mandatory values.
Comment 10 Bernardo Gonzalez Kriegel 2022-04-01 20:24:32 UTC
Created attachment 132874 [details]
Sample UNIMARC install po file with fr-FR translations

After creation of new language xx-YY but before installing it, replace xx-YY-installer-UNIMARC.po with this file, populated with strings from fr-FR sql files.
Comment 11 Jonathan Druart 2022-04-05 11:42:32 UTC
Hola Bernardo,

About the test plan of "Bug 30373: Enable translation of UNIMARC frameworks":

    3. Create (or update) some language
    Ej.
        misc/translation/translate create xx-YY

        verify new file misc/translation/po/xx-YY-installer-UNIMARC.po

The correct path is misc/translator/po/xx-YY-installer-UNIMARC.po

Yes it got created!

But:

    4. Install new language
        misc/translation/translate install xx-YY

        verify new dirs:
            installer/data/mysql/xx-YY/marcflavour/unimarc/{mandatory,optional}

The 'unimarc' directory is not created. Tried with "fr-FR" and "es-ES".
Comment 12 Jonathan Druart 2022-04-05 11:43:39 UTC
kohadev-koha@kohadevbox:/kohadevbox/koha/misc/translator$ perl translate create es-ES
kohadev-koha@kohadevbox:/kohadevbox/koha/misc/translator$ perl translate install es-ES

kohadev-koha@kohadevbox:/kohadevbox/koha/misc/translator$ ls /kohadevbox/koha/installer/data/mysql/es-ES/marcflavour/
marc21
Comment 13 Bernardo Gonzalez Kriegel 2022-04-05 14:56:07 UTC
> 
> The correct path is misc/translator/po/xx-YY-installer-UNIMARC.po
You're right, sorry

> But:
> 
>     4. Install new language
>         misc/translation/translate install xx-YY
> 
>         verify new dirs:
>            
> installer/data/mysql/xx-YY/marcflavour/unimarc/{mandatory,optional}
> 
> The 'unimarc' directory is not created. Tried with "fr-FR" and "es-ES".

My tests using biblibre branch:

$ git status
En la rama bz30373
Tu rama está actualizada con 'biblibre/bz30373'.

$ misc/translator/translate create xx-YY

$ ls -l misc/translator/po/xx-YY-installer*
-rw-rw-r-- 1 bgkriegel bgkriegel 288499 abr  5 07:10 misc/translator/po/xx-YY-installer-MARC21.po
-rw-rw-r-- 1 bgkriegel bgkriegel 107615 abr  5 07:10 misc/translator/po/xx-YY-installer.po
-rw-rw-r-- 1 bgkriegel bgkriegel 201055 abr  5 07:10 misc/translator/po/xx-YY-installer-UNIMARC.po

$ misc/translator/translate install xx-YY
ERROR parsing plural forms header [nplurals=INTEGER; plural=EXPRESSION;]

$ ls -l installer/data/mysql/xx-YY/marcflavour/
total 8
drwxrwxr-x 4 bgkriegel bgkriegel 4096 abr  5 11:37 marc21
drwxrwxr-x 4 bgkriegel bgkriegel 4096 abr  5 11:37 unimarc


NOTE: check that installer/data/mysql/xx-YY (or es-ES) does *not* exists before doing the install phase. By design that dir is not overwritten or updated.
Comment 14 Jonathan Druart 2022-04-06 13:16:11 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #13)
> NOTE: check that installer/data/mysql/xx-YY (or es-ES) does *not* exists
> before doing the install phase. By design that dir is not overwritten or
> updated.

Ha! I didn't know (or forgot) that!
That helped a lot, thanks.

Everything is working for me now! :)
Comment 15 Jonathan Druart 2022-04-06 13:24:12 UTC
How hard would it be to provide patches for languages that had UNIMARC frameworks with the descriptions translated (it-IT, uk-UA, etc.)?

I would like to push this ASAP and let testers test against master.
Comment 16 Jonathan Druart 2022-04-06 13:33:17 UTC
Thibault, you can see the framework with translated descriptions on this sandbox: https://staff-bug30373.sandboxes.biblibre.eu/cgi-bin/koha/admin/marctagstructure.pl?frameworkcode=
Comment 17 Bernardo Gonzalez Kriegel 2022-04-06 14:11:03 UTC
(In reply to Jonathan Druart from comment #15)
> How hard would it be to provide patches for languages that had UNIMARC
> frameworks with the descriptions translated (it-IT, uk-UA, etc.)?
> 
> I would like to push this ASAP and let testers test against master.

Ok, I'll open a new bug with all new *-installer-UNIMARC.po, and fill them as much as possible.
Comment 18 Thibault Keromnès 2022-04-06 15:11:41 UTC
(In reply to Jonathan Druart from comment #16)
> Thibault, you can see the framework with translated descriptions on this
> sandbox:
> https://staff-bug30373.sandboxes.biblibre.eu/cgi-bin/koha/admin/
> marctagstructure.pl?frameworkcode=

Thanks, i'm having a look, and asking others to do it too. 
I noticed search is broken on this link, hope it doesn't have to do with the patch? (maybe some data i imported? but i had an error before importing too). 
Can you take a look ? https://staff-bug30373.sandboxes.biblibre.eu/cgi-bin/koha/catalogue/search.pl?q=test
Comment 19 Jonathan Druart 2022-04-06 15:43:28 UTC
Hum indeed, search is broken.

I saw that in the log:
Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a ->  at /usr/share/perl5/MARC/File/XML.pm line 570

I rebuilt the zebra's index but it's still not working

$ koha-rebuild-zebra --force -v -v -a -b bug30373
Zebra configuration information
================================
Zebra biblio directory      = /var/lib/koha/bug30373/biblios
Zebra authorities directory = /var/lib/koha/bug30373/authorities
Koha directory              = /kohadevbox/koha
Lockfile                    = /var/lock/koha/bug30373/rebuild/rebuild..LCK
BIBLIONUMBER in :     001$@
BIBLIOITEMNUMBER in : 090$a
================================
Job started: 15:40:20
skipping authorities
====================
exporting biblio 15:40:20 [00:00:00]
====================

Records exported: 0 15:40:20 [00:00:00]

Records exported: 0 15:40:20 [00:00:00]
====================
REINDEXING zebra 15:40:20 [00:00:00]
====================
====================
Indexing complete: 15:40:20 [00:00:00]
====================
CLEANING
====================
Zebra configuration information
================================
Zebra biblio directory      = /var/lib/koha/bug30373/biblios
Zebra authorities directory = /var/lib/koha/bug30373/authorities
Koha directory              = /kohadevbox/koha
Lockfile                    = /var/lock/koha/bug30373/rebuild/rebuild..LCK
BIBLIONUMBER in :     001$@
BIBLIOITEMNUMBER in : 090$a
================================
Job started: 15:40:21
====================
exporting authority 15:40:22 [00:00:01]
====================

Records exported: 0 15:40:22 [00:00:01]

Records exported: 0 15:40:22 [00:00:01]
====================
REINDEXING zebra 15:40:22 [00:00:01]
====================
skipping biblios
====================
Indexing complete: 15:40:22 [00:00:01]
====================
CLEANING
====================
Comment 20 Bernardo Gonzalez Kriegel 2022-04-06 21:11:47 UTC
Yep, search is broken but without the patches.

Try master fr-FR or en UNIMARC installs, both shows the same when doing a search (empty catalog)

"Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a -> Rischkowsky, B. at /usr/share/perl5/MARC/File/XML.pm line 570
... etc"
Comment 21 Bernardo Gonzalez Kriegel 2022-04-06 21:14:35 UTC
(In reply to Jonathan Druart from comment #15)
> How hard would it be to provide patches for languages that had UNIMARC
> frameworks with the descriptions translated (it-IT, uk-UA, etc.)?
> 
> I would like to push this ASAP and let testers test against master.

Done, Bug 30477 (and Bug 30476 to remove NORMARC files)
Comment 22 Bernardo Gonzalez Kriegel 2022-04-06 22:35:30 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #20)
> Yep, search is broken but without the patches.
> 
> Try master fr-FR or en UNIMARC installs, both shows the same when doing a
> search (empty catalog)
> 
> "Unsupported UNIMARC character encoding [] for XML output for UNIMARC; 100$a
> -> Rischkowsky, B. at /usr/share/perl5/MARC/File/XML.pm line 570
> ... etc"

Disabling plack it works :(

root@kohadevbox:~$ koha-plack --disable kohadev
Plack disabled for kohadev OPAC
Plack disabled for kohadev Intranet
root@kohadevbox:~$ restart_all 
...
root@kohadevbox:~$ koha-rebuild-zebra --force -v -v -a -b kohadev 
...       

and search gives no results, as expected, but also no errors
Comment 23 Jonathan Druart 2022-04-07 07:56:37 UTC
Yes, and I don't recreate the problem locally.
Comment 24 Jonathan Druart 2022-04-07 08:04:53 UTC
Asking for a quick push and let us time to find bugs before the 22.05 release.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 25 Jonathan Druart 2022-04-07 08:05:41 UTC
Fridolin, patches with signed-off-by lines are available at https://gitlab.com/joubu/Koha/-/commits/bug_30373
Comment 26 Jonathan Druart 2022-04-07 08:10:55 UTC
I would push this along with bug 27619. We will need to adjust koha-misc4dev as well.
Comment 27 Jonathan Druart 2022-04-07 08:12:20 UTC
(In reply to Jonathan Druart from comment #26)
> I would push this along with bug 27619. We will need to adjust koha-misc4dev
> as well.

https://gitlab.com/koha-community/koha-misc4dev/-/issues/54
Comment 28 Thibault Keromnès 2022-04-07 08:29:42 UTC
Arf maybe it's too late to ask for modification on the biblio framework?
Two new things we found : 
    - 995 (items) $5 (dateaccessionned) shouldn't be in the editor, it's automatically filled with the current day and if you type something else than a properly formatted date it creates an error. I checked in master for now it's not in the editor. 
    - 325$a shouldn't be hidden
Comment 29 Jonathan Druart 2022-04-07 08:49:31 UTC
Created attachment 133059 [details] [review]
Bug 30373: Hide items.dateaccessioned from the editor

-6 was used in unimarc_complet/Obligatoire/framework_DEFAULT.sql
Comment 30 Jonathan Druart 2022-04-07 08:49:35 UTC
Created attachment 133060 [details] [review]
Bug 30373: Show 325$a

It was not hidden in unimarc_complet/Obligatoire/framework_DEFAULT.sql
Comment 31 Jonathan Druart 2022-04-07 08:49:50 UTC
(In reply to Thibault Keromnès from comment #28)
> Arf maybe it's too late to ask for modification on the biblio framework?
> Two new things we found : 
>     - 995 (items) $5 (dateaccessionned) shouldn't be in the editor, it's
> automatically filled with the current day and if you type something else
> than a properly formatted date it creates an error. I checked in master for
> now it's not in the editor. 
>     - 325$a shouldn't be hidden

Never too late, thanks!

Bernardo, Do those 2 patches look correct to you?
Comment 32 Bernardo Gonzalez Kriegel 2022-04-07 10:19:51 UTC
(In reply to Jonathan Druart from comment #31)
> 
> Bernardo, Do those 2 patches look correct to you?

Yes! I'm doing a complete check of hidden values (old/new)
Comment 33 Bernardo Gonzalez Kriegel 2022-04-08 22:25:12 UTC
Created attachment 133126 [details] [review]
Bug 30373: Add TYPEDOC authorised_values_category

TYPEDOC authorised value category is needed to permit
correct loading of default UNIMARC bibliographic framework.
Comment 34 Bernardo Gonzalez Kriegel 2022-04-08 22:26:08 UTC
Created attachment 133127 [details] [review]
Bug 30373: Fix visibility, authorised values and descriptions

This patch fixes certain values of default bibliographic
framework.

It fixes hidden values using as reference default framework
from biblibre demo site.

Also adds some missing subfields, authorised values and
fixes capitalization on some descriptions.

To test:
1) On biblibre demo site, run a report
   SELECT tagfield,tagsubfield,hidden
   FROM marc_subfield_structure
   WHERE frameworkcode = ''
   ORDER BY tagfield,tagsubfield;
2) Download as tab separated values, ej. biblibre-hidden.txt
3) Apply the patch, remove bib frameworks, load YAML file
4) Run something like this to check correct hidden values

   while read -r -a l; do
    t="${l[0]}";
    s="${l[1]}";
    h="${l[2]}";
    b=$( mysql koha -N -B -e "select hidden from marc_subfield_structure where frameworkcode = '' and tagfield = '$t' and tagsubfield = '$s'");
    [ -n "$b" -a "$b" != "$h" ] && echo "$t:$s: -> change $b to $h";
   done < <( cat biblibre-hidden.txt )

   there must be no output
Comment 35 Fridolin Somers 2022-04-20 18:54:25 UTC
(In reply to Jonathan Druart from comment #25)
> Fridolin, patches with signed-off-by lines are available at
> https://gitlab.com/joubu/Koha/-/commits/bug_30373

Hi, could you please apply last patch :
 Bug 30373: Fix visibility, authorised values and descriptions

It does not work with 'git bz apply'
Comment 36 Bernardo Gonzalez Kriegel 2022-04-20 19:18:34 UTC
(In reply to Fridolin Somers from comment #35)
> (In reply to Jonathan Druart from comment #25)
> > Fridolin, patches with signed-off-by lines are available at
> > https://gitlab.com/joubu/Koha/-/commits/bug_30373
> 
> Hi, could you please apply last patch :
>  Bug 30373: Fix visibility, authorised values and descriptions
> 
> It does not work with 'git bz apply'

Hi, you need to apply the last four patches over Jonathan's branch.
The last patch depends on #29 and #30
Comment 37 Fridolin Somers 2022-04-21 06:13:01 UTC
Ah sorry does not work.

I see Jonathan's branch contains until patch "Show 325$a".

git bz apply shows error :

Applying: Bug 30373: Fix visibility, authorised values and descriptions
error: sha1 information is lacking or useless (installer/data/mysql/en/marcflavour/unimarc/mandatory/unimarc_framework_DEFAULT.yml).
error: could not build fake ancestor
Comment 38 Bernardo Gonzalez Kriegel 2022-04-21 09:46:51 UTC
(In reply to Fridolin Somers from comment #37)
> Ah sorry does not work.
> 
> I see Jonathan's branch contains until patch "Show 325$a".
> 
> git bz apply shows error :
> 
> Applying: Bug 30373: Fix visibility, authorised values and descriptions
> error: sha1 information is lacking or useless
> (installer/data/mysql/en/marcflavour/unimarc/mandatory/
> unimarc_framework_DEFAULT.yml).
> error: could not build fake ancestor

Strange. Using Jonathan's remote, I don't find any problem.

$ git fetch joubu
$ git checkout -b bz30373 joubu/bug_30373
  Rama 'bz30373' configurada para hacer seguimiento a la rama remota 'bug_30373' de 'joubu'.
  Cambiado a nueva rama 'bz30373'
$ git log --oneline | head -8
  01eba5253d Bug 30373: Show 325$a
  88798576db Bug 30373: Hide items.dateaccessioned from the editor
  5e4e37c704 Bug 30477: Add new UNIMARC installer translation files
  b81b2c107b Bug 30373: Enable translation of UNIMARC frameworks
  b57c4ea2bd Bug 30373: Rewrite authorities_normal_unimarc to YAML
  60106a4780 Bug 30373: Add UNIMARC FA (fast add) framework
  8ca3f24f4b Bug 30373: Rewrite unimarc_framework_DEFAULT to YAML
  3dd12b8ea2 Bug 30476: Remove NORMARC translation files
$ git bz apply 30373

  Bug 30373 - Rewrite UNIMARC installer data to YAML

  132280 - Bug 30373: Add UNIMARC FA (fast add) framework
  132282 - Bug 30373: Enable translation of UNIMARC frameworks
  133059 - Bug 30373: Hide items.dateaccessioned from the editor
  133060 - Bug 30373: Show 325$a
  133126 - Bug 30373: Add TYPEDOC authorised_values_category
  133127 - Bug 30373: Fix visibility, authorised values and descriptions

  Apply? [(y)es, (n)o, (i)nteractive] i
  Aplicando: Bug 30373: Add TYPEDOC authorised_values_category
  Aplicando: Bug 30373: Fix visibility, authorised values and descriptions
Comment 39 Jonathan Druart 2022-04-21 10:12:38 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #38)
> (In reply to Fridolin Somers from comment #37)
> > Ah sorry does not work.
> > 
> > I see Jonathan's branch contains until patch "Show 325$a".
> > 
> > git bz apply shows error :
> > 
> > Applying: Bug 30373: Fix visibility, authorised values and descriptions
> > error: sha1 information is lacking or useless
> > (installer/data/mysql/en/marcflavour/unimarc/mandatory/
> > unimarc_framework_DEFAULT.yml).
> > error: could not build fake ancestor
> 
> Strange. Using Jonathan's remote, I don't find any problem.

I am getting the same error than Fridolin. Can you resubmit the two patches or push a remote branch please?
Comment 40 Bernardo Gonzalez Kriegel 2022-04-21 10:59:00 UTC
(In reply to Jonathan Druart from comment #39)
> 
> I am getting the same error than Fridolin. Can you resubmit the two patches
> or push a remote branch please?

Pushed to https://gitlab.com/bgkriegel/Koha/-/commits/bz30373
Comment 41 Fridolin Somers 2022-04-21 23:28:05 UTC
Super, it works now.
Comment 42 Fridolin Somers 2022-04-21 23:42:08 UTC
Pushed to master for 22.05, thanks to everybody involved ﷐[U+1F984]﷑
Comment 43 Jonathan Druart 2022-04-22 08:17:52 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #34)
> Created attachment 133127 [details] [review] [review]
> Bug 30373: Fix visibility, authorised values and descriptions
> 
> This patch fixes certain values of default bibliographic
> framework.
> 
> It fixes hidden values using as reference default framework
> from biblibre demo site.
> 
> Also adds some missing subfields, authorised values and
> fixes capitalization on some descriptions.

This patch is buggy, it uses authorised values that have not been inserted.
Comment 44 Jonathan Druart 2022-04-22 08:22:19 UTC
Created attachment 133630 [details] [review]
Bug 30373: Fix unimarc authorised values

1. Replace
  authorised_value: ""
with
  authorised_value:
to match what is done for marc21

2. Fix some wrong category names
-          authorised_value: 146a
+          authorised_value: unimarc_146a

-          authorised_value: 182a
+          authorised_value: unimarc_182a

-          authorised_value: 182c
+          authorised_value: unimarc_182c

3. Remove some category that have not been inserted (should we insert
them then?)
-          authorised_value: 183a
+          authorised_value:

-          authorised_value: qualif
+          authorised_value:
Comment 45 Jonathan Druart 2022-04-22 08:22:44 UTC
Bernardo, is this patch correct?
Comment 46 Jonathan Druart 2022-04-22 10:03:02 UTC
Created attachment 133658 [details] [review]
Bug 30373: Fix xt/find-license-problems.t
Comment 47 Bernardo Gonzalez Kriegel 2022-04-22 10:19:27 UTC
(In reply to Jonathan Druart from comment #45)
> Bernardo, is this patch correct?

Unfortunately it has errors.

I'll provide a patch asap. Most of the authorised values are inserted, correctly, at the beginning of the bib framework file, ej. 
        - category_name: "unimarc_146a"
        - category_name: "unimarc_181c"
        - category_name: "unimarc_182a"
        - category_name: "unimarc_182c"
        - category_name: "unimarc_325h"

The values like '146a' are missing the 'unimarc_' part, 'qualif' already exist.

I suppose a new bug should be used for this.
Comment 48 Jonathan Druart 2022-04-22 10:32:40 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #47)
> (In reply to Jonathan Druart from comment #45)
> > Bernardo, is this patch correct?
> 
> Unfortunately it has errors.
> 
> I'll provide a patch asap. Most of the authorised values are inserted,
> correctly, at the beginning of the bib framework file, ej. 
>         - category_name: "unimarc_146a"
>         - category_name: "unimarc_181c"
>         - category_name: "unimarc_182a"
>         - category_name: "unimarc_182c"
>         - category_name: "unimarc_325h"
> 
> The values like '146a' are missing the 'unimarc_' part, 'qualif' already
> exist.
> 
> I suppose a new bug should be used for this.

Nope, please attach a follow-up here. Feel free to scratch mine! Thanks! :)
Comment 49 Fridolin Somers 2022-04-22 18:48:27 UTC
Pushed to master :
 Fix unimarc authorised values
 Fix xt/find-license-problems.t
Comment 50 Bernardo Gonzalez Kriegel 2022-04-24 00:01:06 UTC
Created attachment 133704 [details] [review]
Bug 30373: (followup) Country codes auth values

This patch adds COUNTRY codes (ISO 3166-1) into
mandatory auth_values.yml

This authorized value is used in UNIMARC default bibliographic
framework.

To test:
1) Do a clean install or load the YAML file
2) Verify new country codes are in the database
    SELECT * FROM authorised_values WHERE category = 'COUNTRY';
Comment 51 Bernardo Gonzalez Kriegel 2022-04-24 00:01:12 UTC
Created attachment 133705 [details] [review]
Bug 30373: (followup) Language codes auth values

This patch adds language codes (ISO 639-2) into
mandatory auth_values.yml

This authorized value is used in UNIMARC default bibliographic
framework.

To test:
1) Do a clean install or load the YAML file
2) Verify new language codes are in the database
    SELECT * FROM authorised_values WHERE category = 'LANG';
Comment 52 Bernardo Gonzalez Kriegel 2022-04-24 00:01:18 UTC
Created attachment 133706 [details] [review]
Bug 30373: (followup) UNIMARC relator codes auth values

This patch adds UNIMARC relator codes into
mandatory auth_values.yml

This authorized value is used in UNIMARC default bibliographic
framework.

To test:
1) Do a clean install or load the YAML file
2) Verify new language codes are in the database
    SELECT * FROM authorised_values WHERE category = 'qualif';
Comment 53 Bernardo Gonzalez Kriegel 2022-04-24 00:01:23 UTC
Created attachment 133707 [details] [review]
Bug 30373: (followup) Optional SUGGEST auth values

This patch adds 3 SUGGEST optional auth_val.yml

This authorized value is used in UNIMARC default bibliographic
framework.

To test:
1) Do a clean install with optional files or load the YAML file
2) Verify new SUGGEST values are in the database
    SELECT * FROM authorised_values WHERE category = 'SUGGEST';

    Check new values Budget, Expensive and Poldoc
Comment 54 Bernardo Gonzalez Kriegel 2022-04-24 00:06:34 UTC
(In reply to Jonathan Druart from comment #48)
> Nope, please attach a follow-up here. Feel free to scratch mine! Thanks! :)

Your patch was fine!
I only think that we can add again the entries using qualif's authorised values, they are UNIMARC relator codes

I added 4 follow-ups, all authorised values from fr-FR SQL files needed by the UNIMARC framework
Comment 55 Bernardo Gonzalez Kriegel 2022-04-25 01:54:28 UTC
Created attachment 133723 [details] [review]
Bug 30373: (followup) Enable 'qualif' authorised_value

This patch reinstates the authorised value 'qualif'
to it's corresponding place

This authorized value is used in UNIMARC default bibliographic
framework.

To test:
1) Do a clean install or load the YAML file
2) Verify 'qualif' authorised_value is present in subfield $4 of
   fields 700/1/2/3, 710/1/2/3, 720/1/2/3 and 730
Comment 56 Bernardo Gonzalez Kriegel 2022-04-25 02:32:29 UTC
Created attachment 133724 [details] [review]
Bug 30373: (followup) Add BNF z3950 server

This is not strictly necessary, but is useful to do the testing
Comment 57 Fridolin Somers 2022-04-27 19:49:44 UTC
Follow-ups pushed to master (commit message fixed) :

286a800704 Bug 30373: (follow-up) Add BNF z3950 server
880103371b Bug 30373: (follow-up) Enable 'qualif' authorised_value
5edafaff11 Bug 30373: (follow-up) Optional SUGGEST auth values
20ae18e5ee Bug 30373: (follow-up) UNIMARC relator codes auth values
d4fa096001 Bug 30373: (follow-up) Language codes auth values
4bdaaed17b Bug 30373: (follow-up) Country codes auth values
Comment 58 Jonathan Druart 2023-11-08 14:38:07 UTC
gulpfile.js changes from "Enable translation of UNIMARC frameworks" will be cherry-pick for 21.11.26
Comment 59 Mathieu Saby 2024-08-30 07:26:08 UTC
Hello

I believe Zebra (and maybe ES, I don't know where is the config file) index for 995$a was not updated after this patch.
It is still mapped with "homebranch:w", "Host-item:w" and "item:w". This is no more appropriate, if 995$a now contains the name of the provider ("Source of acquisition").
Comment 60 Katrin Fischer 2024-08-30 07:40:50 UTC
(In reply to Mathieu Saby from comment #59)
> Hello
> 
> I believe Zebra (and maybe ES, I don't know where is the config file) index
> for 995$a was not updated after this patch.
> It is still mapped with "homebranch:w", "Host-item:w" and "item:w". This is
> no more appropriate, if 995$a now contains the name of the provider ("Source
> of acquisition").

Hi Mathieu, would you mind filing a separate bug please? you can use the "depends on" to link it to this and Bugzilla will also email everyone about it :)
Comment 61 Mathieu Saby 2024-08-30 12:22:50 UTC
I am checking Elasticsearch and Zebra index for Unimarc items, and there are a lot of discrepancies, errors and missing subfields :/

I will fill a bz when I am done
Comment 62 Mathieu Saby 2024-09-13 12:38:52 UTC
Should I open 1 ticket for all the changes needed (in ES and Zebra), or 1 for Zebra and 1 for ES?
Comment 63 Katrin Fischer 2024-09-13 12:43:48 UTC
(In reply to Mathieu Saby from comment #62)
> Should I open 1 ticket for all the changes needed (in ES and Zebra), or 1
> for Zebra and 1 for ES?

maybe we could fix the same indexes for ES and Zebra at the same time. Breaking it into smaller portions might be nice, but if you think it's easier maybe making some "bundles" also works? (Like doing all "footnotes" related ones in one patch or so?)
Comment 64 Mathieu Saby 2024-09-13 16:41:48 UTC
So I will open 3 bugs

1- fields wrongly indexed
2- fields indexed in Zebra but not in ES, or in ES and not in Zebra
3- fields not indexed, but their Marc21 equivalent is indexed

The most important to fix is of course 1, then 2, and 3