Bug 28352

Summary: Errors in search_for_data_inconsistencies.pl relating to authorised values and frameworks
Product: Koha Reporter: Andrew Fuerste-Henry <andrew>
Component: Command-line UtilitiesAssignee: Nick Clemens (kidclamp) <nick>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: normal    
Priority: P5 - low CC: fridolin.somers, jonathan.druart, kyle, m.de.rooy, nick, robin, victor
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.11.00,21.05.05,20.11.11
Circulation function:
Bug Depends on: 21466, 27852    
Bug Blocks:    
Attachments: Bug 28352: Only check authorised values mapped to DB fields
Bug 28352: Only check authorised values mapped to DB fields
Bug 28352: Only check authorised values mapped to DB fields

Description Andrew Fuerste-Henry 2021-05-13 19:40:09 UTC
When running search_for_data_inconsistencies.pl on 20.05 and later, I get the following errors:
Use of uninitialized value $tmp_kohafield in pattern match (m//) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 151.
Use of uninitialized value $tmp_kohafield in substitution (s///) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 154.
Can't call method "get_column" on an undefined value at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 157.

The script seems to be looking for something in authorised_values that it't not finding. I've verified that every authorised_value used in a marc_subfield_structure exists and has at least one value.
Comment 1 Nick Clemens (kidclamp) 2021-05-26 11:49:22 UTC
Created attachment 121424 [details] [review]
Bug 28352: Only check authorised values mapped to DB fields

The errors reported seem to be caused by authorised values mapped to MARC fields
but not mapped to a koha field.

We should additionally make sure to check the Default framework

Also, adding comment to indicate we only check records with items, because we do

TO test:
1 - In a framework that is not the default map a MARC field to an authorised value, but not a koha field
2 - In SQL, force the kohafield to NULL for the mapping you just make
    UPDATE marc_subfield_structure SET kohafield = NULL WHERE frameworkcode='BKS' and authorised_value='HINGS_AS'
3 - perl misc/maintenance/search_for_data_inconsistencies.pl
4 - get the following errors:
Use of uninitialized value $tmp_kohafield in pattern match (m//) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 151.
Use of uninitialized value $tmp_kohafield in substitution (s///) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 154.
Can't call method "get_column" on an undefined value at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 157.
5 - Apply patch
6 - Repeat
7 - No more errors
Comment 2 Fridolin Somers 2021-08-04 07:30:43 UTC
Created attachment 123447 [details] [review]
Bug 28352: Only check authorised values mapped to DB fields

The errors reported seem to be caused by authorised values mapped to MARC fields
but not mapped to a koha field.

We should additionally make sure to check the Default framework

Also, adding comment to indicate we only check records with items, because we do

TO test:
1 - In a framework that is not the default map a MARC field to an authorised value, but not a koha field
2 - In SQL, force the kohafield to NULL for the mapping you just make
    UPDATE marc_subfield_structure SET kohafield = NULL WHERE frameworkcode='BKS' and authorised_value='HINGS_AS'
3 - perl misc/maintenance/search_for_data_inconsistencies.pl
4 - get the following errors:
Use of uninitialized value $tmp_kohafield in pattern match (m//) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 151.
Use of uninitialized value $tmp_kohafield in substitution (s///) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 154.
Can't call method "get_column" on an undefined value at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 157.
5 - Apply patch
6 - Repeat
7 - No more errors

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Comment 3 Fridolin Somers 2021-08-04 07:31:49 UTC
I was just about to open the same issue ;)
Thanks a lot Nick :D
Comment 4 Jonathan Druart 2021-08-04 09:01:45 UTC
Highlighed by bug 27852.
Comment 5 Jonathan Druart 2021-08-04 09:02:20 UTC
My fix was 
 143             next unless $kohafield and $av;

But yours is better.
Comment 6 Marcel de Rooy 2021-10-01 09:54:45 UTC
Created attachment 125599 [details] [review]
Bug 28352: Only check authorised values mapped to DB fields

The errors reported seem to be caused by authorised values mapped to MARC fields
but not mapped to a koha field.

We should additionally make sure to check the Default framework

Also, adding comment to indicate we only check records with items, because we do

TO test:
1 - In a framework that is not the default map a MARC field to an authorised value, but not a koha field
2 - In SQL, force the kohafield to NULL for the mapping you just make
    UPDATE marc_subfield_structure SET kohafield = NULL WHERE frameworkcode='BKS' and authorised_value='HINGS_AS'
3 - perl misc/maintenance/search_for_data_inconsistencies.pl
4 - get the following errors:
Use of uninitialized value $tmp_kohafield in pattern match (m//) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 151.
Use of uninitialized value $tmp_kohafield in substitution (s///) at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 154.
Can't call method "get_column" on an undefined value at /kohadevbox/koha/misc/maintenance/search_for_data_inconsistencies.pl line 157.
5 - Apply patch
6 - Repeat
7 - No more errors

Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 7 Jonathan Druart 2021-10-01 14:30:14 UTC
Pushed to master for 21.11, thanks to everybody involved!
Comment 8 Kyle M Hall (khall) 2021-10-08 13:52:49 UTC
Pushed to 21.05.x for 21.05.05
Comment 9 Fridolin Somers 2021-10-21 04:08:05 UTC
Pushed to 20.11.x for 20.11.11
Comment 10 Victor Grousset/tuxayo 2021-10-26 00:43:59 UTC
Not backported to oldoldstable (20.05.x). Feel free to ask if it's needed.