Bug 27730 - Innocent changes to framework silently delete bibliographic data
Summary: Innocent changes to framework silently delete bibliographic data
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P3 enhancement (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
: 35135 (view as bug list)
Depends on:
Blocks:
 
Reported: 2021-02-19 10:36 UTC by Ian Bays
Modified: 2024-04-11 16:17 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ian Bays 2021-02-19 10:36:19 UTC
If you copy and change a framework and either delete a subfield in the new framework or mark the subfield tab as "ignore", then any bib data with data in that subfield, when saved in the new framework will be silently destroyed with no option to undo.

Test Plan to confirm behaviour:

1. Create bib record in default framework with minimal data, but add tag 365 (as it has a number of subfields all visible) and populate subfields a through h with "fa" through "fh". Save the record and with sql check the contents of the metadata.

(I found this SQL gives enough information substituting the correct bib number of the saved record:

select tagfield, tagsubfield, frameworkcode, tab, hidden, (select ExtractValue(metadata, '//datafield[@tag="365"]/subfield[@code>="a"]') from biblio_metadata where biblionumber = 37002 and (select frameworkcode from biblio where biblionumber = 37002) = marc_subfield_structure.frameworkcode) tv from marc_subfield_structure where frameworkcode in ('', 'TEST') and tagfield = '365';
)

2. Copy the default framework to a TEST framework and set (say 365$b to tab ignore and delete 365$d). (I used export the default framework to a csv, create new TEST framework and import from the saved default csv)

3. Edit the bib record and change framework to TEST framework (settings > change framework) and save the record.

4. Run the SQL again and observe the data from subfields b and d have both disappeared from the metadata.

If you were to rerun these tests you may need to reload memcache to ensure the newly changed frameworks are picked up.

I understand it has been broadly known by some long-term Koha users that deleting a subfield from the framework has this effect, but both these situations potentially cause bib data to be silently and permanently delated from the database.  Many users will be unaware and may not discover the problem for some time.

Potential solutions might include:

a) For subfields not in the framework or with tab "ignore" allow the data to be saved as is on saving for edit;

b) When selecting "change framework" detect if data would be lost and alert the user with an option to back out of the change without losing data;

c) Do not allow deletion of a subfield from a framework and do not allow the "ignore" option for the tab.  If a field is to be ignored but hidden there are plenty of visibility options in the framework.

NB: The current delivered default framework has 13 subfields set to the "ignore" tab.  I cannot see what purpose these serve as no data can survive within them.
Comment 1 Jonathan Druart 2021-03-11 11:00:26 UTC
This is the historical behaviour, and how it is intended to work.

It could be nice to have an alert (that could be switched off the a syspref?), but I am considering it an enhancement request.

Katrin, do you agree?
Comment 2 Katrin Fischer 2021-03-23 08:42:31 UTC
Actually I was not aware of the setting "ignore" having this effect. The known thing is that you don't want to delete any field and subfield definitions as that will cause data loss. That's not great, but we have some documentation and warnings.

For the "ignore" tab, I always assumed this was only a display setting as it's used to set the tab something is displayed on. So I think reading "Ignore" there as "don't display" is to be expected from user side.

1) Can we determin if the "ignore" always had this effect? I believe I have set "ignore" for some item fields in one of our oldest installations, but we never noticed any misbehaviour.

2) If we think this is not only to affect display, we definitely need to add at least a hint GUI wise. Maybe we should also add a list of "ignored" to the Bibliographic framework check to help people verify.
Comment 3 Ian Bays 2021-03-23 09:57:00 UTC
Hi. The initial description includes a test plan that shows "ignore" deletes metadata. This was tested on recent versions including 20.11, but for completeness I went back to a 17.11 version and used a simpler method to verify the behaviour:

We often set up a framework for (say) BK that sets (eg) the 022$a to ignore so I first checked this was the case on the system, then added a bib in the default framework and added an 022$a. I checked the biblio_metadata and it was present.  I edited the record and changed frameworkcode from default to BK and saved. On inspection the 022$a has not present.  Here is the SQL from this sequence:

MariaDB [koha_ras]> select metadata from biblio_metadata where biblionumber = 37441;
| <?xml version="1.0" encoding="UTF-8"?>
<record
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd"
    xmlns="http://www.loc.gov/MARC21/slim">

  <leader>00242nam a22001217a 4500</leader>
  <datafield tag="999" ind1=" " ind2=" ">
    <subfield code="c">37441</subfield>
    <subfield code="d">37441</subfield>
  </datafield>
  <controlfield tag="003">OSt</controlfield>
  <controlfield tag="005">20210323092844.0</controlfield>
  <controlfield tag="008">210323b           ||||| |||| 00| 0 eng d</controlfield>
  <datafield tag="022" ind1=" " ind2=" ">
    <subfield code="a">1234</subfield>
  </datafield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="a">Test</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Test</subfield>
  </datafield>
  <datafield tag="942" ind1=" " ind2=" ">
    <subfield code="2">ddc</subfield>
    <subfield code="c">MONO</subfield>
  </datafield>
</record>
 |

MariaDB [koha_ras]> select frameworkcode from biblio where biblionumber = 37441;
+---------------+
| frameworkcode |
+---------------+
|               |
+---------------+

MariaDB [koha_ras]> select tab from marc_subfield_structure where frameworkcode = '' and tagfield='022' and tagsubfield = 'a';
+------+
| tab  |
+------+
|    0 |
+------+

MariaDB [koha_ras]> select frameworkcode from biblio where biblionumber = 37441;
+---------------+
| frameworkcode |
+---------------+
| BK            |
+---------------+

MariaDB [koha_ras]> select tab from marc_subfield_structure where frameworkcode = 'BK' and tagfield='022' and tagsubfield = 'a';
+------+
| tab  |
+------+
|   -1 |
+------+

MariaDB [koha_ras]> select metadata from biblio_metadata where biblionumber = 37441;
| <?xml version="1.0" encoding="UTF-8"?>
<record
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.loc.gov/MARC21/slim http://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd"
    xmlns="http://www.loc.gov/MARC21/slim">

  <leader>00216nam a22001097a 4500</leader>
  <datafield tag="999" ind1=" " ind2=" ">
    <subfield code="c">37441</subfield>
    <subfield code="d">37441</subfield>
  </datafield>
  <controlfield tag="003">OSt</controlfield>
  <controlfield tag="005">20210323094228.0</controlfield>
  <controlfield tag="008">210323b           ||||| |||| 00| 0 eng d</controlfield>
  <datafield tag="100" ind1=" " ind2=" ">
    <subfield code="a">Test</subfield>
  </datafield>
  <datafield tag="245" ind1=" " ind2=" ">
    <subfield code="a">Test</subfield>
  </datafield>
  <datafield tag="942" ind1=" " ind2=" ">
    <subfield code="c">MONO</subfield>
  </datafield>
</record>
 |
1 row in set (0.00 sec)

MariaDB [koha_ras]> select value from systempreferences where variable like '%version%';
+------------+
| value      |
+------------+
| 17.1109000 |
+------------+
Comment 4 Katrin Fischer 2021-03-23 10:04:18 UTC
Thx Ian. 

What do you think we shoudl do? Make this a display thing (only affecting the tab) or a warning/hint be ok?
Comment 5 Ian Bays 2021-03-23 10:28:26 UTC
As yourself I had assumed that "ignore" would leave the data intact, so I initially thought it would be "best" if the data is preserved but it is just not pulled out for display.

Because we understand it may take time to agree and implement we set about scripting changes to our frameworks to set all those that say "ignore" to a real tab but to set "hidden" to a value that hides it.

However we discovered that the default framework has circa 13 subfields set to ignore including 999$a, $b and $c.  If you set these to other than ignore the framework check fails.

We have an amended script that reinstates the real tab number for all set to "ignore" except the 999 subfields.  The script also copies tags and subfields from default where they had been deleted (or possibly not updated from the delivered defaults).

Many people will never notice losing these bits of data as they don't see them, but if you've got your data from a rich source and save in a different framework you would not expect the rich data to be removed.

A warning (if you are changing to a framework that will lose data for this record) might be a good first step until there is consensus on how to deal with it.
Comment 6 Katrin Fischer 2022-11-17 10:58:17 UTC
With Bug 31923 - 'Ignore' tab description is misleading there is no at least a warning in the framework configuration.
Comment 7 Phil Ringnalda 2024-04-05 19:01:48 UTC
*** Bug 35135 has been marked as a duplicate of this bug. ***