Bug 14957 - Write protecting MARC fields based on source of import
Summary: Write protecting MARC fields based on source of import
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Eivin Giske Skaaren
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-10-05 14:07 UTC by Martin Stenberg
Modified: 2018-05-07 12:33 UTC (History)
17 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 14957: Rule based permission for MARC fields (81.46 KB, patch)
2015-10-27 11:31 UTC, Martin Stenberg
Details | Diff | Splinter Review
Bug 14957: Database changes (3.56 KB, patch)
2015-10-27 11:31 UTC, Martin Stenberg
Details | Diff | Splinter Review
Bug 14957: Add filters to relevant scripts (5.84 KB, patch)
2015-10-27 11:31 UTC, Martin Stenberg
Details | Diff | Splinter Review
Bug 14957: Rule based permission for MARC fields (82.59 KB, patch)
2016-05-23 20:53 UTC, Eivin Giske Skaaren
Details | Diff | Splinter Review
Bug 14957: Correct parameters for TransformMarcToKoha (940 bytes, patch)
2016-05-23 20:59 UTC, Eivin Giske Skaaren
Details | Diff | Splinter Review
Bug 14957: Key fixes to patch (19.48 KB, patch)
2016-10-19 04:23 UTC, Aleisha Amohia
Details | Diff | Splinter Review
Bug 14957: Key fixes to patch (19.48 KB, patch)
2016-10-19 04:28 UTC, Aleisha Amohia
Details | Diff | Splinter Review
Bug 14957: Database changes (3.56 KB, patch)
2016-10-19 09:23 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Add filters to relevant scripts (6.07 KB, patch)
2016-10-19 09:23 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Rule based permission for MARC fields (81.96 KB, patch)
2016-10-19 09:23 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Correct parameters for TransformMarcToKoha (940 bytes, patch)
2016-10-19 09:23 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Key fixes to patch (19.48 KB, patch)
2016-10-19 09:23 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Followup - Pod, import_lexile.pl, template and test fixes (3.45 KB, patch)
2016-10-19 09:23 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Database changes (3.56 KB, patch)
2016-11-03 07:06 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Add filters to relevant scripts (6.07 KB, patch)
2016-11-03 07:06 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Rule based permission for MARC fields (81.60 KB, patch)
2016-11-03 07:06 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Correct parameters for TransformMarcToKoha (940 bytes, patch)
2016-11-03 07:06 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Key fixes to patch (19.48 KB, patch)
2016-11-03 07:07 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 14957: Followup - Pod, import_lexile.pl, template and test fixes (3.45 KB, patch)
2016-11-03 07:07 UTC, Josef Moravec
Details | Diff | Splinter Review
Rule based permission for MARC fields (87.02 KB, patch)
2017-02-02 12:44 UTC, David Gustafsson
Details | Diff | Splinter Review
Rule based permission for MARC fields (86.66 KB, patch)
2017-02-02 12:49 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (90.52 KB, patch)
2017-02-03 14:09 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (90.64 KB, patch)
2017-02-27 13:09 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (90.64 KB, patch)
2017-03-06 09:59 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (90.60 KB, patch)
2017-03-06 10:04 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (90.88 KB, patch)
2017-03-14 13:25 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (90.77 KB, patch)
2017-03-27 12:00 UTC, Johan Larsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (91.35 KB, patch)
2017-03-30 12:24 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (92.35 KB, patch)
2017-07-04 14:15 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (68.58 KB, patch)
2018-03-07 10:16 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (68.40 KB, patch)
2018-04-25 19:22 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 14957 - Rule based permission for MARC fields (67.12 KB, patch)
2018-05-07 12:27 UTC, David Gustafsson
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Stenberg 2015-10-05 14:07:53 UTC
Add options for write protecting individual MARC fields based on source of import (Z39.50, stage for import, etc) to avoid overwriting local changes.

This feature is a superset of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11780
Comment 1 Katrin Fischer 2015-10-05 17:34:58 UTC
+100!
Comment 2 Martin Stenberg 2015-10-27 11:31:38 UTC Comment hidden (obsolete)
Comment 3 Martin Stenberg 2015-10-27 11:31:47 UTC
Created attachment 44057 [details] [review]
Bug 14957: Database changes
Comment 4 Martin Stenberg 2015-10-27 11:31:58 UTC
Created attachment 44058 [details] [review]
Bug 14957: Add filters to relevant scripts

Add MARCPermissions filters in scripts which modifies MARC records to enable the
permission system for these scripts.
Comment 5 Magnus Enger 2016-04-21 19:36:23 UTC
Just confirming the patches do not apply. Here is the error:

$ git bz apply 14957
Bug 14957 - Write protecting MARC fields based on source of import

44056 - Bug 14957: Rule based permission for MARC fields
44057 - Bug 14957: Database changes
44058 - Bug 14957: Add filters to relevant scripts

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 14957: Rule based permission for MARC fields
Using index info to reconstruct a base tree...
M	C4/Biblio.pm
M	koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
M	koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/cataloguing.pref
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/cataloguing.pref
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
Auto-merging C4/Biblio.pm
CONFLICT (content): Merge conflict in C4/Biblio.pm
Failed to merge in the changes.
Patch failed at 0001 Bug 14957: Rule based permission for MARC fields
The copy of the patch that failed is found in:
   /home/magnus/scripts/kohaclone/.git/rebase-apply/patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/Bug-14957-Rule-based-permission-for-MARC-fields-IZUxzW.patch
Comment 6 Eivin Giske Skaaren 2016-05-23 20:53:46 UTC
Created attachment 51723 [details] [review]
Bug 14957: Rule based permission for MARC fields

This patch adds a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Apply related database patch
    2. Apply related scripts update patch
    3. Run updatedatabase.pl
    5. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 7 Eivin Giske Skaaren 2016-05-23 20:59:31 UTC
Created attachment 51724 [details] [review]
Bug 14957: Correct parameters for TransformMarcToKoha
Comment 8 Eivin Giske Skaaren 2016-05-23 21:01:05 UTC
I have rebased and merged this and it seems to work. I dont have any MARC experience so please test that this works in some real use case.
Comment 9 David Cook 2016-05-24 23:38:09 UTC
Oh I had forgotten about this one... might have to look at this sometime...
Comment 10 Magnus Enger 2016-06-02 09:39:03 UTC
I am trying to test this, but after applying the patches (and fixing some conflicts), I get this error when I try to do a search, both in the OPAC and the intranet: 

Can't locate object method "fields" via package "DBI::db" at /home/kohaadmin/kohaclone/C4/Biblio.pm line 2673.

Line 2673 looks like this: 

  MARCFIELD: foreach my $field ( $record->fields() ) {

I'll try to figure out the problem, but if anyone got any hunches I'm all ears!
Comment 11 Magnus Enger 2016-06-02 10:13:34 UTC
Never mind me, I was applying the patches on the 3.22.x branch.
Comment 12 Magnus Enger 2016-06-02 11:56:29 UTC
When you are on the "MARC field permissions" page, "MARC field permissions" is not shown in the left hand menu.
Comment 13 Andreas Hedström Mace 2016-07-19 20:47:21 UTC
Failed to apply for me (merge conflicts). Needs a rebase again?
Comment 14 delaye 2016-10-12 12:28:50 UTC
i can't test the patch with a sandbox biblibre
i have this message
The sandbox you've requested is not ready.
Some problems occurred applying patches from bug 14957:
<h1>Something went wrong !</h1>Applying: Bug 14957: Database changes
Applying: Bug 14957: Add filters to relevant scripts
.git/rebase-apply/patch:14: trailing whitespace.
        } 
.git/rebase-apply/patch:16: trailing whitespace.
            ModBiblioframework( ,  ); 
warning: 2 lines add whitespace errors.
Using index info to reconstruct a base tree...
M	cataloguing/addbiblio.pl
M	koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/addbiblio.tt
M	misc/migration_tools/bulkmarcimport.pl
M	tools/batch_record_modification.pl
.git/rebase-apply/patch:14: trailing whitespace.
        } 
.git/rebase-apply/patch:16: trailing whitespace.
            ModBiblioframework( ,  ); 
warning: 2 lines add whitespace errors.
Falling back to patching base and 3-way merge...
Auto-merging tools/batch_record_modification.pl
CONFLICT (content): Merge conflict in tools/batch_record_modification.pl
Auto-merging misc/migration_tools/bulkmarcimport.pl
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/addbiblio.tt
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/addbiblio.tt
Auto-merging cataloguing/addbiblio.pl
error: Failed to merge in the changes.
Patch failed at 0001 Bug 14957: Add filters to relevant scripts
The copy of the patch that failed is found in: .git/rebase-apply/patch
When you have resolved this problem run git bz apply --continue.
If you would prefer to skip this patch, instead run git bz apply --skip.
To restore the original branch and stop patching run git bz apply --abort.
Bug 14957 - Write protecting MARC fields based on source of import

44057 - Bug 14957: Database changes
44058 - Bug 14957: Add filters to relevant scripts
51723 - Bug 14957: Rule based permission for MARC fields
51724 - Bug 14957: Correct parameters for TransformMarcToKoha

Apply? [(y)es, (n)o, (i)nteractive] Patch left in /tmp/Bug-14957-Add-filters-to-relevant-scripts-7vvvqA.patch .
Comment 15 Aleisha Amohia 2016-10-19 04:23:46 UTC
Created attachment 56637 [details] [review]
Bug 14957: Key fixes to patch

- Fixing merge conflicts
- adding String::Similariry to list of dependencies
- font-awesome icons and adding consistency to formatting
- removing original use of onclick
- moving help text out of main page and into a new help file

When testing, follow above test plan but:
1) Remember to install String::Similarity before testing (apt-get
    install libstring-similarity-perl)
2) Instead of doing step 7, click Help in top-right corner instead.
Confirm this works and shows as expected

Sponsored-by: Catalyst IT
Comment 16 Aleisha Amohia 2016-10-19 04:28:15 UTC
Created attachment 56638 [details] [review]
Bug 14957: Key fixes to patch

- Fixing merge conflicts
- adding String::Similariry to list of dependencies
- font-awesome icons and adding consistency to formatting
- removing original use of onclick
- moving help text out of main page and into a new help file

When testing, follow above test plan but:
1) Remember to install String::Similarity before testing (apt-get
    install libstring-similarity-perl)
2) Instead of doing step 7, click Help in top-right corner instead.
Confirm this works and shows as expected

Sponsored-by: Catalyst IT
Comment 17 Josef Moravec 2016-10-19 09:23:03 UTC
Created attachment 56643 [details] [review]
Bug 14957: Database changes
Comment 18 Josef Moravec 2016-10-19 09:23:11 UTC
Created attachment 56644 [details] [review]
Bug 14957: Add filters to relevant scripts

Add MARCPermissions filters in scripts which modifies MARC records to enable the
permission system for these scripts.
Comment 19 Josef Moravec 2016-10-19 09:23:18 UTC
Created attachment 56645 [details] [review]
Bug 14957: Rule based permission for MARC fields

This patch adds a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Apply related database patch
    2. Apply related scripts update patch
    3. Run updatedatabase.pl
    5. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 20 Josef Moravec 2016-10-19 09:23:29 UTC
Created attachment 56646 [details] [review]
Bug 14957: Correct parameters for TransformMarcToKoha
Comment 21 Josef Moravec 2016-10-19 09:23:34 UTC
Created attachment 56647 [details] [review]
Bug 14957: Key fixes to patch

- Fixing merge conflicts
- adding String::Similariry to list of dependencies
- font-awesome icons and adding consistency to formatting
- removing original use of onclick
- moving help text out of main page and into a new help file

When testing, follow above test plan but:
1) Remember to install String::Similarity before testing (apt-get
    install libstring-similarity-perl)
2) Instead of doing step 7, click Help in top-right corner instead.
Confirm this works and shows as expected

Sponsored-by: Catalyst IT
Comment 22 Josef Moravec 2016-10-19 09:23:42 UTC
Created attachment 56648 [details] [review]
Bug 14957: Followup - Pod, import_lexile.pl, template and test fixes
Comment 23 Josef Moravec 2016-10-19 09:24:34 UTC
Rebased on master

Fixed few issues in the follow-up
Comment 24 Andreas Hedström Mace 2016-11-02 21:32:04 UTC
Doesn't apply for me. Needs yet another rebase?
Comment 25 Josef Moravec 2016-11-03 07:06:07 UTC
Created attachment 57114 [details] [review]
Bug 14957: Database changes
Comment 26 Josef Moravec 2016-11-03 07:06:17 UTC
Created attachment 57115 [details] [review]
Bug 14957: Add filters to relevant scripts

Add MARCPermissions filters in scripts which modifies MARC records to enable the
permission system for these scripts.
Comment 27 Josef Moravec 2016-11-03 07:06:51 UTC
Created attachment 57116 [details] [review]
Bug 14957: Rule based permission for MARC fields

This patch adds a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Apply related database patch
    2. Apply related scripts update patch
    3. Run updatedatabase.pl
    5. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 28 Josef Moravec 2016-11-03 07:06:59 UTC
Created attachment 57117 [details] [review]
Bug 14957: Correct parameters for TransformMarcToKoha
Comment 29 Josef Moravec 2016-11-03 07:07:05 UTC
Created attachment 57118 [details] [review]
Bug 14957: Key fixes to patch

- Fixing merge conflicts
- adding String::Similariry to list of dependencies
- font-awesome icons and adding consistency to formatting
- removing original use of onclick
- moving help text out of main page and into a new help file

When testing, follow above test plan but:
1) Remember to install String::Similarity before testing (apt-get
    install libstring-similarity-perl)
2) Instead of doing step 7, click Help in top-right corner instead.
Confirm this works and shows as expected

Sponsored-by: Catalyst IT
Comment 30 Josef Moravec 2016-11-03 07:07:11 UTC
Created attachment 57119 [details] [review]
Bug 14957: Followup - Pod, import_lexile.pl, template and test fixes
Comment 31 Josef Moravec 2016-11-03 07:08:17 UTC
It really needed rebasing again. Done.
Comment 32 Andreas Hedström Mace 2016-12-13 22:42:34 UTC
I started testing this today and after some initial confusion about the rules system (that could use some more documentation!) it seems mostly to work as intended. But am I correct in understanding that Marc field permissions does not currently work with Staged MARC Imports?
Comment 33 David Gustafsson 2017-02-02 12:44:21 UTC
Created attachment 59811 [details] [review]
Rule based permission for MARC fields

Rebased against master as single patch. Behaviour of filter modules slightly tweaked.
Comment 34 David Gustafsson 2017-02-02 12:49:03 UTC
Created attachment 59812 [details] [review]
Rule based permission for MARC fields

Oops, forgot to remove some temporary comments.
Comment 35 David Gustafsson 2017-02-02 13:10:51 UTC
The previously posted version of this patch at least for me did not apply cleanly. Rebased against current HEAD of master, and also slightly altered the behavior of "GetMarcPermissionsAction". The previous behaviour struck me as odd since a module with a higher specificity will always override modules with lower, even for the case when a filter argument for a module with lower specificity has an exact match. So for example, if we have a rule with filter 'source' => 'bulkmarcimport', and another for 'borrower' => '*', the later rule will always have precedence, even when source is matching 'bulkmarcimport'. I seems more flexible to have the option to override on a more specific filter match. 

Generally I think this patch is great, and almost exactly what we need, but the whole filter-system I think is in need of some revamping. It would be nice if the modules where more like separate namespaces, so that one set of rules (belonging to a certain module) was applied at once. Right now rules from several modules might be mixed in, and field-specificity has precedence in rules selection. The code could be simplified and optimized if rules where grouped by module instead (and also easier to admin from a user perspective I would believe, since rules precedence would be more obvious. But have made no such changes in the current patch, as wanted to bring this up for discussion before make any large breaking changes.
Comment 36 Andreas Hedström Mace 2017-02-03 09:38:14 UTC
Patch doesn't apply for my (using Kohadevbox). I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/git-bz", line 2630, in <module>
    do_apply(*args)
  File "/usr/local/bin/git-bz", line 1755, in do_apply
    users = bug.server.get_xmlrpc_proxy().User.get({ 'names': [patch.attacher] })['users']
  File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1591, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1273, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1321, in single_request
    response.msg,
xmlrpclib.ProtocolError: <ProtocolError for bugs.koha-community.org/xmlrpc.cgi: 404 Not Found>
Comment 37 Magnus Enger 2017-02-03 09:49:45 UTC
(In reply to Andreas Hedström Mace from comment #36)
> Patch doesn't apply for my (using Kohadevbox). I get the following error:
...
> xmlrpclib.ProtocolError: <ProtocolError for
> bugs.koha-community.org/xmlrpc.cgi: 404 Not Found>

That looks more like an error from git bz than from git itself. 

404 Not Found - are you sure git bz is setup properly and has all the configs it needs?
Comment 38 David Gustafsson 2017-02-03 12:39:06 UTC
I think I might not have followed the guidelines for submitting patches properly. Will try to see if I can generate a patch using the recommended tools (git bz etc).
Comment 39 Magnus Enger 2017-02-03 12:48:25 UTC
(In reply to David Gustafsson from comment #38)
> I think I might not have followed the guidelines for submitting patches
> properly. Will try to see if I can generate a patch using the recommended
> tools (git bz etc).

Indeed, it looks more like a regular diff than a git patch. It seems odd that this should trigger the error that Andreas got, though. But I tested, an I get the same error on this patch while other patches work as they should. A new patch looks like a good place to start, though. :-)
Comment 40 Andreas Hedström Mace 2017-02-03 13:42:16 UTC
(In reply to Magnus Enger from comment #39)
> Indeed, it looks more like a regular diff than a git patch. It seems odd
> that this should trigger the error that Andreas got, though. But I tested,
> an I get the same error on this patch while other patches work as they
> should. A new patch looks like a good place to start, though. :-)

Yeah, as I have never seen such an error either I made sure to check that other patches applied as intended, which they did. Sounds great if you can provide a follow-up David!
Comment 41 David Gustafsson 2017-02-03 14:09:20 UTC
Created attachment 59885 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 42 David Gustafsson 2017-02-03 14:13:50 UTC
Ok! One more try. A little bit unsure how to handle attribution, right now looks like the patch is attributed to me which is wrong. But hope it will at least apply now (also hope I did not make any mistake during the rebase, when through the patch several times to check everything adds up, but still might have botched something up).
Comment 43 David Gustafsson 2017-02-27 13:09:39 UTC
Created attachment 60711 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 44 David Gustafsson 2017-02-27 13:10:37 UTC
Rebased against master.
Comment 45 David Gustafsson 2017-03-06 09:59:24 UTC
Created attachment 60824 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 46 David Gustafsson 2017-03-06 10:00:30 UTC
Screwed up the last merge. Should be fixed now.
Comment 47 David Gustafsson 2017-03-06 10:03:43 UTC
Nope, still managed to screw it up. Should be ok now.
Comment 48 David Gustafsson 2017-03-06 10:04:35 UTC
Created attachment 60825 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 49 David Gustafsson 2017-03-09 14:06:35 UTC
Have discovered a lot of weirdness in how permissions are applied, not sure if this is something I have introduces or present in the original patch, but don't merge in this quite yet. Will update the patch some time in the coming days.
Comment 50 David Gustafsson 2017-03-14 13:25:23 UTC
Created attachment 61065 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 51 David Gustafsson 2017-03-14 13:32:02 UTC
Made some small adjustments:

- Removed $options-{filter} arguments from ModBiblioMarc in AddBiblio since ApplyMarcPermissions is never run on adding new biblios anyways (since no caller uses it, and marc-xml for current biblionumber will always be empty on save and ApplyMarcPermissions will bail out anyway), and to provide that arguments makes things even more confusing.

- Added source => 'batchimport' for staged import batches. So now ApplyMarcPermissions runs also on staged imports. One thing I did not realize before is that marc permissions are never enforced adding new biblios, this is also the case for staged imports. Only if a match is found permissions will be applied. It would probably be more appropriate to call the current system "Marc merge rules" or something like that, since it is really a framework for defining what happens when two marc records are to be merged (and not "permissions" in the strict sense of the word).
Comment 52 Johan Larsson 2017-03-27 12:00:29 UTC
Created attachment 61641 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administration".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 53 Johan Larsson 2017-03-27 12:01:57 UTC
corrected spelling error in commit message.
Comment 54 David Gustafsson 2017-03-30 12:24:11 UTC
Created attachment 61736 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 55 David Gustafsson 2017-03-30 12:30:45 UTC Comment hidden (obsolete)
Comment 56 David Gustafsson 2017-03-30 12:33:01 UTC
Ops, wrong copy paste. Fixed missing "String::Similarity" include in Biblio.pm, and changed $filter argument in ModBiblio to more flexible $options to avoid future merge conflicts and provide a more stable API.
Comment 57 David Gustafsson 2017-07-04 14:15:50 UTC
Created attachment 64792 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 58 David Gustafsson 2017-07-04 15:04:03 UTC
The last change (adding the $options parameter) contained several mistakes, (hopefully) fixed those now. Have run the tests and did some manual testing (which I apparently did not do last time).
Comment 59 Andreas Hedström Mace 2017-08-25 10:30:32 UTC
Wanted to to try and test this again, but patch doesn't apply for me.
Comment 60 David Gustafsson 2018-01-25 17:31:25 UTC
Have discovered several serious issues with the current patch in this issue, so do not recommend that any one uses it or re-roll it against master. My personal view is that the current approach is based on some invalid assumptions (that positions of fields in marc is stable and carry significance for example). Have rewritten most of the core logic, and could perhaps submit it here when cleaned up. But for our needs (Gothenburg University Library) I'm not sure we want to get this in to Koha, but instead refactor and use as a plugin, but because of time-constraints I'm not sure will have time to submit the code for at least a couple of weeks.
Comment 61 Katrin Fischer 2018-01-30 07:20:41 UTC
Hi David, we've always been keen on this one. If you have something to submit, I'd be interested in testing. As we work in a union catalog setting (union catalog is 'master'), Koha's missing ability to protect fields and subfields from being overwritten is sometimes a problem for us.
Comment 62 David Gustafsson 2018-01-30 10:31:53 UTC
Hi! The code can be found here, but there are some parts (help section, tests, user UI) that has not really been adjusted for the new code. Have performed some limited manual testing and seems to work, but it will probably be confusing to configure for someone who has not looked through the code first: https://github.com/ub-digit/Koha/tree/bug_14957

I will try to clean up, make ui more intuitive, and document the functionality, but right now as previously noted there is not enough time for me to do this.
Comment 63 Katrin Fischer 2018-02-22 21:52:39 UTC
Hi David, thx for the feedback. Moving this to Failed QA as it seems more appropriate for now.
Comment 64 David Gustafsson 2018-03-07 10:16:36 UTC
Created attachment 72498 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Comment 65 paxed 2018-03-07 10:25:09 UTC
Comment on attachment 72498 [details] [review]
Bug 14957 - Rule based permission for MARC fields

$(this).attr('title', 'Invalid regular expression: ' + e.message);

should be

$(this).attr('title', _("Invalid regular expression: %s").format(e.message));
Comment 66 David Gustafsson 2018-03-07 10:26:36 UTC
Attaching the patch with latest changes even though not yet ready for sign off to not block any further contributions by someone else.

Since the model for how rules work has changed since the "old" version of the patch tests and help pages needs updating to reflect these changes, which has not been done.

I will have time to work some more on this in about a month or so, but since the patch is functional for our library in it's current state it will not get much more love from us until then.
Comment 67 Katrin Fischer 2018-04-01 14:00:55 UTC
Hi David, thx for submitting. Peeking at the code I am not quite sure how it works, but notice it adds a lot of code to C4. Would probably need quite a bit of work.
Comment 68 David Gustafsson 2018-04-25 19:22:47 UTC
Created attachment 74855 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Sponsored-by: Catalyst IT
Sponsored-by: Gothenbug University Library
Comment 69 David Gustafsson 2018-04-25 19:23:59 UTC
Removed String::Similarity dependency and fixed an issue with the field data hashing function.
Comment 70 David Gustafsson 2018-04-25 19:25:53 UTC
@Katrin Fischer Yes it could use some work, still have not gotten around to updating the help-pages and tests. We are currently running this in production though and has been doing so for a couple of weeks. It could take some time though before have time to fix the above.
Comment 71 David Gustafsson 2018-05-07 12:27:03 UTC
Created attachment 75118 [details] [review]
Bug 14957 - Rule based permission for MARC fields

Add a rule based permission system for MARC field modifications.

Test plan:
    1. Apply this patch
    2. Run updatedatabase.pl
    3. Log in to staff client
    4. Enable new syspref MARCPermissions
    5. You should see a new link "MARC field permissions" in the "Catalog"
       section under "Koha administrtion".
    6. Go to the "MARC field permissions" page
    7. Next to the page heading ("Manage MARC field permissions") there is a
       button with a question mark, click this for more information about the
       rule system.
    8. Add desired rules to the rule table, press the +-button to add.
    9. Clicking the pen-icon (edit) should allow you to edit corresponding rule.
    10. Clicking the x-icon should remove corresponding rule after confirmation.
    11. Selecting one or more rules followed by clicking the trash-icon
        should remove all selected rules after confirmation.
    12. To test your rules, import or in other way modify a record.
    13. Check that the record was modified as defined by your rules.

Sponsored-by: Halland County Library
Sponsored-by: Catalyst IT
Sponsored-by: Gothenbug University Library
Comment 72 David Gustafsson 2018-05-07 12:30:30 UTC
Rebase against master and remove (probably last of) String::Similarity related code. Have not had time to work on the other issues. We have been running this in production for a couple of weeks now, and have not experienced any issues.
Comment 73 David Gustafsson 2018-05-07 12:33:04 UTC
Which I now saw I already wrote about in previous comment. Still need to fix tests and documentation as the bare minimum before ready for sign-off.