Bug 21959

Summary: Add ability to apply regular expressions to text fields in the batch item modification tool
Product: Koha Reporter: Alex Arnaud <alex.arnaud>
Component: ToolsAssignee: Alex Arnaud <alex.arnaud>
Status: CLOSED FIXED QA Contact: Jonathan Druart <jonathan.druart>
Severity: enhancement    
Priority: P5 - low CC: 1joynelson, andrewfh, anne-cecile.bories, black23, e.betemps, jonathan.druart, josef.moravec, katrin.fischer, martin.renvoize, nick, oleonard, pierre-marc.thibault, sandboxes, severine.queune, victor
Version: master   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23873
Change sponsored?: Sponsored Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
This adds a 'RegEx' link to the fields on the batch item modification form that allows you to rewrite the content of the fields using regular expressions. For example this could be used to add prefix or suffixes to callnumbers and barcodes or to rewrite item URLs.
Version(s) released in:
20.05.00
Bug Depends on:    
Bug Blocks: 27929    
Attachments: Bug 21959: Batch item modification - ability to apply regular expression on text fields
Bug 21959: Batch item modification - ability to apply regular expression on text fields
Bug 21959: Batch item modification - ability to apply regular expression on text fields
Bug 21959: Batch item modification - ability to apply regular expression on text fields
Bug 21959: use a link instead of a checkbox
Bug 21959: allow only i and g regex modifiers
Bug 21959: Batch item modification - ability to apply regular expression on text fields
Bug 21959: use a link instead of a checkbox
Bug 21959: allow only i and g regex modifiers
Bug 21959: Batch item modification - ability to apply regular expression on text fields
Bug 21959: use a link instead of a checkbox
Bug 21959: allow only i and g regex modifiers
Bug 21959: Batch item modification - ability to apply regular expression on text fields
Bug 21959: use a link instead of a checkbox
Bug 21959: allow only i and g regex modifiers

Description Alex Arnaud 2018-12-06 08:54:14 UTC
Here we want the possibility to make regex on text (type text or textarea) fields in the batch item modification form.
Comment 1 Alex Arnaud 2018-12-06 09:07:02 UTC Comment hidden (obsolete)
Comment 2 Pierre-Marc Thibault 2018-12-07 21:48:14 UTC
When I use regex on 2 items that have the same value on a field and their field changes, it's said that 2 fields have been modified. 

Ex : 1) It's a note. 2) It's a note.
s/note/note2/g

However, if it's not the same value, it's said that 3 fields have been modified.

Ex : 1) It's a note. 2) This is a note.
s/note/note2/g

Can you please check why it happens?
Comment 3 Alex Arnaud 2018-12-19 15:08:47 UTC
Created attachment 83392 [details] [review]
Bug 21959: Batch item modification - ability to apply regular expression on text fields

Test plan:
  - go to Tools > Batch item modification,
  - select items to modify,
  - for one or some text fields, check the second checkbox (Use regular
    expression) and type your regex,
  - check the regex has been applied.
Comment 4 Alex Arnaud 2018-12-19 15:10:19 UTC
(In reply to Pierre-Marc Thibault from comment #2)
> When I use regex on 2 items that have the same value on a field and their
> field changes, it's said that 2 fields have been modified. 
> 
> Ex : 1) It's a note. 2) It's a note.
> s/note/note2/g
> 
> However, if it's not the same value, it's said that 3 fields have been
> modified.
> 
> Ex : 1) It's a note. 2) This is a note.
> s/note/note2/g
> 
> Can you please check why it happens?

Please try the replacement patch. The code may be quite tricky but it should work.
Comment 5 Pierre-Marc Thibault 2018-12-21 21:00:03 UTC
Created attachment 83470 [details] [review]
Bug 21959: Batch item modification - ability to apply regular expression on text fields

Test plan:
  - go to Tools > Batch item modification,
  - select items to modify,
  - for one or some text fields, check the second checkbox (Use regular
    expression) and type your regex,
  - check the regex has been applied.

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Comment 6 Katrin Fischer 2019-02-02 21:58:17 UTC
Some things:

1) Tiny typo in "subflied":

$(this).parent().find('.hint').html(_("Enter a regular expression for this subflied"));

2) I am not happy witht he GUI. The delete checkbox always takes some time explaining and I think having now 2 unlabelled checkboxes makes it worse. Maybe a link or small button "RegEx" would work? Adding Owen to see if he has an idea.

3) During our discussion on bug 15836 we came to the conclusion that a separate permission should be added for that page as there are some potential issues with the regex. In our libraries a lot of people have item batch edit permissions, as this is a super useful tool. In wonder if we need to take extra precautions here. Adding some people in hope for a second opinion.
Comment 7 Katrin Fischer 2019-02-02 21:58:44 UTC
I should also say: I've tested it a bit and it all worked ok so far.
Comment 8 Owen Leonard 2019-02-05 15:56:00 UTC
I tried to test this but I keep getting an error: "Can't use string ("0") as an ARRAY ref while "strict refs" in use at /home/vagrant/kohaclone/tools/batchMod.pl line 279, <$fh> line 100."
Comment 9 Owen Leonard 2019-02-05 16:03:17 UTC
(In reply to Owen Leonard from comment #8)
> I tried to test this but I keep getting an error

Never mind, that error is unrelated.
Comment 10 Jonathan Druart 2019-02-16 16:36:59 UTC
(In reply to Katrin Fischer from comment #6)
> Some things:
> 
> 1) Tiny typo in "subflied":
> 
> $(this).parent().find('.hint').html(_("Enter a regular expression for this
> subflied"));
> 
> 2) I am not happy witht he GUI. The delete checkbox always takes some time
> explaining and I think having now 2 unlabelled checkboxes makes it worse.
> Maybe a link or small button "RegEx" would work? Adding Owen to see if he
> has an idea.

Definitely, we should not have a second checkbox

> 3) During our discussion on bug 15836 we came to the conclusion that a
> separate permission should be added for that page as there are some
> potential issues with the regex. In our libraries a lot of people have item
> batch edit permissions, as this is a super useful tool. In wonder if we need
> to take extra precautions here. Adding some people in hope for a second
> opinion.

Nope, not in this case, see how MMT works.
However we should list the available modifiers, see Koha/SimpleMARC.pm:627
  my @available_modifiers = qw( i g );

4) ident must be 4 spaces

5) why do you use eval for the regex? Same as before, I feel like you should mimick Koha::SimpleMARC::_modify_values
Comment 11 Alex Arnaud 2019-04-02 14:15:16 UTC
Created attachment 87323 [details] [review]
Bug 21959: Batch item modification - ability to apply regular expression on text fields

Test plan:
  - go to Tools > Batch item modification,
  - select items to modify,
  - for one or some text fields, check the second checkbox (Use regular
    expression) and type your regex,
  - check the regex has been applied.

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>
Comment 12 Alex Arnaud 2019-04-02 14:15:28 UTC
Created attachment 87324 [details] [review]
Bug 21959: use a link instead of a checkbox
Comment 13 Alex Arnaud 2019-04-02 14:15:35 UTC
Created attachment 87325 [details] [review]
Bug 21959: allow only i and g regex modifiers
Comment 14 Alex Arnaud 2019-04-02 14:25:49 UTC
(In reply to Jonathan Druart from comment #10)
> 
> 5) why do you use eval for the regex? Same as before, I feel like you should
> mimick Koha::SimpleMARC::_modify_values

This is the only way i've found to make the regex work with modifiers in a variable. But i don't need it anymore since i've limited modifiers to i and g like in Koha::SimpleMARC::_modify_values

I re-used a part of the code in Koha::SimpleMARC::_modify_values. And i don't like code duplicating so much. So i'm thinking about a code refactoring:

 - directly use _modify_values (and take into account specificities),
 - create a common apply_regex() sub with unit tests. But where ? In Koha/Util/* ? and use it in _modify_values
 - other ?

Any thougths ?
Comment 15 Séverine Queune 2019-04-16 09:10:11 UTC
I can definitely not answer Alex's question, but only had an opinion : that's a great enhancement !
Unfortunately, I’m really bad at regular expressions (but this feature is to good motivation to learn !), do you have some examples or reference pages to test further ?
Comment 16 Victor Grousset/tuxayo 2019-07-08 15:52:29 UTC
(In reply to Séverine Queune from comment #15)
> Unfortunately, I’m really bad at regular expressions (but this feature is to
> good motivation to learn !), do you have some examples or reference pages to
> test further ?

Thanks for your interest in this :)

You can try the following regex «s/foo/bar/g»
In the content of the subfield, it will replace every occurrences of «foo» by «bar»
Comment 17 Michal Denar 2019-07-09 19:58:04 UTC
Hi guys,
is it this patch ready to test or we should change statu to "In duscussion"?

Thank you
Comment 18 Biblibre Sandboxes 2019-09-17 10:32:11 UTC
Patch tested with a sandbox, by Mathilde Formery <mathilde.formery@ville-nimes.fr>
Comment 19 Biblibre Sandboxes 2019-09-17 10:32:51 UTC
Created attachment 92870 [details] [review]
Bug 21959: Batch item modification - ability to apply regular expression on text fields

Test plan:
  - go to Tools > Batch item modification,
  - select items to modify,
  - for one or some text fields, check the second checkbox (Use regular
    expression) and type your regex,
  - check the regex has been applied.

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>
Comment 20 Biblibre Sandboxes 2019-09-17 10:32:54 UTC
Created attachment 92871 [details] [review]
Bug 21959: use a link instead of a checkbox

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>
Comment 21 Biblibre Sandboxes 2019-09-17 10:32:57 UTC
Created attachment 92872 [details] [review]
Bug 21959: allow only i and g regex modifiers

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>
Comment 22 Katrin Fischer 2019-10-03 11:17:13 UTC
Can you please rebase? Should be easy, but don't want to risk it:

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 21959: Batch item modification - ability to apply regular expression on text fields
Using index info to reconstruct a base tree...
M	koha-tmpl/intranet-tmpl/prog/en/modules/tools/batchMod-edit.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/tools/batchMod-edit.tt
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/tools/batchMod-edit.tt
error: Failed to merge in the changes.
Patch failed at 0001 Bug 21959: Batch item modification - ability to apply regular expression on text fields
The copy of the patch that failed is found in: .git/rebase-apply/patch
When you have resolved this problem run "git bz apply --continue".
If you would prefer to skip this patch, instead run "git bz apply --skip".
To restore the original branch and stop patching run "git bz apply --abort".
Patch left in /tmp/Bug-21959-Batch-item-modification---ability-to-app-HQtELZ.patch
Comment 23 Alex Arnaud 2019-11-14 16:23:53 UTC
Created attachment 95432 [details] [review]
Bug 21959: Batch item modification - ability to apply regular expression on text fields

Test plan:
  - go to Tools > Batch item modification,
  - select items to modify,
  - for one or some text fields, check the second checkbox (Use regular
    expression) and type your regex,
  - check the regex has been applied.

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>
Comment 24 Alex Arnaud 2019-11-14 16:24:03 UTC
Created attachment 95433 [details] [review]
Bug 21959: use a link instead of a checkbox

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>
Comment 25 Alex Arnaud 2019-11-14 16:24:12 UTC
Created attachment 95434 [details] [review]
Bug 21959: allow only i and g regex modifiers

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>
Comment 26 Alex Arnaud 2019-11-14 16:25:31 UTC
Patch rebased
Comment 27 Jonathan Druart 2019-12-26 11:49:58 UTC
Created attachment 96636 [details] [review]
Bug 21959: Batch item modification - ability to apply regular expression on text fields

Test plan:
  - go to Tools > Batch item modification,
  - select items to modify,
  - for one or some text fields, check the second checkbox (Use regular
    expression) and type your regex,
  - check the regex has been applied.

Signed-off-by: Pierre-Marc Thibault <pierre-marc.thibault@inLibro.com>

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 28 Jonathan Druart 2019-12-26 11:50:02 UTC
Created attachment 96637 [details] [review]
Bug 21959: use a link instead of a checkbox

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 29 Jonathan Druart 2019-12-26 11:50:06 UTC
Created attachment 96638 [details] [review]
Bug 21959: allow only i and g regex modifiers

Signed-off-by: Mathilde Formery <mathilde.formery@ville-nimes.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 30 Victor Grousset/tuxayo 2020-01-03 19:01:02 UTC
Additional info: It's sponsored by the City of Nîmes.
Comment 31 Martin Renvoize 2020-01-08 14:43:38 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 32 Martin Renvoize 2020-01-09 16:56:00 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 33 Joy Nelson 2020-01-18 15:53:57 UTC
Enhancement not backported to 19.11.x branch.