Bug 29781 - Allow item batch modification to use capturing groups in regex option
Summary: Allow item batch modification to use capturing groups in regex option
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Martin Renvoize
QA Contact: Nick Clemens
URL:
Keywords: Manual, release-notes-needed
Depends on:
Blocks:
 
Reported: 2022-01-03 19:13 UTC by Andrew Fuerste-Henry
Modified: 2022-12-12 21:23 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement adds support for capture groups in the regular expression option of batch item modification. One may now use `(\d+)` for example to capture digits in the match and then the `$1` placeholder in the replace.
Version(s) released in:
22.05.00


Attachments
Bug 29781: This patch adds a unit test for capture groups support in Koha::Items (2.08 KB, patch)
2022-01-17 13:52 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 29781: Enable capture groups in batch item modification (2.55 KB, patch)
2022-01-17 13:52 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 29781: Unit test (2.09 KB, patch)
2022-01-17 13:53 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 29781: Enable capture groups in batch item modification (2.55 KB, patch)
2022-01-17 13:53 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 29781: Unit test (2.15 KB, patch)
2022-01-18 14:50 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 29781: Enable capture groups in batch item modification (2.61 KB, patch)
2022-01-18 14:50 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 29781: Unit test (2.20 KB, patch)
2022-02-11 14:47 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 29781: Enable capture groups in batch item modification (2.66 KB, patch)
2022-02-11 14:47 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 29781: Unit test (2.25 KB, patch)
2022-02-11 15:24 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 29781: Enable capture groups in batch item modification (2.71 KB, patch)
2022-02-11 15:24 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Andrew Fuerste-Henry 2022-01-03 19:13:38 UTC
Bug 23873 seeks to make capturing groups usable in batch record mod. We should do the same with items.
Comment 1 Martin Renvoize 2022-01-17 13:52:19 UTC
Created attachment 129526 [details] [review]
Bug 29781: This patch adds a unit test for capture groups support in Koha::Items

Test plan
1) Run the test prior to applying the capture groups fix, note it fails.
2) Apply the capture groups fix patch
3) Run the test again, it should now pass
Comment 2 Martin Renvoize 2022-01-17 13:52:23 UTC
Created attachment 129527 [details] [review]
Bug 29781: Enable capture groups in batch item modification

This patch adds support for regular expression capture groups in
MarcModificationTemplates.

We escape double quotes in the replacement string, then quote the whole
string before applying the `ee` modifier to the final regex
application.

See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variables/
for further details.

Test plan
1) Attempt to use capture groups in your regular expression based
   batch item modification and note it does not work.
2) Apply the patch and try again, this time the capture group
   should yield the expected results.
3) Run t/SimpleMARC.t and confirm the tests still pass.
Comment 3 Martin Renvoize 2022-01-17 13:53:23 UTC
Created attachment 129528 [details] [review]
Bug 29781: Unit test

This patch adds a unit test for capture groups support in Koha::Items

Test plan
1) Run the test prior to applying the capture groups fix, note it fails.
2) Apply the capture groups fix patch
3) Run the test again, it should now pass
Comment 4 Martin Renvoize 2022-01-17 13:53:26 UTC
Created attachment 129529 [details] [review]
Bug 29781: Enable capture groups in batch item modification

This patch adds support for regular expression capture groups in
MarcModificationTemplates.

We escape double quotes in the replacement string, then quote the whole
string before applying the `ee` modifier to the final regex
application.

See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variables/
for further details.

Test plan
1) Attempt to use capture groups in your regular expression based
   batch item modification and note it does not work.
2) Apply the patch and try again, this time the capture group
   should yield the expected results.
3) Run t/SimpleMARC.t and confirm the tests still pass.
Comment 5 Andrew Fuerste-Henry 2022-01-18 13:58:14 UTC
A quick hypothetical example using this to find call numbers that don't have volume and copy numbers in the correct order.

To test:
1: have 2 items with these call numbers:
 - KF 123 .A987 c. 11 v. 22
 - KF 123 .A987 v. 3 c. 1
2: put those items in item batch mod
3: do a regex substitution on call number:
 - s/(^[A-Z]+ \d+\b)\s+(\.[A-Z]+\d+\b)\s+(c\.\s\d+\b)\s+(v\.\s\d+\b)/$1 $2 $4 $3/
4: Observe your call numbers are now:
 - KF 123 .A987 v. 22 c.11
 - KF 123 .A987 v. 3 c. 1
Comment 6 Andrew Fuerste-Henry 2022-01-18 14:50:42 UTC
Created attachment 129573 [details] [review]
Bug 29781: Unit test

This patch adds a unit test for capture groups support in Koha::Items

Test plan
1) Run the test prior to applying the capture groups fix, note it fails.
2) Apply the capture groups fix patch
3) Run the test again, it should now pass

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 7 Andrew Fuerste-Henry 2022-01-18 14:50:45 UTC
Created attachment 129574 [details] [review]
Bug 29781: Enable capture groups in batch item modification

This patch adds support for regular expression capture groups in
MarcModificationTemplates.

We escape double quotes in the replacement string, then quote the whole
string before applying the `ee` modifier to the final regex
application.

See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variables/
for further details.

Test plan
1) Attempt to use capture groups in your regular expression based
   batch item modification and note it does not work.
2) Apply the patch and try again, this time the capture group
   should yield the expected results.
3) Run t/SimpleMARC.t and confirm the tests still pass.

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 8 AspenCat Team 2022-01-19 21:39:54 UTC
Signing off.
Comment 9 Nick Clemens 2022-02-11 14:47:46 UTC
Created attachment 130498 [details] [review]
Bug 29781: Unit test

This patch adds a unit test for capture groups support in Koha::Items

Test plan
1) Run the test prior to applying the capture groups fix, note it fails.
2) Apply the capture groups fix patch
3) Run the test again, it should now pass

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Jo Hunter <jhunter@clicweb.org>
Comment 10 Nick Clemens 2022-02-11 14:47:50 UTC
Created attachment 130499 [details] [review]
Bug 29781: Enable capture groups in batch item modification

This patch adds support for regular expression capture groups in
MarcModificationTemplates.

We escape double quotes in the replacement string, then quote the whole
string before applying the `ee` modifier to the final regex
application.

See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variables/
for further details.

Test plan
1) Attempt to use capture groups in your regular expression based
   batch item modification and note it does not work.
2) Apply the patch and try again, this time the capture group
   should yield the expected results.
3) Run t/SimpleMARC.t and confirm the tests still pass.

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Jo Hunter <jhunter@clicweb.org>
Comment 11 Nick Clemens 2022-02-11 15:24:27 UTC
Created attachment 130508 [details] [review]
Bug 29781: Unit test

This patch adds a unit test for capture groups support in Koha::Items

Test plan
1) Run the test prior to applying the capture groups fix, note it fails.
2) Apply the capture groups fix patch
3) Run the test again, it should now pass

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Jo Hunter <jhunter@clicweb.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 12 Nick Clemens 2022-02-11 15:24:31 UTC
Created attachment 130509 [details] [review]
Bug 29781: Enable capture groups in batch item modification

This patch adds support for regular expression capture groups in
MarcModificationTemplates.

We escape double quotes in the replacement string, then quote the whole
string before applying the `ee` modifier to the final regex
application.

See https://blog.james.rcpt.to/2010/10/25/perl-search-and-replace-using-variables/
for further details.

Test plan
1) Attempt to use capture groups in your regular expression based
   batch item modification and note it does not work.
2) Apply the patch and try again, this time the capture group
   should yield the expected results.
3) Run t/SimpleMARC.t and confirm the tests still pass.

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Jo Hunter <jhunter@clicweb.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 13 Nick Clemens 2022-02-11 15:26:30 UTC
I tried executing code and couldn't exploit it

perl -e 'use Koha::Items; warn Koha::Items::apply_regex({value=>q{KF 123 .A987 c. 11  v. 22},search=>q{(^[A-Z]+ \d+\b)\s+(\.[A-Z]+\d+\b)\s+(c\.\s\d+\b)\s+(v\.\s\d+\b)}, replace=>q{ $1 $2 $4 $3 object_class() }});'

If I comment out the lines that quote the replace I get 'Koha::Items' - so that protection seems to work
Comment 14 Fridolin Somers 2022-02-16 20:58:40 UTC
Great :) Looking forward to see Bug 23873 in passed QA
Comment 15 Fridolin Somers 2022-02-16 20:59:48 UTC
It will be important to document this.
Comment 16 Fridolin Somers 2022-02-16 21:52:30 UTC
Pushed to master for 22.05, thanks to everybody involved 🦄