This seems to work quite well! Also tested renew through checkout, renewal tab and OPAC.

When renewing the item from the list of checkouts and using the renewal limit override functionality, the message displayed is:

Renew failed: item_denied_renewal

This could be improved, maybe with a small follow-up?

Created attachment 68146 [details] [review]
Bug 15494 - Unit tests

prove t/db_dependent/Circulation.t

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 68147 [details] [review]
Bug 15494 - Add _items_denied_renewal sub to C4:Circulation::CanBookBeRenewed

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 68148 [details] [review]
Bug 15494 - Update display of renewal errors

To test:
1 - Define a rule to deny renewal
2 - Checkout an item meeting this rule
3 - Attempt renew an item via the OPAC
4 - Attempt renewal via SIP
5 - Attempt renewal via staff client
6 - Verify errors are reasonable

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 68149 [details] [review]
Bug 15494 - Add syspref

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 70295 [details] [review]
Bug 15494 - Followup: Make sure items in tests are not overdue

prove -v t/db_dependent/Circulation.t

+ unless ($item)    { carp "No item passed in!" && return }
Not our preferred coding style

+ return () if (! $yaml =~ /\S/ );
Just return here. The only call is in scalar context. We do not need a list.
Shorter: return if $yaml !~ /\S/;

+    eval {
+        $denyingrules = YAML::Load($yaml);
+    };
How much time are we spending on doing this? What is the performance effect? If the pref always looks like field1: [val]\nfield2: [val2] we can just split on newline and split on ':', etc. Give me some convincing numbers please.

+        return ();
Same as above

+        my $val;
+        if (exists $item->{$field}) {
+            $val = $item->{$field};
+        }
+        $val = '' unless defined $val;
Why not in just one line: my $val = $item->{$field} // '';

+            - How to write these rules is documented on the <a href="http://wiki.koha-community.org/wiki/OpacHiddenItems" target="_blank">Koha wiki</a>.
This is from the preference file. Not sure if this is the preferred way. A wiki page could be gone just as easily. Why not just provide an example as close as possible to the pref itself. Just another 40 chars ?

Just curious too if this supports denying renewal on an empty string? If so, the conversion from undef to empty string might not be what we want. Can we signal empty string or undef to be a blocking value? [No blocker for me btw :)]

Changing status. Please make some adjustments.

Created attachment 70629 [details] [review]
Bug 15494 - Unit tests

prove -v t/db_dependent/Circulation.t

Created attachment 70630 [details] [review]
Bug 15494 - Add _items_denied_renewal sub to C4:Circulation::CanBookBeRenewed

Created attachment 70631 [details] [review]
Bug 15494 - Update display of renewal errors

To test:
1 - Define a rule to deny renewal
2 - Checkout an item meeting this rule
3 - Attempt renew an item via the OPAC
4 - Attempt renewal via SIP
5 - Attempt renewal via staff client
6 - Verify errors are reasonable

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 70632 [details] [review]
Bug 15494 - Add syspref

(In reply to Marcel de Rooy from comment #21)
> + unless ($item)    { carp "No item passed in!" && return }
> Not our preferred coding style

This was just copying others in this file, changed to "return unless $item"

> + return () if (! $yaml =~ /\S/ );
> Just return here. 
Removed

> +    eval {
> +        $denyingrules = YAML::Load($yaml);
> +    };
> How much time are we spending on doing this? What is the performance effect?
Was about .037 seconds (with 2 fields and 1000 values)
Using split is about .014 seconds, okay, I switched

> +        return ();
Yup

> Why not in just one line: my $val = $item->{$field} // '';
Good point

>Why not just provide an example as close as possible to the pref itself?

Okay

> Just curious too if this supports denying renewal on an empty string? If so,
> the conversion from undef to empty string might not be what we want. 
Okay, can use "" or NULL for specific now (see the pref explanation)


> Changing status. Please make some adjustments.
Improved, squashed, sent back for SO

Working on this one now

Created attachment 71129 [details] [review]
Bug 15494 - Unit tests

prove -v t/db_dependent/Circulation.t

Created attachment 71130 [details] [review]
Bug 15494 - Add _items_denied_renewal sub to C4:Circulation::CanBookBeRenewed

Created attachment 71131 [details] [review]
Bug 15494 - Update display of renewal errors

To test:
1 - Define a rule to deny renewal
2 - Checkout an item meeting this rule
3 - Attempt renew an item via the OPAC
4 - Attempt renewal via SIP
5 - Attempt renewal via staff client
6 - Verify errors are reasonable

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 71132 [details] [review]
Bug 15494 - Add syspref

Created attachment 71133 [details] [review]
Bug 15494: (QA follow-up) Additional polishing

[1] Fix two typos in Circulation.t.
    Although the test does not fail, line 2127 contains two typos.
    Changing INVISILE to INVISIBLE :)
    And type should be itype.
[2] Remove $yaml as leftover from older code.
[3] Add a next when the split on /:/ does not give two results. This will
    prevent uninit warnings (although still disabled now in Circulation).
[4] For the same reason we should switch the lines for NULL and empty
    string. The undefs you insert should trigger a warn.
[5] The line for empty string should not insert undef, but empty string.
    For the same reason adding the condition defined($_) ...
    And proving it by adding two tests for the opposite values of
    callnumber and itemnotes.
[6] Adding a strip spaces around the fieldname. User friendly..

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 71134 [details] [review]
Bug 15494 - Unit tests

prove -v t/db_dependent/Circulation.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 71135 [details] [review]
Bug 15494 - Add _items_denied_renewal sub to C4:Circulation::CanBookBeRenewed

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 71136 [details] [review]
Bug 15494 - Update display of renewal errors

To test:
1 - Define a rule to deny renewal
2 - Checkout an item meeting this rule
3 - Attempt renew an item via the OPAC
4 - Attempt renewal via SIP
5 - Attempt renewal via staff client
6 - Verify errors are reasonable

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 71137 [details] [review]
Bug 15494 - Add syspref

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 71138 [details] [review]
Bug 15494: (QA follow-up) Additional polishing

[1] Fix two typos in Circulation.t.
    Although the test does not fail, line 2127 contains two typos.
    Changing INVISILE to INVISIBLE :)
    And type should be itype.
[2] Remove $yaml as leftover from older code.
[3] Add a next when the split on /:/ does not give two results. This will
    prevent uninit warnings (although still disabled now in Circulation).
[4] For the same reason we should switch the lines for NULL and empty
    string. The undefs you insert should trigger a warn.
[5] The line for empty string should not insert undef, but empty string.
    For the same reason adding the condition defined($_) ...
    And proving it by adding two tests for the opposite values of
    callnumber and itemnotes.
[6] Adding a strip spaces around the fieldname. User friendly..

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Has been tested already before, spent sufficient time on changes now.
Looks good to me.
No blocker, but slightly confusing in staff checkouts table:

0 Renewal denied by syspref
(10 of 10 renewals remaining)

If I can't renew, what is the value of these 10? :)

Passed QA with follow-up for some minor issues.

1. In tests, prefer Koha::IssuingRules->search->update($what_you_need_to_be_set);
instead of the DELETE then INSERT. It makes the changes more readable in my opinion.

2. What about "NULL" string values? :)

3. Should not we have the same code for 
OpacHiddenItems and ItemsDeniedRenewal (and UpdateNotForLoanStatusOnCheckin?)
There is also UpdateItemLocationOnCheckin (bug 14576)

The minimum would be a new get_rules_from_prefs sub

Marking as FQA for discussion

We should also tell people that a new item_denied_renewal condition must be added to AUTO_RENEWALS

(In reply to Jonathan Druart from comment #40)
> We should also tell people that a new item_denied_renewal condition must be
> added to AUTO_RENEWALS

Can you explain that one? People?

s/people/Koha administrators

In the update DB process I guess, a warning to tell this new code is added and must be taken into account in the template of the notice.

Ah! Not sure the reason is available in the template tho.

Created attachment 79249 [details] [review]
Bug 15494 - Unit tests

prove -v t/db_dependent/Circulation.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 79250 [details] [review]
Bug 15494 - Add _items_denied_renewal sub to C4:Circulation::CanBookBeRenewed

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 79251 [details] [review]
Bug 15494 - Update display of renewal errors

To test:
1 - Define a rule to deny renewal
2 - Checkout an item meeting this rule
3 - Attempt renew an item via the OPAC
4 - Attempt renewal via SIP
5 - Attempt renewal via staff client
6 - Verify errors are reasonable

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 79252 [details] [review]
Bug 15494 - Add syspref

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 79253 [details] [review]
Bug 15494: (QA follow-up) Additional polishing

[1] Fix two typos in Circulation.t.
    Although the test does not fail, line 2127 contains two typos.
    Changing INVISILE to INVISIBLE :)
    And type should be itype.
[2] Remove $yaml as leftover from older code.
[3] Add a next when the split on /:/ does not give two results. This will
    prevent uninit warnings (although still disabled now in Circulation).
[4] For the same reason we should switch the lines for NULL and empty
    string. The undefs you insert should trigger a warn.
[5] The line for empty string should not insert undef, but empty string.
    For the same reason adding the condition defined($_) ...
    And proving it by adding two tests for the opposite values of
    callnumber and itemnotes.
[6] Adding a strip spaces around the fieldname. User friendly..

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 79254 [details] [review]
Bug 15494: Move yaml syspref code to its own sub

To test:
1 - prove -v t/db_dependent/Koha/Util/SystemPreferences.t

(In reply to Jonathan Druart from comment #39)
> 1. In tests, prefer
> Koha::IssuingRules->search->update($what_you_need_to_be_set);
> instead of the DELETE then INSERT. It makes the changes more readable in my
> opinion.

I generally agree, but circ rules are messy and this file has the same code existing. Should be refactored as a whole

 
> 2. What about "NULL" string values? :)

I think we expect a certain level of input from the user and are fairly thorough here

> 3. Should not we have the same code for 
> OpacHiddenItems and ItemsDeniedRenewal (and UpdateNotForLoanStatusOnCheckin?)
> There is also UpdateItemLocationOnCheckin (bug 14576)
> 
> The minimum would be a new get_rules_from_prefs sub

Moved to its own sub/module - changes for other prefs should be on their own reports

> Ah! Not sure the reason is available in the template tho.

The reasons are stored in the issues table, they should be available to templates

Created attachment 79293 [details] [review]
Bug 15494: Update tests to generate their own data (no delete)

(In reply to Nick Clemens from comment #50)
> (In reply to Jonathan Druart from comment #39)
> > 1. In tests, prefer
> > Koha::IssuingRules->search->update($what_you_need_to_be_set);
> > instead of the DELETE then INSERT. It makes the changes more readable in my
> > opinion.
> 
> I generally agree, but circ rules are messy and this file has the same code
> existing. Should be refactored as a whole
> 
okay, found an easy way to fix, updated.

Should not Koha::Util::SystemPreferences->get_yaml_pref_hash be a method of Koha::Config::SysPref instead?

Created attachment 80327 [details] [review]
Bug 15494 - Unit tests

prove -v t/db_dependent/Circulation.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 80328 [details] [review]
Bug 15494 - Add _items_denied_renewal sub to C4:Circulation::CanBookBeRenewed

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 80329 [details] [review]
Bug 15494 - Update display of renewal errors

To test:
1 - Define a rule to deny renewal
2 - Checkout an item meeting this rule
3 - Attempt renew an item via the OPAC
4 - Attempt renewal via SIP
5 - Attempt renewal via staff client
6 - Verify errors are reasonable

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 80330 [details] [review]
Bug 15494 - Add syspref

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 80331 [details] [review]
Bug 15494: (QA follow-up) Additional polishing

[1] Fix two typos in Circulation.t.
    Although the test does not fail, line 2127 contains two typos.
    Changing INVISILE to INVISIBLE :)
    And type should be itype.
[2] Remove $yaml as leftover from older code.
[3] Add a next when the split on /:/ does not give two results. This will
    prevent uninit warnings (although still disabled now in Circulation).
[4] For the same reason we should switch the lines for NULL and empty
    string. The undefs you insert should trigger a warn.
[5] The line for empty string should not insert undef, but empty string.
    For the same reason adding the condition defined($_) ...
    And proving it by adding two tests for the opposite values of
    callnumber and itemnotes.
[6] Adding a strip spaces around the fieldname. User friendly..

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Created attachment 80332 [details] [review]
Bug 15494: (follow-up) Move yaml syspref code to its own sub in SySpref object

To test:
1 - prove -v t/db_dependent/Koha/Config/SysPrefs.t

Created attachment 80333 [details] [review]
Bug 15494: (follow-up) Update tests to generate their own data (no delete)

Created attachment 81411 [details] [review]
Bug 15494: Unit tests

prove -v t/db_dependent/Circulation.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 81412 [details] [review]
Bug 15494: Add _items_denied_renewal sub to C4:Circulation::CanBookBeRenewed

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 81413 [details] [review]
Bug 15494: Update display of renewal errors

To test:
1 - Define a rule to deny renewal
2 - Checkout an item meeting this rule
3 - Attempt renew an item via the OPAC
4 - Attempt renewal via SIP
5 - Attempt renewal via staff client
6 - Verify errors are reasonable

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 81414 [details] [review]
Bug 15494: Add syspref

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 81415 [details] [review]
Bug 15494: (QA follow-up) Additional polishing

[1] Fix two typos in Circulation.t.
    Although the test does not fail, line 2127 contains two typos.
    Changing INVISILE to INVISIBLE :)
    And type should be itype.
[2] Remove $yaml as leftover from older code.
[3] Add a next when the split on /:/ does not give two results. This will
    prevent uninit warnings (although still disabled now in Circulation).
[4] For the same reason we should switch the lines for NULL and empty
    string. The undefs you insert should trigger a warn.
[5] The line for empty string should not insert undef, but empty string.
    For the same reason adding the condition defined($_) ...
    And proving it by adding two tests for the opposite values of
    callnumber and itemnotes.
[6] Adding a strip spaces around the fieldname. User friendly..

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 81416 [details] [review]
Bug 15494: (follow-up) Move yaml syspref code to its own sub in SySpref object

To test:
1 - prove -v t/db_dependent/Koha/Config/SysPrefs.t

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 81417 [details] [review]
Bug 15494: (follow-up) Update tests to generate their own data (no delete)

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Awesome work all!

Pushed to master for 18.11

Enhancement, will not be backported to 18.05.x series.

*** Bug 20213 has been marked as a duplicate of this bug. ***

Nick, why did you copy/pasted get_yaml_pref_hash in Koha/Config/SysPref.pm and Koha/Util/SystemPreferences.pm?

Koha/Config/SysPref.pm:sub get_yaml_pref_hash {
Koha/Util/SystemPreferences.pm:sub get_yaml_pref_hash {

(In reply to Jonathan Druart from comment #71)
> Nick, why did you copy/pasted get_yaml_pref_hash in Koha/Config/SysPref.pm
> and Koha/Util/SystemPreferences.pm?
> 
> Koha/Config/SysPref.pm:sub get_yaml_pref_hash {
> Koha/Util/SystemPreferences.pm:sub get_yaml_pref_hash {

Opened bug 27154.