Bug 35836 - search_for_data_inconsistencies.pl - Search for loops in dependencies
Summary: search_for_data_inconsistencies.pl - Search for loops in dependencies
Status: Needs documenting
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Matthias Le Gac
QA Contact: Martin Renvoize
URL:
Keywords: no-sandbox
Depends on: 12532
Blocks:
  Show dependency treegraph
 
Reported: 2024-01-18 15:06 UTC by Matthias Le Gac
Modified: 2024-05-29 12:56 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.05.00


Attachments
Bug 35836: Search for loops in dependencies. (6.15 KB, patch)
2024-01-22 15:49 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: Search for loops in dependencies. (6.23 KB, patch)
2024-01-23 16:42 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 35836: Search for loops in dependencies. (6.23 KB, patch)
2024-01-24 16:18 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836 - search_for_data_inconsistencies.pl - Search for loops in dependencies. (4.67 KB, patch)
2024-02-07 17:29 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - test tidiness1 (1.17 KB, patch)
2024-02-07 17:29 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - test tidiness2 (1.17 KB, patch)
2024-02-07 17:29 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (4.65 KB, patch)
2024-02-07 17:49 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (1.15 KB, patch)
2024-02-07 17:49 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (1.15 KB, patch)
2024-02-07 17:50 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (4.65 KB, patch)
2024-02-07 17:52 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (1.15 KB, patch)
2024-02-07 17:52 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (1.15 KB, patch)
2024-02-07 17:52 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - test tidiness (4.67 KB, patch)
2024-02-19 19:57 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - fixed tidiness (21.89 KB, patch)
2024-02-22 17:37 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (4.60 KB, patch)
2024-02-22 19:28 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: change | by , for the print (933 bytes, patch)
2024-03-18 18:06 UTC, Matthias Le Gac
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (4.65 KB, patch)
2024-04-12 09:09 UTC, Philip Orr
Details | Diff | Splinter Review
Bug 35836: change | by , for the print (984 bytes, patch)
2024-04-12 09:09 UTC, Philip Orr
Details | Diff | Splinter Review
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies (4.78 KB, patch)
2024-05-15 07:15 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 35836: change | by , for the print (1.09 KB, patch)
2024-05-15 07:15 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 35836: (follow-up) optimize the script when searching for guarantors (5.26 KB, patch)
2024-05-16 12:58 UTC, Hammat wele
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Le Gac 2024-01-18 15:06:33 UTC
We have guarantors who guarantee individuals who are, in turn, their guarantors. 

It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.

However, it breaks with bz12532, which produces a recursion that disrupts everything.

Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.
Comment 1 Matthias Le Gac 2024-01-22 15:49:56 UTC
Created attachment 161249 [details] [review]
Bug 35836: Search for loops in dependencies.

We have guarantors who guarantee individuals who are, in turn, their guarantors.
It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.
However, it breaks with bz12532, which produces a recursion that disrupts everything.
Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.

This is the first bz I've made. Here's the test plan, let me know if there's anything to improve.

Test Plan:

1 - Apply the patch.
2 - Create 4 different adult users.
3 - In the terminal, navigate to the database.
4 - Insert relationships into the "borrower_relationships" table:
	- Example: INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (1, 52, 53, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (2, 53, 54, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (3, 54, 52, 'father');
	- Ensure that the relationships created form a loop of guarantors.
5 - In the terminal, navigate to your Koha environment's git repository.
6 - Change directory to "cd misc/maintenance."
7 - Run the script to detect guarantor loops: ./search_for_data_inconsistencies.pl
8 - Verify that the output provides the borrower IDs involved in the guarantor loop.

*Have fun creating intricate relationships to test the script's limit
Comment 2 Matt Blenkinsop 2024-01-23 16:42:10 UTC
Created attachment 161307 [details] [review]
Bug 35836: Search for loops in dependencies.

We have guarantors who guarantee individuals who are, in turn, their guarantors.
It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.
However, it breaks with bz12532, which produces a recursion that disrupts everything.
Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.

This is the first bz I've made. Here's the test plan, let me know if there's anything to improve.

Test Plan:

1 - Apply the patch.
2 - Create 4 different adult users.
3 - In the terminal, navigate to the database.
4 - Insert relationships into the "borrower_relationships" table:
	- Example: INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (1, 52, 53, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (2, 53, 54, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (3, 54, 52, 'father');
	- Ensure that the relationships created form a loop of guarantors.
5 - In the terminal, navigate to your Koha environment's git repository.
6 - Change directory to "cd misc/maintenance."
7 - Run the script to detect guarantor loops: ./search_for_data_inconsistencies.pl
8 - Verify that the output provides the borrower IDs involved in the guarantor loop.

*Have fun creating intricate relationships to test the script's limit

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Comment 3 Matthias Le Gac 2024-01-24 16:18:07 UTC
Created attachment 161351 [details] [review]
Bug 35836: Search for loops in dependencies.

We have guarantors who guarantee individuals who are, in turn, their guarantors.
It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.
However, it breaks with bz12532, which produces a recursion that disrupts everything.
Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.

This is the first bz I've made. Here's the test plan, let me know if there's anything to improve.

Test Plan:

1 - Apply the patch.
2 - Create 4 different adult users.
3 - In the terminal, navigate to the database.
4 - Insert relationships into the "borrower_relationships" table:
	- Example: INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (1, 52, 53, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (2, 53, 54, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (3, 54, 52, 'father');
	- Ensure that the relationships created form a loop of guarantors.
5 - In the terminal, navigate to your Koha environment's git repository.
6 - Change directory to "cd misc/maintenance."
7 - Run the script to detect guarantor loops: ./search_for_data_inconsistencies.pl
8 - Verify that the output provides the borrower IDs involved in the guarantor loop.

*Have fun creating intricate relationships to test the script's limit

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Comment 4 Matthias Le Gac 2024-01-24 16:23:00 UTC
I have just change the author.
Comment 5 Marcel de Rooy 2024-01-26 08:19:37 UTC Comment hidden (obsolete)
Comment 6 Matthias Le Gac 2024-02-06 21:43:03 UTC Comment hidden (obsolete)
Comment 7 Matthias Le Gac 2024-02-07 17:29:15 UTC
Created attachment 161836 [details] [review]
Bug 35836 - search_for_data_inconsistencies.pl - Search for loops in dependencies.

We have guarantors who guarantee individuals who are, in turn, their guarantors. 
It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.
However, it breaks with bz12532, which produces a recursion that disrupts everything.
Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.

This is the first bz I've made. Here's the test plan, let me know if there's anything to improve.

Test Plan:

1 - Apply the patch.
2 - Create 4 different adult users.
3 - In the terminal, navigate to the database.
4 - Insert relationships into the "borrower_relationships" table:
	- Example: INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (1, 52, 53, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (2, 53, 54, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (3, 54, 52, 'father');
	- Ensure that the relationships created form a loop of guarantors.
5 - In the terminal, navigate to your Koha environment's git repository.
6 - Change directory to "cd misc/maintenance."
7 - Run the script to detect guarantor loops: ./search_for_data_inconsistencies.pl
8 - Verify that the output provides the borrower IDs involved in the guarantor loop.

*Have fun creating intricate relationships to test the script's limit
Comment 8 Matthias Le Gac 2024-02-07 17:29:17 UTC
Created attachment 161837 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - test tidiness1
Comment 9 Matthias Le Gac 2024-02-07 17:29:19 UTC
Created attachment 161838 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - test tidiness2
Comment 10 Matthias Le Gac 2024-02-07 17:49:56 UTC
Created attachment 161839 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies

We have guarantors who guarantee individuals who are, in turn, their guarantors. 
It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.
However, it breaks with bz12532, which produces a recursion that disrupts everything.
Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.

This is the first bz I've made. Here's the test plan, let me know if there's anything to improve.

Test Plan:

1 - Apply the patch.
2 - Create 4 different adult users.
3 - In the terminal, navigate to the database.
4 - Insert relationships into the "borrower_relationships" table:
	- Example: INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (1, 52, 53, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (2, 53, 54, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (3, 54, 52, 'father');
	- Ensure that the relationships created form a loop of guarantors.
5 - In the terminal, navigate to your Koha environment's git repository.
6 - Change directory to "cd misc/maintenance."
7 - Run the script to detect guarantor loops: ./search_for_data_inconsistencies.pl
8 - Verify that the output provides the borrower IDs involved in the guarantor loop.

*Have fun creating intricate relationships to test the script's limit
Comment 11 Matthias Le Gac 2024-02-07 17:49:58 UTC
Created attachment 161840 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies
Comment 12 Matthias Le Gac 2024-02-07 17:50:00 UTC
Created attachment 161841 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies
Comment 13 Matthias Le Gac 2024-02-07 17:52:00 UTC Comment hidden (obsolete)
Comment 14 Matthias Le Gac 2024-02-07 17:52:01 UTC Comment hidden (obsolete)
Comment 15 Matthias Le Gac 2024-02-07 17:52:03 UTC Comment hidden (obsolete)
Comment 16 Magnus Enger 2024-02-16 14:29:34 UTC Comment hidden (obsolete)
Comment 17 Matthias Le Gac 2024-02-16 14:37:31 UTC Comment hidden (obsolete)
Comment 18 Magnus Enger 2024-02-16 15:06:04 UTC Comment hidden (obsolete)
Comment 19 Matthias Le Gac 2024-02-16 15:25:00 UTC Comment hidden (obsolete)
Comment 20 Matthias Le Gac 2024-02-16 15:59:19 UTC Comment hidden (obsolete)
Comment 21 Matthias Le Gac 2024-02-19 19:57:21 UTC
Created attachment 162284 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - test tidiness

We have guarantors who guarantee individuals who are, in turn, their guarantors. 
It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.
However, it breaks with bz12532, which produces a recursion that disrupts everything.
Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.

This is the first bz I've made. Here's the test plan, let me know if there's anything to improve.

Test Plan:

1 - Apply the patch.
2 - Create 4 different adult users.
3 - In the terminal, navigate to the database.
4 - Insert relationships into the "borrower_relationships" table:
	- Example: INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (1, 52, 53, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (2, 53, 54, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (3, 54, 52, 'father');
	- Ensure that the relationships created form a loop of guarantors.
5 - In the terminal, navigate to your Koha environment's git repository.
6 - Change directory to "cd misc/maintenance."
7 - Run the script to detect guarantor loops: ./search_for_data_inconsistencies.pl
8 - Verify that the output provides the borrower IDs involved in the guarantor loop.

*Have fun creating intricate relationships to test the script's limit
Comment 22 Matthias Le Gac 2024-02-19 20:00:40 UTC Comment hidden (obsolete)
Comment 23 Matthias Le Gac 2024-02-22 17:37:49 UTC
Created attachment 162341 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies - fixed tidiness
Comment 24 Matthias Le Gac 2024-02-22 17:38:54 UTC Comment hidden (obsolete)
Comment 25 Matthias Le Gac 2024-02-22 19:28:49 UTC
Created attachment 162344 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies
Comment 26 Matthias Le Gac 2024-03-18 18:06:21 UTC
Created attachment 163368 [details] [review]
Bug 35836: change | by , for the print

We have guarantors who guarantee individuals who are, in turn, their guarantors. 
It is not clear if this should be allowed (bz35421), but it works, and has been working for a while.
However, it breaks with bz12532, which produces a recursion that disrupts everything.
Creation of a code in search_for_data_inconsistencies.pl that identifies loops in borrower_relationships. This must work in both directions for validation.

This is the first bz I've made. Here's the test plan, let me know if there's anything to improve.

Test Plan:

1 - Apply the patch.
2 - Create 4 different adult users.
3 - In the terminal, navigate to the database.
4 - Insert relationships into the "borrower_relationships" table:
	- Example: INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (1, 52, 53, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (2, 53, 54, 'father');
		   INSERT INTO borrower_relationships (id, guarantor_id, guarantee_id, relationship) VALUES (3, 54, 52, 'father');
	- Ensure that the relationships created form a loop of guarantors.
5 - In the terminal, navigate to your Koha environment's git repository.
6 - Change directory to "cd misc/maintenance."
7 - Run the script to detect guarantor loops: ./search_for_data_inconsistencies.pl
8 - Verify that the output provides the borrower IDs involved in the guarantor loop.

*Have fun creating intricate relationships to test the script's limit
Comment 27 Philip Orr 2024-04-12 09:09:25 UTC
Created attachment 164817 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies

Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Comment 28 Philip Orr 2024-04-12 09:09:27 UTC
Created attachment 164818 [details] [review]
Bug 35836: change | by , for the print

Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Comment 29 Martin Renvoize 2024-05-15 07:15:01 UTC
Created attachment 166751 [details] [review]
Bug 35836: search_for_data_inconsistencies.pl - Search for loops in dependencies

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 30 Martin Renvoize 2024-05-15 07:15:04 UTC
Created attachment 166752 [details] [review]
Bug 35836: change | by , for the print

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 31 Hammat wele 2024-05-16 12:58:10 UTC
Created attachment 166825 [details] [review]
Bug 35836: (follow-up) optimize the script when searching for guarantors
Comment 32 Katrin Fischer 2024-05-17 08:02:04 UTC
This is not a bug fix as is I think, but as an addition to a CLI script I still decided to push.
Comment 33 Katrin Fischer 2024-05-17 08:37:19 UTC
Pushed for 24.05!

Well done everyone, thank you!
Comment 34 Blou 2024-05-17 12:59:02 UTC
Wow, you took me by surprise here, I didn't think anything would be pushed for a while.

The script works well (discover the loops), but I found out it is ressource intensive in its coding when using it on a huge library.  I had put a note to improve it.

I'll open a follow-up bz.
Comment 35 Katrin Fischer 2024-05-17 18:43:52 UTC
(In reply to Blou from comment #34)
> Wow, you took me by surprise here, I didn't think anything would be pushed
> for a while.
> 
> The script works well (discover the loops), but I found out it is ressource
> intensive in its coding when using it on a huge library.  I had put a note
> to improve it.
> 
> I'll open a follow-up bz.

This was marked as a bug (not enh), so I considered it. A follow-up would be good. Maybe it could be optional with a warn if that is a possible 'quicker' fix? I expect no string changes here, so we can still take a follow-up into account.
Comment 36 Blou 2024-05-21 14:15:39 UTC
Ok, seems Hammat already worked his magic, I hadn't noticed he had done the fixes.
It seems all good from my pov.
Comment 37 Fridolin Somers 2024-05-29 12:56:11 UTC
Not backported to 23.11.x