Description
Kyle M Hall (khall)
2024-01-10 16:46:29 UTC
We can accept the host keys when we add a new SFTP server via ssh-keyscan -h <ip or name of host> >> known_hosts when a new SFTP server is added Then, if the key changes we can detect the error ( via Net::SFTP::Foreign ) to flag e given SFTP server has no longer having a matching key. We could then add the option to "reset" the public key via "ssh-keygen -R <ip or name of host>" Really nice suggestion, I'm going to look at progressing this. Jake. I should be in a position to push an attachment this week :-) Jake. Created attachment 174933 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. Whoops, will add a test plan!! Created attachment 174937 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Created attachment 175030 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Created attachment 175052 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Ready for SO :-) Jake. Created attachment 175067 [details] [review] Bug 35761: (follow-up) Increase length of password I was finding that the password was getting truncated way too soon. I've lengthened this field to accomodate longer (encrypted) passwords. Created attachment 175166 [details] [review] Bug 35761: Integrate EDI into new FTP/SFTP management UI This patch integrates the EDI Transport.pm module into the new FTP/SFTP management UI by tweaking the existing methods to look at the new SftpServer object, instead of its own. Included also is a change to the database structure to make use of the new sftp_servers table, as well as an atomic update to copy the schema changes, and existing config to the new sftp_servers table. *** Bug 35622 has been marked as a duplicate of this bug. *** *** Bug 38038 has been marked as a duplicate of this bug. *** When I've entered invalid SFTP credentials and save, I'm no longer able to load the table of SFTP servers. I just get a 404 error and ended up deleting that server via SQL. Users need to be able to delete broken entries themselves from the UI. I had a go at testing, and have a couple of questions - these are probably more related to the SFTP server I tested with, and how it is configured: 1. I can connect to the SFTP server (see testing notes), but I can't write to or delete a file in the root directory: Testing we can write a test file... Failed Error message: Couldn't open remote file '.koha_test_file': Permission denied Testing we can delete test file... Failed Error message: Couldn't delete remote file: No such file 2. In this case, the root folder for the SFTP user is not writeable. However, the upload directory is (/upload). I think this is by design in this case with the atmoz/sftp Docker image. 3. However, if a SFTP server only allows uploads to a specific directory (such as uploads), should this be configurable? 4. When I refresh the FTP/SFTP servers listing page, there is a browser console error - this is probably related to 1 and 2 above: GET http://127.0.0.1:8081/api/v1/sftp_server//test_connection 404 (Not Found) 5. Should reloading or accessing the page actually try to connect, or should this be user initiated with the test action? Testing notes (using KTD): 1. Add these commands to docker-compose-light.yml (before the networks section): sftp: image: atmoz/sftp command: koha:koha:::upload networks: - kohanet 2. Do a ktd pull and ktd up. 3. Identify the IP address of the SFTP server (kohadev-sftp-1): docker inspect -f '{{printf "====================\n"}}{{.Name}} {{range $net,$v := .NetworkSettings.Networks}}{{printf "\n"}}{{printf "\n"}}{{printf "%s\n" $net}}{{.IPAddress}}{{printf "\n"}}{{end}}' $(docker ps -q) 4. Check that you can connect to the server locally (not from the ktd shell, use koha as the password and the IP address for kohadev-sftp-1), for example: sftp koha@172.18.0.2 5. Check that you can connect to the server from within your ktd shell: sftp koha@sftp Thanks for the testing, both. Digesting and will reply with some notes shortly! Jake. Thanks for waiting folks, I've addressed most of the points here. Moving the directories used to the SFTP server config seems sensible, so I've done that and added a few more tests.
Just on the mentioned:
> 5. Should reloading or accessing the page actually try to connect, or should this be user initiated with the test action?
How do we envisage this working? A big red “push me” button on that page? A modal when clicking “test” on the main config table?
Jake.
(In reply to Jake Deery from comment #17) > Thanks for waiting folks, I've addressed most of the points here. Moving the > directories used to the SFTP server config seems sensible, so I've done that > and added a few more tests. > > Just on the mentioned: > > > 5. Should reloading or accessing the page actually try to connect, or should this be user initiated with the test action? > > How do we envisage this working? A big red “push me” button on that page? A > modal when clicking “test” on the main config table? > > Jake. From memory, there was a 'Test' button in the table's action column. Created attachment 175608 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF I've reworked a few aspects of this work to now: 1) Include an upload & download directory in the FTP/SFTP Server config page 2) Include a status which gives an at-a-glance of whether the server is working or not, based on the results of the latest test 3) Include a prescriptive run tests button which prevents the tests from running without prior user confirmatory action 4) Added a warning on the test page for SFTP users, informing them of the risks of using unknown SFTP servers Comment on attachment 175608 [details] [review] Bug 35761: Introduce central FTP/SFTP management Review of attachment 175608 [details] [review]: ----------------------------------------------------------------- ::: Koha/SFTP/Server.pm @@ +182,5 @@ > + > + return $self->set( > + { > + status => $value, > + password => $self->plain_text_password, What's the purpose of updating the password and key here? It looks to me like your setting status and also decrypting the password and key file at the same time? When one updates the configuration details we should reset the status as without running a test we don't know whether the updated details are correct so the 'Passed' or 'Failed' states in the table are misleading ;) I also prefer the behaviour where clicking 'Test' from the table took you to the new page AND ran the tests.. I find it a bit disjointed that I need to click 'Run tests' as a second step. I also find you get 'stuck' on that test page now without a clear way back to the display page.. yes you can click the breadcrumb but I think as it would make sense to have one of the buttons beneath the "form" take you back too for user flow For testing, this is a pretty cool service btw: https://sftpcloud.io/tools/free-sftp-server Created attachment 175916 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> (In reply to Martin Renvoize (ashimema) from comment #23) > I also prefer the behaviour where clicking 'Test' from the table took you to > the new page AND ran the tests.. I find it a bit disjointed that I need to > click 'Run tests' as a second step. I also find you get 'stuck' on that > test page now without a clear way back to the display page.. yes you can > click the breadcrumb but I think as it would make sense to have one of the > buttons beneath the "form" take you back too for user flow I wouldn't mind if it automatically testing upon saving so long as a failed test doesn't prevent the save. Agreed that there could be a clearer post-test button to get back to the server list. But I signed off anyway; those could both be followup issues. Created attachment 176429 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Created attachment 176430 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Made further improvements to this work: * Reverted behaviour, so tests now run when tests button clicked (shortening path to user goal) * Test status now resets when an edit is made * Separated out password and key_file storage functionality, to avoid a bloated store subroutine, and to negate the need to 'work around' password and key_file storing when it is not necessary * Removed key_file from API response Missing some POD: FAIL Koha/Schema/Result/SftpServer.pm FAIL pod coverage POD is missing for koha_object_class POD is missing for koha_objects_class A test is failing, but I think this could just be due to my setup: # Failed test 'SWAGGER3.3.4' # at /kohadevbox/koha/t/db_dependent/api/v1/sftp_servers.t line 436. # got: '{"1_ftp_conn":{"err":"cannot connect to ZuxQPXx5x71QvmLk9O9rtmTOKJGgtbPZn3CJD8g66: Temporary failure in name resolution","msg":null,"passed":false}}' # expected: '{"1_ftp_conn":{"err":"cannot connect to ZuxQPXx5x71QvmLk9O9rtmTOKJGgtbPZn3CJD8g66: Name or service not known","msg":null,"passed":false}}' # Looks like you failed 1 test of 5. Failing for POD. Do we often write POD documents for dbix class files, because I can't see examples of this anywhere else? This was automatically generated by `./misc/devel/update_dbix_class_files.pl`. Jake. Hi, Any chance at another look on this one, please? I can't see anything wrong per se when I'm running QA tools, and I don't believe those missing POD documents are essential. Jake. (In reply to Jake Deery from comment #32) > Hi, > > Any chance at another look on this one, please? I can't see anything wrong > per se when I'm running QA tools, and I don't believe those missing POD > documents are essential. > > Jake. Jake, my fault. I can take a go at QA'ing this again. Can we have the database update/schema put into a separate commit? Created attachment 177577 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Created attachment 177578 [details] [review] Bug 35761: DB schemas and atomicupdate No prob, easy to miss, I know! I've divided that commits up into two patches. Hopefully this helps. :-) Jake. Created attachment 177741 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Created attachment 177742 [details] [review] Bug 35761: DB schemas and atomicupdate Rebased atop main All works as expected from test plan QA failures: FAIL spelling passiv ==> passive Why did you name the field 'passiv' instead of 'passive'? FAIL installer/data/mysql/atomicupdate/bug_35761-add_SFTP_tables_and_perm.pl FAIL file permissions File must have the exec flag Koha/Schema/Result/SftpServer.pm - this should be in its own patch Bug 35761: DO NOT PUSH - Schema changes so that RM can generate the file on their own when they push (In reply to Nick Clemens (kidclamp) from comment #40) > All works as expected from test plan > > QA failures: > FAIL spelling > passiv ==> passive > Why did you name the field 'passiv' instead of 'passive'? > > FAIL installer/data/mysql/atomicupdate/bug_35761-add_SFTP_tables_and_perm.pl > FAIL file permissions > File must have the exec flag > > Koha/Schema/Result/SftpServer.pm - this should be in its own patch > Bug 35761: DO NOT PUSH - Schema changes > so that RM can generate the file on their own when they push I have a whole patchset above this now.. just considering whether it should go here or on it's own bug. I do fix the passiv -> passive spelling in that set though. Created attachment 178552 [details] [review] Bug 35761: Database updates Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 178553 [details] [review] Bug 35761: DBIC Schema Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 178554 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> I added the re-work on top as it's own bug.. it warranted a proper review as it does a few things not seen in Koha before. I've asked Tomas to review it :). Setting this one back to Signed Off so it can be reviewed, but take into consideration I believe this and the follow-up bug should be pushed together as the follow-up changes a fair bit of what's implemented here. Created attachment 179249 [details] [review] Bug 35761: Database updates Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 179250 [details] [review] Bug 35761: DBIC Schema Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 179251 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Getting a "Does not apply" error. Think you've got a bit of an issue here with your patches. I think "Bug 35761: DB schemas and atomicupdate" needs to be obsoleted. But I'll let you double-check things. Yup, somehow a patch got unobsoleted Created attachment 182918 [details] [review] Bug 35761: (follow-up) Add missing import There is a missing import for the encode_json method, as well as an unnecessary call to the Koha::Logger class which throws an error Created attachment 185011 [details] [review] Bug 35761: Database updates Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 185012 [details] [review] Bug 35761: DBIC Schema Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 185013 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Hi all, Marked attachment #182918 [details] [review] as obsolete. I think it was intended for Bug 39190. Ta, Jake. Hi, the DBIC schema seems to be manually tweaked (passiv vs. passive), probably an unintended deletion. It prevents `updatedatabase` to run, but that's not a problem because `dbic --force` fixes it and is not required to provide this RM-generated files. That said, I got a failure: ``` prove xt/api.t ``` which points to a problem with the response definition. It actually looks weird: ```yaml "200": description: Results of FTP/SFTP server test schema: type: object items: $ref: "../swagger.yaml#/definitions/sftp_server" ``` which should probably be: ```yaml "200": description: Results of FTP/SFTP server test schema: $ref: "../swagger.yaml#/definitions/sftp_server" ``` but reading at the controller, it seems that the endpoint is not returning an `sftp_server` object at all, but some connection test results in a JSON. I suggest you add a spec for the response structure (stfp_sever_test_results.yaml?) and follow the second example structure. This is hiding something else I bet, as that response format would not validate as an `sftp_server` object anyway and should be rejected by the OpenAPI plugin. Confirmed: ```shell kohadev-koha@kohadevbox:koha(qa_39190)$ git diff diff --git a/api/v1/swagger/paths/test_sftp_servers.yaml b/api/v1/swagger/paths/test_sftp_servers.yaml index 3fab229fc28..77c0ae46eb6 100644 --- a/api/v1/swagger/paths/test_sftp_servers.yaml +++ b/api/v1/swagger/paths/test_sftp_servers.yaml @@ -14,9 +14,7 @@ "200": description: Results of FTP/SFTP server test schema: - type: object - items: - $ref: "../swagger.yaml#/definitions/sftp_server" + $ref: "../swagger.yaml#/definitions/sftp_server" "400": description: | Bad request. Possible `error_code` attribute values: @@ -45,4 +43,4 @@ $ref: "../swagger.yaml#/definitions/error" x-koha-authorization: permissions: - parameters: manage_sftp_servers \ No newline at end of file + parameters: manage_sftp_servers kohadev-koha@kohadevbox:koha(qa_39190)$ prove t/db_dependent/api/v1/sftp_servers.t t/db_dependent/api/v1/sftp_servers.t .. 4/6 # Failed test 'SWAGGER3.2.4' # at t/db_dependent/api/v1/sftp_servers.t line 436. # got: '500' # expected: '200' # Failed test 'SWAGGER3.3.4' # at t/db_dependent/api/v1/sftp_servers.t line 436. # got: '{"errors":[{"message":"Properties not allowed: 1_ftp_conn.","path":"\/body"}],"status":500}' # expected: '{"1_ftp_conn":{"err":"cannot connect to V7OE4H7DT3dp0w0VmBxVIPF3sOf2dLpT6Js0yHDydBmb4: Name or service not known","msg":null,"passed":false}}' # Looks like you failed 2 tests of 5. # Failed test 'test() tests' # at t/db_dependent/api/v1/sftp_servers.t line 443. # Looks like you failed 1 test of 6. t/db_dependent/api/v1/sftp_servers.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/6 subtests Test Summary Report ------------------- t/db_dependent/api/v1/sftp_servers.t (Wstat: 256 (exited 1) Tests: 6 Failed: 1) Failed test: 6 Non-zero exit status: 1 Files=1, Tests=6, 4 wallclock secs ( 0.02 usr 0.01 sys + 3.50 cusr 0.75 csys = 4.28 CPU) Result: FAIL ``` Created attachment 185960 [details] [review] Bug 35761: Database updates Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Created attachment 185961 [details] [review] Bug 35761: DBIC Schema Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Edit: Added POD to the defined methods to make tests pass: ``` prove xt/author/pod_coverage.t ``` Created attachment 185962 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Created attachment 185963 [details] [review] Bug 35761: Fix API spec for sftp_servers test endpoint response The /sftp_server/{id}/test_connection endpoint was incorrectly documented as returning an sftp_server object with items array structure. Fixed by: - Creating proper sftp_test_result.yaml definition - Documenting the actual response structure with test step objects - Each test step contains: passed (boolean), err (string), msg (string) - Updated test_sftp_servers.yaml to reference correct schema - Added sftp_test_result definition to main swagger.yaml The response structure matches what the controller actually returns and what the existing tests expect. I also corrected a missing newline error highlighted by: ``` prove xt/find-missing-new-lines-at-the-end-of-file.t ``` Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Created attachment 185964 [details] [review] Bug 35761: (follow-up) Add NoWarnings Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Created attachment 185965 [details] [review] Bug 35761: Fix stray </form> entries Highlighted by the QA script, this patch addresses the wrong HTML structure in the template that was alerted by the tidy script. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> (In reply to Tomás Cohen Arazi (tcohen) from comment #56) > This is hiding something else I bet, as that response format would not > validate as an `sftp_server` object anyway and should be rejected by the > OpenAPI plugin... This has been hanging for too long and I realized you're already gone for the weekend... so I went ahead and provided the needed fixes :-D Happy to see this moving forward Once I applied bug 39190 I understood my follow-ups weren't needed. I agree with keeping Jake's work history, and then Martin's rewrite as it is part of the process and is interesting for the project. There is a stray </form> tag in your modal. I also see this failure: [FAIL] installer/data/mysql/kohastructure.sql FAIL tinyint_has_boolean_flag The new column sftp_servers.passive is using TINYINT(1) as type but is not defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), see the SQL12 coding guideline The new column sftp_servers.debug is using TINYINT(1) as type but is not defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), see the SQL12 coding guideline (In reply to Lucas Gass (lukeg) from comment #66) > There is a stray </form> tag in your modal. I also see this failure: > > > [FAIL] installer/data/mysql/kohastructure.sql > FAIL tinyint_has_boolean_flag > The new column sftp_servers.passive is using TINYINT(1) as type but is not > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > see the SQL12 coding guideline > The new column sftp_servers.debug is using TINYINT(1) as type but is not > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > see the SQL12 coding guideline My previous follow-ups addressed those issues amongst others. I marked them as obsolete once I understood that Martin's follow-up bug basically rewrites it all. Sorry all, but I still see the same stray <form> tag with everything applied. (In reply to Tomás Cohen Arazi (tcohen) from comment #67) > (In reply to Lucas Gass (lukeg) from comment #66) > > There is a stray </form> tag in your modal. I also see this failure: > > > > > > [FAIL] installer/data/mysql/kohastructure.sql > > FAIL tinyint_has_boolean_flag > > The new column sftp_servers.passive is using TINYINT(1) as type but is not > > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > > see the SQL12 coding guideline > > The new column sftp_servers.debug is using TINYINT(1) as type but is not > > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > > see the SQL12 coding guideline > > My previous follow-ups addressed those issues amongst others. I marked them > as obsolete once I understood that Martin's follow-up bug basically rewrites > it all. Sorry all, I don't believe this is fixed. I certainly still see the stray form tag and the QA tool tells me prog/en/modules/admin/sftp_servers.tt is not tidy. I cannot tidy it with the tool because of the stray tag. Please fix. (In reply to Tomás Cohen Arazi (tcohen) from comment #67) > (In reply to Lucas Gass (lukeg) from comment #66) > > There is a stray </form> tag in your modal. I also see this failure: > > > > > > [FAIL] installer/data/mysql/kohastructure.sql > > FAIL tinyint_has_boolean_flag > > The new column sftp_servers.passive is using TINYINT(1) as type but is not > > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > > see the SQL12 coding guideline > > The new column sftp_servers.debug is using TINYINT(1) as type but is not > > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > > see the SQL12 coding guideline > > My previous follow-ups addressed those issues amongst others. I marked them > as obsolete once I understood that Martin's follow-up bug basically rewrites > it all. ^^^ Lucas Don't look at this bug alone. Apply the them together with 39190. Because of what I said. I already posted fixes to all the issues, but it turns out it all gets rewritten there. (In reply to Tomás Cohen Arazi (tcohen) from comment #69) > (In reply to Tomás Cohen Arazi (tcohen) from comment #67) > > (In reply to Lucas Gass (lukeg) from comment #66) > > > There is a stray </form> tag in your modal. I also see this failure: > > > > > > > > > [FAIL] installer/data/mysql/kohastructure.sql > > > FAIL tinyint_has_boolean_flag > > > The new column sftp_servers.passive is using TINYINT(1) as type but is not > > > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > > > see the SQL12 coding guideline > > > The new column sftp_servers.debug is using TINYINT(1) as type but is not > > > defined as boolean in the schema file (Koha/Schema/Result/SftpServer.pm), > > > see the SQL12 coding guideline > > > > My previous follow-ups addressed those issues amongst others. I marked them > > as obsolete once I understood that Martin's follow-up bug basically rewrites > > it all. > > ^^^ Lucas > > Don't look at this bug alone. Apply the them together with 39190. Because of > what I said. I already posted fixes to all the issues, but it turns out it > all gets rewritten there. Ah, my misunderstanding. Sorry! I was reviewing this bug without the others applied. |