Description
Kyle M Hall (khall)
2024-01-10 16:46:29 UTC
We can accept the host keys when we add a new SFTP server via ssh-keyscan -h <ip or name of host> >> known_hosts when a new SFTP server is added Then, if the key changes we can detect the error ( via Net::SFTP::Foreign ) to flag e given SFTP server has no longer having a matching key. We could then add the option to "reset" the public key via "ssh-keygen -R <ip or name of host>" Really nice suggestion, I'm going to look at progressing this. Jake. I should be in a position to push an attachment this week :-) Jake. Created attachment 174933 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. Whoops, will add a test plan!! Created attachment 174937 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Created attachment 175030 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Created attachment 175052 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Ready for SO :-) Jake. Created attachment 175067 [details] [review] Bug 35761: (follow-up) Increase length of password I was finding that the password was getting truncated way too soon. I've lengthened this field to accomodate longer (encrypted) passwords. Created attachment 175166 [details] [review] Bug 35761: Integrate EDI into new FTP/SFTP management UI This patch integrates the EDI Transport.pm module into the new FTP/SFTP management UI by tweaking the existing methods to look at the new SftpServer object, instead of its own. Included also is a change to the database structure to make use of the new sftp_servers table, as well as an atomic update to copy the schema changes, and existing config to the new sftp_servers table. *** Bug 35622 has been marked as a duplicate of this bug. *** *** Bug 38038 has been marked as a duplicate of this bug. *** When I've entered invalid SFTP credentials and save, I'm no longer able to load the table of SFTP servers. I just get a 404 error and ended up deleting that server via SQL. Users need to be able to delete broken entries themselves from the UI. I had a go at testing, and have a couple of questions - these are probably more related to the SFTP server I tested with, and how it is configured: 1. I can connect to the SFTP server (see testing notes), but I can't write to or delete a file in the root directory: Testing we can write a test file... Failed Error message: Couldn't open remote file '.koha_test_file': Permission denied Testing we can delete test file... Failed Error message: Couldn't delete remote file: No such file 2. In this case, the root folder for the SFTP user is not writeable. However, the upload directory is (/upload). I think this is by design in this case with the atmoz/sftp Docker image. 3. However, if a SFTP server only allows uploads to a specific directory (such as uploads), should this be configurable? 4. When I refresh the FTP/SFTP servers listing page, there is a browser console error - this is probably related to 1 and 2 above: GET http://127.0.0.1:8081/api/v1/sftp_server//test_connection 404 (Not Found) 5. Should reloading or accessing the page actually try to connect, or should this be user initiated with the test action? Testing notes (using KTD): 1. Add these commands to docker-compose-light.yml (before the networks section): sftp: image: atmoz/sftp command: koha:koha:::upload networks: - kohanet 2. Do a ktd pull and ktd up. 3. Identify the IP address of the SFTP server (kohadev-sftp-1): docker inspect -f '{{printf "====================\n"}}{{.Name}} {{range $net,$v := .NetworkSettings.Networks}}{{printf "\n"}}{{printf "\n"}}{{printf "%s\n" $net}}{{.IPAddress}}{{printf "\n"}}{{end}}' $(docker ps -q) 4. Check that you can connect to the server locally (not from the ktd shell, use koha as the password and the IP address for kohadev-sftp-1), for example: sftp koha@172.18.0.2 5. Check that you can connect to the server from within your ktd shell: sftp koha@sftp Thanks for the testing, both. Digesting and will reply with some notes shortly! Jake. Thanks for waiting folks, I've addressed most of the points here. Moving the directories used to the SFTP server config seems sensible, so I've done that and added a few more tests.
Just on the mentioned:
> 5. Should reloading or accessing the page actually try to connect, or should this be user initiated with the test action?
How do we envisage this working? A big red “push me” button on that page? A modal when clicking “test” on the main config table?
Jake.
(In reply to Jake Deery from comment #17) > Thanks for waiting folks, I've addressed most of the points here. Moving the > directories used to the SFTP server config seems sensible, so I've done that > and added a few more tests. > > Just on the mentioned: > > > 5. Should reloading or accessing the page actually try to connect, or should this be user initiated with the test action? > > How do we envisage this working? A big red “push me” button on that page? A > modal when clicking “test” on the main config table? > > Jake. From memory, there was a 'Test' button in the table's action column. Created attachment 175608 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF I've reworked a few aspects of this work to now: 1) Include an upload & download directory in the FTP/SFTP Server config page 2) Include a status which gives an at-a-glance of whether the server is working or not, based on the results of the latest test 3) Include a prescriptive run tests button which prevents the tests from running without prior user confirmatory action 4) Added a warning on the test page for SFTP users, informing them of the risks of using unknown SFTP servers Comment on attachment 175608 [details] [review] Bug 35761: Introduce central FTP/SFTP management Review of attachment 175608 [details] [review]: ----------------------------------------------------------------- ::: Koha/SFTP/Server.pm @@ +182,5 @@ > + > + return $self->set( > + { > + status => $value, > + password => $self->plain_text_password, What's the purpose of updating the password and key here? It looks to me like your setting status and also decrypting the password and key file at the same time? When one updates the configuration details we should reset the status as without running a test we don't know whether the updated details are correct so the 'Passed' or 'Failed' states in the table are misleading ;) I also prefer the behaviour where clicking 'Test' from the table took you to the new page AND ran the tests.. I find it a bit disjointed that I need to click 'Run tests' as a second step. I also find you get 'stuck' on that test page now without a clear way back to the display page.. yes you can click the breadcrumb but I think as it would make sense to have one of the buttons beneath the "form" take you back too for user flow For testing, this is a pretty cool service btw: https://sftpcloud.io/tools/free-sftp-server Created attachment 175916 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> (In reply to Martin Renvoize (ashimema) from comment #23) > I also prefer the behaviour where clicking 'Test' from the table took you to > the new page AND ran the tests.. I find it a bit disjointed that I need to > click 'Run tests' as a second step. I also find you get 'stuck' on that > test page now without a clear way back to the display page.. yes you can > click the breadcrumb but I think as it would make sense to have one of the > buttons beneath the "form" take you back too for user flow I wouldn't mind if it automatically testing upon saving so long as a failed test doesn't prevent the save. Agreed that there could be a clearer post-test button to get back to the server list. But I signed off anyway; those could both be followup issues. Created attachment 176429 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Created attachment 176430 [details] [review] Bug 35761: Introduce central FTP/SFTP management This patch adds a central FTP/SFTP management UI, with an associated database object and class, plus the beginnings of an API. We can use this code as a building block for adding FTP/SFTP connections to other parts of the system, as well as to improve the usability of the EDI transports. It is also now possible to test these connections from within the Koha UI. To test:- You will need access to an FTP or SFTP server Contact me if you need a disposable FTP/SFTP server a) Go to Koha Administration b) Notice no FTP/SFTP referenced on this page c) APPLY PATCH, reset_all d) Create new user, let's say sftpuser e) Set permissions for new user to include: 1) catalogue 2) parameters -> manage_sftp_servers f) Log in with this new user, go back to Koha Administration 1) Notice the FTP/SFTP servers option g) Undo step e2) for the sftpuser via the koha admin user 1) Refresh the Koha Administration page as the sftpuser 2) Notice how FTP/SFTP servers has disappeared h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New i) Complete form depending on the FTP or SFTP server you have access to, Submit 1) Notice you are prompted to confirm before saving j) Select Edit, observe your details are the same, and Submit again k) Repeat step J one additional time l) Select Test, observe all tests passing m) Select Delete, confirm deletion, and observe server disappear n) Optionally, run prove on new tests 1) t/Koha/Auth/Permissions.t 2) t/db_dependent/Koha/SFTP/Server.t 3) t/db_dependent/api/v1/sftp_servers.t o) SIGN-OFF Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com> Made further improvements to this work: * Reverted behaviour, so tests now run when tests button clicked (shortening path to user goal) * Test status now resets when an edit is made * Separated out password and key_file storage functionality, to avoid a bloated store subroutine, and to negate the need to 'work around' password and key_file storing when it is not necessary * Removed key_file from API response Missing some POD: FAIL Koha/Schema/Result/SftpServer.pm FAIL pod coverage POD is missing for koha_object_class POD is missing for koha_objects_class A test is failing, but I think this could just be due to my setup: # Failed test 'SWAGGER3.3.4' # at /kohadevbox/koha/t/db_dependent/api/v1/sftp_servers.t line 436. # got: '{"1_ftp_conn":{"err":"cannot connect to ZuxQPXx5x71QvmLk9O9rtmTOKJGgtbPZn3CJD8g66: Temporary failure in name resolution","msg":null,"passed":false}}' # expected: '{"1_ftp_conn":{"err":"cannot connect to ZuxQPXx5x71QvmLk9O9rtmTOKJGgtbPZn3CJD8g66: Name or service not known","msg":null,"passed":false}}' # Looks like you failed 1 test of 5. Failing for POD. Do we often write POD documents for dbix class files, because I can't see examples of this anywhere else? This was automatically generated by `./misc/devel/update_dbix_class_files.pl`. Jake. Hi, Any chance at another look on this one, please? I can't see anything wrong per se when I'm running QA tools, and I don't believe those missing POD documents are essential. Jake. (In reply to Jake Deery from comment #32) > Hi, > > Any chance at another look on this one, please? I can't see anything wrong > per se when I'm running QA tools, and I don't believe those missing POD > documents are essential. > > Jake. Jake, my fault. I can take a go at QA'ing this again. Can we have the database update/schema put into a separate commit? |