Description
Jonathan Druart
2025-05-12 13:40:49 UTC
Created attachment 182308 [details] [review] Bug 39877: Incremental test runs The idea of this patchset is to allow increment runs for some of our tests. The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15). One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc). From a previous run (last week) I noted this top 5: xt/vue_tidy.t 105446 ms xt/perltidy.t 175970 ms xt/js_tidy.t 239615 ms xt/author/codespell.t 467041 ms xt/pl_valid.t 2502354 ms We are going to merge js_tidy.t and vue_tidy.t as it does no longer make sense to have a difference between them (we had at the beginning of the vue work, because of differences in the options, now everything is handled in the .prettierrc config file) Once this is pushed, we will be able to create a new project on gitlab and pass some env variables from Jenkins. Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results When everything will be setup, jenkins will run the tests one the whole test suite, then the next run will be "incremental": we are going to run the tests on the files that were failing and the files modified by the last push. Example: * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm * koha-ci-results/perltidy/abc is created and contains ['Koha.pm'] * "commit def" has a modification on about.pl * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy * koha-ci-results/perltidy/def is created and contains [] Test plan: * Create an empty project on gitlab eg. https://gitlab.com/me/koha-ci-results * Generate an access token with the developer role (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens) Copy the token (in the following noted as TOKEN_REPLACE_ME) * Unprotect the "main" branch * Run the following prove command: ``` KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \ prove xt/perltidy.t ``` => a new commit is created on your repo * Add some mess in one of your perl file, commit using `--no-verify` to bypass the git hook * Run the prove command again => only the file you have modified is tested => a new commit is created and the file is reported as failing * Add another commit, fix the failure, add other mess => confirm that what you are seeing makes sense :) * Ideally redo with the different tests modified by this patchset Note that you can also export the following env vars to set git author detail: * GIT_COMMITTER_NAME * GIT_COMMITTER_EMAIL * GIT_AUTHOR_NAME * GIT_AUTHOR_EMAIL Question: What are we going to use on Jenkins? Created attachment 182309 [details] [review] Bug 39877: Remove files that have been deleted Ignore deleted or renamed files Created attachment 182310 [details] [review] Bug 39877: Merge vue and js tidy tests Created attachment 182311 [details] [review] Bug 39877: Apply to other slow tests Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main). Maybe later we will also use it for stable branches. An important note I forgot in the commit message. You can use the incremental run without the report (the report will be only for Jenkins). So it will help RM as xt tests will run (way) faster. Note that we may miss some bad bugs. For instance if a version of codespell changes and that catch more typo, we will miss them with the incremental run even if the test fails in a regular run. Could also happen with pl_valid I guess if an external lib introduces changes that breaks our code. But they really are edge cases we cannot avoid and that should be caught by the qa script anyway. Created attachment 182347 [details] [review] Bug 39877: Incremental test runs The idea of this patchset is to allow increment runs for some of our tests. The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15). One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc). From a previous run (last week) I noted this top 5: xt/vue_tidy.t 105446 ms xt/perltidy.t 175970 ms xt/js_tidy.t 239615 ms xt/author/codespell.t 467041 ms xt/pl_valid.t 2502354 ms We are going to merge js_tidy.t and vue_tidy.t as it does no longer make sense to have a difference between them (we had at the beginning of the vue work, because of differences in the options, now everything is handled in the .prettierrc config file) Once this is pushed, we will be able to create a new project on gitlab and pass some env variables from Jenkins. Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results When everything will be setup, jenkins will run the tests one the whole test suite, then the next run will be "incremental": we are going to run the tests on the files that were failing and the files modified by the last push. Example: * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm * koha-ci-results/perltidy/abc is created and contains ['Koha.pm'] * "commit def" has a modification on about.pl * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy * koha-ci-results/perltidy/def is created and contains [] Test plan: * Create an empty project on gitlab eg. https://gitlab.com/me/koha-ci-results * Generate an access token with the developer role (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens) Copy the token (in the following noted as TOKEN_REPLACE_ME) * Unprotect the "main" branch * Run the following prove command: ``` KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \ prove xt/perltidy.t ``` => a new commit is created on your repo * Add some mess in one of your perl file, commit using `--no-verify` to bypass the git hook * Run the prove command again => only the file you have modified is tested => a new commit is created and the file is reported as failing * Add another commit, fix the failure, add other mess => confirm that what you are seeing makes sense :) * Ideally redo with the different tests modified by this patchset * Note that you can use the incremental run without the report (the report will be only for Jenkins). It will help RM as xt tests will run (way) faster. Note that you can also export the following env vars to set git author detail: * GIT_COMMITTER_NAME * GIT_COMMITTER_EMAIL * GIT_AUTHOR_NAME * GIT_AUTHOR_EMAIL Question: What are we going to use on Jenkins? Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main). Maybe later we will also use it for stable branches. Created attachment 182348 [details] [review] Bug 39877: Remove files that have been deleted Ignore deleted or renamed files Created attachment 182349 [details] [review] Bug 39877: Merge vue and js tidy tests Created attachment 182350 [details] [review] Bug 39877: Apply to other slow tests Created attachment 182355 [details] [review] Bug 39877: Incremental test runs The idea of this patchset is to allow increment runs for some of our tests. The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15). One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc). From a previous run (last week) I noted this top 5: xt/vue_tidy.t 105446 ms xt/perltidy.t 175970 ms xt/js_tidy.t 239615 ms xt/author/codespell.t 467041 ms xt/pl_valid.t 2502354 ms We are going to merge js_tidy.t and vue_tidy.t as it does no longer make sense to have a difference between them (we had at the beginning of the vue work, because of differences in the options, now everything is handled in the .prettierrc config file) Once this is pushed, we will be able to create a new project on gitlab and pass some env variables from Jenkins. Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results When everything will be setup, jenkins will run the tests one the whole test suite, then the next run will be "incremental": we are going to run the tests on the files that were failing and the files modified by the last push. Example: * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm * koha-ci-results/perltidy/abc is created and contains ['Koha.pm'] * "commit def" has a modification on about.pl * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy * koha-ci-results/perltidy/def is created and contains [] Test plan: * Create an empty project on gitlab eg. https://gitlab.com/me/koha-ci-results * Generate an access token with the developer role (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens) Copy the token (in the following noted as TOKEN_REPLACE_ME) * Unprotect the "main" branch * Run the following prove command: ``` KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \ prove xt/perltidy.t ``` => a new commit is created on your repo * Add some mess in one of your perl file, commit using `--no-verify` to bypass the git hook * Run the prove command again => only the file you have modified is tested => a new commit is created and the file is reported as failing * Add another commit, fix the failure, add other mess => confirm that what you are seeing makes sense :) * Ideally redo with the different tests modified by this patchset * Note that you can use the incremental run without the report (the report will be only for Jenkins). It will help RM as xt tests will run (way) faster. Note that you can also export the following env vars to set git author detail: * GIT_COMMITTER_NAME * GIT_COMMITTER_EMAIL * GIT_AUTHOR_NAME * GIT_AUTHOR_EMAIL Question: What are we going to use on Jenkins? Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main). Maybe later we will also use it for stable branches. Created attachment 182356 [details] [review] Bug 39877: Remove files that have been deleted Ignore deleted or renamed files Created attachment 182357 [details] [review] Bug 39877: Merge vue and js tidy tests Created attachment 182358 [details] [review] Bug 39877: Apply to other slow tests > xt/pl_valid.t 2502354 ms We can also save some time by not checking *.t files, since they are executed anyway. *.t files represent approximately 25% of the total time of pl_valid.t for me. (In reply to Jonathan Druart from comment #7) > Could also happen with pl_valid I > guess if an external lib introduces changes that breaks our code. But they > really are edge cases we cannot avoid and that should be caught by the qa > script anyway. If an exported subroutine is removed/renamed, code that still use/import it will fail to compile, but this won't get caught if we are only checking modified files. Will it be caught by the QA script ? (In reply to Julian Maurice from comment #16) > > xt/pl_valid.t 2502354 ms > We can also save some time by not checking *.t files, since they are > executed anyway. *.t files represent approximately 25% of the total time of > pl_valid.t for me. Could be, yes. > (In reply to Jonathan Druart from comment #7) > > Could also happen with pl_valid I > > guess if an external lib introduces changes that breaks our code. But they > > really are edge cases we cannot avoid and that should be caught by the qa > > script anyway. > If an exported subroutine is removed/renamed, code that still use/import it > will fail to compile, but this won't get caught if we are only checking > modified files. Will it be caught by the QA script ? I have no idea why I have written that, it does not make sense. Created attachment 183795 [details] [review] Bug 39877: Incremental test runs The idea of this patchset is to allow increment runs for some of our tests. The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15). One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc). From a previous run (last week) I noted this top 5: xt/vue_tidy.t 105446 ms xt/perltidy.t 175970 ms xt/js_tidy.t 239615 ms xt/author/codespell.t 467041 ms xt/pl_valid.t 2502354 ms We are going to merge js_tidy.t and vue_tidy.t as it does no longer make sense to have a difference between them (we had at the beginning of the vue work, because of differences in the options, now everything is handled in the .prettierrc config file) Once this is pushed, we will be able to create a new project on gitlab and pass some env variables from Jenkins. Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results When everything will be setup, jenkins will run the tests one the whole test suite, then the next run will be "incremental": we are going to run the tests on the files that were failing and the files modified by the last push. Example: * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm * koha-ci-results/perltidy/abc is created and contains ['Koha.pm'] * "commit def" has a modification on about.pl * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy * koha-ci-results/perltidy/def is created and contains [] Test plan: * Create an empty project on gitlab eg. https://gitlab.com/me/koha-ci-results * Generate an access token with the developer role (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens) Copy the token (in the following noted as TOKEN_REPLACE_ME) * Unprotect the "main" branch * Run the following prove command: ``` KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \ prove xt/perltidy.t ``` => a new commit is created on your repo * Add some mess in one of your perl file, commit using `--no-verify` to bypass the git hook * Run the prove command again => only the file you have modified is tested => a new commit is created and the file is reported as failing * Add another commit, fix the failure, add other mess => confirm that what you are seeing makes sense :) * Ideally redo with the different tests modified by this patchset * Note that you can use the incremental run without the report (the report will be only for Jenkins). It will help RM as xt tests will run (way) faster. Note that you can also export the following env vars to set git author detail: * GIT_COMMITTER_NAME * GIT_COMMITTER_EMAIL * GIT_AUTHOR_NAME * GIT_AUTHOR_EMAIL Question: What are we going to use on Jenkins? Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main). Maybe later we will also use it for stable branches. Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 183796 [details] [review] Bug 39877: Remove files that have been deleted Ignore deleted or renamed files Created attachment 183797 [details] [review] Bug 39877: Merge vue and js tidy tests Created attachment 183798 [details] [review] Bug 39877: Apply to other slow tests Rebased and fixed conflicts. Double-checked with a diff between the old and new patches to confirm there are no unexpected changes introduced. I'll attempt signing off. (In reply to Jonathan Druart from comment #5) > Discussion for QA: How should we handle concurrency? It's crucial to prevent > multiple tests from pushing simultaneously. To minimize the risk of the > repository becoming outdated, we might need to perform a pull right before > each push. Is that enough? Keep in mind that only one job will be run at the > same time (Koha_Main). Looks good to me. If some mess occasionally comes up, someone will just need to manually commit changes to add some files that might not have been added to the list of files to recheck. Is that all? Or just rerun locally and that's it without bothering with the repo. > Maybe later we will also use it for stable branches. Which could have their own branch in koha-ci-results right? So it shouldn't cause race conditions. (In reply to Victor Grousset/tuxayo from comment #23) > (In reply to Jonathan Druart from comment #5) > > Maybe later we will also use it for stable branches. > > Which could have their own branch in koha-ci-results right? So it shouldn't > cause race conditions. We should not need separate branches. It seems the initial clone expects an https:// prefix. kohadev-koha@kohadevbox:koha(testing-SO-QA)$ KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/tuxayo/koha-ci-results.git KOHA_CI_INCREMENTAL_RUNS=1 KOHA_CI_INCREMENTAL_RUNS_REPORT=1 KOHA_CI_INCREMENTAL_RUNS_TOKEN=XXXXXX prove xt/perltidy.t xt/perltidy.t .. fatal: repository 'gitlab.com/tuxayo/koha-ci-results.git' does not exist fatal: cannot change to '/tmp/koha-ci-results': No such file or directory xt/perltidy.t .. 3/3031 ^C (cleanup) So it goes further with it. But then (the push?) ends up with https://https:// kohadev-koha@kohadevbox:koha(testing-SO-QA)$ KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/tuxayo/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=XXXXXXXXXXX \ prove xt/perltidy.t xt/perltidy.t .. Cloning into '/tmp/koha-ci-results'... xt/perltidy.t .. 3007/3031 fatal: unable to access 'https://https://gitlab.com/tuxayo/koha-ci-results.git/': Could not resolve host: https xt/perltidy.t .. ok All tests successful. Files=1, Tests=3031, 216 wallclock secs ( 0.41 usr 0.04 sys + 207.40 cusr 1.50 csys = 209.35 CPU) Result: PASS Created attachment 184462 [details] [review] Bug 39877: Incremental test runs The idea of this patchset is to allow increment runs for some of our tests. The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15). One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc). From a previous run (last week) I noted this top 5: xt/vue_tidy.t 105446 ms xt/perltidy.t 175970 ms xt/js_tidy.t 239615 ms xt/author/codespell.t 467041 ms xt/pl_valid.t 2502354 ms We are going to merge js_tidy.t and vue_tidy.t as it does no longer make sense to have a difference between them (we had at the beginning of the vue work, because of differences in the options, now everything is handled in the .prettierrc config file) Once this is pushed, we will be able to create a new project on gitlab and pass some env variables from Jenkins. Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results When everything will be setup, jenkins will run the tests one the whole test suite, then the next run will be "incremental": we are going to run the tests on the files that were failing and the files modified by the last push. Example: * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm * koha-ci-results/perltidy/abc is created and contains ['Koha.pm'] * "commit def" has a modification on about.pl * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy * koha-ci-results/perltidy/def is created and contains [] Test plan: * Create an empty project on gitlab eg. https://gitlab.com/me/koha-ci-results * Generate an access token with the developer role (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens) Copy the token (in the following noted as TOKEN_REPLACE_ME) * Unprotect the "main" branch * Run the following prove command: ``` KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \ prove xt/perltidy.t ``` => a new commit is created on your repo * Add some mess in one of your perl file, commit using `--no-verify` to bypass the git hook * Run the prove command again => only the file you have modified is tested => a new commit is created and the file is reported as failing * Add another commit, fix the failure, add other mess => confirm that what you are seeing makes sense :) * Ideally redo with the different tests modified by this patchset * Note that you can use the incremental run without the report (the report will be only for Jenkins). It will help RM as xt tests will run (way) faster. Note that you can also export the following env vars to set git author detail: * GIT_COMMITTER_NAME * GIT_COMMITTER_EMAIL * GIT_AUTHOR_NAME * GIT_AUTHOR_EMAIL Question: What are we going to use on Jenkins? Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main). Maybe later we will also use it for stable branches. Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184463 [details] [review] Bug 39877: Remove files that have been deleted Ignore deleted or renamed files Created attachment 184464 [details] [review] Bug 39877: Merge vue and js tidy tests Created attachment 184465 [details] [review] Bug 39877: Apply to other slow tests Created attachment 184466 [details] [review] Bug 39877: Fix confusion with URL prefix git clone needs https://, but we need to remove it to build the push url (In reply to Victor Grousset/tuxayo from comment #25) > It seems the initial clone expects an https:// prefix. Indeed, I mixed up things it seems. Should be fixed now. I've amended the commit message of the first patch and added a fix ("Fix confusion with URL prefix") Created attachment 184779 [details] [review] Bug 39877: Incremental test runs The idea of this patchset is to allow increment runs for some of our tests. The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15). One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc). From a previous run (last week) I noted this top 5: xt/vue_tidy.t 105446 ms xt/perltidy.t 175970 ms xt/js_tidy.t 239615 ms xt/author/codespell.t 467041 ms xt/pl_valid.t 2502354 ms We are going to merge js_tidy.t and vue_tidy.t as it does no longer make sense to have a difference between them (we had at the beginning of the vue work, because of differences in the options, now everything is handled in the .prettierrc config file) Once this is pushed, we will be able to create a new project on gitlab and pass some env variables from Jenkins. Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results When everything will be setup, jenkins will run the tests one the whole test suite, then the next run will be "incremental": we are going to run the tests on the files that were failing and the files modified by the last push. Example: * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm * koha-ci-results/perltidy/abc is created and contains ['Koha.pm'] * "commit def" has a modification on about.pl * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy * koha-ci-results/perltidy/def is created and contains [] Test plan: * Create an empty project on gitlab (keep the default readme) eg. https://gitlab.com/me/koha-ci-results * Generate an access token with the developer role (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens) Copy the token (in the following noted as TOKEN_REPLACE_ME) * Unprotect the "main" branch * Run the following prove command: ``` KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \ prove xt/perltidy.t ``` => a new commit is created on your repo * Add some mess in one of your perl file, commit using `--no-verify` to bypass the git hook * Run the prove command again => only the file you have modified is tested => a new commit is created and the file is reported as failing * Add another commit, fix the failure, add other mess => confirm that what you are seeing makes sense :) * Ideally redo with the different tests modified by this patchset * Note that you can use the incremental run without the report (the report will be only for Jenkins). It will help RM as xt tests will run (way) faster. Note that you can also export the following env vars to set git author detail: * GIT_COMMITTER_NAME * GIT_COMMITTER_EMAIL * GIT_AUTHOR_NAME * GIT_AUTHOR_EMAIL Question: What are we going to use on Jenkins? Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main). Maybe later we will also use it for stable branches. Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184780 [details] [review] Bug 39877: Remove files that have been deleted Ignore deleted or renamed files Created attachment 184781 [details] [review] Bug 39877: Merge vue and js tidy tests Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184782 [details] [review] Bug 39877: Apply to other slow tests Created attachment 184783 [details] [review] Bug 39877: Fix confusion with URL prefix git clone needs https://, but we need to remove it to build the push url > * Create an empty project on gitlab
> eg. https://gitlab.com/me/koha-ci-results
Let it create a readme, don't untick the readme creation. Otherwise, there will be no branch to unprotect and there won't be an actual repo. (clone will fail)
----
rebased and fixed conflicts (diffs compared to confirm no unwanted change introduced)
> * Add another commit, fix the failure, add other mess If adding the other mess in the same file where I fixed it. The new issue is outputted twice. Is that a relevant issue? https://paste.evolix.org/?6e80d183272f48c9#3e6Gab54Gzuh5NWLfcEsjV3qMuchJimNEAoRgQyyhy85 Same in a vue file. --- other testing: On main from a week ago, I applied the patch changes without committing. So HEAD is from main. But the new tests are there. (actual code doesn't match the commit but it just a hack to test the incremental run in another situation) The full run happened as expected. After a proper application of the patches, the next test run only processed a subset of the files: ☑️ Created attachment 184785 [details] [review] Bug 39877: Prevent files to be processed twice If the file had a failure and still has, we should not list it twice. (In reply to Victor Grousset/tuxayo from comment #37) > > * Create an empty project on gitlab > > eg. https://gitlab.com/me/koha-ci-results > > Let it create a readme, don't untick the readme creation. Otherwise, there > will be no branch to unprotect and there won't be an actual repo. (clone > will fail) I will deal with the setup once this is pushed to prevent such problems. (In reply to Victor Grousset/tuxayo from comment #38) > > * Add another commit, fix the failure, add other mess > > If adding the other mess in the same file where I fixed it. The new issue is > outputted twice. Is that a relevant issue? > https://paste.evolix.org/ > ?6e80d183272f48c9#3e6Gab54Gzuh5NWLfcEsjV3qMuchJimNEAoRgQyyhy85 > > Same in a vue file. Good catch, Victor. The file was indeed listed twice in this situation. Created attachment 184875 [details] [review] Bug 39877: Incremental test runs The idea of this patchset is to allow increment runs for some of our tests. The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15). One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc). From a previous run (last week) I noted this top 5: xt/vue_tidy.t 105446 ms xt/perltidy.t 175970 ms xt/js_tidy.t 239615 ms xt/author/codespell.t 467041 ms xt/pl_valid.t 2502354 ms We are going to merge js_tidy.t and vue_tidy.t as it does no longer make sense to have a difference between them (we had at the beginning of the vue work, because of differences in the options, now everything is handled in the .prettierrc config file) Once this is pushed, we will be able to create a new project on gitlab and pass some env variables from Jenkins. Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results When everything will be setup, jenkins will run the tests one the whole test suite, then the next run will be "incremental": we are going to run the tests on the files that were failing and the files modified by the last push. Example: * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm * koha-ci-results/perltidy/abc is created and contains ['Koha.pm'] * "commit def" has a modification on about.pl * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy * koha-ci-results/perltidy/def is created and contains [] Test plan: * Create an empty project on gitlab (keep the default readme) eg. https://gitlab.com/me/koha-ci-results * Generate an access token with the developer role (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens) Copy the token (in the following noted as TOKEN_REPLACE_ME) * Unprotect the "main" branch * Run the following prove command: ``` KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \ KOHA_CI_INCREMENTAL_RUNS=1 \ KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \ KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \ prove xt/perltidy.t ``` => a new commit is created on your repo * Add some mess in one of your perl file, commit using `--no-verify` to bypass the git hook * Run the prove command again => only the file you have modified is tested => a new commit is created and the file is reported as failing * Add another commit, fix the failure, add other mess => confirm that what you are seeing makes sense :) * Ideally redo with the different tests modified by this patchset * Note that you can use the incremental run without the report (the report will be only for Jenkins). It will help RM as xt tests will run (way) faster. Note that you can also export the following env vars to set git author detail: * GIT_COMMITTER_NAME * GIT_COMMITTER_EMAIL * GIT_AUTHOR_NAME * GIT_AUTHOR_EMAIL Question: What are we going to use on Jenkins? Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main). Maybe later we will also use it for stable branches. Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184876 [details] [review] Bug 39877: Remove files that have been deleted Ignore deleted or renamed files Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184877 [details] [review] Bug 39877: Merge vue and js tidy tests Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184878 [details] [review] Bug 39877: Apply to other slow tests Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184879 [details] [review] Bug 39877: Fix confusion with URL prefix git clone needs https://, but we need to remove it to build the push url Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Created attachment 184880 [details] [review] Bug 39877: Prevent files to be processed twice If the file had a failure and still has, we should not list it twice. Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> >> Let it create a readme, don't untick the readme creation. Otherwise, there >> will be no branch to unprotect and there won't be an actual repo. (clone >> will fail) > I will deal with the setup once this is pushed to prevent such problems. oops, I forgot to say it was a testing note and that I amended the test plan. Anyway, it works! It super cool, thanks! :D |