Bug 39877 - CI - Incremental runs
Summary: CI - Incremental runs
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Test Suite (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement
Assignee: Jonathan Druart
QA Contact: Kyle M Hall (khall)
URL:
Keywords:
Depends on: 39876
Blocks:
  Show dependency treegraph
 
Reported: 2025-05-12 13:40 UTC by Jonathan Druart
Modified: 2025-09-19 13:18 UTC (History)
5 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 39877: Incremental test runs (12.90 KB, patch)
2025-05-12 20:35 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.08 KB, patch)
2025-05-12 20:35 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.62 KB, patch)
2025-05-12 20:35 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.15 KB, patch)
2025-05-12 20:36 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.30 KB, patch)
2025-05-13 10:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.08 KB, patch)
2025-05-13 10:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.62 KB, patch)
2025-05-13 10:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.15 KB, patch)
2025-05-13 10:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.33 KB, patch)
2025-05-13 12:32 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.08 KB, patch)
2025-05-13 12:32 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.62 KB, patch)
2025-05-13 12:32 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.15 KB, patch)
2025-05-13 12:32 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.45 KB, patch)
2025-07-04 16:31 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.08 KB, patch)
2025-07-04 16:31 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.62 KB, patch)
2025-07-04 16:31 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.15 KB, patch)
2025-07-04 16:31 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.46 KB, patch)
2025-07-22 07:52 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.08 KB, patch)
2025-07-22 07:52 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.62 KB, patch)
2025-07-22 07:52 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.15 KB, patch)
2025-07-22 07:52 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Fix confusion with URL prefix (2.49 KB, patch)
2025-07-22 07:52 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.48 KB, patch)
2025-07-29 05:08 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.08 KB, patch)
2025-07-29 05:08 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.69 KB, patch)
2025-07-29 05:08 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.10 KB, patch)
2025-07-29 05:08 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Fix confusion with URL prefix (2.49 KB, patch)
2025-07-29 05:08 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Prevent files to be processed twice (1.06 KB, patch)
2025-07-29 08:04 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.53 KB, patch)
2025-07-30 12:26 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.13 KB, patch)
2025-07-30 12:26 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.74 KB, patch)
2025-07-30 12:26 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.16 KB, patch)
2025-07-30 12:26 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Fix confusion with URL prefix (2.55 KB, patch)
2025-07-30 12:26 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Prevent files to be processed twice (1.12 KB, patch)
2025-07-30 12:26 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.60 KB, patch)
2025-09-10 12:19 UTC, Paul Derscheid
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.19 KB, patch)
2025-09-10 12:19 UTC, Paul Derscheid
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.80 KB, patch)
2025-09-10 12:19 UTC, Paul Derscheid
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.23 KB, patch)
2025-09-10 12:19 UTC, Paul Derscheid
Details | Diff | Splinter Review
Bug 39877: Fix confusion with URL prefix (2.61 KB, patch)
2025-09-10 12:19 UTC, Paul Derscheid
Details | Diff | Splinter Review
Bug 39877: Prevent files to be processed twice (1.18 KB, patch)
2025-09-10 12:19 UTC, Paul Derscheid
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.65 KB, patch)
2025-09-10 14:17 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.25 KB, patch)
2025-09-10 14:17 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.86 KB, patch)
2025-09-10 14:18 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.28 KB, patch)
2025-09-10 14:18 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 39877: Fix confusion with URL prefix (2.66 KB, patch)
2025-09-10 14:18 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 39877: Prevent files to be processed twice (1.24 KB, patch)
2025-09-10 14:18 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 39877: Incremental test runs (13.65 KB, patch)
2025-09-11 08:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Remove files that have been deleted (1.25 KB, patch)
2025-09-11 08:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Merge vue and js tidy tests (2.86 KB, patch)
2025-09-11 08:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Apply to other slow tests (5.28 KB, patch)
2025-09-11 08:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Fix confusion with URL prefix (2.66 KB, patch)
2025-09-11 08:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Prevent files to be processed twice (1.23 KB, patch)
2025-09-11 08:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Do not clone if env var is not set (1.98 KB, patch)
2025-09-11 08:21 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Use KOHA_IMAGE to separate OSes (3.30 KB, patch)
2025-09-11 09:41 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 39877: Use OS codename to separate OSes (3.94 KB, patch)
2025-09-19 13:05 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2025-05-12 13:40:49 UTC
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

We should improve that.

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).


From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms
Comment 1 Jonathan Druart 2025-05-12 20:35:53 UTC
Created attachment 182308 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?
Comment 2 Jonathan Druart 2025-05-12 20:35:56 UTC
Created attachment 182309 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files
Comment 3 Jonathan Druart 2025-05-12 20:35:58 UTC
Created attachment 182310 [details] [review]
Bug 39877: Merge vue and js tidy tests
Comment 4 Jonathan Druart 2025-05-12 20:36:05 UTC
Created attachment 182311 [details] [review]
Bug 39877: Apply to other slow tests
Comment 5 Jonathan Druart 2025-05-12 20:40:20 UTC
Discussion for QA: How should we handle concurrency? It's crucial to prevent multiple tests from pushing simultaneously. To minimize the risk of the repository becoming outdated, we might need to perform a pull right before each push. Is that enough? Keep in mind that only one job will be run at the same time (Koha_Main).

Maybe later we will also use it for stable branches.
Comment 6 Jonathan Druart 2025-05-12 20:42:42 UTC
An important note I forgot in the commit message. You can use the incremental run without the report (the report will be only for Jenkins).
So it will help RM as xt tests will run (way) faster.
Comment 7 Jonathan Druart 2025-05-12 21:06:24 UTC
Note that we may miss some bad bugs. For instance if a version of codespell changes and that catch more typo, we will miss them with the incremental run even if the test fails in a regular run. Could also happen with pl_valid I guess if an external lib introduces changes that breaks our code. But they really are edge cases we cannot avoid and that should be caught by the qa script anyway.
Comment 8 Jonathan Druart 2025-05-13 10:11:44 UTC
Created attachment 182347 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.
Comment 9 Jonathan Druart 2025-05-13 10:11:47 UTC
Created attachment 182348 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files
Comment 10 Jonathan Druart 2025-05-13 10:11:50 UTC
Created attachment 182349 [details] [review]
Bug 39877: Merge vue and js tidy tests
Comment 11 Jonathan Druart 2025-05-13 10:11:53 UTC
Created attachment 182350 [details] [review]
Bug 39877: Apply to other slow tests
Comment 12 Jonathan Druart 2025-05-13 12:32:30 UTC
Created attachment 182355 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.
Comment 13 Jonathan Druart 2025-05-13 12:32:33 UTC
Created attachment 182356 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files
Comment 14 Jonathan Druart 2025-05-13 12:32:35 UTC
Created attachment 182357 [details] [review]
Bug 39877: Merge vue and js tidy tests
Comment 15 Jonathan Druart 2025-05-13 12:32:38 UTC
Created attachment 182358 [details] [review]
Bug 39877: Apply to other slow tests
Comment 16 Julian Maurice 2025-05-13 12:39:01 UTC
> xt/pl_valid.t          2502354 ms
We can also save some time by not checking *.t files, since they are executed anyway. *.t files represent approximately 25% of the total time of pl_valid.t for me.

(In reply to Jonathan Druart from comment #7)
> Could also happen with pl_valid I
> guess if an external lib introduces changes that breaks our code. But they
> really are edge cases we cannot avoid and that should be caught by the qa
> script anyway.
If an exported subroutine is removed/renamed, code that still use/import it will fail to compile, but this won't get caught if we are only checking modified files. Will it be caught by the QA script ?
Comment 17 Jonathan Druart 2025-05-13 12:51:04 UTC
(In reply to Julian Maurice from comment #16)
> > xt/pl_valid.t          2502354 ms
> We can also save some time by not checking *.t files, since they are
> executed anyway. *.t files represent approximately 25% of the total time of
> pl_valid.t for me.

Could be, yes.

> (In reply to Jonathan Druart from comment #7)
> > Could also happen with pl_valid I
> > guess if an external lib introduces changes that breaks our code. But they
> > really are edge cases we cannot avoid and that should be caught by the qa
> > script anyway.
> If an exported subroutine is removed/renamed, code that still use/import it
> will fail to compile, but this won't get caught if we are only checking
> modified files. Will it be caught by the QA script ?

I have no idea why I have written that, it does not make sense.
Comment 18 Victor Grousset/tuxayo 2025-07-04 16:31:16 UTC
Created attachment 183795 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 19 Victor Grousset/tuxayo 2025-07-04 16:31:18 UTC
Created attachment 183796 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files
Comment 20 Victor Grousset/tuxayo 2025-07-04 16:31:20 UTC
Created attachment 183797 [details] [review]
Bug 39877: Merge vue and js tidy tests
Comment 21 Victor Grousset/tuxayo 2025-07-04 16:31:22 UTC
Created attachment 183798 [details] [review]
Bug 39877: Apply to other slow tests
Comment 22 Victor Grousset/tuxayo 2025-07-04 16:45:18 UTC
Rebased and fixed conflicts. Double-checked with a diff between the old and new patches to confirm there are no unexpected changes introduced.

I'll attempt signing off.
Comment 23 Victor Grousset/tuxayo 2025-07-04 18:53:29 UTC
(In reply to Jonathan Druart from comment #5)
> Discussion for QA: How should we handle concurrency? It's crucial to prevent
> multiple tests from pushing simultaneously. To minimize the risk of the
> repository becoming outdated, we might need to perform a pull right before
> each push. Is that enough? Keep in mind that only one job will be run at the
> same time (Koha_Main).

Looks good to me. If some mess occasionally comes up, someone will just need to manually commit changes to add some files that might not have been added to the list of files to recheck. Is that all? Or just rerun locally and that's it without bothering with the repo.


> Maybe later we will also use it for stable branches.

Which could have their own branch in koha-ci-results right? So it shouldn't cause race conditions.
Comment 24 Jonathan Druart 2025-07-07 13:02:56 UTC
(In reply to Victor Grousset/tuxayo from comment #23)
> (In reply to Jonathan Druart from comment #5)
> > Maybe later we will also use it for stable branches.
> 
> Which could have their own branch in koha-ci-results right? So it shouldn't
> cause race conditions.

We should not need separate branches.
Comment 25 Victor Grousset/tuxayo 2025-07-22 01:31:32 UTC
It seems the initial clone expects an https:// prefix.

kohadev-koha@kohadevbox:koha(testing-SO-QA)$ KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/tuxayo/koha-ci-results.git KOHA_CI_INCREMENTAL_RUNS=1 KOHA_CI_INCREMENTAL_RUNS_REPORT=1 KOHA_CI_INCREMENTAL_RUNS_TOKEN=XXXXXX prove xt/perltidy.t
xt/perltidy.t .. 
fatal: repository 'gitlab.com/tuxayo/koha-ci-results.git' does not exist
fatal: cannot change to '/tmp/koha-ci-results': No such file or directory
xt/perltidy.t .. 3/3031 ^C

(cleanup)

So it goes further with it. But then (the push?) ends up with https://https://

kohadev-koha@kohadevbox:koha(testing-SO-QA)$ KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/tuxayo/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=XXXXXXXXXXX \
prove xt/perltidy.t
xt/perltidy.t .. Cloning into '/tmp/koha-ci-results'...
xt/perltidy.t .. 3007/3031 fatal: unable to access 'https://https://gitlab.com/tuxayo/koha-ci-results.git/': Could not resolve host: https
xt/perltidy.t .. ok
All tests successful.
Files=1, Tests=3031, 216 wallclock secs ( 0.41 usr  0.04 sys + 207.40 cusr  1.50 csys = 209.35 CPU)
Result: PASS
Comment 26 Jonathan Druart 2025-07-22 07:52:43 UTC
Created attachment 184462 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 27 Jonathan Druart 2025-07-22 07:52:46 UTC
Created attachment 184463 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files
Comment 28 Jonathan Druart 2025-07-22 07:52:48 UTC
Created attachment 184464 [details] [review]
Bug 39877: Merge vue and js tidy tests
Comment 29 Jonathan Druart 2025-07-22 07:52:50 UTC
Created attachment 184465 [details] [review]
Bug 39877: Apply to other slow tests
Comment 30 Jonathan Druart 2025-07-22 07:52:53 UTC
Created attachment 184466 [details] [review]
Bug 39877: Fix confusion with URL prefix

git clone needs https://, but we need to remove it to build the push url
Comment 31 Jonathan Druart 2025-07-22 07:54:06 UTC
(In reply to Victor Grousset/tuxayo from comment #25)
> It seems the initial clone expects an https:// prefix.

Indeed, I mixed up things it seems.

Should be fixed now. I've amended the commit message of the first patch and added a fix ("Fix confusion with URL prefix")
Comment 32 Victor Grousset/tuxayo 2025-07-29 05:08:01 UTC
Created attachment 184779 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab (keep the default readme)
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 33 Victor Grousset/tuxayo 2025-07-29 05:08:04 UTC
Created attachment 184780 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files
Comment 34 Victor Grousset/tuxayo 2025-07-29 05:08:07 UTC
Created attachment 184781 [details] [review]
Bug 39877: Merge vue and js tidy tests

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 35 Victor Grousset/tuxayo 2025-07-29 05:08:10 UTC
Created attachment 184782 [details] [review]
Bug 39877: Apply to other slow tests
Comment 36 Victor Grousset/tuxayo 2025-07-29 05:08:12 UTC
Created attachment 184783 [details] [review]
Bug 39877: Fix confusion with URL prefix

git clone needs https://, but we need to remove it to build the push url
Comment 37 Victor Grousset/tuxayo 2025-07-29 05:09:30 UTC
> * Create an empty project on gitlab
>  eg. https://gitlab.com/me/koha-ci-results

Let it create a readme, don't untick the readme creation. Otherwise, there will be no branch to unprotect and there won't be an actual repo. (clone will fail)


----


rebased and fixed conflicts (diffs compared to confirm no unwanted change introduced)
Comment 38 Victor Grousset/tuxayo 2025-07-29 06:04:47 UTC
> * Add another commit, fix the failure, add other mess

If adding the other mess in the same file where I fixed it. The new issue is outputted twice. Is that a relevant issue?
https://paste.evolix.org/?6e80d183272f48c9#3e6Gab54Gzuh5NWLfcEsjV3qMuchJimNEAoRgQyyhy85

Same in a vue file.


---


other testing:
On main from a week ago, I applied the patch changes without committing. So HEAD is from main. But the new tests are there. (actual code doesn't match the commit but it just a hack to test the incremental run in another situation)
The full run happened as expected.
After a proper application of the patches, the next test run only processed a subset of the files: ☑️
Comment 39 Jonathan Druart 2025-07-29 08:04:14 UTC
Created attachment 184785 [details] [review]
Bug 39877: Prevent files to be processed twice

If the file had a failure and still has, we should not list it twice.
Comment 40 Jonathan Druart 2025-07-29 08:05:31 UTC
(In reply to Victor Grousset/tuxayo from comment #37)
> > * Create an empty project on gitlab
> >  eg. https://gitlab.com/me/koha-ci-results
> 
> Let it create a readme, don't untick the readme creation. Otherwise, there
> will be no branch to unprotect and there won't be an actual repo. (clone
> will fail)

I will deal with the setup once this is pushed to prevent such problems.


(In reply to Victor Grousset/tuxayo from comment #38)
> > * Add another commit, fix the failure, add other mess
> 
> If adding the other mess in the same file where I fixed it. The new issue is
> outputted twice. Is that a relevant issue?
> https://paste.evolix.org/
> ?6e80d183272f48c9#3e6Gab54Gzuh5NWLfcEsjV3qMuchJimNEAoRgQyyhy85
> 
> Same in a vue file.

Good catch, Victor. The file was indeed listed twice in this situation.
Comment 41 Victor Grousset/tuxayo 2025-07-30 12:26:37 UTC
Created attachment 184875 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab (keep the default readme)
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 42 Victor Grousset/tuxayo 2025-07-30 12:26:40 UTC
Created attachment 184876 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 43 Victor Grousset/tuxayo 2025-07-30 12:26:42 UTC
Created attachment 184877 [details] [review]
Bug 39877: Merge vue and js tidy tests

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 44 Victor Grousset/tuxayo 2025-07-30 12:26:45 UTC
Created attachment 184878 [details] [review]
Bug 39877: Apply to other slow tests

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 45 Victor Grousset/tuxayo 2025-07-30 12:26:47 UTC
Created attachment 184879 [details] [review]
Bug 39877: Fix confusion with URL prefix

git clone needs https://, but we need to remove it to build the push url

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 46 Victor Grousset/tuxayo 2025-07-30 12:26:50 UTC
Created attachment 184880 [details] [review]
Bug 39877: Prevent files to be processed twice

If the file had a failure and still has, we should not list it twice.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 47 Victor Grousset/tuxayo 2025-07-30 12:31:28 UTC
>> Let it create a readme, don't untick the readme creation. Otherwise, there
>> will be no branch to unprotect and there won't be an actual repo. (clone
>> will fail)

> I will deal with the setup once this is pushed to prevent such problems.

oops, I forgot to say it was a testing note and that I amended the test plan.

Anyway, it works! It super cool, thanks! :D
Comment 48 Victor Grousset/tuxayo 2025-08-19 15:07:24 UTC
@QA: Redirecting `git log --pretty=format:"%h"` in a file takes 0.2 sec for the 58000 commits of the koha repo, on my slow laptop.
And it's already large and will take like 15 years to double.

The ls in koha-ci-results/$test_name shouldn't become slow with time.

Cloning the repo from scratch might eventually be a bit slow. But then the same approach could be done as for cloning the Koha repo.

Is there anything else that could become slow as time passes and the history of tested commits grows? Nothing else found on my side.
Comment 49 Jonathan Druart 2025-09-01 13:15:33 UTC
(In reply to Victor Grousset/tuxayo from comment #48)
> @QA: Redirecting `git log --pretty=format:"%h"` in a file takes 0.2 sec for
> the 58000 commits of the koha repo, on my slow laptop.
> And it's already large and will take like 15 years to double.
> 
> The ls in koha-ci-results/$test_name shouldn't become slow with time.
> 
> Cloning the repo from scratch might eventually be a bit slow. But then the
> same approach could be done as for cloning the Koha repo.
> 
> Is there anything else that could become slow as time passes and the history
> of tested commits grows? Nothing else found on my side.

We will see when the problem appears... :D
We can still empty the repo or create branches. I won't try to fix problem that we don't have yet. The problem now is the execution time of the test suite, and it's been 4 months that I am waiting for this to be pushed.
Comment 50 Kyle M Hall (khall) 2025-09-05 12:29:04 UTC
I seem to be getting errors. I'm testing in koha-testing-docker. It appears to be writing the files to temp, but doesn't seem to be cloning the git repo into that path? The dir exists, but I can confirm it is not a git repo, nor is my gitlab repo being updated. I did a manual close to that path but it didn't seem to make a difference.

kohadev-koha@kohadevbox:koha(bug39877-qa)$ KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/kylemhall/koha-ci-results.git KOHA_CI_INCREMENTAL_RUNS=1 KOHA_CI_INCREMENTAL_RUNS_REPORT=1 KOHA_CI_INCREMENTAL_RUNS_TOKEN=glpat-XXX.01.120je4tjd prove xt/perltidy.t
xt/perltidy.t .. fatal: repository 'gitlab.com/kylemhall/koha-ci-results.git' does not exist
fatal: cannot change to '/tmp/koha-ci-results': No such file or directory
xt/perltidy.t .. 3044/3054 fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
xt/perltidy.t .. ok
All tests successful.
Files=1, Tests=3054, 103 wallclock secs ( 0.22 usr  0.04 sys + 100.72 cusr  0.76 csys = 101.74 CPU)
Result: PASS
kohadev-koha@kohadevbox:koha(bug39877-qa)$ KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/kylemhall/koha-ci-results.git KOHA_CI_INCREMENTAL_RUNS=1 KOHA_CI_INCREMENTAL_RUNS_REPORT=1 KOHA_CI_INCREMENTAL_RUNS_TOKEN=glpat-XXX.01.120je4tjd prove xt/perltidy.t
xt/perltidy.t .. fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
xt/perltidy.t .. ok
All tests successful.
Files=1, Tests=1,  1 wallclock secs ( 0.01 usr  0.00 sys +  0.33 cusr  0.26 csys =  0.60 CPU)
Result: PASS
kohadev-koha@kohadevbox:koha(bug39877-qa)$ cat /tmp/koha-ci-results/perltidy/dae0cf02d1f
[]kohadev-koha@kohadevbox:koha(bug39877-qa)$ #make a change and commit
kohadev-koha@kohadevbox:koha(bug39877-qa)$ git show HEAD
commit 154f96a18d6292e51d074a5aad218d1c01f0c682 (HEAD -> bug39877-qa)
Author: Kyle M Hall <kyle@bywatersolutions.com>
Date:   Fri Sep 5 08:23:25 2025 -0400

    Test commit

diff --git a/mainpage.pl b/mainpage.pl
index 87f784631f6..205bc8ad7dd 100755
--- a/mainpage.pl
+++ b/mainpage.pl
@@ -36,7 +36,7 @@ use Koha::Quotes;
 use Koha::Suggestions;
 use Koha::BackgroundJobs;
 use Koha::CurbsidePickups;
-use Koha::Tickets;
+   use Koha::Tickets;

 my $query = CGI->new;

kohadev-koha@kohadevbox:koha(bug39877-qa)$ KOHA_CI_INCREMENTAL_RUN_REPO_URL=gitlab.com/kylemhall/koha-ci-results.git KOHA_CI_INCREMENTAL_RUNS=1 KOHA_CI_INCREMENTAL_RUNS_REPORT=1 KOHA_CI_INCREMENTAL_RUNS_TOKEN=glpat-XXX.01.120je4tjd prove xt/perltidy.t
xt/perltidy.t .. fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
# The file 'mainpage.pl' is not tidy
# +---+----------------------------+----------------------------+
# | 36|use Koha::Suggestions;      |use Koha::Suggestions;      |
# | 37|use Koha::BackgroundJobs;   |use Koha::BackgroundJobs;   |
# | 38|use Koha::CurbsidePickups;  |use Koha::CurbsidePickups;  |
# * 39|   use Koha::Tickets;       |use Koha::Tickets;          *
# | 40|                            |                            |
# | 41|my $query = CGI->new;       |my $query = CGI->new;       |
# | 42|                            |                            |
# +---+----------------------------+----------------------------+
xt/perltidy.t .. 1/2
#   Failed test at xt/perltidy.t line 16.
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
# Looks like you failed 1 test of 2.
xt/perltidy.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests

Test Summary Report
-------------------
xt/perltidy.t (Wstat: 256 (exited 1) Tests: 2 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=1, Tests=2,  1 wallclock secs ( 0.01 usr  0.00 sys +  0.42 cusr  0.26 csys =  0.69 CPU)
Result: FAIL
kohadev-koha@kohadevbox:koha(bug39877-qa)$ cat /tmp/koha-ci-results/perltidy/
154f96a18d6  dae0cf02d1f
kohadev-koha@kohadevbox:koha(bug39877-qa)$ cat /tmp/koha-ci-results/perltidy/154f96a18d6
["mainpage.pl"]kohadev-koha@kohadevbox:koha(bug39877-qa)$
Comment 51 Jonathan Druart 2025-09-05 12:45:50 UTC
Hi Kyle,
Make sure you are using the test plan from the last version of the patches

    KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
    KOHA_CI_INCREMENTAL_RUNS=1 \
    KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
    KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
    prove xt/perltidy.t

You are missing "https://" in the REPO_URL.
Comment 52 Paul Derscheid 2025-09-10 12:19:36 UTC
Created attachment 186320 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab (keep the default readme)
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Comment 53 Paul Derscheid 2025-09-10 12:19:38 UTC
Created attachment 186321 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Comment 54 Paul Derscheid 2025-09-10 12:19:40 UTC
Created attachment 186322 [details] [review]
Bug 39877: Merge vue and js tidy tests

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Comment 55 Paul Derscheid 2025-09-10 12:19:43 UTC
Created attachment 186323 [details] [review]
Bug 39877: Apply to other slow tests

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Comment 56 Paul Derscheid 2025-09-10 12:19:45 UTC
Created attachment 186324 [details] [review]
Bug 39877: Fix confusion with URL prefix

git clone needs https://, but we need to remove it to build the push url

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Comment 57 Paul Derscheid 2025-09-10 12:19:47 UTC
Created attachment 186325 [details] [review]
Bug 39877: Prevent files to be processed twice

If the file had a failure and still has, we should not list it twice.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Comment 58 Paul Derscheid 2025-09-10 12:21:07 UTC
This is awesome! I would pass QA, but we need more eyes on this.

For the next QA reviewer, you can skip running the tests in the QA script. They all pass.
Comment 59 Kyle M Hall (khall) 2025-09-10 13:41:04 UTC
Comment on attachment 186320 [details] [review]
Bug 39877: Incremental test runs

Review of attachment 186320 [details] [review]:
-----------------------------------------------------------------

::: Koha/Devel/CI/IncrementalRuns.pm
@@ +92,5 @@
> +        $self->{test_name} =~ s|\..*$||g;
> +    }
> +
> +    if ( $self->{git_repo_dir} && $self->{repo_url} ) {
> +        unless ( -d $self->{git_repo_dir} ) {

Should check to see if the dir exists *and* that it is a git dir. Exit with an error or init git in the dir?
Comment 60 Kyle M Hall (khall) 2025-09-10 14:17:48 UTC
Created attachment 186327 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab (keep the default readme)
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: John Doe <you@example.com>
Comment 61 Kyle M Hall (khall) 2025-09-10 14:17:58 UTC
Created attachment 186328 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: John Doe <you@example.com>
Comment 62 Kyle M Hall (khall) 2025-09-10 14:18:01 UTC
Created attachment 186329 [details] [review]
Bug 39877: Merge vue and js tidy tests

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: John Doe <you@example.com>
Comment 63 Kyle M Hall (khall) 2025-09-10 14:18:05 UTC
Created attachment 186330 [details] [review]
Bug 39877: Apply to other slow tests

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: John Doe <you@example.com>
Comment 64 Kyle M Hall (khall) 2025-09-10 14:18:09 UTC
Created attachment 186331 [details] [review]
Bug 39877: Fix confusion with URL prefix

git clone needs https://, but we need to remove it to build the push url

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: John Doe <you@example.com>
Comment 65 Kyle M Hall (khall) 2025-09-10 14:18:12 UTC
Created attachment 186332 [details] [review]
Bug 39877: Prevent files to be processed twice

If the file had a failure and still has, we should not list it twice.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: John Doe <you@example.com>
Comment 66 Kyle M Hall (khall) 2025-09-10 14:22:29 UTC
Comment on attachment 186320 [details] [review]
Bug 39877: Incremental test runs

Review of attachment 186320 [details] [review]:
-----------------------------------------------------------------

Other than this one line the only question I have is should we be using something like Git.pm for git actions instead of system commands? Perl has to spawn a shell for each git command, but it's only a handful of commands at startup so I imagine the difference wouldn't be very noticeable if at all.
Comment 67 Kyle M Hall (khall) 2025-09-10 14:29:40 UTC
> Discussion for QA: How should we handle concurrency?
> It's crucial to prevent multiple tests from pushing simultaneously.
> To minimize the risk of the repository becoming outdated, we might need to
> perform
> a pull right before each push. Is that enough? Keep in mind that only one
> job will
> be run at the same time (Koha_Main).
> Maybe later we will also use it for stable branches.

Should each running specify a name, and maybe a branch? So
koha-ci-results/perltidy
would be koha-ci-results/$branch/$runner/perltidy/$rev

Branch knowability would be a possible issue, `git branch -vv` lists the tracked branch iirc.
Comment 68 Paul Derscheid 2025-09-10 14:37:04 UTC
Hmm, the graceful handling of an error in git dir init makes sense.
We should probably abort then so that we don't overlook any problems that this might mask if we just initialized?
Comment 69 Kyle M Hall (khall) 2025-09-10 15:18:18 UTC
(In reply to Paul Derscheid from comment #68)
> Hmm, the graceful handling of an error in git dir init makes sense.
> We should probably abort then so that we don't overlook any problems that
> this might mask if we just initialized?

That sounds good to me!
Comment 70 Jonathan Druart 2025-09-11 08:10:51 UTC
(In reply to Kyle M Hall (khall) from comment #66)
> Comment on attachment 186320 [details] [review] [review]
> Bug 39877: Incremental test runs
> 
> Review of attachment 186320 [details] [review] [review]:
> -----------------------------------------------------------------
> 
> Other than this one line the only question I have is should we be using
> something like Git.pm for git actions instead of system commands? Perl has
> to spawn a shell for each git command, but it's only a handful of commands
> at startup so I imagine the difference wouldn't be very noticeable if at all.

Hum, I have initially started with Git::Wrapper (that we already use in the auto_rebase.pl script), but I think I found a blocker and decided to go with straight qx calls. But to be honest I don't remember what was the problem.
Comment 71 Jonathan Druart 2025-09-11 08:21:19 UTC
Created attachment 186362 [details] [review]
Bug 39877: Incremental test runs

The idea of this patchset is to allow increment runs for some of our
tests.
The current (full) test suite runs in 2h20 (Koha_Main/3253 on Docker_15).

One of the main reason is the recent introduction of new xt tests that are run on all files (valid, tidy, pod, codespell, etc).

From a previous run (last week) I noted this top 5:

xt/vue_tidy.t          105446 ms
xt/perltidy.t          175970 ms
xt/js_tidy.t           239615 ms
xt/author/codespell.t  467041 ms
xt/pl_valid.t          2502354 ms

We are going to merge js_tidy.t and vue_tidy.t as it does no longer make
sense to have a difference between them (we had at the beginning of the
vue work, because of differences in the options, now everything is
handled in the .prettierrc config file)

Once this is pushed, we will be able to create a new project on gitlab
and pass some env variables from Jenkins.
Suggestion: project would be named https://gitlab.com/koha-community/koha-ci-results
When everything will be setup, jenkins will run the tests one the whole
test suite, then the next run will be "incremental": we are going to run
the tests on the files that were failing and the files modified by the
last push.
Example:
 * "commit abc": all the files are tested, xt/perltidy.t is failing on Koha.pm
 * koha-ci-results/perltidy/abc is created and contains ['Koha.pm']
 * "commit def" has a modification on about.pl
 * xt/perltidy.t is checking tidy on Koha.pm and about.pl, both are tidy
 * koha-ci-results/perltidy/def is created and contains []

Test plan:
* Create an empty project on gitlab (keep the default readme)
  eg. https://gitlab.com/me/koha-ci-results
* Generate an access token with the developer role
  (https://gitlab.com/me/koha-ci-results/-/settings/access_tokens)
  Copy the token (in the following noted as TOKEN_REPLACE_ME)
* Unprotect the "main" branch
* Run the following prove command:
```
KOHA_CI_INCREMENTAL_RUN_REPO_URL=https://gitlab.com/me/koha-ci-results.git \
KOHA_CI_INCREMENTAL_RUNS=1 \
KOHA_CI_INCREMENTAL_RUNS_REPORT=1 \
KOHA_CI_INCREMENTAL_RUNS_TOKEN=TOKEN_REPLACE_ME \
prove xt/perltidy.t
```
=> a new commit is created on your repo
* Add some mess in one of your perl file, commit using `--no-verify` to
  bypass the git hook
* Run the prove command again
=> only the file you have modified is tested
=> a new commit is created and the file is reported as failing
* Add another commit, fix the failure, add other mess
=> confirm that what you are seeing makes sense :)

* Ideally redo with the different tests modified by this patchset

* Note that you can use the incremental run without the report (the report will
be only for Jenkins).
It will help RM as xt tests will run (way) faster.

Note that you can also export the following env vars to set git author
detail:
 * GIT_COMMITTER_NAME
 * GIT_COMMITTER_EMAIL
 * GIT_AUTHOR_NAME
 * GIT_AUTHOR_EMAIL

Question: What are we going to use on Jenkins?

Discussion for QA: How should we handle concurrency?
It's crucial to prevent multiple tests from pushing simultaneously.
To minimize the risk of the repository becoming outdated, we might need to perform
a pull right before each push. Is that enough? Keep in mind that only one job will
be run at the same time (Koha_Main).
Maybe later we will also use it for stable branches.

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 72 Jonathan Druart 2025-09-11 08:21:22 UTC
Created attachment 186363 [details] [review]
Bug 39877: Remove files that have been deleted

Ignore deleted or renamed files

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 73 Jonathan Druart 2025-09-11 08:21:24 UTC
Created attachment 186364 [details] [review]
Bug 39877: Merge vue and js tidy tests

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 74 Jonathan Druart 2025-09-11 08:21:27 UTC
Created attachment 186365 [details] [review]
Bug 39877: Apply to other slow tests

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 75 Jonathan Druart 2025-09-11 08:21:29 UTC
Created attachment 186366 [details] [review]
Bug 39877: Fix confusion with URL prefix

git clone needs https://, but we need to remove it to build the push url

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 76 Jonathan Druart 2025-09-11 08:21:32 UTC
Created attachment 186367 [details] [review]
Bug 39877: Prevent files to be processed twice

If the file had a failure and still has, we should not list it twice.

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 77 Jonathan Druart 2025-09-11 08:21:34 UTC
Created attachment 186368 [details] [review]
Bug 39877: Do not clone if env var is not set

If we don't want the incremental run we should return early in ->new to
prevent the git clone.
Comment 78 Jonathan Druart 2025-09-11 08:22:28 UTC
(In reply to Kyle M Hall (khall) from comment #65)
> Signed-off-by: John Doe <you@example.com>

replaced with Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

And added a QA check https://gitlab.com/koha-community/qa-test-tools/-/issues/97
Comment 79 Jonathan Druart 2025-09-11 08:23:15 UTC
(In reply to Kyle M Hall (khall) from comment #69)
> (In reply to Paul Derscheid from comment #68)
> > Hmm, the graceful handling of an error in git dir init makes sense.
> > We should probably abort then so that we don't overlook any problems that
> > this might mask if we just initialized?
> 
> That sounds good to me!

I think "Bug 39877: Do not clone if env var is not set" is what we want here. If the env var is not set we should not try to clone actually.
Comment 80 Jonathan Druart 2025-09-11 08:24:21 UTC
(In reply to Kyle M Hall (khall) from comment #67)
> > Discussion for QA: How should we handle concurrency?
> > It's crucial to prevent multiple tests from pushing simultaneously.
> > To minimize the risk of the repository becoming outdated, we might need to
> > perform
> > a pull right before each push. Is that enough? Keep in mind that only one
> > job will
> > be run at the same time (Koha_Main).
> > Maybe later we will also use it for stable branches.
>
> Should each running specify a name, and maybe a branch? So
> koha-ci-results/perltidy
> would be koha-ci-results/$branch/$runner/perltidy/$rev

Yes, we need to adjust that now, before this gets pushed.
Actually it is a problem and we want to catch stuff like bug 40680 (xt/pl_valid.t crashes on U24 only), and the codespell problem described in comment 7.

However I don't think it should be $runner. Maybe we need $os: remember that we want RM and RMaints to run those slow tests before they push, and we don't want them to specify the jenkins's job name, we could pass the "OS code" automatically from ktd (D12, D13, U24).

> Branch knowability would be a possible issue, `git branch -vv` lists the
> tracked branch iirc.

Regarding the branch I think we can go with a single "main" branch for now, and list all commit ids. It's a problem we can fix later if it needs to be fixed (and we will create branches on koha-ci-results).
Comment 81 Jonathan Druart 2025-09-11 09:41:48 UTC
Created attachment 186370 [details] [review]
Bug 39877: Use KOHA_IMAGE to separate OSes
Comment 82 Jonathan Druart 2025-09-11 09:43:22 UTC
(In reply to Jonathan Druart from comment #81)
> Created attachment 186370 [details] [review] [review]
> Bug 39877: Use KOHA_IMAGE to separate OSes

We could do this. Would that be enough?

KOHA_IMAGE is exported by jenkins, then pass to the container. It's available in the root env var list, not for the kohadev user (more tweak needed here, but it would work for jenkins as a first step).
Comment 83 Jonathan Druart 2025-09-11 09:44:47 UTC
(In reply to Jonathan Druart from comment #82)
> (In reply to Jonathan Druart from comment #81)
> > Created attachment 186370 [details] [review] [review] [review]
> > Bug 39877: Use KOHA_IMAGE to separate OSes
> 
> We could do this. Would that be enough?
> 
> KOHA_IMAGE is exported by jenkins, then pass to the container. It's
> available in the root env var list, not for the kohadev user (more tweak
> needed here, but it would work for jenkins as a first step).

Hum, it won't work when we adjust the OS to track for "main" (ie. when we increment the default Debian version).
Comment 84 Jonathan Druart 2025-09-19 13:05:12 UTC
Created attachment 186622 [details] [review]
Bug 39877: Use OS codename to separate OSes
Comment 85 Jonathan Druart 2025-09-19 13:07:05 UTC
(In reply to Jonathan Druart from comment #84)
> Created attachment 186622 [details] [review] [review]
> Bug 39877: Use OS codename to separate OSes

Discussed with Tomas and we decided to use lsb_release to get the OS's codename.

koha-ci-results/${test_name}/${codename}/${commit_id}

eg.

koha-ci-results/perltidy/bookworm/d2657382c93
Comment 86 Jonathan Druart 2025-09-19 13:09:53 UTC
(Tomas is going to install lsb-release in ktd)
Comment 87 Jonathan Druart 2025-09-19 13:18:00 UTC
(In reply to Jonathan Druart from comment #84)
> Created attachment 186622 [details] [review] [review]
> Bug 39877: Use OS codename to separate OSes

So we don't need to add a new variable in ktd, or to pass from the jenkins job. 

And everything is seamless for the users.