Bug 36381 - "not a tree object" error from Jenkins
Summary: "not a tree object" error from Jenkins
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Test Suite (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low major
Assignee: Chris Cormack
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-03-21 08:53 UTC by Jonathan Druart
Modified: 2024-09-23 07:21 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2024-03-21 08:53:04 UTC
"not a tree object" warning on Jenkins.

It's like the test are not run inside a git repository:

koha_1       | error: object directory /srv/jenkins/workspace/Koha_Source/.git/objects does not exist; check .git/objects/info/alternates
koha_1       | error: Could not read 3288f3a2129207626916d7a7b49f580c16d097a1
koha_1       | Nothing to critique at t/00-testcritic.t line 18.
koha_1       | [18:52:15] t/00-testcritic.t ........................................................... 
koha_1       | Dubious, test returned 25 (wstat 6400, 0x1900)
koha_1       | No subtests run 


In this case we have t/00-testcritic.t that is failing.

But even on green runs we see:
koha_1       | error: object directory /srv/jenkins/workspace/Koha_Source/.git/objects does not exist; check .git/objects/info/alternates
koha_1       | fatal: not a tree object
koha_1       | [18:17:56] t/00-testcritic.t

And lot of other "not a tree object" error (I guess everytime we run 'git' commands from tests).
Comment 1 Jonathan Druart 2024-03-21 08:56:18 UTC
From #koha-bots, earlier this year

[08:43:33] <Joubu> So I was looking at: https://jenkins.koha-community.org/job/Koha_Master/2796/consoleText
[08:43:35] <ashimema> haha, no worries 
[08:43:36] <Joubu> the last one
[08:43:37] <Joubu> and noticed:.
[08:43:43] <Joubu> koha_1       | error: object directory /srv/jenkins/workspace/Koha_Source/.git/objects does not exist; check .git/objects/info/alternates
[08:43:46] <Joubu> koha_1       | fatal: not a tree object                                                                                                                                                           
[08:44:00] <Joubu> and I thought it was because or Koha_Source not existing at the beginning of the run



[09:02:29] <Joubu> Does `git ls-tree -d --name-only HEAD` work?
[09:02:55] <Joubu> it should return the list of the directories at the root of Koha
[09:03:56] <ashimema> yup
[09:04:02] <ashimema> looks present and correct
[09:04:10] <Joubu> ok good
[09:04:57] <Joubu> so, Koha_Master is currently running and git command works correctly. We will wait for the end of the run, where the "not a tree object" errors appear
[09:05:36] <Joubu> if they do not appear, it means Koha_Source is not correct (corrupt, or bare repo?) on the nodes where the errors pop up
[09:06:48] <Joubu> look at this one:
[09:06:54] <Joubu> https://jenkins.koha-community.org/job/Koha_23.05_U22/155/consoleText on Docker_7
[09:07:05] <Joubu> => no "not a tree" errors
[09:07:21] <Joubu> the next run https://jenkins.koha-community.org/job/Koha_23.05_U22/156/consoleText on Docker_15
[09:07:26] <Joubu> => lot of "not a tree" errors
[09:07:55] <ashimema> I think jake recloned a bunch yesterday afternoon using github instead of gittea
[09:08:45] <Joubu> this last run of from 2024-02-07 19:27:28.
[09:14:07] <Joubu> ashimema: There is a current run on Docker_15
[09:14:43] <Joubu> the errors didn't appear yet but it would be interesting to inspect the clone there
[09:14:54] <Joubu> and run `git ls-tree -d --name-only HEAD`
[09:16:35] <ashimema> all looks good there too
[09:17:38] <Joubu> hum ok, weird
[09:17:56] <Joubu> let's see if the errors appear then
Comment 2 Jonathan Druart 2024-03-22 15:25:12 UTC
Not getting this error on Docker_7 but the warning 

error: unable to normalize alternate object path: /srv/jenkins/workspace/Koha_Source/.git/objects

Indeed: 

/srv/jenkins/workspace/xxx_test$ grep -r Koha_Source .git
.git/objects/info/alternates:/srv/jenkins/workspace/Koha_Source/.git/objects

The tests are not failing but there warnings is not expected.

I have tried to:
1. Remove "Honor refspec on initial clone"
2. Remove "Shallow clone"

None removed the warning.
Comment 3 Jonathan Druart 2024-03-22 15:25:58 UTC
(Not sure it's related to the "not a tree object" error however)
Comment 4 Katrin Fischer 2024-03-26 09:43:21 UTC
The testcritic failed again, more eyes/brains on this problem welcome.
Comment 5 Jonathan Druart 2024-03-26 09:55:19 UTC
(In reply to Katrin Fischer from comment #4)
> The testcritic failed again, more eyes/brains on this problem welcome.

Koha_Master/2823 on Docker_15
Comment 6 Jonathan Druart 2024-04-03 14:15:32 UTC
I run out of ideas, we need somebody else to look into this.
Comment 7 Jonathan Druart 2024-04-03 14:19:01 UTC
This is happening all the time with Docker_15 (last one is Koha_Master/2827)

It's coming from the 'git ls-tree' calls

t/00-testcritic.t:my @files = map { chomp; $_ } `git ls-tree -r HEAD --name-only`;    # only files part of git
t/Makefile.t:my @files = `git ls-tree HEAD|tr -s '\t' ' '|cut -d' ' -f4`;
xt/check_makefile.t:my @dirs = `git ls-tree -d --name-only HEAD`;

My guess is that it's coming from PTFS-e's nodes.
Comment 8 Tomás Cohen Arazi (tcohen) 2024-04-03 14:20:00 UTC
Have we been able to reproduce one of this errors/warnings locally? It certainly comes from the fact Jenkins is making a shallow clone from a directory.

The tip of the iceberg is the 'Koha_Source', as that name shouldn't be mentioned at all on the Jenkins runs: the clone is named as the task.

Maybe it is a git version thing?
Comment 9 Jonathan Druart 2024-04-03 14:53:26 UTC
(In reply to Tomás Cohen Arazi from comment #8)
> Have we been able to reproduce one of this errors/warnings locally? It
> certainly comes from the fact Jenkins is making a shallow clone from a
> directory.

No, see comment 2. I have tried to remove that.
Comment 10 Jonathan Druart 2024-04-05 07:48:01 UTC
Koha_Master_D12/819 Docker_10

koha_1       | error: unable to normalize alternate object path: /srv/jenkins/workspace/Koha_Source/.git/objects
koha_1       | error: Could not read edda4e1bdb0943e43eb80faeb8b71de397327d04
koha_1       | Nothing to critique at t/00-testcritic.t line 18.
koha_1       | [07:34:05] t/00-testcritic.t 
koha_1       | Dubious, test returned 25 (wstat 6400, 0x1900)
koha_1       | No subtests run
Comment 11 Jonathan Druart 2024-04-05 07:48:39 UTC
Koha_Master/2831 Docker_15
Comment 12 Jonathan Druart 2024-04-05 07:49:52 UTC
They all failed

Koha_Master_U20/1160 Docker_13
Koha_Master_D10/1443 Docker_14
Comment 13 Jonathan Druart 2024-04-05 07:54:47 UTC
With the same id edda4e1bdb0943e43eb80faeb8b71de397327d04:

% git show edda4e1bdb0943e43eb80faeb8b71de397327d04
tree edda4e1bdb0943e43eb80faeb8b71de397327d04

I am wondering if PTFS-e's nodes are not tracking gitlab which could be behind github/git.k-c.org?
Comment 14 Jonathan Druart 2024-04-05 07:58:54 UTC
(In reply to Jonathan Druart from comment #13)
> With the same id edda4e1bdb0943e43eb80faeb8b71de397327d04:
> 
> % git show edda4e1bdb0943e43eb80faeb8b71de397327d04
> tree edda4e1bdb0943e43eb80faeb8b71de397327d04
> 
> I am wondering if PTFS-e's nodes are not tracking gitlab which could be
> behind github/git.k-c.org?

The trigger on (at least) U20 is on gitlab, so not sure this makes sense.
Comment 15 Katrin Fischer 2024-05-21 09:34:52 UTC
*** Bug 36903 has been marked as a duplicate of this bug. ***
Comment 16 Martin Renvoize (ashimema) 2024-05-21 12:23:42 UTC
(In reply to Jonathan Druart from comment #13)
> With the same id edda4e1bdb0943e43eb80faeb8b71de397327d04:
> 
> % git show edda4e1bdb0943e43eb80faeb8b71de397327d04
> tree edda4e1bdb0943e43eb80faeb8b71de397327d04
> 
> I am wondering if PTFS-e's nodes are not tracking gitlab which could be
> behind github/git.k-c.org?

origin https://github.com/Koha-Community/Koha.git (fetch)

So, no.. it's not that.
Comment 17 Jonathan Druart 2024-05-22 12:19:53 UTC
Worked with Jake today to tackle this one down.

The "not a tree object" / "Nothing to critique" appeared on PTFS-E nodes because the content of the repository had been removed at some point, only .git existed.
A full new clone has been done and this error is now gone.

The "error: object directory" / "Could not read" has been fixed by removing the "Honor refspec" from the job's config (unlike what I stated in comment 2.

This change has been done on all Koha_Master* job. Not stables (yet).

Before: https://snipboard.io/RCkJuS.jpg
After : https://snipboard.io/vuVscO.jpg
Comment 18 Jonathan Druart 2024-05-22 13:11:32 UTC
(In reply to Jonathan Druart from comment #17)
> This change has been done on all Koha_Master* job. Not stables (yet).

And jobs within "security".
Comment 19 Jonathan Druart 2024-09-05 09:35:21 UTC
Considering this done. However RMaints should update the jobs config on Jenkins.
Comment 20 Jonathan Druart 2024-09-05 09:36:43 UTC
(In reply to Jonathan Druart from comment #19)
> Considering this done. However RMaints should update the jobs config on
> Jenkins.

Well actually I am not sure it's a good idea to let people edit the config of the jobs.

It's something for "Continuous integration infrastructure maintainer", which is... nobody this cycle.
Comment 21 Jonathan Druart 2024-09-23 07:21:20 UTC
(In reply to Jonathan Druart from comment #18)
> (In reply to Jonathan Druart from comment #17)
> > This change has been done on all Koha_Master* job. Not stables (yet).
> 
> And jobs within "security".

Done now for all jobs marked as "Enabled" for 24.05, 23.11, 23.05 and 22.11.

tl;dr; for RMaints: now the t/00-testcritic.t should no longer fail with "Nothing to critique".