Skip to content

fix(keeper): do not treat a pod without container statuses as started - #2162

Open
levin-kitty wants to merge 1 commit into
Altinity:0.28.0from
levin-kitty:fix/pod-started-empty-container-statuses
Open

levin-kitty wants to merge 1 commit into
Altinity:0.28.0from
levin-kitty:fix/pod-started-empty-container-statuses

Conversation

@levin-kitty

Copy link
Copy Markdown

Problem

When a pod is not scheduled yet, status.containerStatuses is empty.
In this case PodHasAllContainersStarted returns true, because the loop does not run.

A new Keeper host waits only for this check. So when the Keeper pod is Pending, the CHK still becomes Completed.

I saw this on 0.27.4. The Keeper pod was Pending because the PVC was not bound. The operator logged Host pod started after about 5 seconds, and the CHK became Completed.

Fix

Return false when there are no container statuses.
IsPodOK in the same file already has the same check.

After this fix, the wait should time out and the CHK should become Aborted.

Tests

I added TestPodHasAllContainersStarted. The pending pod case fails without the fix.

I ran unit tests only (go test ./pkg/model/k8s/ ./pkg/controller/...). I did not run e2e tests.

Note

I found that #2114 (draft) changes the same function with a stricter check.
This PR is a small fix only. If you don't need it, please close it.

Important items to consider before making a Pull Request

Please check items PR complies to:

  • All commits in the PR are squashed. More info
  • The PR is made into dedicated next-release branch, not into master branch1. More info
  • The PR is signed. More info

--

1 If you feel your PR does not affect any Go-code or any testable functionality (for example, PR contains docs only or supplementary materials), PR can be made into master branch, but it has to be confirmed by project's maintainer.

A pod that is not scheduled yet has an empty status.containerStatuses,
so PodHasAllContainersStarted returned true for it. A new Keeper host
waits only for this check, so a CHK whose pod was Pending was marked
Completed.

Signed-off-by: levin-kitty <levin.de.kitty@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant