Sitelet https://github.com/Altinity/clickhouse-operator/commit/a7b34e0ee64c2ace9b95d6303ca5039265ed3612
Skip to content

Commit a7b34e0

Browse files
committed
test: cover error propagation with tests
1 parent f7e91c5 commit a7b34e0

5 files changed

Lines changed: 293 additions & 47 deletions

File tree

‎release_notes.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,14 @@
6363

6464
Two related changes come with it: the hook now execs `kubectl` directly rather than wrapping it in `/bin/sh`, because `registry.k8s.io/kubectl` is distroless and ships no shell; and it runs as root (uid 0) where the Bitnami image ran as uid 1001 — set `crdHook.containerSecurityContext` if your admission policies require a non-root uid.
6565

66+
* **A failed schema migration no longer reports the CHI as `Completed`** ([PR #2077](https://github.com/Altinity/clickhouse-operator/pull/2077)). When `HostCreateTables` returned an error the operator logged it, emitted `CreateCompleted` anyway and recorded the host in `hostsWithTablesCreated`. That listing is persisted on the CR and feeds `HasData()`, which makes `shouldMigrateTables()` skip the host — so the failure was not merely hidden, it was made permanent across operator restarts, and the schema-less replica kept answering Distributed queries with `There is no table ...`. The failure now propagates as a deferral: sibling shards in the cluster still reconcile, the CR-final phase still runs, and the pass ends `Aborted`. Present since 0.21.0. Reported in [#2021](https://github.com/Altinity/clickhouse-operator/issues/2021) and diagnosed by [@Tyagiquamar](https://github.com/Tyagiquamar).
67+
68+
**Action required before upgrading.** A CHI that has quietly tolerated an un-creatable schema object — a Dictionary with an unreachable source, a Materialized View over a dropped table — turns `Aborted` on its first reconcile after the upgrade, where it previously reported `Completed`. There is no opt-out. Read `kubectl get chi <name> -o jsonpath='{.status.errors}'` for the underlying ClickHouse error, which names the object that could not be created, then fix or drop it. Anything gating on `.status.status == Completed` — CI pipelines, `kubectl wait`, alerting — should be checked against clusters carrying such objects before you upgrade.
69+
70+
A host that was already serving keeps its place in the entrypoint, cluster and shard Services while the failure persists, so an image upgrade that trips this does not take the replica out of rotation. A newly added host stays out of them until its schema is actually created — an empty replica must not receive traffic. Migration is retried on every subsequent pass, since the host is deliberately not recorded as tables-created.
71+
72+
**Cleanup is postponed while the failure persists.** A pass that ends this way skips the end-of-reconcile cleanup, so if the same edit also scales the cluster down, the removed hosts keep their StatefulSets, PVCs and ZooKeeper replica paths until a pass completes. This is deliberate: the cleanup derives what to purge from what the pass managed to reconcile, so running it on an incomplete pass could delete a StatefulSet that is still serving. A `CleanupPostponed` warning event names the hosts still on disk.
73+
6674
### Fixed
6775
* **Host deletion is no longer reported as completed when the operator could not tell whether the host still exists.** Deleting a host starts by fetching its StatefulSet, and *any* error from that fetch was treated as "already deleted" — a `DeleteCompleted` event was emitted and the rest of the cleanup was skipped. Only `NotFound` actually means the host is gone; a `Forbidden`, a throttled request or a transport failure took the same branch. The skipped cleanup covers the ZooKeeper table drop and the deletion of the host's StatefulSet, PVCs, ConfigMap and Service. The PVCs matter most: they carry no owner reference, so by default nothing else reclaims them. Now only `NotFound` reports completion; every other error raises `DeleteFailed` and returns the error. Present since 0.10.0.
6876

0 commit comments

Comments
 (0)