Sitelet https://github.com/Altinity/clickhouse-sink-connector/releases
Skip to content

Releases: Altinity/clickhouse-sink-connector

Release v2.10.3

Choose a tag to compare

@github-actions github-actions released this 31 Aug 21:54
e7da4d6

What's Changed

  • urgent fix: a keyless source table must not prevent the connector from starting by @minguyen9988 in #1426

Full Changelog: 2.10.2...2.10.3

Release v2.10.2

Choose a tag to compare

@github-actions github-actions released this 31 Aug 02:45
689fd32

What's Changed

  • fix: /metrics served with no Content-Type, breaking Prometheus 3.x scrapes by @minguyen9988 in #1413
  • fix: INT8 (Byte) hard-cast to Integer fails every TINYINT batch by @minguyen9988 in #1414
  • fix: ARRAY binding hard-casts to ArrayList, breaking every other List type by @minguyen9988 in #1415
  • fix: initial snapshot never finishes -- NPE on every unparseable record by @minguyen9988 in #1416
  • fix: MySQL DOUBLE auto-created as Float32, silently truncating precision by @minguyen9988 in #1417
  • fix: support PostgreSQL infinity values in timestamptz by @minguyen9988 in #1418
  • fix: batch containing a missing ClickHouse table is silently dropped and the offset advances by @minguyen9988 in #1419
  • fix: _version stores UInt64 max when the source has no GTID, permanently winning RMT dedup by @minguyen9988 in #1420
  • Added release notes for 2.10.1 release by @subkanthi in #1412

Full Changelog: 2.10.1...2.10.2

Release v2.10.1

Choose a tag to compare

@github-actions github-actions released this 27 Aug 05:52
d41888a

What's Changed

  • fix: history mode loses ~95% of rows when a CREATE is replayed by @minguyen9988 in #1410
  • fix: history-mode DELETE lost -- _operation dropped from INSERT, and the delete races its own batch by @minguyen9988 in #1411

Full Changelog: 2.10.0...2.10.1

Release v2.10.0

Choose a tag to compare

@github-actions github-actions released this 25 Aug 22:17
b65433a

ClickHouse Sink Connector Release v2.10.0 Notes

2.10.0 is a large reliability release. It stops several classes of silent data loss (DDL ordering, TRUNCATE timing, keyless tables, BIT/DOUBLE mapping, history-mode row drops, offset commits), adds dump/load/compare tooling, and hardens how the connector fails.

Full Changelog: 2.9.1...2.10.0


What's New

  • ch_sink_tools Python toolkit — dump, load, and compare MySQL/PostgreSQL against ClickHouse (ch-checksum, ch-pg-dump, ch-mysql-dump, and related commands). Passwords are masked in logs, eval() is removed from checksum formatting, and table locks are held only for the duration of each table checksum. (#1272, #1356, #1357, #1318)
  • Ansible + systemd deployment for the lightweight connector (no Docker, Kubernetes, or Kafka required). See deploy/ansible-systemd/. (#1338)
  • Optional REST API Basic Auth — set rest.api.auth.enabled=true with rest.api.auth.username / rest.api.auth.password. Auth remains off unless enabled. (#1300)
  • Non-locking snapshot options — additional snapshot locking modes so initial snapshots can avoid holding source table locks. (#1380)
  • db_time column on binlog-history tables (DateTime MATERIALIZED now()) for when the row was written to ClickHouse. (#1283)
  • ClickHouse JDBC driver 2.10. (#1278)
  • Fatal ClickHouse error classification — authentication, schema mismatch, type mismatch, too-many-parts, and similar deterministic failures stop the connector instead of retrying the same batch forever and blocking every table. Transient errors still retry. (#1315)
  • Helm chart — connector.snapshotMode (default schema_only) and connector.disableDropTruncate (default true) are now configurable. (#1359)
  • Updated show_replica_status view. (#1328)
  • Sysbench automated tests and pytest checksum integration tests. (#1354, #1351)

Breaking Changes

Upgrade notes — read these before rolling 2.10.0 into production.

  1. Keyless MySQL tables can refuse startup (#1401, #1405, #1406, #1407)

    A table with no PRIMARY KEY and no non-null UNIQUE key has no row identity in the binlog. The connector now:

    • Refuses to start if it finds such tables on the source (and prints the exact ALTER TABLE ... ADD COLUMN my_row_id ... fix).
    • Refuses keyless tables on MySQL < 8.0.30 and MariaDB (Generated Invisible Primary Key is not available).
    • Does not enable GIPK on the source. The connector is read-only against MySQL.
    • Does not block startup for tables that are already excluded from replication.

    What to do: add a GIPK (or a real primary key) on the source, or set keyless.table.check.skip=true if you accept silent row collapse.

  2. Helm defaults no longer snapshot existing data or apply DROP/TRUNCATE (#1359)

    Setting Previous chart behavior 2.10.0 default
    connector.snapshotMode pinned to initial schema_only
    connector.disableDropTruncate not configurable true

    Fresh Helm deploys will not load existing source rows and will ignore DROP / TRUNCATE. Override in values.yaml if you need the old behavior:

    connector:
      snapshotMode: "initial"
      disableDropTruncate: "false"
  3. Fatal ClickHouse errors stop the connector (#1315)

    Auth failures, unknown table/database/column, type mismatch, TOO_MANY_PARTS, and similar errors used to retry forever and stall the whole pipeline. They now fail the task. Configure alerting/restarts around a stopped connector; do not expect an infinite retry loop to hide a bad schema or credential.

  4. Sample Docker config no longer enables replicated table engines (#1301)

    auto.create.tables.replicated is commented out / false in the sample config.yml. New tables are created as ReplacingMergeTree, not ReplicatedReplacingMergeTree. Set auto.create.tables.replicated: "true" if you need replicated engines.

  5. Auto-created column types changed for new tables

    Existing ClickHouse tables are not rewritten. Newly auto-created tables use:

    • BIT(n) for n > 1 → binary string, not Bool (#1386, #1387, #1390)
    • DOUBLE / DOUBLE PRECISION / FLOAT8 → Float64 (was Float32) (#1386)
    • REAL → does not claim Float64 precision it never receives (#1397)
    • Unsigned integer mapping workaround for a Debezium bug (#1306)

    Recreate (or MODIFY COLUMN) any table that was auto-created under the old mapping if you need the corrected types.


Bug Fixes

CDC / DDL ordering

  • Apply DDL at its position in the stream so pre-ALTER rows are not written after the schema change. (#1385)
  • Execute TRUNCATE at its binlog position, not after the batch inserts. (#1382)
  • ALGORITHM / LOCK clauses no longer discard later operations in the same ALTER. (#1381)
  • Fix double database prefix on TRUNCATE, DROP, and RENAME. (#1316)
  • Invalidate the cached writer for the source table of a RENAME. (#1384)
  • Wait for ALTER TABLE visibility before cache invalidation. (#1321)
  • Fix cache invalidation logic (with integration test). (#1333)
  • Generated ADD / DROP / RENAME DDL is replay-safe after a restart (IF EXISTS / IF NOT EXISTS). Replay safety uses version ordering, not equality. (#1402, #1403, #1404)
  • DDL waiter no longer parses IF NOT EXISTS as the column name. (#1408)
  • Default MODIFY / CHANGE COLUMN to Nullable when ClickHouse schema metadata is unavailable. (#1339, #1341)
  • A column rename no longer NULLs rows still in flight. (#1395)
  • Pre-ALTER rows NULL-fill newly added columns instead of failing. (#1389)

Data integrity

  • Delete operations are persisted. (#1281)
  • _version is anchored to the source commit timestamp so a re-delivered DELETE cannot out-rank a later re-INSERT. Formula stays the 2.8.0 ts_ms * 1_000_000 + counter domain (upgrade- and rollback-safe). (#1378)
  • History mode no longer drops all but the first row of a multi-row binlog event. (#1391)
  • Kafka Connect preCommit only advances offsets after a durable ClickHouse insert. (#1345)
  • Retry offset flushing on process error. (#1335)
  • A nullable UNIQUE key no longer stalls the entire stream. (#1396)
  • Keyless / no-PK tables no longer collapse to a single row (see Breaking Changes for the GIPK requirement). (#1389, #1394, #1401)

Types and schema

Read more

Release v2.9.1

Choose a tag to compare

@github-actions github-actions released this 14 Apr 02:42
999b63d

Lightweight
Docker: altinity/clickhouse-sink-connector:2.9.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.9.1-lt/images/sha256-bc181406a7169418c0bfe623d820bc2ae239db532c8ba8b5a833cf762fe4341b

What's Changed

  • Changed _time column in binloghistory from DateTime64 to DateTime to … by @subkanthi in #1239
  • enable publishing lightweight jar and disable kafka jar and docker pu… by @subkanthi in #1240
  • Fix NullPointerException when config properties are missing by @subkanthi in #1273
  • Added logic to cache serialized results. by @subkanthi in #1247
  • Fix DST transition offset. by @subkanthi in #1271

Full Changelog: 2.9.0...2.9.1

Release v2.9.0

Choose a tag to compare

@github-actions github-actions released this 16 Feb 04:04
546b3e4

Lightweight
Docker: altinity/clickhouse-sink-connector:2.9.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.9.0-lt/images/sha256-75bfc3a63d5b57bd712bcf41958f3bce3bc612d5e94a51ff70bf697e2f0e2c49

What's New

What's Changed

Full Changelog: 2.8.0...2.9.0

Release v2.8.0

Choose a tag to compare

@github-actions github-actions released this 06 Nov 02:07
75b7bc9

Lightweight
Docker: altinity/clickhouse-sink-connector:2.8.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.8.0-lt/images/sha256-3799c182de1e91a1dd1434d2f2035a6bdbb036d9a1f74bad64c8b537446db7f5

What's New

  • Multi-platform(ARM) builds
  • Oracle support(Experimental)
  • Version History(Temporal tables) (Experimental)
  • Schema Override
  • support for more MySQL DDL statements.
  • Debezium upgrade to 3.1.3.Final

What's Changed

New Contributors

Full Changelog: 2.7.1...2.8.0

Release v2.7.1

Choose a tag to compare

@github-actions github-actions released this 09 Jul 22:31

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.1-lt/images/sha256-6aeb1dcbcd7b2f0e0f43f3f31cdc198a594977b2c2e338552465ad50eebd02ef

What's Changed

New Contributors

Full Changelog: 2.7.0...2.7.1

Release v2.7.0

Choose a tag to compare

@github-actions github-actions released this 30 Jun 16:35

Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.0-lt/images/sha256-2e0f1a0487a71a11b19200320707545ec5c5992b0abc37ad09ec954c36352f37

Breaking Changes

Configuration Changes
offset.storage.jdbc.offset.table.name to offset.storage.jdbc.table.name
offset.storage.jdbc.offset.table.ddl to offset.storage.jdbc.table.ddl
offset.storage.jdbc.offset.table.delete to offset.storage.jdbc.table.delete

What's new

What's Changed

New Contributors

Full Changelog: 2.6.0...2.7.0

Release v2.6.0

Choose a tag to compare

@github-actions github-actions released this 14 Apr 04:05
459cd71

Lightweight
Docker: altinity/clickhouse-sink-connector:2.6.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.6.0-lt/images/sha256-1114ee20b6ba76f78c2d4858ee3805115753aeecd26221875d68538ba1aeb6a8

What's New

MySQL SET Data Type #881
ARM support for docker builds (Experimental) - #955
Debezium upgraded to 3.1.0.AlphaFinal - #1001
PostgreSQL time with timezone mapping added - #951

What's Changed

Full Changelog: 2.5.1...2.6.0