Releases: Altinity/clickhouse-sink-connector
Release list
Release v2.10.3
What's Changed
- urgent fix: a keyless source table must not prevent the connector from starting by @minguyen9988 in #1426
Full Changelog: 2.10.2...2.10.3
Release v2.10.2
What's Changed
- fix: /metrics served with no Content-Type, breaking Prometheus 3.x scrapes by @minguyen9988 in #1413
- fix: INT8 (Byte) hard-cast to Integer fails every TINYINT batch by @minguyen9988 in #1414
- fix: ARRAY binding hard-casts to ArrayList, breaking every other List type by @minguyen9988 in #1415
- fix: initial snapshot never finishes -- NPE on every unparseable record by @minguyen9988 in #1416
- fix: MySQL DOUBLE auto-created as Float32, silently truncating precision by @minguyen9988 in #1417
- fix: support PostgreSQL infinity values in timestamptz by @minguyen9988 in #1418
- fix: batch containing a missing ClickHouse table is silently dropped and the offset advances by @minguyen9988 in #1419
- fix: _version stores UInt64 max when the source has no GTID, permanently winning RMT dedup by @minguyen9988 in #1420
- Added release notes for 2.10.1 release by @subkanthi in #1412
Full Changelog: 2.10.1...2.10.2
Release v2.10.1
What's Changed
- fix: history mode loses ~95% of rows when a CREATE is replayed by @minguyen9988 in #1410
- fix: history-mode DELETE lost -- _operation dropped from INSERT, and the delete races its own batch by @minguyen9988 in #1411
Full Changelog: 2.10.0...2.10.1
Release v2.10.0
ClickHouse Sink Connector Release v2.10.0 Notes
2.10.0 is a large reliability release. It stops several classes of silent data loss (DDL ordering, TRUNCATE timing, keyless tables, BIT/DOUBLE mapping, history-mode row drops, offset commits), adds dump/load/compare tooling, and hardens how the connector fails.
Full Changelog: 2.9.1...2.10.0
What's New
ch_sink_toolsPython toolkit — dump, load, and compare MySQL/PostgreSQL against ClickHouse (ch-checksum,ch-pg-dump,ch-mysql-dump, and related commands). Passwords are masked in logs,eval()is removed from checksum formatting, and table locks are held only for the duration of each table checksum. (#1272, #1356, #1357, #1318)- Ansible + systemd deployment for the lightweight connector (no Docker, Kubernetes, or Kafka required). See
deploy/ansible-systemd/. (#1338) - Optional REST API Basic Auth — set
rest.api.auth.enabled=truewithrest.api.auth.username/rest.api.auth.password. Auth remains off unless enabled. (#1300) - Non-locking snapshot options — additional snapshot locking modes so initial snapshots can avoid holding source table locks. (#1380)
db_timecolumn on binlog-history tables (DateTime MATERIALIZED now()) for when the row was written to ClickHouse. (#1283)- ClickHouse JDBC driver 2.10. (#1278)
- Fatal ClickHouse error classification — authentication, schema mismatch, type mismatch, too-many-parts, and similar deterministic failures stop the connector instead of retrying the same batch forever and blocking every table. Transient errors still retry. (#1315)
- Helm chart —
connector.snapshotMode(defaultschema_only) andconnector.disableDropTruncate(defaulttrue) are now configurable. (#1359) - Updated
show_replica_statusview. (#1328) - Sysbench automated tests and pytest checksum integration tests. (#1354, #1351)
Breaking Changes
Upgrade notes — read these before rolling 2.10.0 into production.
-
Keyless MySQL tables can refuse startup (#1401, #1405, #1406, #1407)
A table with no
PRIMARY KEYand no non-nullUNIQUEkey has no row identity in the binlog. The connector now:- Refuses to start if it finds such tables on the source (and prints the exact
ALTER TABLE ... ADD COLUMN my_row_id ...fix). - Refuses keyless tables on MySQL < 8.0.30 and MariaDB (Generated Invisible Primary Key is not available).
- Does not enable GIPK on the source. The connector is read-only against MySQL.
- Does not block startup for tables that are already excluded from replication.
What to do: add a GIPK (or a real primary key) on the source, or set
keyless.table.check.skip=trueif you accept silent row collapse. - Refuses to start if it finds such tables on the source (and prints the exact
-
Helm defaults no longer snapshot existing data or apply DROP/TRUNCATE (#1359)
Setting Previous chart behavior 2.10.0 default connector.snapshotModepinned to initialschema_onlyconnector.disableDropTruncatenot configurable trueFresh Helm deploys will not load existing source rows and will ignore
DROP/TRUNCATE. Override invalues.yamlif you need the old behavior:connector: snapshotMode: "initial" disableDropTruncate: "false"
-
Fatal ClickHouse errors stop the connector (#1315)
Auth failures, unknown table/database/column, type mismatch,
TOO_MANY_PARTS, and similar errors used to retry forever and stall the whole pipeline. They now fail the task. Configure alerting/restarts around a stopped connector; do not expect an infinite retry loop to hide a bad schema or credential. -
Sample Docker config no longer enables replicated table engines (#1301)
auto.create.tables.replicatedis commented out / false in the sampleconfig.yml. New tables are created asReplacingMergeTree, notReplicatedReplacingMergeTree. Setauto.create.tables.replicated: "true"if you need replicated engines. -
Auto-created column types changed for new tables
Existing ClickHouse tables are not rewritten. Newly auto-created tables use:
BIT(n)forn > 1→ binary string, notBool(#1386, #1387, #1390)DOUBLE/DOUBLE PRECISION/FLOAT8→Float64(wasFloat32) (#1386)REAL→ does not claimFloat64precision it never receives (#1397)- Unsigned integer mapping workaround for a Debezium bug (#1306)
Recreate (or
MODIFY COLUMN) any table that was auto-created under the old mapping if you need the corrected types.
Bug Fixes
CDC / DDL ordering
- Apply DDL at its position in the stream so pre-ALTER rows are not written after the schema change. (#1385)
- Execute
TRUNCATEat its binlog position, not after the batch inserts. (#1382) ALGORITHM/LOCKclauses no longer discard later operations in the sameALTER. (#1381)- Fix double database prefix on
TRUNCATE,DROP, andRENAME. (#1316) - Invalidate the cached writer for the source table of a
RENAME. (#1384) - Wait for
ALTER TABLEvisibility before cache invalidation. (#1321) - Fix cache invalidation logic (with integration test). (#1333)
- Generated
ADD/DROP/RENAMEDDL is replay-safe after a restart (IF EXISTS/IF NOT EXISTS). Replay safety uses version ordering, not equality. (#1402, #1403, #1404) - DDL waiter no longer parses
IF NOT EXISTSas the column name. (#1408) - Default
MODIFY/CHANGE COLUMNtoNullablewhen ClickHouse schema metadata is unavailable. (#1339, #1341) - A column rename no longer NULLs rows still in flight. (#1395)
- Pre-ALTER rows NULL-fill newly added columns instead of failing. (#1389)
Data integrity
- Delete operations are persisted. (#1281)
_versionis anchored to the source commit timestamp so a re-deliveredDELETEcannot out-rank a later re-INSERT. Formula stays the 2.8.0ts_ms * 1_000_000 + counterdomain (upgrade- and rollback-safe). (#1378)- History mode no longer drops all but the first row of a multi-row binlog event. (#1391)
- Kafka Connect
preCommitonly advances offsets after a durable ClickHouse insert. (#1345) - Retry offset flushing on process error. (#1335)
- A nullable
UNIQUEkey no longer stalls the entire stream. (#1396) - Keyless / no-PK tables no longer collapse to a single row (see Breaking Changes for the GIPK requirement). (#1389, #1394, #1401)
Types and schema
BIT(n)poison-batch stall, byte-reversal, and binary safety. ([#1386](https://github.com...
Release v2.9.1
Lightweight
Docker: altinity/clickhouse-sink-connector:2.9.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.9.1-lt/images/sha256-bc181406a7169418c0bfe623d820bc2ae239db532c8ba8b5a833cf762fe4341b
What's Changed
- Changed _time column in binloghistory from DateTime64 to DateTime to … by @subkanthi in #1239
- enable publishing lightweight jar and disable kafka jar and docker pu… by @subkanthi in #1240
- Fix NullPointerException when config properties are missing by @subkanthi in #1273
- Added logic to cache serialized results. by @subkanthi in #1247
- Fix DST transition offset. by @subkanthi in #1271
Full Changelog: 2.9.0...2.9.1
Release v2.9.0
Lightweight
Docker: altinity/clickhouse-sink-connector:2.9.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.9.0-lt/images/sha256-75bfc3a63d5b57bd712bcf41958f3bce3bc612d5e94a51ff70bf697e2f0e2c49
What's New
- Version history & replication (https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/version_history.md)
- Binlog-only replication mode (#1228).
- Checksum utility improvements(mysql_table_checksum.py)
- Additional MySQL DDL Handling
- JAR file releases.
What's Changed
- Fixed links to quickstart_kafka.md by @subkanthi in #1124
- Improve the checksum + top level job by @aadant in #1128
- Update production_setup.md by @subkanthi in #1144
- Various improvements to checksums by @aadant in #1139
- Adding support for default file in top_level_table_checksum.py by @aadant in #1154
- Parse source_offset ts_sec and set it to time and valid_to columns by @subkanthi in #1161
- Added if exists to drop constraint. by @subkanthi in #1163
- set NODE_OPTIONS to avoid javascript out of memory error. by @subkanthi in #1164
- Removed JUNIT test report. by @subkanthi in #1165
- 2.8.0 by @subkanthi in #1122
- Update version history link to experimental by @subkanthi in #1166
- Enable running publish.yml separately by @MyroTk in #1159
- Added version history documentation. by @subkanthi in #1168
- Added version history documentation. by @subkanthi in #1170
- Integration test binlog history by @subkanthi in #1172
- Removed deduplication.policy by @subkanthi in #1179
- Updated configuration.md by @subkanthi in #1181
- Updated configmap.yaml by @subkanthi in #1183
- Fix illegal collation issues in mysql_table_checksum.py by @aadant in #1184
- added license compliance plugin by @subkanthi in #1180
- Updated replica.status.view. by @subkanthi in #1187
- Fixed offset.storage.jdbc.table.delete by @subkanthi in #1188
- Added step to publish sink-connector-client general and arm architecture by @subkanthi in #1189
- 979 retries use debezium errormaxretries as a configurable variable by @subkanthi in #1192
- updated snapshot.mode from schema_only to no_data by @subkanthi in #1190
- Added logic to ignore ALTER TABLE AUTOINCREMENT MySQL DDL to internal ddl ignore list. by @subkanthi in #1193
- Pass the original DDL to ANTLR parser to fallback on the regex to get… by @subkanthi in #1194
- Added change to publish lightweight jar. by @subkanthi in #1182
- Fix nullable timestamp load with timezone different from UTC by @aadant in #1195
- support table_include_list and database_override_map by @aadant in #1201
- 1198 compressed binlog history by @subkanthi in #1199
- table_include_list is a list of regex by @aadant in #1203
- ability to reconcile tables with different partition schemes on CH an… by @aadant in #1204
- fix import by @aadant in #1208
- Added support for partition by function range. by @subkanthi in #1207
- Added logic to change __is_deleted to _is_deleted when there is an existing column in the source schema. by @subkanthi in #1206
- Replaced bitnami image by @subkanthi in #1210
- fix table checksum by @aadant in #1211
- Adding a where override per table for the top_level_table_checksum job by @aadant in #1216
- Exclude floating points from checksums by @aadant in #1217
- Update runner labels to more available types by @MyroTk in #1221
- Added logic to invalidate cache of column names to data type mapping after DDL. by @subkanthi in #1223
- Added function to construct input query for updating inplace. by @subkanthi in #1171
- Add FINAL to select for binlog history update, changed operation data… by @subkanthi in #1225
- Fix tests. by @subkanthi in #1226
- Fixed auto creation of tables in replication.history.mode. by @subkanthi in #1227
- Update roadmap link from 2025 to 2026 by @subkanthi in #1234
- Added logic for binlog only replication mode. by @subkanthi in #1228
Full Changelog: 2.8.0...2.9.0
Release v2.8.0
Lightweight
Docker: altinity/clickhouse-sink-connector:2.8.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.8.0-lt/images/sha256-3799c182de1e91a1dd1434d2f2035a6bdbb036d9a1f74bad64c8b537446db7f5
What's New
- Multi-platform(ARM) builds
- Oracle support(Experimental)
- Version History(Temporal tables) (Experimental)
- Schema Override
- support for more MySQL DDL statements.
- Debezium upgrade to 3.1.3.Final
What's Changed
- Fix table name substitution for creating schema history table. by @subkanthi in #1103
- 2.7.1 by @subkanthi in #1104
- Add max_memory_usage support in clickhouse_table_checksum.py by @aadant in #1119
- Support for partition expression in checksum command lines by @aadant in #1118
- Fixed parser when generated columns have isnotnull by @subkanthi in #1115
- schema override functionality. by @subkanthi in #1072
- Update postgres docker configuration. by @subkanthi in #1123
- Upgraded debezium to 3.1.3.Final by @subkanthi in #1117
- Fix time order for inserted rows panel by @mohaidoss in #1095
- [1097] Support longtext MySQL DDL. by @subkanthi in #1101
- Added oracle docker files. by @subkanthi in #1112
- Fixed typo in commented-out config.yml parameter by @PlzStandBy in #1105
- Added logic to write history tables in a separate connector mode. by @subkanthi in #1063
- Implement blocking put for back-pressure in DebeziumChangeEvent in sink connector lightweight by @captainpro-eng in #1108
- Updated Debezium configuration. by @subkanthi in #1147
- Added integration test for binlog history. by @subkanthi in #1143
- Support create database on cluster by @subkanthi in #1135
- Mergetree history fixes. by @subkanthi in #1149
- Replaced bitnami mysql docker images with official mysql images. by @subkanthi in #1152
- Fixed mapping of bigint unsigned. by @subkanthi in #1153
- Enable kafka connect json and added scripts for running without schem… by @subkanthi in #1130
- Fix alter table add column not null tests. by @subkanthi in #1155
New Contributors
- @PlzStandBy made their first contribution in #1105
- @captainpro-eng made their first contribution in #1108
Full Changelog: 2.7.1...2.8.0
Release v2.7.1
Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.1-lt/images/sha256-6aeb1dcbcd7b2f0e0f43f3f31cdc198a594977b2c2e338552465ad50eebd02ef
What's Changed
- 2.7.0 by @subkanthi in #1058
- Some documentation improvements for the production_setup page by @mohaidoss in #1094
- Fixed shell script path to get release tag. by @subkanthi in #1099
- Added maven wrapper by @subkanthi in #1100
- Fix table name substitution for creating schema history table. by @subkanthi in #1103
New Contributors
- @mohaidoss made their first contribution in #1094
Full Changelog: 2.7.0...2.7.1
Release v2.7.0
Lightweight
Docker: altinity/clickhouse-sink-connector:2.7.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.7.0-lt/images/sha256-2e0f1a0487a71a11b19200320707545ec5c5992b0abc37ad09ec954c36352f37
Breaking Changes
Configuration Changes
offset.storage.jdbc.offset.table.name to offset.storage.jdbc.table.name
offset.storage.jdbc.offset.table.ddl to offset.storage.jdbc.table.ddl
offset.storage.jdbc.offset.table.delete to offset.storage.jdbc.table.delete
What's new
- Debezium upgrade to 3.1.2.Final
- Various bug fixes
What's Changed
- fix: clickhouse_loader port and secure options by @philippevezina in #1041
- Added workaround to fix the debezium timestamp for DATETIME4, DATETIM… by @subkanthi in #1057
- [1049] Fixed typo in config file. by @subkanthi in #1050
- Added logic to display the retry count. by @subkanthi in #1023
- Fix integration tests 2 7 0 by @subkanthi in #1067
- Reenable disabled tests. by @subkanthi in #1033
- Fix/1046 nullpointerexception dbsource by @subkanthi in #1068
- [1046] Fix NullPointerException and Refactor Connection Pool Initialization Logic by @chimeyrock999 in #1048
- Reverted back network_mode: host with ports. by @subkanthi in #1070
- Rolled back network_mode: Host by @subkanthi in #1081
- Revert "Fix/1046 nullpointerexception dbsource" by @subkanthi in #1084
- NULL Primary Key from MySQL needs to be translated to CH. by @subkanthi in #1087
- Fixed encoding logic when password has special characters. by @subkanthi in #1086
- Added logic to persist error logs in a separate database. by @subkanthi in #1054
- Replaced redpanda images. by @subkanthi in #1073
- Upgraded debezium to 3.1.2.Final, Added integration test to definer SQL. by @subkanthi in #1055
- Fixed REST API test to primarily test sink-connector-client. by @subkanthi in #1047
New Contributors
- @philippevezina made their first contribution in #1041
- @chimeyrock999 made their first contribution in #1048
Full Changelog: 2.6.0...2.7.0
Release v2.6.0
Lightweight
Docker: altinity/clickhouse-sink-connector:2.6.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.6.0-lt/images/sha256-1114ee20b6ba76f78c2d4858ee3805115753aeecd26221875d68538ba1aeb6a8
What's New
MySQL SET Data Type #881
ARM support for docker builds (Experimental) - #955
Debezium upgraded to 3.1.0.AlphaFinal - #1001
PostgreSQL time with timezone mapping added - #951
What's Changed
- 2.5.1 by @subkanthi in #995
- Renable disabled integration tests. by @subkanthi in #1004
- Added connection pool documentation by @subkanthi in #1000
- Update README.md by @subkanthi in #1009
- Update connection_pool.md by @subkanthi in #1010
- Updated config doc by @subkanthi in #1011
- Pass jdbc parameters to datasource, removed okhttp dependencies. by @subkanthi in #1012
- Update debezium version to 3.1.0.AlphaFinal by @subkanthi in #1001
- Add ddl translation rest api by @subkanthi in #936
- Added support for create table for MySQL Set Data type by @subkanthi in #881
- Added step to install ps command for sink-connector lightweight by @subkanthi in #878
- 866 incorrect precision and scale parsing due to column comment interference by @subkanthi in #868
- Build cross-platform docker image for arm and amd by @subkanthi in #955
- Added test case for parsing ghost SQL statements by @subkanthi in #817
- Added integration test to validate replication of DDL with partitions. by @subkanthi in #953
- Add steps to start a second connector. by @subkanthi in #959
- Update README.md by @subkanthi in #1016
- Local environment setup(IntelliJ) by @subkanthi in #1017
- set errors.max.retries in config.yml by @subkanthi in #1018
- Revert database in config.yml to test by @subkanthi in #1020
- 993 datetime columns with non utc tz on source and target are not replicated properly by @subkanthi in #1022
- Added logic to set clickhouse settings from configuration parameter. by @subkanthi in #1024
- Added retry logic for acknowledging records. by @subkanthi in #1015
- Added mapping of time with timezone postgres datatype to clickhouse string and added integration test. by @subkanthi in #951
- Update ClickHouseDataTypeMapper.java: fix for working with bytebuffer by @salisbury-espinosa in #897
- Revert "Added retry logic for acknowledging records." by @subkanthi in #1031
- maven checkstyle fixes. by @subkanthi in #1030
- Fix datetime integration tests. by @subkanthi in #1029
- Added network_mode:host to docker compose by @subkanthi in #1038
- Testflows tests for 2 6 0 by @Selfeer in #1036
- Instructions to install JAR file for kafka sink connector by @subkanthi in #879
- Added integration test to cover MySQL demo. by @subkanthi in #984
- Added logic to CREATE on Cluster when replicated mode is enabled by @subkanthi in #880
- Update ddl docs by @subkanthi in #1025
- K8s manifest files by @subkanthi in #1032
- Update PreparedStatementExecutor.java: fix updates from postgres (update in postgres -> kafka -> clickhouse) by @salisbury-espinosa in #900
- Added monitoring of sink connector using docker compose by @subkanthi in #1044
Full Changelog: 2.5.1...2.6.0