Sink Connector 2025 Roadmap #401

This issue tracks deliverables on the 2024 2025 roadmap for Sink Connector. Feel free to comment on current deliverables and suggest new ones in comments.

Upcoming Releases

2.2.1 - https://github.com/Altinity/clickhouse-sink-connector/issues?q=is%3Aopen+is%3Aissue+milestone%3A2.2.1
2.6.0 - https://github.com/Altinity/clickhouse-sink-connector/issues?q=is%3Aopen+is%3Aissue+milestone%3A2.6.0
2.7.0 - https://github.com/Altinity/clickhouse-sink-connector/issues?q=is%3Aopen+is%3Aissue+milestone%3A2.7.0
2.8.0 - https://github.com/Altinity/clickhouse-sink-connector/issues?q=is%3Aopen+is%3Aissue+milestone%3A2.8.0

Completed

• ARM64 support – ARM containers for lightweight image, publish ARM targets, sink-connector-client on ARM64. (#955, #1158, #1189)
• Version history / Replication history (SCD Type 2) – ReplacingMergeTree history tables, _valid_from / _valid_to, version history docs, binlog history integration tests, timestamp/timezone handling for history. (#1063, #1168, #1170, #1172, #1156, #1161)
• PostgreSQL – time with time zone mapping, numeric/BigDecimal fix, show_replica_status view, config updates. (#951, #981, #970, #1123)
• Oracle support – Experimental Oracle source support. (#1112)
• MySQL & DDL – Partition-by-range translation, create-database-on-cluster, DDL parser fixes, ALTER AUTO INCREMENT ignore, commented partitions, _is_deleted column naming when source has is_deleted. (#953, #880, #1135, #1001, #1115, #1193, #1194, #1206)
• Timezone & datetime – Nullable timestamp with timezone, non-UTC datetime replication, DateTime64 behavior, ReplacingMergeTree timestamp/tz fixes. (#951, #1022, #1057, #972, #1195)
• Connection pool & config – Connection pool disable option, JDBC params, empty-password handling, config bug fixes. (#992, #1012, #1181, #1183, #1000)
• Debezium & Kafka – Debezium version upgrade, 3.12 config fix, configurable errors.max.retries, Docker Compose JSON converter (no schema registry). (#1055, #1117, #1192, #1130)
• Monitoring & observability – Show-replica-status–style view, instrumentation, Grafana panel, replica status for multiple connectors. (#1054, #1187, #1147, #1149, #1095)
• Binlog history – Compressed binlog history, table/list filters (table_include_list, database_override_map). (#1199, #1201, #1203)
• Publishing & CI – Publish JAR via GitHub Actions, publish ARM artifacts, run publish manually, license compliance plugin. (#1182, #1158, #1159, #1180)
• Stability & bug fixes – Connection leak fix, offset storage fix, NullPointerException fixes, collation fixes, reserved-words/checksum fixes, and others. (#893, #1188, #1068, #1084, #1184, #1211)
• Releases – 2.5.1, 2.6.0, 2.7.0, 2.7.1, 2.8.0. (#995, #998, #999, #1008, #1058, #1104, #1122)

Source Database Support

• Production PostgreSQL support (currently experimental)
• Production MongoDB support

ClickHouse Target Table Types

* [Replicated]MergeTree - Map change log records to ClickHouse, not final state

Replication Capabilities

* Replicate multiple databases in one connector(#531)

• Add existing tables to replication
• Schema overrides to control mapping of upstream schema to ClickHouse (clickhouse_loader.py : support schema overrides in yaml #323)
• Support MergeTree with version timestamp (Request: Support MergeTree with version timestamp. #445)

Security Improvements

?* Remove passwords from command line invocations (#398)

Deployment Configurations and Packaging

• Production Kafka Sink Connector
• Production Helm Chart for Kubernetes
• Tarball releases

Reliability and Performance

• Expanded DDL handling tests
• Ghost testing (MySQL)

Miscellaneous

• Move SQL test data to separate project
• Move dump/load utilities to separate project