[cdc] Parse MySQL SET bitmaps as unsigned 64-bit in canal/aliyun - #10372
jackylee-ch wants to merge 1 commit into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED for the Canal CDC path. Implementation: FINDINGS.
I ran both new SET test classes with JDK 8, -Pflink1 and the normal Maven checks: 6/6 passed. However, an additional test feeding a real Canal INSERT event into CanalRecordParser still fails for the 64th SET member; an unsigned-parsing control passes and produces [c63].
Scope note: AliyunFieldParser currently has no production callers; AliyunRecordParser.extractRowData does not invoke it. Its helper tests therefore do not establish that a DTS job is fixed. Please keep the claim/scope tied to the actual parser path, or demonstrate and exercise the missing Aliyun connection.
| // mysql set type value can be filled with more than one, value is a bit string conversion | ||
| // from the long | ||
| int indexes = Integer.parseInt(value); | ||
| long indexes = Long.parseLong(value); |
There was a problem hiding this comment.
[P2] Parse the SET bitmap as unsigned so the 64th member is supported. MySQL permits 64 SET members (https://dev.mysql.com/doc/refman/8.4/en/set.html). Canal's RowsLogBuffer uses getUlong64 for 8-byte SET values, and LogBuffer.getUlong64 returns BigInteger; LogEventConvert serializes it using String.valueOf, so selecting member 64 produces "9223372036854775808", not a negative signed-long string. This new Long.parseLong still throws NumberFormatException for that valid event, leaving the claimed wide-SET CDC failure unfixed at the upper boundary. I reproduced it through CanalRecordParser.buildSchema with mysqlType set('c0',...,'c63') and INSERT data s="9223372036854775808". Replacing this call with Long.parseUnsignedLong makes the same event decode to [c63]; the existing >>> shift loop already supports the resulting bits. Please use unsigned parsing and add actual-event tests for bit 63 and all 64 bits. Upstream encoding: https://github.com/alibaba/canal/blob/master/dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/event/RowsLogBuffer.java#L868 and https://github.com/alibaba/canal/blob/master/dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/LogBuffer.java#L996 .
5de28a7 to
cf19f7f
Compare
CanalFieldParser decoded a MySQL SET value from its bitmap with a signed parse, so a SET with 32+ members overflowed int, and even after widening to long the 64th member (bit 63, decimal 2^63 > Long.MAX_VALUE) still overflowed a signed long and threw NumberFormatException, failing the sync. Parse the bitmap with Long.parseUnsignedLong so the full 64-member range decodes; the decoding loop already shifts unsigned (>>>). The production path is Canal (CanalRecordParser -> convertSet). AliyunFieldParser carries the identical duplicated code and is fixed in parallel so it stays correct for when it is wired to a record parser; its unit test exercises the helper, not a DTS job.
cf19f7f to
50c03c1
Compare
|
Good catch. The 64th SET member sets bit 63 (decimal 2^63 > Long.MAX_VALUE), so even the signed-long parse threw NumberFormatException. Switched to On scope: the production path is Canal ( |
Purpose
CanalFieldParserandAliyunFieldParserdecode a MySQLSETvalue from itsbitmap representation with
Integer.parseInt(value). A MySQLSETcolumn maydeclare up to 64 members, so selecting a member at bit 31 or higher yields a
bitmap value greater than
Integer.MAX_VALUE.Integer.parseIntthen threwNumberFormatException, failing the whole canal / aliyun-DTS sync job for anotherwise valid row.
This parses the bitmap as a
longand widensgetSetValuesByIndexto match. Thedecoding loop already operated in long terms (
indexes != 0L,indexes % 2L,indexes >>> 1), and the method's own comment already states the value is"a bit string conversion from the long" — the parse was the only place still
narrowed to
int. Values within the int range decode exactly as before.Tests
CanalFieldParserSetTest/AliyunFieldParserSetTestdecode a 40-memberSET:a high bit (2^32) and a low+high combination both resolve to the correct members,
and a small in-range value is unchanged. The high-bit cases throw
NumberFormatExceptionbefore the change.API and Format
No change.
Documentation
No change.