The behavior for unassigned codepoint of Shift_JIS is incompatible with WHATWG spec

Version

v18.5.0

Platform

No response

Subsystem

No response

What steps will reproduce the bug?

const decoder = new TextDecoder('Shift_JIS');
const s = decoder.decode(new Uint8Array([255]));

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior?

const decoder = new TextDecoder('Shift_JIS');
const s = decoder.decode(new Uint8Array([255]));
console.log(s) // '�' === '\ufffd'

According to WHATWG spec, any decoder should use �(U+FFFD) when an unassigned codepoint is found during decoding.

What do you see instead?

const decoder = new TextDecoder('Shift_JIS');
const s = decoder.decode(new Uint8Array([255]));
console.log(s) // '\x1A'

From my investigation, ICU intentionally uses \x1A for unassigned codepoint on Shift_JIS encoding, and Node.js uses it as it is.
Conversion Data - ICU Documentation
Which substitution character is used if a character cannot be converted?

Additional information

ICU provides the utility ucnv_setSubstChars to specify substitution characters for any encoding, and Node.js already has it in library. I'm working on this.

@cola119

Able to reproduce this on v19.0.0-pre:

Welcome to Node.js v19.0.0-pre.
Type ".help" for more information.
> const decoder = new TextDecoder('Shift_JIS');
> const s = decoder.decode(new Uint8Array([255]));
> s
'\x1A'

@cola119 are you looking into ucnv.cpp for the fix?

@hemanth

@hemanth
I'm thinking ConverterObject can set ? as a substitution character explicitly since node::Converter already have the method to change it.

node/src/node_i18n.cc

Lines 370 to 377 in 7ef069e

    
           void Converter::set_subst_chars(const char* sub) { 
        
             CHECK(conv_); 
        
             UErrorCode status = U_ZERO_ERROR; 
        
             if (sub != nullptr) { 
        
               ucnv_setSubstChars(conv_.get(), sub, strlen(sub), &status); 
        
               CHECK(U_SUCCESS(status)); 
        
             } 
        
           }

src: fix to use replacement character

Navigation Menu

Repository navigation

The behavior for unassigned codepoint of Shift_JIS is incompatible with WHATWG spec #43962

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	void Converter::set_subst_chars(const char* sub) {
	CHECK(conv_);
	UErrorCode status = U_ZERO_ERROR;
	if (sub != nullptr) {
	ucnv_setSubstChars(conv_.get(), sub, strlen(sub), &status);
	CHECK(U_SUCCESS(status));
	}
	}

The behavior for unassigned codepoint of Shift_JIS is incompatible with WHATWG spec #43962

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior?

What do you see instead?

Additional information

Activity

hemanth commented on Jul 23, 2022

cola119 commented on Jul 24, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions