Sitelet https://github.com/gpuweb/gpuweb/issues/2092
Skip to content

context-aware tokenization: Sometimes >> is better parsed as two copies of > (similar for ]] ) #2092

Description

@dneto0

Example for the >> case:

[[block]] struct B { a:  array<array<i32,2>>; } 
// That >> should two copies of GREATER_THAN, not one copy of SHIFT_RIGHT

Example for the ]] case:

fn foo() {
  var a : array<i32,10>;
  let v = a[a[0]]; // That ]] should be two copies of BRACKET_RIGHT instead of ATTR_RIGHT.
}

Tint parses these provisionally as the longer token, then backtracks to split them if needed.

I think the group has already discussed this but I can't find the tracking issue.

Related: #1063 validating the grammar is unambiguous

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    wgslWebGPU Shading Language Issueswgsl:tokensWGSL tokenization and literals

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions