Issues

• Unexpected token with no matched text at the end of tokenstream

  cedricrupb/code_tokenize#5 · Filter by author LakshyAAAgrawal opened on Feb 26, 2023
  4

• string literals lost during tokenization for C/C++ code.

  cedricrupb/code_tokenize#4 · Filter by author 5c4lar opened on Jul 20, 2022
  2