Switch cmeta from chibicc to a new homegrown lexer

This lexer is being written as a chunklet, not quite quite complete yet
as it lacks some functionality to make it generally useful for things,
but good enough for the gluegen use case now. So it's in the repo now
and we can go ahead and use it for this instead of having this
hacked-to-pieces third party thing that's nowhere near as efficient. In
the future the goal is to have a decently usable library for any sort of
C metaprogramming needs, not just within this project.

This design does the data-oriented thing of storing as little about each
token as possible (just 5 bytes), and re-lexing specific pieces only
when necessary. In some cases, full validation of correct syntax is
only possible through this secondary step. Since the use case is
metaprogramming and code-generation rather than the reimplementation of
Clang, this slight sloppiness in validation doesn't seem too bad.

It's been a while since I did a proper performance measurement of this
code but in some crude tests I did in the past the primary tokenisation
step was running at well over 500MB/s, which is fast enough for me. It's
possible that this version is a tiny bit slower due to the added
complexity of making UCNs work. UCNs, incidentally, are one of the
dumbest features of C by far. However, unlike trigraphs - which this
lexer does not handle - UCNs are still in the language, so it's kind of
sort of necessary to support them.

It's probably still possible to come up with a faster design with some
SIMD trickery, but the main loop is currently branchless and the lookup
tables are too big for the SIMD lookup things, so that seems kind of
hard. A separate SIMD path for whitespace or comment runs seems dubious
as it would introduce branch mispredictions everywhere. I also made
previous attempts to unroll the main loop and every attempt just made it
slower, so I guess code size is a significant factor.

Optimising the secondary tokenisation of identifiers (and later numerals
and string/character literals once those are handled) is still on the
cards, but since that happens less often, I don't know how much
difference it'll make.

At any rate, in a multithreaded context this thing would already come
pretty close to SSD speeds, if open-read-close syscall overhead doesn't
get in the way first. I imagine it's fast enough for anyone who hasn't
*already* written something faster.

14 files changed, 2717 insertions, 1737 deletions

diff --git a/compile.bat b/compile.bat
index 3cd8c84..518fd5d 100644
--- a/compile.bat
+++ b/compile.bat
@@ -124,7 +124,7 @@ if %host64%==1 (
 %CC% -fuse-ld=lld -shared -O0 -w -o .build/vstdlib.dll src/stubs/vstdlib.c

 

 %HOSTCC% -fuse-ld=lld -O2 %warnings% %stdflags% -include stdbool.h ^

--L.build %lbcryptprimitives_host% -o .build/gluegen.exe src/build/gluegen.c src/build/cmeta.c src/os.c || goto :end

+-L.build %lbcryptprimitives_host% -o .build/gluegen.exe src/build/gluegen.c src/build/cmeta.c src/os.c src/chunklets/clex.c || goto :end

 %HOSTCC% -fuse-ld=lld -O2 %warnings% %stdflags% -include stdbool.h ^

 -L.build %lbcryptprimitives_host% -o .build/mkgamedata.exe src/build/mkgamedata.c src/os.c || goto :end

 %HOSTCC% -fuse-ld=lld -O2 %warnings% %stdflags% -include stdbool.h ^

diff --git a/src/3p/README b/src/3p/README
index ab11ade..427bc1b 100644
--- a/src/3p/README
+++ b/src/3p/README
@@ -7,10 +7,6 @@ Used in SST itself:
 Used in debug builds, but not compiled into releases:
   - udis86
 
-Used at build time:
-  - chibicc (somewhat hacked up for use as a lexer)
-  - asprintf() from OpenBSD (for compatibility on Windows)
-
 Most of the C sources have wrappers in the parent directory to build proper
 objects for use in the project, and wrapper headers to get the full APIs as
 conveniently as possible. In other words, most of these files aren't built or
@@ -18,8 +14,10 @@ used directly.
 
 It is possible that these libraries may end up lightly modified; we err on the
 side of changing things to fit our use case rather than working around problems
-in outer layers. A couple of the libraries are pretty old and don't see much
-upstream change, but are small enough to be comfortably maintained as vendored.
+in outer layers. Sometimes libraries don't see updates for a while, other times
+we don't bother updating them even when they do. Generally all this stuff is
+stuff we're happy to maintain in-tree, and often stuff we don't need to update
+much either because it already works fine.
 
 IMPORTANT! Libraries are distributed subject to their copyright notices; please
 refer to those!
diff --git a/src/3p/chibicc/LICENSE b/src/3p/chibicc/LICENSE
deleted file mode 100644
index 2d1fd94..0000000
--- a/src/3p/chibicc/LICENSE
+++ /dev/null
@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2019 Rui Ueyama
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
diff --git a/src/3p/chibicc/chibicc.h b/src/3p/chibicc/chibicc.h
deleted file mode 100644
index f3f87ab..0000000
--- a/src/3p/chibicc/chibicc.h
+++ /dev/null
@@ -1,264 +0,0 @@
-// include guards: upstream doesn't have these but we add them so we can cat
-// source files together (or #include them, in particular)
-#ifndef INC_CHIBICC_H
-#define INC_CHIBICC_H
-
-#include <assert.h>
-#include <ctype.h>
-#include <errno.h>
-#include <stdarg.h>
-//#include <stdbool.h>
-#include <stdint.h>
-#include <stdio.h>
-#include <stdlib.h>
-// mike: stdnoreturn means we can't use our noreturn (_Noreturn void)
-// there are no noreturns in tokenize.c anyway, and the ones in this header have
-// been changed to just _Noreturn to avoid any possible conflict
-//#include <stdnoreturn.h>
-#include <string.h>
-
-// exists on all Unixes but normally hidden behind _GNU_SOURCE on Linux.
-// missing entirely on Windows (implemented in 3p/openbsd/asprintf.c for compat)
-int vasprintf(char **str, const char *fmt, va_list ap);
-
-#if !defined(__GNUC__) && !defined(__clang__)
-# define __attribute__(x)
-#endif
-
-typedef struct Type Type;
-typedef struct Member Member;
-typedef struct Node Node;
-typedef struct Hideset Hideset;
-
-//
-// strings.c
-//
-
-typedef struct {
-  char **data;
-  int capacity;
-  int len;
-} StringArray;
-
-void strarray_push(StringArray *arr, char *s);
-
-//
-// tokenize.c
-//
-
-// Token
-typedef enum {
-  TK_IDENT,   // Identifiers
-  TK_PUNCT,   // Punctuators
-  TK_KEYWORD, // Keywords
-  TK_STR,     // String literals
-  TK_NUM,     // Numeric literals
-  TK_PP_NUM,  // Preprocessing numbers
-  TK_EOF,     // End-of-file markers
-} TokenKind;
-
-typedef struct {
-  char *name;
-  int file_no;
-  char *contents;
-
-  // For #line directive
-  char *display_name;
-  int line_delta;
-} File;
-
-// Token type
-typedef struct Token Token;
-struct Token {
-  TokenKind kind;   // Token kind
-  Token *next;      // Next token
-  int64_t val;      // If kind is TK_NUM, its value
-  long double fval; // If kind is TK_NUM, its value
-  char *loc;        // Token location
-  int len;          // Token length
-  Type *ty;         // Used if TK_NUM or TK_STR
-  char *str;        // String literal contents including terminating '\0'
-
-  File *file;       // Source location
-  char *filename;   // Filename
-  int line_no;      // Line number
-  int line_delta;   // Line number
-  bool at_bol;      // True if this token is at beginning of line
-  bool has_space;   // True if this token follows a space character
-  Hideset *hideset; // For macro expansion
-  Token *origin;    // If this is expanded from a macro, the original token
-};
-
-_Noreturn void error(char *fmt, ...) __attribute__((format(printf, 1, 2)));
-_Noreturn void error_at(char *loc, char *fmt, ...) __attribute__((format(printf, 2, 3)));
-_Noreturn void error_tok(Token *tok, char *fmt, ...) __attribute__((format(printf, 2, 3)));
-void warn_tok(Token *tok, char *fmt, ...) __attribute__((format(printf, 2, 3)));
-bool equal(const Token *tok, const char *op);
-Token *skip(Token *tok, char *op);
-bool consume(Token **rest, Token *tok, char *str);
-void convert_pp_tokens(Token *tok);
-File **get_input_files(void);
-File *new_file(char *name, int file_no, char *contents);
-Token *tokenize_string_literal(Token *tok, Type *basety);
-Token *tokenize(File *file);
-//Token *tokenize_file(char *filename);
-Token *tokenize_buf(const char *name, char *p);
-
-// note: replacing memstream-based format with asprintf version. moved down here
-// as error() is declared above.
-//char *format(char *fmt, ...) __attribute__((format(printf, 1, 2)));
-__attribute__((format(printf, 1, 2)))
-static inline char *format(const char *fmt, ...) {
-  char *ret;
-  va_list va;
-  va_start(va, fmt);
-  if (vasprintf(&ret, fmt, va) == -1) error("couldn't allocate memory");
-  va_end(va);
-  return ret;
-}
-
-//
-// type.c
-//
-
-typedef enum {
-  TY_VOID,
-  TY_BOOL,
-  TY_CHAR,
-  TY_SHORT,
-  TY_INT,
-  TY_LONG,
-  TY_FLOAT,
-  TY_DOUBLE,
-  TY_LDOUBLE,
-  TY_ENUM,
-  TY_PTR,
-  TY_FUNC,
-  TY_ARRAY,
-  TY_VLA, // variable-length array
-  TY_STRUCT,
-  TY_UNION,
-} TypeKind;
-
-struct Type {
-  TypeKind kind;
-  int size;           // sizeof() value
-  int align;          // alignment
-  bool is_unsigned;   // unsigned or signed
-  bool is_atomic;     // true if _Atomic
-  Type *origin;       // for type compatibility check
-
-  // Pointer-to or array-of type. We intentionally use the same member
-  // to represent pointer/array duality in C.
-  //
-  // In many contexts in which a pointer is expected, we examine this
-  // member instead of "kind" member to determine whether a type is a
-  // pointer or not. That means in many contexts "array of T" is
-  // naturally handled as if it were "pointer to T", as required by
-  // the C spec.
-  Type *base;
-
-  // Declaration
-  Token *name;
-  Token *name_pos;
-
-  // Array
-  int array_len;
-
-  // Variable-length array
-  //Node *vla_len; // # of elements
-  //Obj *vla_size; // sizeof() value
-
-  // Struct
-  Member *members;
-  bool is_flexible;
-  bool is_packed;
-
-  // Function type
-  Type *return_ty;
-  Type *params;
-  bool is_variadic;
-  Type *next;
-};
-
-// Struct member
-struct Member {
-  Member *next;
-  Type *ty;
-  Token *tok; // for error message
-  Token *name;
-  int idx;
-  int align;
-  int offset;
-
-  // Bitfield
-  bool is_bitfield;
-  int bit_offset;
-  int bit_width;
-};
-
-extern Type *ty_void;
-extern Type *ty_bool;
-
-extern Type *ty_char;
-extern Type *ty_short;
-extern Type *ty_int;
-extern Type *ty_long;
-
-extern Type *ty_uchar;
-extern Type *ty_ushort;
-extern Type *ty_uint;
-extern Type *ty_ulong;
-
-extern Type *ty_float;
-extern Type *ty_double;
-extern Type *ty_ldouble;
-
-bool is_integer(Type *ty);
-bool is_flonum(Type *ty);
-bool is_numeric(Type *ty);
-bool is_compatible(Type *t1, Type *t2);
-Type *copy_type(Type *ty);
-Type *pointer_to(Type *base);
-Type *func_type(Type *return_ty);
-Type *array_of(Type *base, int size);
-Type *vla_of(Type *base, Node *expr);
-Type *enum_type(void);
-Type *struct_type(void);
-void add_type(Node *node);
-
-//
-// unicode.c
-//
-
-int encode_utf8(char *buf, uint32_t c);
-uint32_t decode_utf8(char **new_pos, char *p);
-bool is_ident1(uint32_t c);
-bool is_ident2(uint32_t c);
-int display_width(char *p, int len);
-
-//
-// hashmap.c
-//
-
-typedef struct {
-  char *key;
-  int keylen;
-  void *val;
-} HashEntry;
-
-typedef struct {
-  HashEntry *buckets;
-  int capacity;
-  int used;
-} HashMap;
-
-void *hashmap_get(HashMap *map, char *key);
-void *hashmap_get2(HashMap *map, char *key, int keylen);
-void hashmap_put(HashMap *map, char *key, void *val);
-void hashmap_put2(HashMap *map, char *key, int keylen, void *val);
-void hashmap_delete(HashMap *map, char *key);
-void hashmap_delete2(HashMap *map, char *key, int keylen);
-void hashmap_test(void);
-
-#endif
diff --git a/src/3p/chibicc/hashmap.c b/src/3p/chibicc/hashmap.c
deleted file mode 100644
index 2090274..0000000
--- a/src/3p/chibicc/hashmap.c
+++ /dev/null
@@ -1,137 +0,0 @@
-// This is an implementation of the open-addressing hash table.
-
-#include "chibicc.h"
-
-// mike: moved from chibicc.h and also renamed, to avoid conflicts with langext.h
-#define chibi_unreachable() error("internal error at %s:%d", __FILE__, __LINE__)
-
-// Initial hash bucket size
-#define INIT_SIZE 16
-
-// Rehash if the usage exceeds 70%.
-#define HIGH_WATERMARK 70
-
-// We'll keep the usage below 50% after rehashing.
-#define LOW_WATERMARK 50
-
-// Represents a deleted hash entry
-#define TOMBSTONE ((void *)-1)
-
-static uint64_t fnv_hash(char *s, int len) {
-  uint64_t hash = 0xcbf29ce484222325;
-  for (int i = 0; i < len; i++) {
-    hash *= 0x100000001b3;
-    hash ^= (unsigned char)s[i];
-  }
-  return hash;
-}
-
-// Make room for new entires in a given hashmap by removing
-// tombstones and possibly extending the bucket size.
-static void rehash(HashMap *map) {
-  // Compute the size of the new hashmap.
-  int nkeys = 0;
-  for (int i = 0; i < map->capacity; i++)
-    if (map->buckets[i].key && map->buckets[i].key != TOMBSTONE)
-      nkeys++;
-
-  int cap = map->capacity;
-  while ((nkeys * 100) / cap >= LOW_WATERMARK)
-    cap = cap * 2;
-  assert(cap > 0);
-
-  // Create a new hashmap and copy all key-values.
-  HashMap map2 = {0};
-  map2.buckets = calloc(cap, sizeof(HashEntry));
-  map2.capacity = cap;
-
-  for (int i = 0; i < map->capacity; i++) {
-    HashEntry *ent = &map->buckets[i];
-    if (ent->key && ent->key != TOMBSTONE)
-      hashmap_put2(&map2, ent->key, ent->keylen, ent->val);
-  }
-
-  assert(map2.used == nkeys);
-  *map = map2;
-}
-
-static bool match(HashEntry *ent, char *key, int keylen) {
-  return ent->key && ent->key != TOMBSTONE &&
-         ent->keylen == keylen && memcmp(ent->key, key, keylen) == 0;
-}
-
-static HashEntry *get_entry(HashMap *map, char *key, int keylen) {
-  if (!map->buckets)
-    return NULL;
-
-  uint64_t hash = fnv_hash(key, keylen);
-
-  for (int i = 0; i < map->capacity; i++) {
-    HashEntry *ent = &map->buckets[(hash + i) % map->capacity];
-    if (match(ent, key, keylen))
-      return ent;
-    if (ent->key == NULL)
-      return NULL;
-  }
-  chibi_unreachable();
-}
-
-static HashEntry *get_or_insert_entry(HashMap *map, char *key, int keylen) {
-  if (!map->buckets) {
-    map->buckets = calloc(INIT_SIZE, sizeof(HashEntry));
-    map->capacity = INIT_SIZE;
-  } else if ((map->used * 100) / map->capacity >= HIGH_WATERMARK) {
-    rehash(map);
-  }
-
-  uint64_t hash = fnv_hash(key, keylen);
-
-  for (int i = 0; i < map->capacity; i++) {
-    HashEntry *ent = &map->buckets[(hash + i) % map->capacity];
-
-    if (match(ent, key, keylen))
-      return ent;
-
-    if (ent->key == TOMBSTONE) {
-      ent->key = key;
-      ent->keylen = keylen;
-      return ent;
-    }
-
-    if (ent->key == NULL) {
-      ent->key = key;
-      ent->keylen = keylen;
-      map->used++;
-      return ent;
-    }
-  }
-  chibi_unreachable();
-}
-
-void *hashmap_get(HashMap *map, char *key) {
-  return hashmap_get2(map, key, strlen(key));
-}
-
-void *hashmap_get2(HashMap *map, char *key, int keylen) {
-  HashEntry *ent = get_entry(map, key, keylen);
-  return ent ? ent->val : NULL;
-}
-
-void hashmap_put(HashMap *map, char *key, void *val) {
-   hashmap_put2(map, key, strlen(key), val);
-}
-
-void hashmap_put2(HashMap *map, char *key, int keylen, void *val) {
-  HashEntry *ent = get_or_insert_entry(map, key, keylen);
-  ent->val = val;
-}
-
-void hashmap_delete(HashMap *map, char *key) {
-  hashmap_delete2(map, key, strlen(key));
-}
-
-void hashmap_delete2(HashMap *map, char *key, int keylen) {
-  HashEntry *ent = get_entry(map, key, keylen);
-  if (ent)
-    ent->key = TOMBSTONE;
-}
diff --git a/src/3p/chibicc/strings.c b/src/3p/chibicc/strings.c
deleted file mode 100644
index 0538fef..0000000
--- a/src/3p/chibicc/strings.c
+++ /dev/null
@@ -1,17 +0,0 @@
-#include "chibicc.h"
-
-void strarray_push(StringArray *arr, char *s) {
-  if (!arr->data) {
-    arr->data = calloc(8, sizeof(char *));
-    arr->capacity = 8;
-  }
-
-  if (arr->capacity == arr->len) {
-    arr->data = realloc(arr->data, sizeof(char *) * arr->capacity * 2);
-    arr->capacity *= 2;
-    for (int i = arr->len; i < arr->capacity; i++)
-      arr->data[i] = NULL;
-  }
-
-  arr->data[arr->len++] = s;
-}
diff --git a/src/3p/chibicc/tokenize.c b/src/3p/chibicc/tokenize.c
deleted file mode 100644
index 6738206..0000000
--- a/src/3p/chibicc/tokenize.c
+++ /dev/null
@@ -1,785 +0,0 @@
-#include "chibicc.h"
-
-#ifdef _WIN32
-#define strncasecmp _strnicmp
-#endif
-
-// Input file
-static File *current_file;
-
-// A list of all input files.
-static File **input_files;
-
-// True if the current position is at the beginning of a line
-static bool at_bol;
-
-// True if the current position follows a space character
-static bool has_space;
-
-// Reports an error and exit.
-void error(char *fmt, ...) {
-  va_list ap;
-  va_start(ap, fmt);
-  fprintf(stderr, "cmeta: chibicc: ");
-  vfprintf(stderr, fmt, ap);
-  fprintf(stderr, "\n");
-  exit(1);
-}
-
-// Reports an error message in the following format.
-//
-// foo.c:10: x = y + 1;
-//               ^ <error message here>
-static void verror_at(char *filename, char *input, int line_no,
-                      char *loc, char *fmt, va_list ap) {
-  // Find a line containing `loc`.
-  char *line = loc;
-  while (input < line && line[-1] != '\n')
-    line--;
-
-  char *end = loc;
-  while (*end && *end != '\n')
-    end++;
-
-  // Print out the line.
-  int indent = fprintf(stderr, "%s:%d: ", filename, line_no);
-  fprintf(stderr, "%.*s\n", (int)(end - line), line);
-
-  // Show the error message.
-  int pos = display_width(line, loc - line) + indent;
-
-  fprintf(stderr, "%*s", pos, ""); // print pos spaces.
-  fprintf(stderr, "^ ");
-  vfprintf(stderr, fmt, ap);
-  fprintf(stderr, "\n");
-}
-
-void error_at(char *loc, char *fmt, ...) {
-  int line_no = 1;
-  for (char *p = current_file->contents; p < loc; p++)
-    if (*p == '\n')
-      line_no++;
-
-  va_list ap;
-  va_start(ap, fmt);
-  verror_at(current_file->name, current_file->contents, line_no, loc, fmt, ap);
-  exit(1);
-}
-
-void error_tok(Token *tok, char *fmt, ...) {
-  va_list ap;
-  va_start(ap, fmt);
-  verror_at(tok->file->name, tok->file->contents, tok->line_no, tok->loc, fmt, ap);
-  exit(1);
-}
-
-void warn_tok(Token *tok, char *fmt, ...) {
-  va_list ap;
-  va_start(ap, fmt);
-  verror_at(tok->file->name, tok->file->contents, tok->line_no, tok->loc, fmt, ap);
-  va_end(ap);
-}
-
-// Consumes the current token if it matches `op`.
-bool equal(const Token *tok, const char *op) {
-  return memcmp(tok->loc, op, tok->len) == 0 && op[tok->len] == '\0';
-}
-
-// Ensure that the current token is `op`.
-Token *skip(Token *tok, char *op) {
-  if (!equal(tok, op))
-    error_tok(tok, "expected '%s'", op);
-  return tok->next;
-}
-
-bool consume(Token **rest, Token *tok, char *str) {
-  if (equal(tok, str)) {
-    *rest = tok->next;
-    return true;
-  }
-  *rest = tok;
-  return false;
-}
-
-// Create a new token.
-static Token *new_token(TokenKind kind, char *start, char *end) {
-  Token *tok = calloc(1, sizeof(Token));
-  tok->kind = kind;
-  tok->loc = start;
-  tok->len = end - start;
-  tok->file = current_file;
-  tok->filename = current_file->display_name;
-  tok->at_bol = at_bol;
-  tok->has_space = has_space;
-
-  at_bol = has_space = false;
-  return tok;
-}
-
-static bool startswith(char *p, char *q) {
-  return strncmp(p, q, strlen(q)) == 0;
-}
-
-// Read an identifier and returns the length of it.
-// If p does not point to a valid identifier, 0 is returned.
-static int read_ident(char *start) {
-  char *p = start;
-  uint32_t c = decode_utf8(&p, p);
-  if (!is_ident1(c))
-    return 0;
-
-  for (;;) {
-    char *q;
-    c = decode_utf8(&q, p);
-    if (!is_ident2(c))
-      return p - start;
-    p = q;
-  }
-}
-
-static int from_hex(char c) {
-  if ('0' <= c && c <= '9')
-    return c - '0';
-  if ('a' <= c && c <= 'f')
-    return c - 'a' + 10;
-  return c - 'A' + 10;
-}
-
-// Read a punctuator token from p and returns its length.
-static int read_punct(char *p) {
-  static char *kw[] = {
-    "<<=", ">>=", "...", "==", "!=", "<=", ">=", "->", "+=",
-    "-=", "*=", "/=", "++", "--", "%=", "&=", "|=", "^=", "&&",
-    "||", "<<", ">>", "##",
-  };
-
-  for (int i = 0; i < sizeof(kw) / sizeof(*kw); i++)
-    if (startswith(p, kw[i]))
-      return strlen(kw[i]);
-
-  return ispunct(*p) ? 1 : 0;
-}
-
-static bool is_keyword(Token *tok) {
-  static HashMap map;
-
-  if (map.capacity == 0) {
-    static char *kw[] = {
-      "return", "if", "else", "for", "while", "int", "sizeof", "char",
-      "struct", "union", "short", "long", "void", "typedef", "_Bool",
-      "enum", "static", "goto", "break", "continue", "switch", "case",
-      "default", "extern", "_Alignof", "_Alignas", "do", "signed",
-      "unsigned", "const", "volatile", "auto", "register", "restrict",
-      "__restrict", "__restrict__", "_Noreturn", "float", "double",
-      "typeof", "asm", "_Thread_local", "__thread", "_Atomic",
-      "__attribute__",
-    };
-
-    for (int i = 0; i < sizeof(kw) / sizeof(*kw); i++)
-      hashmap_put(&map, kw[i], (void *)1);
-  }
-
-  return hashmap_get2(&map, tok->loc, tok->len);
-}
-
-static int read_escaped_char(char **new_pos, char *p) {
-  if ('0' <= *p && *p <= '7') {
-    // Read an octal number.
-    int c = *p++ - '0';
-    if ('0' <= *p && *p <= '7') {
-      c = (c << 3) + (*p++ - '0');
-      if ('0' <= *p && *p <= '7')
-        c = (c << 3) + (*p++ - '0');
-    }
-    *new_pos = p;
-    return c;
-  }
-
-  if (*p == 'x') {
-    // Read a hexadecimal number.
-    p++;
-    if (!isxdigit(*p))
-      error_at(p, "invalid hex escape sequence");
-
-    int c = 0;
-    for (; isxdigit(*p); p++)
-      c = (c << 4) + from_hex(*p);
-    *new_pos = p;
-    return c;
-  }
-
-  *new_pos = p + 1;
-
-  // Escape sequences are defined using themselves here. E.g.
-  // '\n' is implemented using '\n'. This tautological definition
-  // works because the compiler that compiles our compiler knows
-  // what '\n' actually is. In other words, we "inherit" the ASCII
-  // code of '\n' from the compiler that compiles our compiler,
-  // so we don't have to teach the actual code here.
-  //
-  // This fact has huge implications not only for the correctness
-  // of the compiler but also for the security of the generated code.
-  // For more info, read "Reflections on Trusting Trust" by Ken Thompson.
-  // https://github.com/rui314/chibicc/wiki/thompson1984.pdf
-  switch (*p) {
-  case 'a': return '\a';
-  case 'b': return '\b';
-  case 't': return '\t';
-  case 'n': return '\n';
-  case 'v': return '\v';
-  case 'f': return '\f';
-  case 'r': return '\r';
-  // [GNU] \e for the ASCII escape character is a GNU C extension.
-  case 'e': return 27;
-  default: return *p;
-  }
-}
-
-// Find a closing double-quote.
-static char *string_literal_end(char *p) {
-  char *start = p;
-  for (; *p != '"'; p++) {
-    if (*p == '\n' || *p == '\0')
-      error_at(start, "unclosed string literal");
-    if (*p == '\\')
-      p++;
-  }
-  return p;
-}
-
-static Token *read_string_literal(char *start, char *quote) {
-  char *end = string_literal_end(quote + 1);
-  char *buf = calloc(1, end - quote);
-  int len = 0;
-
-  for (char *p = quote + 1; p < end;) {
-    if (*p == '\\')
-      buf[len++] = read_escaped_char(&p, p + 1);
-    else
-      buf[len++] = *p++;
-  }
-
-  Token *tok = new_token(TK_STR, start, end + 1);
-  tok->ty = array_of(ty_char, len + 1);
-  tok->str = buf;
-  return tok;
-}
-
-// Read a UTF-8-encoded string literal and transcode it in UTF-16.
-//
-// UTF-16 is yet another variable-width encoding for Unicode. Code
-// points smaller than U+10000 are encoded in 2 bytes. Code points
-// equal to or larger than that are encoded in 4 bytes. Each 2 bytes
-// in the 4 byte sequence is called "surrogate", and a 4 byte sequence
-// is called a "surrogate pair".
-static Token *read_utf16_string_literal(char *start, char *quote) {
-  char *end = string_literal_end(quote + 1);
-  uint16_t *buf = calloc(2, end - start);
-  int len = 0;
-
-  for (char *p = quote + 1; p < end;) {
-    if (*p == '\\') {
-      buf[len++] = read_escaped_char(&p, p + 1);
-      continue;
-    }
-
-    uint32_t c = decode_utf8(&p, p);
-    if (c < 0x10000) {
-      // Encode a code point in 2 bytes.
-      buf[len++] = c;
-    } else {
-      // Encode a code point in 4 bytes.
-      c -= 0x10000;
-      buf[len++] = 0xd800 + ((c >> 10) & 0x3ff);
-      buf[len++] = 0xdc00 + (c & 0x3ff);
-    }
-  }
-
-  Token *tok = new_token(TK_STR, start, end + 1);
-  tok->ty = array_of(ty_ushort, len + 1);
-  tok->str = (char *)buf;
-  return tok;
-}
-
-// Read a UTF-8-encoded string literal and transcode it in UTF-32.
-//
-// UTF-32 is a fixed-width encoding for Unicode. Each code point is
-// encoded in 4 bytes.
-static Token *read_utf32_string_literal(char *start, char *quote, Type *ty) {
-  char *end = string_literal_end(quote + 1);
-  uint32_t *buf = calloc(4, end - quote);
-  int len = 0;
-
-  for (char *p = quote + 1; p < end;) {
-    if (*p == '\\')
-      buf[len++] = read_escaped_char(&p, p + 1);
-    else
-      buf[len++] = decode_utf8(&p, p);
-  }
-
-  Token *tok = new_token(TK_STR, start, end + 1);
-  tok->ty = array_of(ty, len + 1);
-  tok->str = (char *)buf;
-  return tok;
-}
-
-static Token *read_char_literal(char *start, char *quote, Type *ty) {
-  char *p = quote + 1;
-  if (*p == '\0')
-    error_at(start, "unclosed char literal");
-
-  int c;
-  if (*p == '\\')
-    c = read_escaped_char(&p, p + 1);
-  else
-    c = decode_utf8(&p, p);
-
-  char *end = p;
-  for (; *end != '\''; ++end) {
-    if (!*end || *end == '\n')
-      error_at(p, "unclosed char literal");
-  }
-
-  Token *tok = new_token(TK_NUM, start, end + 1);
-  tok->val = c;
-  tok->ty = ty;
-  return tok;
-}
-
-static bool convert_pp_int(Token *tok) {
-  char *p = tok->loc;
-
-  // Read a binary, octal, decimal or hexadecimal number.
-  int base = 10;
-  if (!strncasecmp(p, "0x", 2) && isxdigit(p[2])) {
-    p += 2;
-    base = 16;
-  } else if (!strncasecmp(p, "0b", 2) && (p[2] == '0' || p[2] == '1')) {
-    p += 2;
-    base = 2;
-  } else if (*p == '0') {
-    base = 8;
-  }
-
-  int64_t val = strtoul(p, &p, base);
-
-  // Read U, L or LL suffixes.
-  bool l = false;
-  bool u = false;
-
-  if (startswith(p, "LLU") || startswith(p, "LLu") ||
-      startswith(p, "llU") || startswith(p, "llu") ||
-      startswith(p, "ULL") || startswith(p, "Ull") ||
-      startswith(p, "uLL") || startswith(p, "ull")) {
-    p += 3;
-    l = u = true;
-  } else if (!strncasecmp(p, "lu", 2) || !strncasecmp(p, "ul", 2)) {
-    p += 2;
-    l = u = true;
-  } else if (startswith(p, "LL") || startswith(p, "ll")) {
-    p += 2;
-    l = true;
-  } else if (*p == 'L' || *p == 'l') {
-    p++;
-    l = true;
-  } else if (*p == 'U' || *p == 'u') {
-    p++;
-    u = true;
-  }
-
-  if (p != tok->loc + tok->len)
-    return false;
-
-  // Infer a type.
-  Type *ty;
-  if (base == 10) {
-    if (l && u)
-      ty = ty_ulong;
-    else if (l)
-      ty = ty_long;
-    else if (u)
-      ty = (val >> 32) ? ty_ulong : ty_uint;
-    else
-      ty = (val >> 31) ? ty_long : ty_int;
-  } else {
-    if (l && u)
-      ty = ty_ulong;
-    else if (l)
-      ty = (val >> 63) ? ty_ulong : ty_long;
-    else if (u)
-      ty = (val >> 32) ? ty_ulong : ty_uint;
-    else if (val >> 63)
-      ty = ty_ulong;
-    else if (val >> 32)
-      ty = ty_long;
-    else if (val >> 31)
-      ty = ty_uint;
-    else
-      ty = ty_int;
-  }
-
-  tok->kind = TK_NUM;
-  tok->val = val;
-  tok->ty = ty;
-  return true;
-}
-
-// The definition of the numeric literal at the preprocessing stage
-// is more relaxed than the definition of that at the later stages.
-// In order to handle that, a numeric literal is tokenized as a
-// "pp-number" token first and then converted to a regular number
-// token after preprocessing.
-//
-// This function converts a pp-number token to a regular number token.
-static void convert_pp_number(Token *tok) {
-  // Try to parse as an integer constant.
-  if (convert_pp_int(tok))
-    return;
-
-  // If it's not an integer, it must be a floating point constant.
-  char *end;
-  long double val = strtold(tok->loc, &end);
-
-  Type *ty;
-  if (*end == 'f' || *end == 'F') {
-    ty = ty_float;
-    end++;
-  } else if (*end == 'l' || *end == 'L') {
-    ty = ty_ldouble;
-    end++;
-  } else {
-    ty = ty_double;
-  }
-
-  if (tok->loc + tok->len != end)
-    error_tok(tok, "invalid numeric constant");
-
-  tok->kind = TK_NUM;
-  tok->fval = val;
-  tok->ty = ty;
-}
-
-void convert_pp_tokens(Token *tok) {
-  for (Token *t = tok; t->kind != TK_EOF; t = t->next) {
-    if (is_keyword(t))
-      t->kind = TK_KEYWORD;
-    else if (t->kind == TK_PP_NUM)
-      convert_pp_number(t);
-  }
-}
-
-// Initialize line info for all tokens.
-static void add_line_numbers(Token *tok) {
-  char *p = current_file->contents;
-  int n = 1;
-
-  do {
-    if (p == tok->loc) {
-      tok->line_no = n;
-      tok = tok->next;
-    }
-    if (*p == '\n')
-      n++;
-  } while (*p++);
-}
-
-Token *tokenize_string_literal(Token *tok, Type *basety) {
-  Token *t;
-  if (basety->size == 2)
-    t = read_utf16_string_literal(tok->loc, tok->loc);
-  else
-    t = read_utf32_string_literal(tok->loc, tok->loc, basety);
-  t->next = tok->next;
-  return t;
-}
-
-// Tokenize a given string and returns new tokens.
-Token *tokenize(File *file) {
-  current_file = file;
-
-  char *p = file->contents;
-  Token head = {0};
-  Token *cur = &head;
-
-  at_bol = true;
-  has_space = false;
-
-  while (*p) {
-    // Skip line comments.
-    if (startswith(p, "//")) {
-      p += 2;
-      while (*p != '\n')
-        p++;
-      has_space = true;
-      continue;
-    }
-
-    // Skip block comments.
-    if (startswith(p, "/*")) {
-      char *q = strstr(p + 2, "*/");
-      if (!q)
-        error_at(p, "unclosed block comment");
-      p = q + 2;
-      has_space = true;
-      continue;
-    }
-
-    // Skip newline.
-    if (*p == '\n') {
-      p++;
-      at_bol = true;
-      has_space = false;
-      continue;
-    }
-
-    // Skip whitespace characters.
-    if (isspace(*p)) {
-      p++;
-      has_space = true;
-      continue;
-    }
-
-    // Numeric literal
-    if (isdigit(*p) || (*p == '.' && isdigit(p[1]))) {
-      char *q = p++;
-      for (;;) {
-        if (p[0] && p[1] && strchr("eEpP", p[0]) && strchr("+-", p[1]))
-          p += 2;
-        else if (isalnum(*p) || *p == '.')
-          p++;
-        else
-          break;
-      }
-      cur = cur->next = new_token(TK_PP_NUM, q, p);
-      continue;
-    }
-
-    // String literal
-    if (*p == '"') {
-      cur = cur->next = read_string_literal(p, p);
-      p += cur->len;
-      continue;
-    }
-
-    // UTF-8 string literal
-    if (startswith(p, "u8\"")) {
-      cur = cur->next = read_string_literal(p, p + 2);
-      p += cur->len;
-      continue;
-    }
-
-    // UTF-16 string literal
-    if (startswith(p, "u\"")) {
-      cur = cur->next = read_utf16_string_literal(p, p + 1);
-      p += cur->len;
-      continue;
-    }
-
-    // Wide string literal
-    if (startswith(p, "L\"")) {
-      cur = cur->next = read_utf32_string_literal(p, p + 1, ty_int);
-      p += cur->len;
-      continue;
-    }
-
-    // UTF-32 string literal
-    if (startswith(p, "U\"")) {
-      cur = cur->next = read_utf32_string_literal(p, p + 1, ty_uint);
-      p += cur->len;
-      continue;
-    }
-
-    // Character literal
-    if (*p == '\'') {
-      cur = cur->next = read_char_literal(p, p, ty_int);
-      cur->val = (char)cur->val;
-      p += cur->len;
-      continue;
-    }
-
-    // UTF-16 character literal
-    if (startswith(p, "u'")) {
-      cur = cur->next = read_char_literal(p, p + 1, ty_ushort);
-      cur->val &= 0xffff;
-      p += cur->len;
-      continue;
-    }
-
-    // Wide character literal
-    if (startswith(p, "L'")) {
-      cur = cur->next = read_char_literal(p, p + 1, ty_int);
-      p += cur->len;
-      continue;
-    }
-
-    // UTF-32 character literal
-    if (startswith(p, "U'")) {
-      cur = cur->next = read_char_literal(p, p + 1, ty_uint);
-      p += cur->len;
-      continue;
-    }
-
-    // Identifier or keyword
-    int ident_len = read_ident(p);
-    if (ident_len) {
-      cur = cur->next = new_token(TK_IDENT, p, p + ident_len);
-      p += cur->len;
-      continue;
-    }
-
-    // Punctuators
-    int punct_len = read_punct(p);
-    if (punct_len) {
-      cur = cur->next = new_token(TK_PUNCT, p, p + punct_len);
-      p += cur->len;
-      continue;
-    }
-
-    error_at(p, "invalid token");
-  }
-
-  cur = cur->next = new_token(TK_EOF, p, p);
-  add_line_numbers(head.next);
-  return head.next;
-}
-
-// Returns the contents of a given file.
-/* this function is a bit wonkily implemented, and relies on a non-windows
- * thing, so we have our own in src/cmeta.c.
-static char *read_file(char *path) {
-  FILE *fp;
-
-  if (strcmp(path, "-") == 0) {
-    // By convention, read from stdin if a given filename is "-".
-    fp = stdin;
-  } else {
-    fp = fopen(path, "r");
-    if (!fp)
-      return NULL;
-  }
-
-  char *buf;
-  size_t buflen;
-  FILE *out = open_memstream(&buf, &buflen);
-
-  // Read the entire file.
-  for (;;) {
-    char buf2[4096];
-    int n = fread(buf2, 1, sizeof(buf2), fp);
-    if (n == 0)
-      break;
-    fwrite(buf2, 1, n, out);
-  }
-
-  if (fp != stdin)
-    fclose(fp);
-
-  // Make sure that the last line is properly terminated with '\n'.
-  fflush(out);
-  if (buflen == 0 || buf[buflen - 1] != '\n')
-    fputc('\n', out);
-  fputc('\0', out);
-  fclose(out);
-  return buf;
-}
-*/
-
-File **get_input_files(void) {
-  return input_files;
-}
-
-File *new_file(char *name, int file_no, char *contents) {
-  File *file = calloc(1, sizeof(File));
-  file->name = name;
-  file->display_name = name;
-  file->file_no = file_no;
-  file->contents = contents;
-  return file;
-}
-
-// Removes backslashes followed by a newline.
-static void remove_backslash_newline(char *p) {
-  int i = 0, j = 0;
-
-  // We want to keep the number of newline characters so that
-  // the logical line number matches the physical one.
-  // This counter maintain the number of newlines we have removed.
-  int n = 0;
-
-  while (p[i]) {
-    if (p[i] == '\\' && p[i + 1] == '\n') {
-      i += 2;
-      n++;
-    } else if (p[i] == '\n') {
-      p[j++] = p[i++];
-      for (; n > 0; n--)
-        p[j++] = '\n';
-    } else {
-      p[j++] = p[i++];
-    }
-  }
-
-  for (; n > 0; n--)
-    p[j++] = '\n';
-  p[j] = '\0';
-}
-
-static uint32_t read_universal_char(char *p, int len) {
-  uint32_t c = 0;
-  for (int i = 0; i < len; i++) {
-    if (!isxdigit(p[i]))
-      return 0;
-    c = (c << 4) | from_hex(p[i]);
-  }
-  return c;
-}
-
-// Replace \u or \U escape sequences with corresponding UTF-8 bytes.
-static void convert_universal_chars(char *p) {
-  char *q = p;
-
-  while (*p) {
-    if (startswith(p, "\\u")) {
-      uint32_t c = read_universal_char(p + 2, 4);
-      if (c) {
-        p += 6;
-        q += encode_utf8(q, c);
-      } else {
-        *q++ = *p++;
-      }
-    } else if (startswith(p, "\\U")) {
-      uint32_t c = read_universal_char(p + 2, 8);
-      if (c) {
-        p += 10;
-        q += encode_utf8(q, c);
-      } else {
-        *q++ = *p++;
-      }
-    } else if (p[0] == '\\') {
-      *q++ = *p++;
-      *q++ = *p++;
-    } else {
-      *q++ = *p++;
-    }
-  }
-
-  *q = '\0';
-}
-
-// NOTE modified API from upstream
-Token *tokenize_buf(const char *name, char *p) {
-  remove_backslash_newline(p);
-  convert_universal_chars(p);
-
-  // Save the filename for assembler .file directive.
-  static int file_no;
-  File *file = new_file((char *)name, file_no + 1, p);
-
-  // Save the filename for assembler .file directive.
-  input_files = realloc(input_files, sizeof(char *) * (file_no + 2));
-  input_files[file_no] = file;
-  input_files[file_no + 1] = NULL;
-  file_no++;
-
-  return tokenize(file);
-}
diff --git a/src/3p/chibicc/unicode.c b/src/3p/chibicc/unicode.c
deleted file mode 100644
index 6db1ad7..0000000
--- a/src/3p/chibicc/unicode.c
+++ /dev/null
@@ -1,189 +0,0 @@
-#include "chibicc.h"
-
-// Encode a given character in UTF-8.
-int encode_utf8(char *buf, uint32_t c) {
-  if (c <= 0x7F) {
-    buf[0] = c;
-    return 1;
-  }
-
-  if (c <= 0x7FF) {
-    buf[0] = 0xC0 | (c >> 6);
-    buf[1] = 0x80 | ((c >> 6) & 0x3F);
-    return 2;
-  }
-
-  if (c <= 0xFFFF) {
-    buf[0] = 0xE0 | (c >> 12);
-    buf[1] = 0x80 | ((c >> 6) & 0x3F);
-    buf[2] = 0x80 | (c & 0x3F);
-    return 3;
-  }
-
-  buf[0] = 0xF0 | (c >> 18);
-  buf[1] = 0x80 | ((c >> 12) & 0x3F);
-  buf[2] = 0x80 | ((c >> 6) & 0x3F);
-  buf[3] = 0x80 | (c & 0x3F);
-  return 4;
-}
-
-// Read a UTF-8-encoded Unicode code point from a source file.
-// We assume that source files are always in UTF-8.
-//
-// UTF-8 is a variable-width encoding in which one code point is
-// encoded in one to four bytes. One byte UTF-8 code points are
-// identical to ASCII. Non-ASCII characters are encoded using more
-// than one byte.
-uint32_t decode_utf8(char **new_pos, char *p) {
-  if ((unsigned char)*p < 128) {
-    *new_pos = p + 1;
-    return *p;
-  }
-
-  char *start = p;
-  int len;
-  uint32_t c;
-
-  if ((unsigned char)*p >= 0xF0) {
-    len = 4;
-    c = *p & 7;
-  } else if ((unsigned char)*p >= 0xE0) {
-    len = 3;
-    c = *p & 15;
-  } else if ((unsigned char)*p >= 0xC0) {
-    len = 2;
-    c = *p & 31;
-  } else {
-    error_at(start, "invalid UTF-8 sequence");
-  }
-
-  for (int i = 1; i < len; i++) {
-    if ((unsigned char)p[i] >> 6 != 2)
-      error_at(start, "invalid UTF-8 sequence");
-    c = (c << 6) | (p[i] & 63);
-  }
-
-  *new_pos = p + len;
-  return c;
-}
-
-static bool in_range(uint32_t *range, uint32_t c) {
-  for (int i = 0; range[i] != -1; i += 2)
-    if (range[i] <= c && c <= range[i + 1])
-      return true;
-  return false;
-}
-
-// [https://www.sigbus.info/n1570#D] C11 allows not only ASCII but
-// some multibyte characters in certan Unicode ranges to be used in an
-// identifier.
-//
-// This function returns true if a given character is acceptable as
-// the first character of an identifier.
-//
-// For example, ¾ (U+00BE) is a valid identifier because characters in
-// 0x00BE-0x00C0 are allowed, while neither ⟘ (U+27D8) nor '　'
-// (U+3000, full-width space) are allowed because they are out of range.
-bool is_ident1(uint32_t c) {
-  static uint32_t range[] = {
-    '_', '_', 'a', 'z', 'A', 'Z', '$', '$',
-    0x00A8, 0x00A8, 0x00AA, 0x00AA, 0x00AD, 0x00AD, 0x00AF, 0x00AF,
-    0x00B2, 0x00B5, 0x00B7, 0x00BA, 0x00BC, 0x00BE, 0x00C0, 0x00D6,
-    0x00D8, 0x00F6, 0x00F8, 0x00FF, 0x0100, 0x02FF, 0x0370, 0x167F,
-    0x1681, 0x180D, 0x180F, 0x1DBF, 0x1E00, 0x1FFF, 0x200B, 0x200D,
-    0x202A, 0x202E, 0x203F, 0x2040, 0x2054, 0x2054, 0x2060, 0x206F,
-    0x2070, 0x20CF, 0x2100, 0x218F, 0x2460, 0x24FF, 0x2776, 0x2793,
-    0x2C00, 0x2DFF, 0x2E80, 0x2FFF, 0x3004, 0x3007, 0x3021, 0x302F,
-    0x3031, 0x303F, 0x3040, 0xD7FF, 0xF900, 0xFD3D, 0xFD40, 0xFDCF,
-    0xFDF0, 0xFE1F, 0xFE30, 0xFE44, 0xFE47, 0xFFFD,
-    0x10000, 0x1FFFD, 0x20000, 0x2FFFD, 0x30000, 0x3FFFD, 0x40000, 0x4FFFD,
-    0x50000, 0x5FFFD, 0x60000, 0x6FFFD, 0x70000, 0x7FFFD, 0x80000, 0x8FFFD,
-    0x90000, 0x9FFFD, 0xA0000, 0xAFFFD, 0xB0000, 0xBFFFD, 0xC0000, 0xCFFFD,
-    0xD0000, 0xDFFFD, 0xE0000, 0xEFFFD, -1,
-  };
-
-  return in_range(range, c);
-}
-
-// Returns true if a given character is acceptable as a non-first
-// character of an identifier.
-bool is_ident2(uint32_t c) {
-  static uint32_t range[] = {
-    '0', '9', '$', '$', 0x0300, 0x036F, 0x1DC0, 0x1DFF, 0x20D0, 0x20FF,
-    0xFE20, 0xFE2F, -1,
-  };
-
-  return is_ident1(c) || in_range(range, c);
-}
-
-// Returns the number of columns needed to display a given
-// character in a fixed-width font.
-//
-// Based on https://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
-static int char_width(uint32_t c) {
-  static uint32_t range1[] = {
-    0x0000, 0x001F, 0x007f, 0x00a0, 0x0300, 0x036F, 0x0483, 0x0486,
-    0x0488, 0x0489, 0x0591, 0x05BD, 0x05BF, 0x05BF, 0x05C1, 0x05C2,
-    0x05C4, 0x05C5, 0x05C7, 0x05C7, 0x0600, 0x0603, 0x0610, 0x0615,
-    0x064B, 0x065E, 0x0670, 0x0670, 0x06D6, 0x06E4, 0x06E7, 0x06E8,
-    0x06EA, 0x06ED, 0x070F, 0x070F, 0x0711, 0x0711, 0x0730, 0x074A,
-    0x07A6, 0x07B0, 0x07EB, 0x07F3, 0x0901, 0x0902, 0x093C, 0x093C,
-    0x0941, 0x0948, 0x094D, 0x094D, 0x0951, 0x0954, 0x0962, 0x0963,
-    0x0981, 0x0981, 0x09BC, 0x09BC, 0x09C1, 0x09C4, 0x09CD, 0x09CD,
-    0x09E2, 0x09E3, 0x0A01, 0x0A02, 0x0A3C, 0x0A3C, 0x0A41, 0x0A42,
-    0x0A47, 0x0A48, 0x0A4B, 0x0A4D, 0x0A70, 0x0A71, 0x0A81, 0x0A82,
-    0x0ABC, 0x0ABC, 0x0AC1, 0x0AC5, 0x0AC7, 0x0AC8, 0x0ACD, 0x0ACD,
-    0x0AE2, 0x0AE3, 0x0B01, 0x0B01, 0x0B3C, 0x0B3C, 0x0B3F, 0x0B3F,
-    0x0B41, 0x0B43, 0x0B4D, 0x0B4D, 0x0B56, 0x0B56, 0x0B82, 0x0B82,
-    0x0BC0, 0x0BC0, 0x0BCD, 0x0BCD, 0x0C3E, 0x0C40, 0x0C46, 0x0C48,
-    0x0C4A, 0x0C4D, 0x0C55, 0x0C56, 0x0CBC, 0x0CBC, 0x0CBF, 0x0CBF,
-    0x0CC6, 0x0CC6, 0x0CCC, 0x0CCD, 0x0CE2, 0x0CE3, 0x0D41, 0x0D43,
-    0x0D4D, 0x0D4D, 0x0DCA, 0x0DCA, 0x0DD2, 0x0DD4, 0x0DD6, 0x0DD6,
-    0x0E31, 0x0E31, 0x0E34, 0x0E3A, 0x0E47, 0x0E4E, 0x0EB1, 0x0EB1,
-    0x0EB4, 0x0EB9, 0x0EBB, 0x0EBC, 0x0EC8, 0x0ECD, 0x0F18, 0x0F19,
-    0x0F35, 0x0F35, 0x0F37, 0x0F37, 0x0F39, 0x0F39, 0x0F71, 0x0F7E,
-    0x0F80, 0x0F84, 0x0F86, 0x0F87, 0x0F90, 0x0F97, 0x0F99, 0x0FBC,
-    0x0FC6, 0x0FC6, 0x102D, 0x1030, 0x1032, 0x1032, 0x1036, 0x1037,
-    0x1039, 0x1039, 0x1058, 0x1059, 0x1160, 0x11FF, 0x135F, 0x135F,
-    0x1712, 0x1714, 0x1732, 0x1734, 0x1752, 0x1753, 0x1772, 0x1773,
-    0x17B4, 0x17B5, 0x17B7, 0x17BD, 0x17C6, 0x17C6, 0x17C9, 0x17D3,
-    0x17DD, 0x17DD, 0x180B, 0x180D, 0x18A9, 0x18A9, 0x1920, 0x1922,
-    0x1927, 0x1928, 0x1932, 0x1932, 0x1939, 0x193B, 0x1A17, 0x1A18,
-    0x1B00, 0x1B03, 0x1B34, 0x1B34, 0x1B36, 0x1B3A, 0x1B3C, 0x1B3C,
-    0x1B42, 0x1B42, 0x1B6B, 0x1B73, 0x1DC0, 0x1DCA, 0x1DFE, 0x1DFF,
-    0x200B, 0x200F, 0x202A, 0x202E, 0x2060, 0x2063, 0x206A, 0x206F,
-    0x20D0, 0x20EF, 0x302A, 0x302F, 0x3099, 0x309A, 0xA806, 0xA806,
-    0xA80B, 0xA80B, 0xA825, 0xA826, 0xFB1E, 0xFB1E, 0xFE00, 0xFE0F,
-    0xFE20, 0xFE23, 0xFEFF, 0xFEFF, 0xFFF9, 0xFFFB, 0x10A01, 0x10A03,
-    0x10A05, 0x10A06, 0x10A0C, 0x10A0F, 0x10A38, 0x10A3A, 0x10A3F, 0x10A3F,
-    0x1D167, 0x1D169, 0x1D173, 0x1D182, 0x1D185, 0x1D18B, 0x1D1AA, 0x1D1AD,
-    0x1D242, 0x1D244, 0xE0001, 0xE0001, 0xE0020, 0xE007F, 0xE0100, 0xE01EF,
-    -1,
-  };
-
-  if (in_range(range1, c))
-    return 0;
-
-  static uint32_t range2[] = {
-    0x1100, 0x115F, 0x2329, 0x2329, 0x232A, 0x232A, 0x2E80, 0x303E,
-    0x3040, 0xA4CF, 0xAC00, 0xD7A3, 0xF900, 0xFAFF, 0xFE10, 0xFE19,
-    0xFE30, 0xFE6F, 0xFF00, 0xFF60, 0xFFE0, 0xFFE6, 0x1F000, 0x1F644,
-    0x20000, 0x2FFFD, 0x30000, 0x3FFFD, -1,
-  };
-
-  if (in_range(range2, c))
-    return 2;
-  return 1;
-}
-
-// Returns the number of columns needed to display a given
-// string in a fixed-width font.
-int display_width(char *p, int len) {
-  char *start = p;
-  int w = 0;
-  while (p - start < len) {
-    uint32_t c = decode_utf8(&p, p);
-    w += char_width(c);
-  }
-  return w;
-}
diff --git a/src/3p/openbsd/asprintf.c b/src/3p/openbsd/asprintf.c
deleted file mode 100644
index 195efce..0000000
--- a/src/3p/openbsd/asprintf.c
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * Copyright (c) 2004 Darren Tucker.
- *
- * Based originally on asprintf.c from OpenBSD:
- * Copyright (c) 1997 Todd C. Miller <Todd.Miller@courtesan.com>
- *
- * Permission to use, copy, modify, and distribute this software for any
- * purpose with or without fee is hereby granted, provided that the above
- * copyright notice and this permission notice appear in all copies.
- *
- * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
- * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
- * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
- * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
- * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
- * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
- * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
- */
-
-#include <errno.h>
-#include <limits.h> /* for INT_MAX */
-#include <stdarg.h>
-#include <stdio.h> /* for vsnprintf */
-#include <stdlib.h>
-
-#define INIT_SZ	128
-
-int
-vasprintf(char **str, const char *fmt, va_list ap)
-{
-	int ret;
-	va_list ap2;
-	char *string, *newstr;
-	size_t len;
-
-	if ((string = malloc(INIT_SZ)) == NULL)
-		goto fail;
-
-	va_copy(ap2, ap);
-	ret = vsnprintf(string, INIT_SZ, fmt, ap2);
-	va_end(ap2);
-	if (ret >= 0 && ret < INIT_SZ) { /* succeeded with initial alloc */
-		*str = string;
-	} else if (ret == INT_MAX || ret < 0) { /* Bad length */
-		free(string);
-		goto fail;
-	} else {	/* bigger than initial, realloc allowing for nul */
-		len = (size_t)ret + 1;
-		if ((newstr = realloc(string, len)) == NULL) {
-			free(string);
-			goto fail;
-		}
-		va_copy(ap2, ap);
-		ret = vsnprintf(newstr, len, fmt, ap2);
-		va_end(ap2);
-		if (ret < 0 || (size_t)ret >= len) { /* failed with realloc'ed string */
-			free(newstr);
-			goto fail;
-		}
-		*str = newstr;
-	}
-	return (ret);
-
-fail:
-	*str = NULL;
-	errno = ENOMEM;
-	return (-1);
-}
-
-int asprintf(char **str, const char *fmt, ...)
-{
-	va_list ap;
-	int ret;
-	
-	*str = NULL;
-	va_start(ap, fmt);
-	ret = vasprintf(str, fmt, ap);
-	va_end(ap);
-
-	return ret;
-}
diff --git a/src/build/cmeta.c b/src/build/cmeta.c
index 434be76..88ac327 100644
--- a/src/build/cmeta.c
+++ b/src/build/cmeta.c
@@ -17,56 +17,12 @@
 #include <stdio.h>
 #include <stdlib.h>
 
+#include "../chunklets/clex.h"
 #include "../intdefs.h"
 #include "../langext.h"
 #include "../os.h"
 #include "cmeta.h"
 
-// lazy inlined 3rd party stuff {{{
-// too lazy to write a C tokenizer at the moment, or indeed probably ever, so
-// let's just yoink some code from a hacked-up copy of chibicc, a nice minimal C
-// compiler with code that's pretty easy to work with. it does leak memory by
-// design, but build stuff is all one-shot so that's fine.
-#include "../3p/chibicc/chibicc.h"
-#include "../3p/chibicc/unicode.c"
-// type sentinels from type.c (don't bring in the rest of type.c because it
-// circularly depends on other stuff and we really only want tokenize here)
-Type *ty_void = &(Type){TY_VOID, 1, 1};
-Type *ty_bool = &(Type){TY_BOOL, 1, 1};
-Type *ty_char = &(Type){TY_CHAR, 1, 1};
-Type *ty_short = &(Type){TY_SHORT, 2, 2};
-Type *ty_int = &(Type){TY_INT, 4, 4};
-Type *ty_long = &(Type){TY_LONG, 8, 8};
-Type *ty_uchar = &(Type){TY_CHAR, 1, 1, true};
-Type *ty_ushort = &(Type){TY_SHORT, 2, 2, true};
-Type *ty_uint = &(Type){TY_INT, 4, 4, true};
-Type *ty_ulong = &(Type){TY_LONG, 8, 8, true};
-Type *ty_float = &(Type){TY_FLOAT, 4, 4};
-Type *ty_double = &(Type){TY_DOUBLE, 8, 8};
-Type *ty_ldouble = &(Type){TY_LDOUBLE, 16, 16};
-// inline just a couple more things, super lazy, but whatever
-static Type *new_type(TypeKind kind, int size, int align) {
-	Type *ty = calloc(1, sizeof(Type));
-	ty->kind = kind;
-	ty->size = size;
-	ty->align = align;
-	return ty;
-}
-Type *array_of(Type *base, int len) {
-	Type *ty = new_type(TY_ARRAY, base->size * len, base->align);
-	ty->base = base;
-	ty->array_len = len;
-	return ty;
-}
-#include "../3p/chibicc/hashmap.c"
-#include "../3p/chibicc/strings.c"
-#include "../3p/chibicc/tokenize.c"
-// }}}
-
-#ifdef _WIN32
-#include "../3p/openbsd/asprintf.c" // missing from libc; plonked here for now
-#endif
-
 static cold noreturn die(int status, const char *s) {
 	fprintf(stderr, "cmeta: fatal: %s\n", s);
 	exit(status);
@@ -76,188 +32,314 @@ struct cmeta cmeta_loadfile(const os_char *path) {
 	int f = os_open_read(path);
 	if_cold (f == -1) die(100, "couldn't open file");
 	vlong len = os_fsize(f);
-	if_cold (len > 1u << 30 - 1) die(2, "input file is far too large");
+	if_cold (len == 0) die(2, "empty source file");
+	// limit this a little lower so that clex_memreq doesn't overflow if (for
+	// some reason!?) the host compiler target is 32-bit. no point worrying as
+	// we should never have a 256MiB source file anyway!
+	if_cold (len > 1u << 28 - 1) die(2, "input file is far too large");
 	struct cmeta ret;
-	ret.sbase = malloc(len + 1);
-	ret.sbase[len] = '\0'; // chibicc needs a null terminator
-	if_cold (!ret.sbase) die(100, "couldn't allocate memory");
-	if_cold (os_read(f, ret.sbase, len) != len) die(100, "couldn't read file");
-	int maxitems = len / 4; // shortest word is "END"
+	usize lexmemreq = clex_memreq(len, os_strlen(path));
+	// smallest possible item is END{} (5 chars), but it's nice to be able to go
+	// >> 2, so pretend it's 4 chars. for each item we store 1 32-bit ints and
+	// an 8-bit int, so the memory requirement ends up being clex_memreq() +
+	// len + len >> 2. in total, including the file buffer, that should be 7.25x
+	// the file size. which is pretty reasonable for normal files.
+	usize memreq = lexmemreq + (len << 1) + (len >> 2);
+	void *mem = malloc(memreq);
 	ret.nitems = 0;
-	// eventual overall memory requirement: file size * 6. seems fine to me.
-	// current memory requirement: file size * 10, + all the chibicc linked list
-	// crap. not as good but we'll continue tolerating it... probably for years!
-	//ret.itemoffs = malloc(maxitems * sizeof(*ret.itemoffs));
-	//if (!ret.itemoffs) die(100, "couldn't allocate memory");
-	ret.itemtoks = malloc(maxitems * sizeof(*ret.itemtoks));
-	if_cold (!ret.itemtoks) die(100, "couldn't allocate memory");
-	ret.itemtypes = malloc(maxitems * sizeof(*ret.itemtypes));
-	if_cold (!ret.itemtypes) die(100, "couldn't allocate memory");
+	ret.itemtoks = mem;
+	// put the string and item bytes at the end so the cmeta stuff is aligned.
+	ret.sbase = (char *)mem + memreq - len;
+	ret.items = (struct cmeta_item *)mem + memreq - (len << 1);
+	if_cold (!mem) die(100, "couldn't allocate memory");
+	if_cold (os_read(f, ret.sbase, len) != len) die(100, "couldn't read file");
 	os_close(f);
 #ifdef _WIN32
-	char *realname = malloc(wcslen(path) + 1);
-	if_cold (!realname) die(100, "couldn't allocate memory");
+	char *asciiname = malloc(wcslen(path) + 1);
+	if_cold (!asciiname) die(100, "couldn't allocate memory");
 	// XXX: being lazy about Unicode right now; a general purpose tool should
 	// implement WTF8 or something. SST itself doesn't have any unicode paths
 	// though, so we don't really care as much. this code still sucks though.
-	*realname = *path;
-	for (const ushort *p = path + 1; p[-1]; ++p) realname[p - path] = *p;
+	*asciiname = *path;
+	for (const ushort *p = path + 1; p[-1]; ++p) asciiname[p - path] = *p;
 #else
-	const char *realname = f;
+	const char *asciiname = f;
 #endif
-	struct Token *t = tokenize_buf(realname, ret.sbase);
-	// everything is THING() or THING {} so we need at least 3 tokens ahead - if
-	// we have fewer tokens left in the file we can bail
-	if (t && t->next) while (t->next->next) {
-		if (!t->at_bol) {
-			t = t->next;
-			continue;
+	ret.lexer = clex(ret.sbase, len, (u32 *)mem + (len >> 2), asciiname);
+	if (ret.lexer.err) die(2, ret.lexer.err);
+	// everything is THING() or THING {}, and file also ends in an EOL, so once
+	// there's less than 4 tokens left in the file, we can bail.
+	for (u32 i = 0, end = ret.lexer.ntoks - 4; i < end; ++i) {
+		if_hot (ret.lexer.toks[i] != CLEX_TOK_IDENT) continue;
+		// technically we don't have to validate *every* token, but doing so
+		// gives less confusing syntax errors.
+		struct clex_ident_validate_ret val = clex_ident_validate(
+				&ret.lexer, ret.sbase, i);
+		if (val.err) {
+			char buf[CLEX_IDENT_ERRSTR_MEMREQ(PATH_MAX)];
+			clex_ident_errstr(buf, ret.sbase, val.err, val.err_off, asciiname);
+			die(2, buf);
 		}
+		// everything we match has to be followed by either ( or {
+		if (ret.lexer.toks[i + 1] != CLEX_TOK_OP1) { ++i; continue; }
+		if (val.len > 24) continue; // longer than the longest string below
+		char name[24];
+		clex_ident(&ret.lexer, ret.sbase, i, name);
 		int type;
-		if ((equal(t, "DEF_CVAR") || equal(t, "DEF_CVAR_MIN") ||
-				equal(t, "DEF_CVAR_MAX") || equal(t, "DEF_CVAR_MINMAX") ||
-				equal(t, "DEF_CVAR_UNREG") || equal(t, "DEF_CVAR_MIN_UNREG") ||
-				equal(t, "DEF_CVAR_MAX_UNREG") ||
-				equal(t, "DEF_CVAR_MINMAX_UNREG") ||
-				equal(t, "DEF_FEAT_CVAR") || equal(t, "DEF_FEAT_CVAR_MIN") ||
-				equal(t, "DEF_FEAT_CVAR_MAX") ||
-				equal(t, "DEF_FEAT_CVAR_MINMAX")) && equal(t->next, "(")) {
-			type = CMETA_ITEM_DEF_CVAR;
-		}
-		else if ((equal(t, "DEF_CCMD") || equal(t, "DEF_CCMD_HERE") ||
-				equal(t, "DEF_CCMD_UNREG") || equal(t, "DEF_CCMD_HERE_UNREG") ||
-				equal(t, "DEF_CCMD_PLUSMINUS") ||
-				equal(t, "DEF_CCMD_PLUSMINUS_UNREG") ||
-				equal(t, "DEF_FEAT_CCMD") || equal(t, "DEF_FEAT_CCMD_HERE") ||
-				equal(t, "DEF_FEAT_CCMD_PLUSMINUS")) && equal(t->next, "(")) {
-			type = CMETA_ITEM_DEF_CCMD;
-		}
-		else if ((equal(t, "DEF_EVENT") || equal(t, "DEF_PREDICATE")) &&
-				equal(t->next, "(")) {
-			type = CMETA_ITEM_DEF_EVENT;
-		}
-		else if (equal(t, "HANDLE_EVENT") && equal(t->next, "(")) {
-			type = CMETA_ITEM_HANDLE_EVENT;
-		}
-		else if (equal(t, "FEATURE") && equal(t->next, "(")) {
-			type = CMETA_ITEM_FEATURE;
-		}
-		else if ((equal(t, "REQUIRE") || equal(t, "REQUIRE_GAMEDATA") ||
-				equal(t, "REQUIRE_GLOBAL") || equal(t, "REQUEST")) &&
-				equal(t->next, "(")) {
-			type = CMETA_ITEM_REQUIRE;
-		}
-		else if (equal(t, "GAMESPECIFIC") && equal(t->next, "(")) {
-			type = CMETA_ITEM_GAMESPECIFIC;
-		}
-		else if (equal(t, "PREINIT") && equal(t->next, "{")) {
-			type = CMETA_ITEM_PREINIT;
+		int flags = 0;
+		char nextop = '(';
+		// this is kind of dumb code. oh well, good enough probably.
+		switch (val.len) {
+			case 3:
+				if (!memcmp(name, "END", 3)) {
+					type = CMETA_ITEM_END;
+					nextop = '{';
+					break;
+				}
+				continue;
+			case 4:
+				if (!memcmp(name, "INIT", 4)) {
+					type = CMETA_ITEM_INIT;
+					nextop = '{';
+					break;
+				}
+				continue;
+			case 7:
+				if (!memcmp(name, "FEATURE", 7)) {
+					type = CMETA_ITEM_FEATURE;
+					break;
+				}
+				if (!memcmp(name, "PREINIT", 7)) {
+					type = CMETA_ITEM_PREINIT;
+					nextop = '{';
+					break;
+				}
+				if (!memcmp(name, "REQUIRE", 7)) {
+					type = CMETA_ITEM_REQUIRE;
+					break;
+				}
+				if (!memcmp(name, "REQUEST", 7)) {
+					type = CMETA_ITEM_REQUIRE;
+					flags = CMETA_REQUIRE_OPTIONAL;
+					break;
+				}
+				continue;
+			case 8:
+				if (!memcmp(name, "DEF_CCMD", 8)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					break;
+				}
+				if (!memcmp(name, "DEF_CVAR", 8)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					break;
+				}
+				continue;
+			case 9:
+				if (!memcmp(name, "DEF_EVENT", 9)) {
+					type = CMETA_ITEM_DEF_EVENT;
+					break;
+				}
+				continue;
+			case 12:
+				if (!memcmp(name, "DEF_CVAR_MAX", 12) ||
+						!memcmp(name, "DEF_CVAR_MIN", 12)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					break;
+				}
+				if (!memcmp(name, "GAMESPECIFIC", 12)) {
+					type = CMETA_ITEM_GAMESPECIFIC;
+					break;
+				}
+				if (!memcmp(name, "HANDLE_EVENT", 12)) {
+					type = CMETA_ITEM_HANDLE_EVENT;
+					break;
+				}
+				continue;
+			case 13:
+				if (!memcmp(name, "DEF_CCMD_HERE", 13)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					break;
+				}
+				if (!memcmp(name, "DEF_FEAT_CCMD", 13)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					flags = CMETA_CVAR_FEAT;
+					break;
+				}
+				if (!memcmp(name, "DEF_FEAT_CVAR", 13)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					flags = CMETA_CVAR_FEAT;
+					break;
+				}
+				if (!memcmp(name, "DEF_PREDICATE", 13)) {
+					type = CMETA_ITEM_DEF_EVENT;
+					flags = CMETA_EVENT_ISPREDICATE;
+					break;
+				}
+				continue;
+			case 14:
+				if (!memcmp(name, "DEF_CCMD_UNREG", 14)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					flags = CMETA_CCMD_UNREG;
+					break;
+				}
+				if (!memcmp(name, "DEF_CVAR_UNREG", 14)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					flags = CMETA_CVAR_UNREG;
+					break;
+				}
+				if (!memcmp(name, "REQUIRE_GLOBAL", 14)) {
+					type = CMETA_ITEM_REQUIRE;
+					flags = CMETA_REQUIRE_GLOBAL;
+					break;
+				}
+				continue;
+			case 15:
+				if (!memcmp(name, "DEF_CVAR_MINMAX", 15)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					break;
+				}
+				continue;
+			case 16:
+				if (!memcmp(name, "REQUIRE_GAMEDATA", 16)) {
+					type = CMETA_ITEM_REQUIRE;
+					flags = CMETA_REQUIRE_GAMEDATA;
+					break;
+				}
+				continue;
+			case 17:
+				if (!memcmp(name, "DEF_FEAT_CVAR_MAX", 17) ||
+						!memcmp(name, "DEF_FEAT_CVAR_MIN", 17)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					flags = CMETA_CVAR_FEAT;
+					break;
+				}
+				continue;
+			case 18:
+				if (!memcmp(name, "DEF_CCMD_PLUSMINUS", 18)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					flags = CMETA_CCMD_PLUSMINUS;
+					break;
+				}
+				if (!memcmp(name, "DEF_CVAR_MAX_UNREG", 18) ||
+						!memcmp(name, "DEF_CVAR_MIN_UNREG", 18)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					flags = CMETA_CVAR_UNREG;
+					break;
+				}
+				if (!memcmp(name, "DEF_FEAT_CCMD_HERE", 18)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					flags = CMETA_CCMD_FEAT;
+					break;
+				}
+				continue;
+			case 19:
+				if (!memcmp(name, "DEF_CCMD_HERE_UNREG", 19)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					flags = CMETA_CCMD_UNREG;
+					break;
+				}
+				continue;
+			case 20:
+				if (!memcmp(name, "DEF_FEAT_CVAR_MINMAX", 20)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					flags = CMETA_CVAR_FEAT;
+					break;
+				}
+				continue;
+			case 21:
+				if (!memcmp(name, "DEF_CVAR_MINMAX_UNREG", 21)) {
+					type = CMETA_ITEM_DEF_CVAR;
+					flags = CMETA_CVAR_UNREG;
+					break;
+				}
+				continue;
+			case 23:
+				if (!memcmp(name, "DEF_FEAT_CCMD_PLUSMINUS", 23)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					flags = CMETA_CCMD_FEAT | CMETA_CCMD_PLUSMINUS;
+					break;
+				}
+				continue;
+			case 24:
+				if (!memcmp(name, "DEF_CCMD_PLUSMINUS_UNREG", 24)) {
+					type = CMETA_ITEM_DEF_CCMD;
+					flags = CMETA_CCMD_UNREG | CMETA_CCMD_PLUSMINUS;
+					break;
+				}
+			default:
+				continue;
 		}
-		else if (equal(t, "INIT") && equal(t->next, "{")) {
-			type = CMETA_ITEM_INIT;
-		}
-		else if (equal(t, "END") && equal(t->next, "{")) {
-			type = CMETA_ITEM_END;
-		}
-		else {
-			t = t->next;
+		if (ret.sbase[ret.lexer.tokoffs[i + 1]] != nextop) {
+			// bump i a little further as we've already looked at at least 2
+			// tokens. this is technically kind of inefficient; in most cases we
+			// can skip more stuff, but we're always scanning for something
+			// specific, so who cares actually, this is good enough.
+			++i;
 			continue;
 		}
-		ret.itemtoks[ret.nitems] = t;
-		ret.itemtypes[ret.nitems] = type;
+		ret.itemtoks[ret.nitems] = i;
+		ret.items[ret.nitems] = (struct cmeta_item){type, flags};
 		++ret.nitems;
-		// this is kind of inefficient; in most cases we can skip more stuff,
-		// but then also, we're always scanning for something specific, so who
-		// cares actually, this will do for now.
-		t = t->next->next;
+		++i;
 	}
 	return ret;
 }
 
-int cmeta_flags_cvar(const struct cmeta *cm, u32 i) {
-	struct Token *t = cm->itemtoks[i];
-	switch_exhaust (t->len) {
-		// It JUST so happens all of the possible tokens here have a unique
-		// length. I swear this wasn't planned. But it IS convenient!
-		case 8: case 12: case 15: return 0;
-		case 14: case 18: case 21: return CMETA_CVAR_UNREG;
-		case 13: case 17: case 20: return CMETA_CVAR_FEAT;
-	}
-}
-
-int cmeta_flags_ccmd(const struct cmeta *cm, u32 i) {
-	struct Token *t = cm->itemtoks[i];
-	switch_exhaust (t->len) {
-		case 13: if (t->loc[4] == 'F') return CMETA_CCMD_FEAT;
-		case 8: return 0;
-		case 18: if (t->loc[4] == 'F') return CMETA_CCMD_FEAT;
-			return CMETA_CCMD_PLUSMINUS;
-		case 14: case 19: return CMETA_CCMD_UNREG;
-		case 23: return CMETA_CCMD_FEAT | CMETA_CCMD_PLUSMINUS;
-		case 24: return CMETA_CCMD_UNREG | CMETA_CCMD_PLUSMINUS;
-	}
-}
-
-int cmeta_flags_event(const struct cmeta *cm, u32 i) {
-	// assuming CMETA_EVENT_ISPREDICATE remains 1, the ternary should
-	// optimise out
-	return cm->itemtoks[i]->len == 13 ? CMETA_EVENT_ISPREDICATE : 0;
-}
-
-int cmeta_flags_require(const struct cmeta *cm, u32 i) {
-	struct Token *t = cm->itemtoks[i];
-	// NOTE: this is somewhat more flexible to enable REQUEST_GAMEDATA or
-	// something in future, although that's kind of useless currently
-	int optflag = t->loc[4] == 'E'; // REQU[E]ST
-	switch_exhaust (t->len) {
-		case 7: return optflag;
-		case 16: return optflag | CMETA_REQUIRE_GAMEDATA;
-		case 14: return optflag | CMETA_REQUIRE_GLOBAL;
-	};
-}
-
-int cmeta_nparams(const struct cmeta *cm, u32 i) {
+int cmeta_nparams(const struct cmeta *cm, u32 item) {
 	int argc = 1, nest = 0;
-	struct Token *t = cm->itemtoks[i]->next->next;
-	if (equal(t, ")")) return 0; // XXX: stupid special case, surely improvable?
-	for (; t; t = t->next) {
-		if (equal(t, "(")) { ++nest; continue; }
-		if (!nest && equal(t, ",")) ++argc;
-		else if (equal(t, ")") && !nest--) break;
+	int i = cm->itemtoks[item] + 2; // get past the first (
+	// handle immediate ) - XXX: stupid special case, surely improvable?
+	if (clex_isrparen(&cm->lexer, cm->sbase, i)) return 0;
+	for (; i < cm->lexer.ntoks; ++i) {
+		if (clex_islparen(&cm->lexer, cm->sbase, i)) { ++nest; continue; }
+		if (!nest && clex_iscomma(&cm->lexer, cm->sbase, i)) ++argc;
+		else if (clex_isrparen(&cm->lexer, cm->sbase, i) && !nest--) break;
 	}
 	if (nest != -1) return 0; // XXX: any need to do anything better here?
 	return argc;
 }
 
 struct cmeta_param_iter cmeta_param_iter_init(const struct cmeta *cm, u32 i) {
-	return (struct cmeta_param_iter){cm->itemtoks[i]->next->next};
+	return (struct cmeta_param_iter){cm->itemtoks[i] + 2};
 }
 
-struct cmeta_slice cmeta_param_iter(struct cmeta_param_iter *it) {
+struct cmeta_slice cmeta_param_iter(const struct cmeta *cm,
+		struct cmeta_param_iter *it) {
 	int nest = 0;
-	const char *start = it->cur->loc;
-	for (struct Token *last = 0; it->cur;
-			last = it->cur, it->cur = it->cur->next) {
-		if (equal(it->cur, "(")) { ++nest; continue; }
-		if (!nest && equal(it->cur, ",")) {
-			if (!last) { // , immediately after (, for some reason. treat as ""
-				return (struct cmeta_slice){start, 0};
-			}
-			it->cur = it->cur->next;
+	const char *start = cm->sbase + cm->lexer.tokoffs[it->i];
+	for (; it->i < cm->lexer.ntoks; ++it->i) {
+		if (clex_islparen(&cm->lexer, cm->sbase, it->i)) {
+			++nest;
+			continue;
 		}
-		else if (equal(it->cur, ")") && !nest--) {
-			if (!last) break;
+		if (!nest && clex_iscomma(&cm->lexer, cm->sbase, it->i)) {
+			const char *end = cm->sbase + cm->lexer.tokoffs[it->i];
+			// XXX: to avoid picking up random whitespace we should get the
+			// previous token and ask clex for its extent, but I've not yet
+			// implemented full parsing of all variable-length tokens (i.e.
+			// numbers and string/char literals). not worrying about it too much
+			// at the moment since it's not really a problem anyway in practice
+			++it->i; // skip past comma for next time
+			return (struct cmeta_slice){start, end - start};
 		}
-		else {
-			continue;
+		else if (clex_isrparen(&cm->lexer, cm->sbase, it->i) && !nest--) {
+			// annoying case: 0 args is different from ", )" (empty arg)
+			if (clex_islparen(&cm->lexer, cm->sbase, it->i - 1)) break;
+			const char *end = cm->sbase + cm->lexer.tokoffs[it->i];
+			it->i = -1u; // force next call to return {0, 0}
+			return (struct cmeta_slice){start, end - start};
 		}
-		return (struct cmeta_slice){start, last->loc - start + last->len};
 	}
 	return (struct cmeta_slice){0, 0};
 }
 
-u32 cmeta_line(const struct cmeta *cm, u32 i) {
-	return cm->itemtoks[i]->line_no;
+cold u32 cmeta_line(const struct cmeta *cm, u32 i) {
+	u32 line = 1;
+	for (u32 off = 0, end = cm->lexer.tokoffs[cm->itemtoks[i]];
+			off != end; ++off) {
+		line += cm->sbase[off] == '\n';
+	}
+	return line;
 }
 
 // vi: sw=4 ts=4 noet tw=80 cc=80 fdm=marker
diff --git a/src/build/cmeta.h b/src/build/cmeta.h
index e7c275d..c18adbb 100644
--- a/src/build/cmeta.h
+++ b/src/build/cmeta.h
@@ -17,33 +17,23 @@
 #ifndef INC_CMETA_H
 #define INC_CMETA_H
 
+#include "../chunklets/clex.h" /* XXX: only need this for struct, kinda leaky */
 #include "../intdefs.h"
 #include "../os.h"
 
-// XXX: leaking chibicc internals. won't matter after we do away with that
-typedef struct Token Token;
-
-enum cmeta_item {
-	CMETA_ITEM_DEF_CVAR, // includes all min/max/unreg variants
-	CMETA_ITEM_DEF_CCMD, // includes plusminus/unreg variants
-	CMETA_ITEM_DEF_EVENT, // includes predicates
+enum cmeta_itemtype {
+	CMETA_ITEM_DEF_CVAR, /* includes all min/max/unreg variants */
+	CMETA_ITEM_DEF_CCMD, /* includes plusminus/unreg variants */
+	CMETA_ITEM_DEF_EVENT, /* includes predicates */
 	CMETA_ITEM_HANDLE_EVENT,
 	CMETA_ITEM_FEATURE,
-	CMETA_ITEM_REQUIRE, // includes all REQUIRE_*/REQUEST variants
+	CMETA_ITEM_REQUIRE, /* includes all REQUIRE_* and REQUEST variants */
 	CMETA_ITEM_GAMESPECIFIC,
 	CMETA_ITEM_PREINIT,
 	CMETA_ITEM_INIT,
 	CMETA_ITEM_END
 };
 
-struct cmeta {
-	char *sbase;
-	u32 nitems; // number of interesting macros
-	//u32 *itemoffs; // file offsets of interesting macros (ONE DAY!)
-	Token **itemtoks; // crappy linked token structures, for the time being
-	u8 *itemtypes; // CMETA_ITEM_* enum values
-};
-
 enum cmeta_flag_cvar {
 	CMETA_CVAR_UNREG = 1,
 	CMETA_CVAR_FEAT = 2,
@@ -57,28 +47,37 @@ enum cmeta_flag_event {
 	CMETA_EVENT_ISPREDICATE = 1
 };
 enum cmeta_flag_require {
-	CMETA_REQUIRE_OPTIONAL = 1, // i.e. REQUEST() macro, could be extended
+	CMETA_REQUIRE_OPTIONAL = 1, /* i.e. REQUEST() macro, could be extended */
 	CMETA_REQUIRE_GAMEDATA = 2,
 	CMETA_REQUIRE_GLOBAL = 4
 };
 
+struct cmeta {
+	char *sbase;
+	u32 nitems; /* number of interesting macros */
+	//char pad[4];
+	struct cmeta_item {
+		u8 type : 4; /* CMETA_ITEM_* enum value */
+		u8 flags : 4; /* relevant flags (see above) */
+	} *items;
+	u32 *itemtoks; /* internal */
+	struct clex lexer; /* internal */
+};
+
 struct cmeta_slice { const char *s; int len; };
 
 struct cmeta cmeta_loadfile(const os_char *path);
-int cmeta_flags_cvar(const struct cmeta *cm, u32 i);
-int cmeta_flags_ccmd(const struct cmeta *cm, u32 i);
-int cmeta_flags_event(const struct cmeta *cm, u32 i);
-int cmeta_flags_require(const struct cmeta *cm, u32 i);
 
 int cmeta_nparams(const struct cmeta *cm, u32 i);
-struct cmeta_param_iter { Token *cur; };
+struct cmeta_param_iter { u32 i; };
 struct cmeta_param_iter cmeta_param_iter_init(const struct cmeta *cm, u32 i);
-struct cmeta_slice cmeta_param_iter(struct cmeta_param_iter *it);
+struct cmeta_slice cmeta_param_iter(const struct cmeta *cm,
+		struct cmeta_param_iter *it);
 
-#define cmeta_param_foreach(varname, cm, u32) \
+#define cmeta_param_foreach(varname, cm, i) \
 	switch (0) for (struct cmeta_slice varname; 0;) default: \
 		for (struct cmeta_param_iter _it = cmeta_param_iter_init(cm, i); \
-				varname = cmeta_param_iter(&_it), varname.s;) \
+				varname = cmeta_param_iter(cm, &_it), varname.s;) \
 			/* {...} */
 
 u32 cmeta_line(const struct cmeta *cm, u32 i);
diff --git a/src/build/gluegen.c b/src/build/gluegen.c
index a055241..33ba600 100644
--- a/src/build/gluegen.c
+++ b/src/build/gluegen.c
@@ -324,21 +324,23 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 	struct list_chunkhdr *tail_gamedata = &mod_gamedata[mod].hdr;
 	struct list_chunkhdr *tail_globals = &mod_globals[mod].hdr;
 	for (u32 i = 0; i != cm->nitems; ++i) {
-		switch_exhaust_enum(cmeta_item, cm->itemtypes[i]) {
+		struct cmeta_item item = cm->items[i];
+		switch_exhaust_enum(cmeta_itemtype, item.type) {
 			case CMETA_ITEM_DEF_CVAR:
 				if_cold (!cmeta_nparams(cm, i)) {
 					diefile(2, file, cmeta_line(cm, i),
 							"cvar macro missing required parameter");
 				}
-				int flags = cmeta_flags_cvar(cm, i);
 				s16 idx = cvar_new();
-				cvar_flags[idx] = flags;
+				cvar_flags[idx] = item.flags;
 				// NOTE: always hooking cvar/ccmd up to a feat entry for
 				// GAMESPECIFIC checks, even if it's not a DEF_FEAT_*.
 				// If a module doesn't declare FEATURE, its cvars/ccmds will
 				// end up getting initialised/registered unconditionally.
 				cvar_feats[idx] = mod;
-				if (flags & CMETA_CVAR_FEAT) needfeat = "feature cvar defined";
+				if (item.flags & CMETA_CVAR_FEAT) {
+					needfeat = "feature cvar defined";
+				}
 				cmeta_param_foreach (name, cm, i) {
 					cvar_names[idx] = name;
 					break; // ignore subsequent args
@@ -349,15 +351,14 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 					diefile(2, file, cmeta_line(cm, i),
 							"ccmd macro missing required parameter");
 				}
-				flags = cmeta_flags_ccmd(cm, i);
-				if (flags & CMETA_CCMD_FEAT) {
+				if (item.flags & CMETA_CCMD_FEAT) {
 					needfeat = "feature ccmd defined";
 				}
-				if (flags & CMETA_CCMD_PLUSMINUS) {
+				if (item.flags & CMETA_CCMD_PLUSMINUS) {
 					// split PLUSMINUS entries in two; makes stuff easier later.
-					flags &= ~CMETA_CCMD_PLUSMINUS;
+					item.flags &= ~CMETA_CCMD_PLUSMINUS;
 					idx = ccmd_new();
-					ccmd_flags[idx] = flags;
+					ccmd_flags[idx] = item.flags;
 					ccmd_feats[idx] = mod;
 					cmeta_param_foreach (name, cm, i) {
 						char *s = arena + arena_new(5 + name.len);
@@ -367,7 +368,7 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 						break;
 					}
 					idx = ccmd_new();
-					ccmd_flags[idx] = flags;
+					ccmd_flags[idx] = item.flags;
 					ccmd_feats[idx] = mod;
 					cmeta_param_foreach (name, cm, i) {
 						char *s = arena + arena_new(6 + name.len);
@@ -379,7 +380,7 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 				}
 				else {
 					idx = ccmd_new();
-					ccmd_flags[idx] = flags;
+					ccmd_flags[idx] = item.flags;
 					ccmd_feats[idx] = mod;
 					cmeta_param_foreach (name, cm, i) {
 						ccmd_names[idx] = name;
@@ -392,9 +393,8 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 					diefile(2, file, cmeta_line(cm, i),
 							"event macro missing required parameter");
 				}
-				flags = cmeta_flags_event(cm, i);
 				struct cmeta_param_iter it = cmeta_param_iter_init(cm, i);
-				struct cmeta_slice evname = cmeta_param_iter(&it);
+				struct cmeta_slice evname = cmeta_param_iter(cm, &it);
 				struct radix_insert_ret r = radix_insert(events, event_names,
 						evname.s, evname.len, false);
 				int e;
@@ -412,10 +412,11 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 					}
 				}
 				event_owners[e] = mod;
-				event_predicateflags[e] = !!(flags & CMETA_EVENT_ISPREDICATE);
+				event_predicateflags[e] = !!(
+						item.flags & CMETA_EVENT_ISPREDICATE);
 				struct list_chunkhdr *tail = &event_params[e].hdr;
-				for (struct cmeta_slice param; param = cmeta_param_iter(&it),
-						param.s;) {
+				for (struct cmeta_slice param;
+						param = cmeta_param_iter(cm, &it), param.s;) {
 					list_append(&tail, param);
 				}
 				break;
@@ -426,7 +427,7 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 							"event handler macro missing required parameter");
 				}
 				it = cmeta_param_iter_init(cm, i);
-				evname = cmeta_param_iter(&it);
+				evname = cmeta_param_iter(cm, &it);
 				r = radix_insert(events, event_names, evname.s, evname.len,
 						false);
 				if (r.isnew) {
@@ -459,9 +460,8 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 					diefile(2, file, cmeta_line(cm, i),
 							"dependency macro missing required parameter");
 				}
-				flags = cmeta_flags_require(cm, i);
 				struct list_chunkhdr **tailp;
-				switch_exhaust(flags) {
+				switch_exhaust(item.flags) {
 					case 0: tailp = &tail_needs; break;
 					case CMETA_REQUIRE_OPTIONAL: tailp = &tail_wants; break;
 					case CMETA_REQUIRE_GAMEDATA: tailp = &tail_gamedata; break;
@@ -469,7 +469,7 @@ static inline void handle(s16 mod, s16 mods, s16 *featdescs, s16 *events,
 				}
 				cmeta_param_foreach(param, cm, i) {
 					int modflags = 0;
-					switch_exhaust(flags) {
+					switch_exhaust(item.flags) {
 						case CMETA_REQUIRE_OPTIONAL:
 							modflags = HAS_OPTDEPS;
 						case 0:
diff --git a/src/chunklets/clex.c b/src/chunklets/clex.c
new file mode 100644
index 0000000..d20986c
--- /dev/null
+++ b/src/chunklets/clex.c
@@ -0,0 +1,1664 @@
+/*
+ * Copyright © Michael Smith <mikesmiffy128@gmail.com>
+ *
+ * Permission to use, copy, modify, and/or distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+ * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+ * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+ * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+ * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+ * OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+ * PERFORMANCE OF THIS SOFTWARE.
+ */
+
+/* WARNING: this library is incomplete and still subject to change! */
+
+// NYI:
+// - numeric literal parsing
+// - string/char literal parsing
+// - more performance measurements
+//     - if needed: more optimisation (target: >500MB/s give or take)
+// - security/robustness tests (e.g. fuzzing)
+// - UTF-8 aware column numbers (assuming this is what most tools do/expect?)
+
+#ifdef __cplusplus
+#error This file should not be compiled as C++. It relies on numerous C-only \
+features.
+#endif
+
+#if defined(_MSC_VER) && !defined(__clang__)
+#if !defined(_MSVC_TRADITIONAL)
+#error This version of MSVC is too old: upgrade to 2019 or newer and use \
+`-std:c17`, or better yet use Clang.
+#elif _MSVC_TRADITIONAL
+#error This file must be compiled using `-std:c17` when using MSVC.
+#endif
+#endif
+
+#ifndef _WIN32
+_Static_assert(
+	(unsigned char)-1 == 255 &&
+	sizeof(short) == 2 &&
+	sizeof(int) == 4 &&
+	sizeof(long long) == 8 &&
+	sizeof(void *) == 4 || sizeof(void *) == 8 &&
+	sizeof(long) == sizeof(void *),
+	"this code is only designed for relatively sane environments, plus Windows"
+);
+#endif
+
+#ifndef __clang__ // see `#define copy` further down
+#include <string.h>
+#endif
+
+#include "clex.h"
+
+#if defined(__GNUC__) || defined(__clang__)
+#define if_cold(x) if (__builtin_expect(!!(x), 0))
+#else
+#define if_cold(x) if (x)
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define cold __attribute__((cold, noinline))
+#elif defined(_MSC_VER)
+#define cold __declspec(noinline)
+#else
+#define cold
+#endif
+
+// duping this from clex.h because it's undef'd there to pollute a little less
+#if defined(__GNUC__) || defined(__clang__)
+#define unreachable __builtin_unreachable()
+#elif defined(_MSC_VER)
+#define unreachable __assume(0)
+#else
+static inline _Noreturn void _clex_invoke_ub(void) {}
+#define unreachable (_clex_invoke_ub())
+#endif
+
+// note: a couple of these have been moved from seemingly more neat/logical
+// positions in order to make statetoks below more compact. classes and state
+// transitions use designated initialisers so this stuff can be moved around
+// without ruining the state transitions. the visual grouping here shows how
+// statetoks[] will be indexed, using halved state values.
+#define STATES(S, s) \
+	S(BOL) \
+	s(OP1) s(WS) \
+	s(OP2) s(WS_BS) \
+	S(LITPFX) \
+	S(uPFX) \
+	S(WORD) \
+	S(NUM) \
+	S(MIGHTEQ) \
+	S(MINUS) \
+	S(PLUS) \
+	S(AND) \
+	S(OR) \
+	S(HASH) \
+	S(COLON) \
+	S(FS) \
+	S(LT) \
+	S(GT) \
+	S(DOT) \
+	S(SHIFT) \
+	S(LNCOMM) \
+	s(BLKCOMM) s(BLKSTAR) \
+	s(OPEQ) s(BLKSTAR_BS) \
+	s(SHIFTEQ) s(STRLIT_BS_WS) /* < STUPID case, needed for gcc/clang compat! */ \
+	s(STRLIT) s(STRLIT_BS) \
+	s(CHRLIT) s(CHRLIT_BS) \
+	s(ERROR) /* special case for bad syntax */ \
+	s(CHRLIT_BS_WS) /* also stupid case */
+
+#define ENUMSTATE(name) STATE_##name,
+#define ENUMSTATEBS(name) STATE_##name, STATE_##name##_BS,
+#define ENUMTRANS(name) STATE_TRANS_##name = STATE_##name << 1,
+#define ENUMTRANSBS(name) \
+		ENUMTRANS(name) STATE_TRANS_##name##_BS = STATE_##name##_BS << 1,
+enum {
+	STATES(ENUMSTATEBS, ENUMSTATE)
+	NSTATES,
+	STATES(ENUMTRANSBS, ENUMTRANS)
+};
+#undef SHIFTENUMBS
+#undef SHIFTENUM
+#undef ENUMSTATEBS
+#undef ENUMSTATE
+
+#define CLASSES(X) \
+	X(OTHER) /* includes (most) alpha, underscores, anything unicode */ \
+	X(LU) /* L and U (lit prefixes) */ \
+	X(u) /* u prefix (could become u8) */ \
+	X(DIGIT) /* excluding 8 */ \
+	X(8) \
+	/* ^^ those must be before symbols */ \
+	X(OP1) /* 1 char ops/symbols */ \
+	X(MIGHTEQ) /* chars that could be followed by = */ \
+	X(MINUS) \
+	X(PLUS) \
+	X(EQ) \
+	X(LT) \
+	X(GT) \
+	X(AND) \
+	X(OR) \
+	X(DOT) \
+	X(STAR) \
+	X(HASH) \
+	X(COLON) \
+	X(FS) \
+	X(BS) \
+	X(SQ) \
+	X(DQ) \
+	X(SP) \
+	X(NL)
+
+// premultiply class values for transition table.
+#define CLASSBASEVALUE(name) CLASSBASEVAL_##name,
+enum { CLASSES(CLASSBASEVALUE) NCLASSES };
+#undef CLASSBASEVALUE
+#define CLASSREALVALUE(name) CLASS_##name = CLASSBASEVAL_##name * NSTATES,
+enum { CLASSES(CLASSREALVALUE) };
+#undef CLASSREALVALUE
+
+#define CLS(cls, c) [c] = CLASS_##cls,
+#define CLS2(cls, c1, c2) CLS(cls, c1) CLS(cls, c2)
+#define CLS3(cls, c1, ...) CLS(cls, c1) CLS2(cls, __VA_ARGS__)
+#define CLS4(cls, c1, ...) CLS(cls, c1) CLS3(cls, __VA_ARGS__)
+#define CLS5(cls, c1, ...) CLS(cls, c1) CLS4(cls, __VA_ARGS__)
+#define CLS6(cls, c1, ...) CLS(cls, c1) CLS5(cls, __VA_ARGS__)
+#define CLS7(cls, c1, ...) CLS(cls, c1) CLS6(cls, __VA_ARGS__)
+#define CLS8(cls, c1, ...) CLS(cls, c1) CLS7(cls, __VA_ARGS__)
+#define CLS9(cls, c1, ...) CLS(cls, c1) CLS8(cls, __VA_ARGS__)
+#define CLS10(cls, c1, ...) CLS(cls, c1) CLS9(cls, __VA_ARGS__)
+
+static const short classes[256] = {
+	/*CLSX(OTHER, everything, else)*/
+	CLS2(LU, 'L', 'U')
+	CLS(u, 'u')
+	CLS9(DIGIT, '0', '1', '2', '3', '4', '5', '6', '7', '9')
+	CLS(8, '8')
+	CLS10(OP1, '(', ')', '[', ']', '{', '}', ',', '?', ';', '~')
+	CLS3(MIGHTEQ, '!', '^', '%')
+	CLS(PLUS, '+')
+	CLS(MINUS, '-')
+	CLS(EQ, '=')
+	CLS(LT, '<')
+	CLS(GT, '>')
+	CLS(AND, '&')
+	CLS(OR, '|')
+	CLS(DOT, '.')
+	CLS(STAR, '*')
+	CLS(HASH, '#')
+	CLS(COLON, ':')
+	CLS(FS, '/')
+	CLS(BS, '\\')
+	CLS(SQ, '\'')
+	CLS(DQ, '"')
+	CLS3(SP, ' ', '\t', '\r')
+	CLS(NL, '\n')
+};
+
+#define FWD 1 // advance token index (i.e. create new token)
+#define FIXPOS 128 // in \u cases, token starts before the u. fixup file offset
+
+// abstracting transition table definition to allow fiddling with the layout.
+#define TRANS(from, class, to) \
+	[CLASS_##class + STATE_##from] = STATE_TRANS_##to,
+#define ST(s, s1, s2, s3, s4, s5, s6, s7, s8, s9, s10, s11, s12, \
+		s13, s14, s15, s16, s17, s18, s19, s20, s21, s22, s23, s24) \
+	TRANS(s, OTHER, s1) \
+	TRANS(s, LU, s2) \
+	TRANS(s, u, s3) \
+	TRANS(s, DIGIT, s4) \
+	TRANS(s, 8, s5) \
+	TRANS(s, OP1, s6) \
+	TRANS(s, MIGHTEQ, s7) \
+	TRANS(s, PLUS, s8) \
+	TRANS(s, MINUS, s9) \
+	TRANS(s, EQ, s10) \
+	TRANS(s, LT, s11) \
+	TRANS(s, GT, s12) \
+	TRANS(s, AND, s13) \
+	TRANS(s, OR, s14) \
+	TRANS(s, DOT, s15) \
+	TRANS(s, STAR, s16) \
+	TRANS(s, HASH, s17) \
+	TRANS(s, COLON, s18) \
+	TRANS(s, FS, s19) \
+	TRANS(s, BS, s20) \
+	TRANS(s, SQ, s21) \
+	TRANS(s, DQ, s22) \
+	TRANS(s, SP, s23) \
+	TRANS(s, NL, s24)
+
+#define ST_BS(s) \
+	ST(s##_BS, \
+		/* OTHER	=> */ ERROR | FWD, \
+		/* LU		=> */ WORD | FWD | FIXPOS, /* *could* be UCN (unlikely) */ \
+		/* u		=> */ WORD | FWD | FIXPOS, /* same */ \
+		/* DIGIT	=> */ ERROR | FWD, \
+		/* 8		=> */ ERROR | FWD, \
+		/* OP1		=> */ ERROR | FWD, \
+		/* MIGHTEQ	=> */ ERROR | FWD, \
+		/* PLUS		=> */ ERROR | FWD, \
+		/* MINUS	=> */ ERROR | FWD, \
+		/* EQ		=> */ ERROR | FWD, \
+		/* LT		=> */ ERROR | FWD, \
+		/* GT		=> */ ERROR | FWD, \
+		/* AND		=> */ ERROR | FWD, \
+		/* OR		=> */ ERROR | FWD, \
+		/* DOT		=> */ ERROR | FWD, \
+		/* STAR		=> */ ERROR | FWD, \
+		/* HASH		=> */ ERROR | FWD, \
+		/* COLON	=> */ ERROR | FWD, \
+		/* FS		=> */ ERROR | FWD, \
+		/* BS		=> */ ERROR | FWD, \
+		/* SQ		=> */ ERROR | FWD, \
+		/* DQ		=> */ ERROR | FWD, \
+		/* SP		=> */ s##_BS, /* nonstandard crap done by GCC and Clang */ \
+		/* NL		=> */ s \
+	)
+
+#define ST_TERM(s) /* "terminal" state (i.e. not mid-token) */ \
+	ST(s, \
+		/* OTHER	=> */ WORD | FWD, \
+		/* LU		=> */ LITPFX | FWD, \
+		/* u		=> */ uPFX | FWD, \
+		/* DIGIT	=> */ NUM | FWD, \
+		/* 8		=> */ NUM | FWD, \
+		/* OP1		=> */ OP1 | FWD, \
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD, \
+		/* PLUS		=> */ PLUS | FWD, \
+		/* MINUS	=> */ MINUS | FWD, \
+		/* EQ		=> */ MIGHTEQ | FWD, \
+		/* LT		=> */ LT | FWD, \
+		/* GT		=> */ GT | FWD, \
+		/* AND		=> */ AND | FWD, \
+		/* OR		=> */ OR | FWD, \
+		/* DOT		=> */ DOT | FWD, \
+		/* STAR		=> */ MIGHTEQ | FWD, \
+		/* HASH		=> */ HASH | FWD, \
+		/* COLON	=> */ COLON | FWD, \
+		/* FS		=> */ FS | FWD, \
+		/* BS		=> */ WS_BS, \
+		/* SQ		=> */ CHRLIT | FWD, \
+		/* DQ		=> */ STRLIT | FWD, \
+		/* SP		=> */ s, \
+		/* NL		=> */ BOL | FWD \
+	)
+
+// VERY long table {{{
+static const unsigned char trans[NSTATES * NCLASSES] = {
+	ST(BOL, // note: same as ST_TERM but without FWD so we don't repeat NLs
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ BOL_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ BOL,
+		/* NL		=> */ BOL
+	)
+	ST(BOL_BS,
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ WORD | FWD | FIXPOS, // maybe UCN
+		/* u		=> */ WORD | FWD | FIXPOS, // "
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ WORD_BS | FWD | FIXPOS, // maybe UCN split over lines
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ BOL_BS,
+		/* NL		=> */ BOL
+	)
+	ST_TERM(WS)
+	ST(WS_BS,
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ WORD | FWD | FIXPOS, // maybe UCN
+		/* u		=> */ WORD | FWD | FIXPOS, // "
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ WORD_BS | FWD | FIXPOS, // maybe UCN split over liens
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ WS_BS,
+		/* NL		=> */ WS
+	)
+	ST(uPFX,
+		/* OTHER	=> */ WORD,
+		/* LU		=> */ WORD,
+		/* u		=> */ WORD,
+		/* DIGIT	=> */ WORD,
+		/* 8		=> */ LITPFX,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ uPFX_BS,
+		/* SQ		=> */ CHRLIT,
+		/* DQ		=> */ STRLIT,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST(uPFX_BS,
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ WORD, // maybe UCN
+		/* u		=> */ WORD, // "
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ WORD_BS, // maybe UCN split over lines
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ uPFX_BS,
+		/* NL		=> */ uPFX
+	)
+	ST(LITPFX,
+		/* OTHER	=> */ WORD,
+		/* LU		=> */ WORD,
+		/* u		=> */ WORD,
+		/* DIGIT	=> */ WORD,
+		/* 8		=> */ WORD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ LITPFX_BS,
+		/* SQ		=> */ CHRLIT,
+		/* DQ		=> */ STRLIT,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST(LITPFX_BS,
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ WORD, // maybe UCN
+		/* u		=> */ WORD, // "
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ WORD_BS, // maybe UCN split over lines
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ LITPFX_BS,
+		/* NL		=> */ LITPFX
+	)
+	ST(WORD,
+		/* OTHER	=> */ WORD,
+		/* LU		=> */ WORD,
+		/* u		=> */ WORD,
+		/* DIGIT	=> */ WORD,
+		/* 8		=> */ WORD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ WORD_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST(WORD_BS,
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ WORD, // maybe UCN
+		/* u		=> */ WORD, // "
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ WORD_BS, // maybe UCN split over lines
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ WORD_BS,
+		/* NL		=> */ WORD
+	)
+	ST(NUM,
+		/* OTHER	=> */ NUM,
+		/* LU		=> */ NUM,
+		/* u		=> */ NUM,
+		/* DIGIT	=> */ NUM,
+		/* 8		=> */ NUM,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ NUM,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ NUM_BS,
+		/* SQ		=> */ NUM, // C23
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST(NUM_BS,
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ ERROR | FWD, // UCN mid-number makes no sense
+		/* u		=> */ ERROR | FWD,
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ ERROR | FWD,
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ NUM_BS,
+		/* NL		=> */ NUM
+	)
+	ST(MIGHTEQ,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ OPEQ,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ MIGHTEQ_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(MIGHTEQ)
+	ST(MINUS,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ OP2,
+		/* EQ		=> */ OPEQ,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ OP2,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ MINUS_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(MINUS)
+	ST(PLUS,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ DOT | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ OP2,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ OPEQ,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ MIGHTEQ | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ PLUS_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(PLUS)
+	ST(AND,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ OPEQ,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ OP2,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ AND_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(AND)
+	ST(OR,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ OPEQ,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OP2,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ OR_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(OR)
+	ST(HASH,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ OP2,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ HASH_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(HASH)
+	ST(COLON,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ OP2,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ COLON_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(COLON)
+	ST(FS,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ BLKCOMM,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ LNCOMM,
+		/* BS		=> */ FS_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(FS)
+	ST(LT,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ OPEQ,
+		/* LT		=> */ SHIFT,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ LT_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(LT)
+	ST(GT,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ OPEQ,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ SHIFT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ GT_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(GT)
+	ST(DOT,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM,
+		/* 8		=> */ NUM,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ MIGHTEQ | FWD,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ DOT_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(DOT)
+	ST(SHIFT,
+		/* OTHER	=> */ WORD | FWD,
+		/* LU		=> */ LITPFX | FWD,
+		/* u		=> */ uPFX | FWD,
+		/* DIGIT	=> */ NUM | FWD,
+		/* 8		=> */ NUM | FWD,
+		/* OP1		=> */ OP1 | FWD,
+		/* MIGHTEQ	=> */ MIGHTEQ | FWD,
+		/* PLUS		=> */ PLUS | FWD,
+		/* MINUS	=> */ MINUS | FWD,
+		/* EQ		=> */ SHIFTEQ,
+		/* LT		=> */ LT | FWD,
+		/* GT		=> */ GT | FWD,
+		/* AND		=> */ AND | FWD,
+		/* OR		=> */ OR | FWD,
+		/* DOT		=> */ DOT | FWD,
+		/* STAR		=> */ MIGHTEQ | FWD,
+		/* HASH		=> */ HASH | FWD,
+		/* COLON	=> */ COLON | FWD,
+		/* FS		=> */ FS | FWD,
+		/* BS		=> */ SHIFT_BS,
+		/* SQ		=> */ CHRLIT | FWD,
+		/* DQ		=> */ STRLIT | FWD,
+		/* SP		=> */ WS,
+		/* NL		=> */ BOL | FWD
+	)
+	ST_BS(SHIFT)
+	ST(LNCOMM,
+		/* OTHER	=> */ LNCOMM,
+		/* LU		=> */ LNCOMM,
+		/* u		=> */ LNCOMM,
+		/* DIGIT	=> */ LNCOMM,
+		/* 8		=> */ LNCOMM,
+		/* OP1		=> */ LNCOMM,
+		/* MIGHTEQ	=> */ LNCOMM,
+		/* PLUS		=> */ LNCOMM,
+		/* MINUS	=> */ LNCOMM,
+		/* EQ		=> */ LNCOMM,
+		/* LT		=> */ LNCOMM,
+		/* GT		=> */ LNCOMM,
+		/* AND		=> */ LNCOMM,
+		/* OR		=> */ LNCOMM,
+		/* DOT		=> */ LNCOMM,
+		/* STAR		=> */ LNCOMM,
+		/* HASH		=> */ LNCOMM,
+		/* COLON	=> */ LNCOMM,
+		/* FS		=> */ LNCOMM,
+		/* BS		=> */ LNCOMM_BS,
+		/* SQ		=> */ LNCOMM,
+		/* DQ		=> */ LNCOMM,
+		/* SP		=> */ LNCOMM,
+		/* NL		=> */ BOL | FWD
+	)
+	ST(LNCOMM_BS,
+		/* OTHER	=> */ LNCOMM,
+		/* LU		=> */ LNCOMM,
+		/* u		=> */ LNCOMM,
+		/* DIGIT	=> */ LNCOMM,
+		/* 8		=> */ LNCOMM,
+		/* OP1		=> */ LNCOMM,
+		/* MIGHTEQ	=> */ LNCOMM,
+		/* PLUS		=> */ LNCOMM,
+		/* MINUS	=> */ LNCOMM,
+		/* EQ		=> */ LNCOMM,
+		/* LT		=> */ LNCOMM,
+		/* GT		=> */ LNCOMM,
+		/* AND		=> */ LNCOMM,
+		/* OR		=> */ LNCOMM,
+		/* DOT		=> */ LNCOMM,
+		/* STAR		=> */ LNCOMM,
+		/* HASH		=> */ LNCOMM,
+		/* COLON	=> */ LNCOMM,
+		/* FS		=> */ LNCOMM,
+		/* BS		=> */ LNCOMM_BS,
+		/* SQ		=> */ LNCOMM,
+		/* DQ		=> */ LNCOMM,
+		/* SP		=> */ LNCOMM_BS,
+		/* NL		=> */ LNCOMM
+	)
+	ST(BLKCOMM,
+		/* OTHER	=> */ BLKCOMM,
+		/* LU		=> */ BLKCOMM,
+		/* u		=> */ BLKCOMM,
+		/* DIGIT	=> */ BLKCOMM,
+		/* 8		=> */ BLKCOMM,
+		/* OP1		=> */ BLKCOMM,
+		/* MIGHTEQ	=> */ BLKCOMM,
+		/* PLUS		=> */ BLKCOMM,
+		/* MINUS	=> */ BLKCOMM,
+		/* EQ		=> */ BLKCOMM,
+		/* LT		=> */ BLKCOMM,
+		/* GT		=> */ BLKCOMM,
+		/* AND		=> */ BLKCOMM,
+		/* OR		=> */ BLKCOMM,
+		/* DOT		=> */ BLKCOMM,
+		/* STAR		=> */ BLKSTAR,
+		/* HASH		=> */ BLKCOMM,
+		/* COLON	=> */ BLKCOMM,
+		/* FS		=> */ BLKCOMM,
+		/* BS		=> */ BLKCOMM,
+		/* SQ		=> */ BLKCOMM,
+		/* DQ		=> */ BLKCOMM,
+		/* SP		=> */ BLKCOMM,
+		/* NL		=> */ BLKCOMM
+	)
+	ST(BLKSTAR,
+		/* OTHER	=> */ BLKCOMM,
+		/* LU		=> */ BLKCOMM,
+		/* u		=> */ BLKCOMM,
+		/* DIGIT	=> */ BLKCOMM,
+		/* 8		=> */ BLKCOMM,
+		/* OP1		=> */ BLKCOMM,
+		/* MIGHTEQ	=> */ BLKCOMM,
+		/* PLUS		=> */ BLKCOMM,
+		/* MINUS	=> */ BLKCOMM,
+		/* EQ		=> */ BLKCOMM,
+		/* LT		=> */ BLKCOMM,
+		/* GT		=> */ BLKCOMM,
+		/* AND		=> */ BLKCOMM,
+		/* OR		=> */ BLKCOMM,
+		/* DOT		=> */ BLKCOMM,
+		/* STAR		=> */ BLKSTAR,
+		/* HASH		=> */ BLKCOMM,
+		/* COLON	=> */ BLKCOMM,
+		/* FS		=> */ WS,
+		/* BS		=> */ BLKSTAR_BS,
+		/* SQ		=> */ BLKCOMM,
+		/* DQ		=> */ BLKCOMM,
+		/* SP		=> */ BLKCOMM,
+		/* NL		=> */ BLKCOMM
+	)
+	ST(BLKSTAR_BS,
+		/* OTHER	=> */ BLKCOMM,
+		/* LU		=> */ BLKCOMM,
+		/* u		=> */ BLKCOMM,
+		/* DIGIT	=> */ BLKCOMM,
+		/* 8		=> */ BLKCOMM,
+		/* OP1		=> */ BLKCOMM,
+		/* MIGHTEQ	=> */ BLKCOMM,
+		/* PLUS		=> */ BLKCOMM,
+		/* MINUS	=> */ BLKCOMM,
+		/* EQ		=> */ BLKCOMM,
+		/* LT		=> */ BLKCOMM,
+		/* GT		=> */ BLKCOMM,
+		/* AND		=> */ BLKCOMM,
+		/* OR		=> */ BLKCOMM,
+		/* DOT		=> */ BLKCOMM,
+		/* STAR		=> */ BLKSTAR,
+		/* HASH		=> */ BLKCOMM,
+		/* COLON	=> */ BLKCOMM,
+		/* FS		=> */ WS,
+		/* BS		=> */ BLKCOMM,
+		/* SQ		=> */ BLKCOMM,
+		/* DQ		=> */ BLKCOMM,
+		/* SP		=> */ BLKSTAR_BS,
+		/* NL		=> */ BLKSTAR
+	)
+	ST_TERM(OP1)
+	ST_TERM(OP2)
+	ST_TERM(OPEQ)
+	ST_TERM(SHIFTEQ)
+	ST(STRLIT,
+		/* OTHER	=> */ STRLIT,
+		/* LU		=> */ STRLIT,
+		/* u		=> */ STRLIT,
+		/* DIGIT	=> */ STRLIT,
+		/* 8		=> */ STRLIT,
+		/* OP1		=> */ STRLIT,
+		/* MIGHTEQ	=> */ STRLIT,
+		/* PLUS		=> */ STRLIT,
+		/* MINUS	=> */ STRLIT,
+		/* EQ		=> */ STRLIT,
+		/* LT		=> */ STRLIT,
+		/* GT		=> */ STRLIT,
+		/* AND		=> */ STRLIT,
+		/* OR		=> */ STRLIT,
+		/* DOT		=> */ STRLIT,
+		/* STAR		=> */ STRLIT,
+		/* HASH		=> */ STRLIT,
+		/* COLON	=> */ STRLIT,
+		/* FS		=> */ STRLIT,
+		/* BS		=> */ STRLIT_BS,
+		/* SQ		=> */ STRLIT,
+		/* DQ		=> */ WS,
+		/* SP		=> */ STRLIT,
+		/* NL		=> */ ERROR | FWD
+	)
+	ST(STRLIT_BS,
+		/* OTHER	=> */ STRLIT,
+		/* LU		=> */ STRLIT,
+		/* u		=> */ STRLIT,
+		/* DIGIT	=> */ STRLIT,
+		/* 8		=> */ STRLIT,
+		/* OP1		=> */ STRLIT,
+		/* MIGHTEQ	=> */ STRLIT,
+		/* PLUS		=> */ STRLIT,
+		/* MINUS	=> */ STRLIT,
+		/* EQ		=> */ STRLIT,
+		/* LT		=> */ STRLIT,
+		/* GT		=> */ STRLIT,
+		/* AND		=> */ STRLIT,
+		/* OR		=> */ STRLIT,
+		/* DOT		=> */ STRLIT,
+		/* STAR		=> */ STRLIT,
+		/* HASH		=> */ STRLIT,
+		/* COLON	=> */ STRLIT,
+		/* FS		=> */ STRLIT,
+		/* BS		=> */ STRLIT,
+		/* SQ		=> */ STRLIT,
+		/* DQ		=> */ STRLIT,
+		/* SP		=> */ STRLIT_BS_WS,
+		/* NL		=> */ STRLIT
+	)
+	ST(STRLIT_BS_WS, // ugh this really does suck
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ ERROR | FWD,
+		/* u		=> */ ERROR | FWD,
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ ERROR | FWD,
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ STRLIT_BS_WS,
+		/* NL		=> */ STRLIT
+	)
+	ST(CHRLIT,
+		/* OTHER	=> */ CHRLIT,
+		/* LU		=> */ CHRLIT,
+		/* u		=> */ CHRLIT,
+		/* DIGIT	=> */ CHRLIT,
+		/* 8		=> */ CHRLIT,
+		/* OP1		=> */ CHRLIT,
+		/* MIGHTEQ	=> */ CHRLIT,
+		/* PLUS		=> */ CHRLIT,
+		/* MINUS	=> */ CHRLIT,
+		/* EQ		=> */ CHRLIT,
+		/* LT		=> */ CHRLIT,
+		/* GT		=> */ CHRLIT,
+		/* AND		=> */ CHRLIT,
+		/* OR		=> */ CHRLIT,
+		/* DOT		=> */ CHRLIT,
+		/* STAR		=> */ CHRLIT,
+		/* HASH		=> */ CHRLIT,
+		/* COLON	=> */ CHRLIT,
+		/* FS		=> */ CHRLIT,
+		/* BS		=> */ CHRLIT_BS,
+		/* SQ		=> */ WS,
+		/* DQ		=> */ CHRLIT,
+		/* SP		=> */ CHRLIT,
+		/* NL		=> */ ERROR | FWD
+	)
+	ST(CHRLIT_BS,
+		/* OTHER	=> */ CHRLIT,
+		/* LU		=> */ CHRLIT,
+		/* u		=> */ CHRLIT,
+		/* DIGIT	=> */ CHRLIT,
+		/* 8		=> */ CHRLIT,
+		/* OP1		=> */ CHRLIT,
+		/* MIGHTEQ	=> */ CHRLIT,
+		/* PLUS		=> */ CHRLIT,
+		/* MINUS	=> */ CHRLIT,
+		/* EQ		=> */ CHRLIT,
+		/* LT		=> */ CHRLIT,
+		/* GT		=> */ CHRLIT,
+		/* AND		=> */ CHRLIT,
+		/* OR		=> */ CHRLIT,
+		/* DOT		=> */ CHRLIT,
+		/* STAR		=> */ CHRLIT,
+		/* HASH		=> */ CHRLIT,
+		/* COLON	=> */ CHRLIT,
+		/* FS		=> */ CHRLIT,
+		/* BS		=> */ CHRLIT,
+		/* SQ		=> */ CHRLIT,
+		/* DQ		=> */ CHRLIT,
+		/* SP		=> */ CHRLIT_BS_WS,
+		/* NL		=> */ CHRLIT
+	)
+	ST(CHRLIT_BS_WS, // ugh this really does also suck
+		/* OTHER	=> */ ERROR | FWD,
+		/* LU		=> */ ERROR | FWD,
+		/* u		=> */ ERROR | FWD,
+		/* DIGIT	=> */ ERROR | FWD,
+		/* 8		=> */ ERROR | FWD,
+		/* OP1		=> */ ERROR | FWD,
+		/* MIGHTEQ	=> */ ERROR | FWD,
+		/* PLUS		=> */ ERROR | FWD,
+		/* MINUS	=> */ ERROR | FWD,
+		/* EQ		=> */ ERROR | FWD,
+		/* LT		=> */ ERROR | FWD,
+		/* GT		=> */ ERROR | FWD,
+		/* AND		=> */ ERROR | FWD,
+		/* OR		=> */ ERROR | FWD,
+		/* DOT		=> */ ERROR | FWD,
+		/* STAR		=> */ ERROR | FWD,
+		/* HASH		=> */ ERROR | FWD,
+		/* COLON	=> */ ERROR | FWD,
+		/* FS		=> */ ERROR | FWD,
+		/* BS		=> */ ERROR | FWD,
+		/* SQ		=> */ ERROR | FWD,
+		/* DQ		=> */ ERROR | FWD,
+		/* SP		=> */ CHRLIT_BS_WS,
+		/* NL		=> */ CHRLIT
+	)
+	ST(ERROR, // dead end, caught outside the branchless main loop
+		/* OTHER	=> */ ERROR,
+		/* LU		=> */ ERROR,
+		/* u		=> */ ERROR,
+		/* DIGIT	=> */ ERROR,
+		/* 8		=> */ ERROR,
+		/* OP1		=> */ ERROR,
+		/* MIGHTEQ	=> */ ERROR,
+		/* PLUS		=> */ ERROR,
+		/* MINUS	=> */ ERROR,
+		/* EQ		=> */ ERROR,
+		/* LT		=> */ ERROR,
+		/* GT		=> */ ERROR,
+		/* AND		=> */ ERROR,
+		/* OR		=> */ ERROR,
+		/* DOT		=> */ ERROR,
+		/* STAR		=> */ ERROR,
+		/* HASH		=> */ ERROR,
+		/* COLON	=> */ ERROR,
+		/* FS		=> */ ERROR,
+		/* BS		=> */ ERROR,
+		/* SQ		=> */ ERROR,
+		/* DQ		=> */ ERROR,
+		/* SP		=> */ ERROR,
+		/* NL		=> */ ERROR
+	)
+};
+// }}}
+
+static const unsigned char statetoks[] = {
+	// comments here cover corresponding tokens. stuff in parentheses doesn't
+	// matter (could have any value)
+	CLEX_TOK_EOL, // BOL, (BOL_BS)
+	CLEX_TOK_OP1, // OP1, (WS)
+	CLEX_TOK_OP2, // OP2, (WS_BS)
+	CLEX_TOK_IDENT, // LITPFX, (LITPFX_BS)
+	CLEX_TOK_IDENT, // uPFX, (uPFX_BS)
+	CLEX_TOK_IDENT, // WORD, (WORD_BS)
+	CLEX_TOK_NUM, // NUM, (NUM_BS)
+	CLEX_TOK_OP1, // MIGHTEQ, (MIGHTEQ_BS)
+	CLEX_TOK_OP1, // MINUS, (MINUS_BS)
+	CLEX_TOK_OP1, // PLUS, (PLUS_BS)
+	CLEX_TOK_OP1, // AND, (AND_BS)
+	CLEX_TOK_OP1, // OR, (OR_BS)
+	CLEX_TOK_OP1, // HASH, (HASH_BS)
+	CLEX_TOK_OP1, // COLON, (COLON_BS)
+	CLEX_TOK_OP1, // FS, (FS_BS)
+	CLEX_TOK_OP1, // LT, (LT_BS)
+	CLEX_TOK_OP1, // GT, (GT_BS)
+	CLEX_TOK_OP1, // DOT, (DOT_BS)
+	CLEX_TOK_SHIFT, // SHIFT, (SHIFT_BS)
+	CLEX_TOK_LNCOMM, // LNCOMM, (LNCOMM_BS)
+	CLEX_TOK_BLKCOMM, // BLKCOMM, (BLKSTAR)
+	CLEX_TOK_OPEQ, // OPEQ, (BLKSTAR_BS)
+	CLEX_TOK_SHIFTEQ, // SHIFTEQ, (STRLIT_BS_WS)
+	CLEX_TOK_STR, // STRLIT, (STRLIT_BS)
+	CLEX_TOK_CHAR, // CHRLIT, (CHRLIT_BS)
+	200 // (ERROR), (CHRLIT_BS_WS) // completely arbitrary value!
+};
+
+// Uncomment and LSP-inspect this for an idea of how much space is used:
+//enum { TABLESPACE = sizeof(classes) + sizeof(trans) + sizeof(statetoks) };
+// => 1786 bytes.
+
+// somewhat inefficient number formatter - size matters more than speed here.
+// also assumes there's extra space in the buffer, because there always is.
+static cold int err_fmtnum(char *out, unsigned int n) {
+	int i = 0, j = 10;
+	do {
+		// explicitly optimised division by 10 to make sure we NEVER have a
+		// divide instruction because integer division is evil.
+		unsigned int div10 = n * 3435973837ull >> 35;
+		// apparently doing * 10 here emits a more efficient lea-based shift-add
+		// thingy than attempting to shift-add by hand, at least with clang.
+		unsigned int remainder = n - div10 * 10;
+		out[--j] = '0' + remainder;
+		n = div10;
+	} while (n);
+	do out[i++] = out[j++]; while (j < 10);
+	return i;
+}
+
+static cold struct linecol {
+	unsigned int ln, col;
+} getlinecol(const char *buf, unsigned int off) {
+	// we don't store line and col for every token as it wastes a lot of memory.
+	// instead we can simply re-scan and count the newlines.
+	// FIXME: grapheme widths for columns? horrendous, but technically correct!
+	// if not that, then probably at least count utf8 codepoints.
+	unsigned int ln = 1, col = 0;
+	for (unsigned int i = 0;; ++i, ++col) {
+		if (buf[i] == '\n') { ++ln; col = 0; }
+		if (i == off) break;
+	}
+	return (struct linecol) {ln, col};
+}
+
+static cold char *err_putprefix(char *restrict out, const char *f,
+		const char *restrict buf, unsigned int off) {
+	struct linecol lc = getlinecol(buf, off);
+	// "filename:line:col: "
+	while (*f) *out++ = *f++;
+	*out++ = ':';
+	out += err_fmtnum(out, lc.ln);
+	*out++ = ':';
+	out += err_fmtnum(out, lc.col);
+	*out++ = ':';
+	*out++ = ' ';
+	return out;
+}
+
+// if we have clang we can avoid even doing a library call here. otherwise we
+// still only depend on memcpy which is pretty reasonable. we could also in
+// theory do some manual word-wise copying of the strings below, but eww.
+#ifdef __clang__
+#define copy __builtin_memcpy_inline
+#else
+#define copy memcpy
+#endif
+
+static cold void err(struct clex *c, const char *p, unsigned int off, int state,
+		int prevtok, const char *f) {
+	if (state == STATE_ERROR) {
+		// all transitions to ERROR have FWD, so we get a dummy token at the
+		// point where the error was detected. if this token points at a
+		// newline, then we have an unterminated literal. if it does *not* point
+		// at a newline, we have a non-newline after a backslash. in the latter
+		// case, backtrack to the backslash to report it as unexpected. in the
+		// former case, move back one character to point to the end of the line.
+		if (p[off--] != '\n') while (p[off] != '\\') --off;
+	}
+	// format the message as f:ln:col: msg. pretty verbose due to not using
+	// stdio or any other convenient formatting library, but also pretty simple
+	// IMPORTANT: clex_memreq() must be recalculated after changing any of this!
+	char *msg = (char *)c->tokoffs, *msgp = msg;
+	c->err = msg;
+	msgp = err_putprefix(msgp, f, p, off);
+	static const char strblock[78] =
+		"unterminated "		// + 0, len 13
+		"string"			// +13, len 6
+		"character"			// +19, len 9
+		" literal"			// +28, len 8
+		"block comment"		// +36, len 13
+		"expected"			// +49, len 8
+		" line after"		// +57, len 11
+		" backslash";		// +68, len 10
+	// minor code size trick: write a little too much first, then replace parts
+	copy(msgp, strblock, 19); // "unterminated string"
+	if (state == STATE_ERROR) {
+		// note: ERROR can only follow another token-producing state. prevtok
+		// could otherwise contain garbage from one of the dummy slots (see
+		// dotok() below) but in this scenario that will never be the case.
+		if (prevtok == CLEX_TOK_CHAR) {
+			// s/string /character literal/
+			copy(msgp + 13, strblock + 19, 9 + 8);
+			msgp += 13 + 9 + 8;
+		}
+		else if (prevtok == CLEX_TOK_STR) {
+			// append "literal" -> "
+			copy(msgp + 19, strblock + 29, 8);
+			msgp += 13 + 6 + 8;
+		}
+		else /* prevtok == 200 (backslash case above) */ {
+			// s/terminated/expected/ -> unexpected
+			copy(msgp + 2, strblock + 49, 8);
+			// append " backslash" -> "unexpected backslash"
+			copy(msgp + 10, strblock + 68, 10);
+			msgp += 10 + 10;
+		}
+	}
+	else if (state == STATE_BLKCOMM) {
+		// s/string/block comment/ -> "unterminated block comment"
+		copy(msgp + 13, strblock + 36, 13);
+		msgp += 13 + 13;
+	}
+	else /* state == STATE_BS */ {
+		// s/unterminated string/expected line after backslash/
+		copy(msgp, strblock + 49, 8 + 11 + 10);
+		msgp += 8 + 11 + 10;
+	}
+	*msgp = '\0';
+	c->errlen = msgp - msg;
+}
+
+static inline void dotok(struct clex *c, const unsigned char *restrict p,
+		unsigned int sz, const char *restrict f) {
+	unsigned char *toks = c->toks;
+	unsigned int *tokoffs = c->tokoffs;
+	int state = STATE_BOL;
+	unsigned int off = 0;
+	unsigned int ntoks = 1; // skip first slot, see below
+	for (; off != sz; ++off) {
+		unsigned char state_trans = trans[classes[p[off]] + state];
+		unsigned int lower7 = state_trans & 127;
+		unsigned int off_adj = off - (state_trans >> 7); // see FIXPOS
+		unsigned int fwdbit = state_trans & 1; // see FWD
+		// bit 2 (low state bit): set -> 1; unset -> -1.
+		// => only even-indexed states (which are terminal) update the type
+		// note: this means we have to waste the bottom two slots. oh well!
+		unsigned int idxmask = (state_trans | -3u) + 2;
+		// set the starting offset for the *next* token, that way if we're not
+		// starting a new one we don't clobber the existing position.
+		tokoffs[ntoks + 1] = off_adj;
+		// if FWD bit is set then we have a new token; at this point we increase
+		// ntoks meaning the first token will be at position 2. hence the waste.
+		ntoks += fwdbit; // advance if we have a new token
+		state = lower7 >> 1; // middle 6 bits are our new state
+		toks[ntoks & idxmask] = statetoks[lower7 >> 2];
+	}
+	// hide the wasted slots from the caller
+	c->ntoks = ntoks - 2; c->toks += 2; c->tokoffs += 2;
+	if_cold (state != STATE_BOL) {
+		if (f) {
+			err(c, (char *)p, tokoffs[ntoks], state, toks[ntoks - 1], f);
+		}
+		else {
+			c->err = "syntax error";
+			c->errlen = 12;
+		}
+	}
+}
+
+struct clex clex(const char *restrict buf, unsigned int sz,
+		void *restrict outmem, const char *restrict filename) {
+	struct clex c = {
+		.tokoffs = outmem,
+		.toks = (unsigned char *)(c.tokoffs + sz)
+	};
+	if_cold (sz != 0 && buf[sz - 1] != '\n') {
+		c.err = "input is not a valid text file (must end with newline)";
+		return c;
+	}
+	dotok(&c, (unsigned char *)buf, sz, filename);
+	return c;
+}
+
+static inline bool ishex(char c) {
+	char lower = c | 32;
+	return c >= '0' & c <= '9' | lower >= 'a' & lower <= 'f';
+}
+static inline int hexval(char c) {
+	unsigned char u = c;
+	return (u & 15) + (u >> 6) * 9; // assumes ishex(c)
+}
+
+static inline int utf8len_ucs2(int codepoint) { // assumes codepoint <= 0xFFFF
+	if (codepoint > 0x7FF) return 3;
+	// would be weird to use \u for ascii, so do this branch last
+	if (codepoint <= 0x7F) return 1;
+	return 2;
+}
+static inline int utf8len(int codepoint) { // assumes codepoint <= 0x10FFFF
+    if (codepoint <= 0xFFFF) { // most likely use for \U over \u
+		if (codepoint <= 0x7FF) {
+			if (codepoint <= 0x7F) return 1; // should still be least likely
+			return 2;
+		}
+		return 3;
+	}
+	return 4;
+}
+
+static inline bool isbadcodepoint_ucs2(int codepoint) { // assumes <= 0xFFFF
+	return codepoint >= 0xD800 && codepoint <= 0xDFFF || // surrogates
+			codepoint >= 0xFDD0 && codepoint <= 0xFDEF || // non-characters
+			(codepoint & 0x0FFE) == 0x0FFE; // more non-characters
+}
+static inline bool isbadcodepoint(int codepoint) {
+	return isbadcodepoint_ucs2(codepoint) ||
+			codepoint >= 0x110000 && codepoint <= 0x1FFFFF || // more non-chars
+			codepoint > 0x10FFFF; // max valid codepoint
+}
+
+static struct bsiter_ret { char c; const char *next; } bsiter(const char *p) {
+	char c;
+	// skip line continuations, but don't ignore backslashes mid-line. may need
+	// some backtracking/re-scanning - oh well. this is the slow path.
+	while ((c = *p) == '\\') {
+		const char *q = p;
+		while (classes[(unsigned char)*++q] == CLASS_SP);
+		if (*q != '\n') break; // backslash is significant, return it
+		p = q + 1; // skip newline and keep scanning forward
+	}
+	return (struct bsiter_ret){c, p + 1};
+}
+
+struct clex_ident_validate_ret clex_ident_validate(const struct clex *c,
+		const char *restrict buf, unsigned int idx) {
+	const char *p = buf + c->tokoffs[idx], *start = p;
+	// N.B. file always ends in an EOL, so no need for bounds check on this loop
+	struct clex_ident_validate_ret ret;
+	for (int namelen = 0;;) {
+		struct bsiter_ret iter = bsiter(p); char c = iter.c; p = iter.next;
+		// first 5 entries in CLASSES() are alphanumeric; if it's a symbol or
+		// space, we're past this token...
+		if (classes[(unsigned char)c] > 4 * NSTATES) {
+			if (c == '\\') { // ... unless it's a UCN, or bad syntax
+				const char *pbs = p;
+				iter = bsiter(p); c = iter.c; p = iter.next;
+				if (c == 'u') {
+					char hex[4];
+					iter = bsiter(p); hex[0] = iter.c;
+					const char *p0 = iter.next;
+					for (int i = 0;;) {
+						if_cold (!ishex(hex[i])) {
+							ret.err = CLEX_IDENT_BADUCNHEX4;
+							ret.err_off = p - buf;
+							return ret;
+						}
+						p = iter.next;
+						if (++i == 4) break;
+						iter = bsiter(p); hex[i] = iter.c;
+					}
+					int codepoint = hexval(hex[0]) << 12 | hexval(hex[1]) << 8 |
+							hexval(hex[2]) << 4 | hexval(hex[3]);
+					if_cold (isbadcodepoint_ucs2(codepoint)) {
+						ret.err = CLEX_IDENT_BADCODEPOINT;
+						ret.err_off = p0 - buf;
+						return ret;
+					}
+					namelen += utf8len_ucs2(codepoint);
+				}
+				else if (c == 'U') {
+					char hex[6];
+					iter = bsiter(p); hex[0] = iter.c;
+					const char *p0 = iter.next;
+					for (int i = 0;;) {
+						if_cold (!ishex(hex[i])) {
+							ret.err = CLEX_IDENT_BADUCNHEX6;
+							ret.err_off = p - buf;
+							return ret;
+						}
+						p = iter.next;
+						if (++i == 6) break;
+						iter = bsiter(p); hex[i] = iter.c;
+					}
+					int codepoint = hexval(hex[0]) << 20 | hexval(hex[1]) << 16 |
+							hexval(hex[2]) << 12 | hexval(hex[3] << 8) |
+							hexval(hex[4]) << 4 | hexval(hex[5]);
+					if_cold (isbadcodepoint(codepoint)) {
+						ret.err = CLEX_IDENT_BADCODEPOINT;
+						ret.err_off = p0 - buf;
+						return ret;
+					}
+					namelen += utf8len(codepoint);
+				}
+				else { // we skipped line conts already; must be bad syntax here
+					ret.err = CLEX_IDENT_UNEXPBACKSLASH;
+					ret.err_off = pbs - buf;
+					return ret;
+				}
+			}
+			else {
+				ret.err = 0; ret.ext = p - start; ret.len = namelen;
+				return ret;
+			}
+		}
+		else {
+			++namelen;
+		}
+	}
+}
+
+cold int clex_ident_errstr(char *restrict out, const char *restrict buf,
+		enum clex_ident_validate_err err, unsigned int err_off,
+		const char *restrict filename) {
+	char *p = out;
+	static const char strblock[81] =
+		"illegal character in 3-digit UCN" // (becomes 4 or 6, see below)
+		"Unicode codepoint"
+		"identifier"
+		"unexpected end of line";
+	if (buf[err_off] == '\n') {
+		p = err_putprefix(p, filename, buf, err_off - 1);
+		copy(p, strblock + 59, 22);
+		p[22] = '\0';
+		return p + 23 - out;
+	}
+	switch (err) {
+		case CLEX_IDENT_BADUCNHEX4:
+		case CLEX_IDENT_BADUCNHEX6:
+			copy(p, strblock, 32);
+			p[21] += err; // 3 -> 4 or 6 (N.B. relies on enum value!!!)
+			p += 32;
+			break;
+		case CLEX_IDENT_BADCODEPOINT:
+			copy(p, strblock, 8); // "illegal "
+			copy(p + 8, strblock + 32, 17); // "Unicode codepoint"
+			p += 25;
+			break;
+		case CLEX_IDENT_UNEXPBACKSLASH:
+			copy(p, strblock, 18); // "illegal character in "
+			copy(p + 18, strblock + 49, 10); // "identifier"
+			p += 28;
+			break;
+		default:
+			unreachable;
+	}
+	*p = '\0';
+	return p - out;
+}
+
+static int utf8put_ucs2(char *out, int codepoint) { // assumes valid UCS2
+	if (codepoint > 0x7FF) {
+		out[0] = 0xE0 | codepoint >> 12;
+		out[1] = 0x80 | codepoint >> 6 & 0x3F;
+		out[2] = 0x80 | codepoint & 0x3F;
+		return 3;
+	}
+    if (codepoint <= 0x7F) {
+		out[0] = codepoint;
+		return 1;
+	}
+	out[0] = 0xC0 | codepoint >> 6;
+	out[1] = 0x80 | codepoint & 0x3F;
+	return 2;
+}
+static int utf8put(char *out, int codepoint) { // assumes valid Unicode
+	if (codepoint <= 0xFFFF) return utf8put_ucs2(out, codepoint);
+	out[0] = 0xF0 | codepoint >> 18;
+	out[1] = 0x80 | codepoint >> 12 & 0x3F;
+	out[2] = 0x80 | codepoint >> 6 & 0x3F;
+	out[3] = 0x80 | codepoint & 0x3F;
+	return 4;
+}
+
+void clex_ident(const struct clex *c, const char *restrict buf,
+		unsigned int idx, char *restrict out) {
+	const char *p = buf + c->tokoffs[idx];
+	struct bsiter_ret iter = bsiter(p); char ch = iter.c; p = iter.next;
+	for (;;) {
+		if (classes[(unsigned char)ch] > 4 * NSTATES) {
+			if (ch == '\\') {
+				iter = bsiter(p); ch = iter.c; p = iter.next;
+				if (ch == 'u') {
+					iter = bsiter(p); char c0 = iter.c; p = iter.next;
+					iter = bsiter(p); char c1 = iter.c; p = iter.next;
+					iter = bsiter(p); char c2 = iter.c; p = iter.next;
+					iter = bsiter(p); char c3 = iter.c; p = iter.next;
+					int codepoint = hexval(c0) << 12 | hexval(c1) << 8 |
+							hexval(c2) << 4 | hexval(c3);
+					out += utf8put_ucs2(out, codepoint);
+				}
+				else /* ch == 'U' */ {
+					iter = bsiter(p); char c0 = iter.c; p = iter.next;
+					iter = bsiter(p); char c1 = iter.c; p = iter.next;
+					iter = bsiter(p); char c2 = iter.c; p = iter.next;
+					iter = bsiter(p); char c3 = iter.c; p = iter.next;
+					iter = bsiter(p); char c4 = iter.c; p = iter.next;
+					iter = bsiter(p); char c5 = iter.c; p = iter.next;
+					int codepoint = hexval(c0) << 20 | hexval(c1) << 16 |
+							hexval(c2) << 12 | (hexval(c3) << 8) |
+							hexval(c4) << 4 | hexval(c5);
+					out += utf8put(out, codepoint);
+				}
+			}
+			else {
+				return;
+			}
+		}
+		else {
+			*out++ = ch;
+			iter = bsiter(p); ch = iter.c; p = iter.next;
+		}
+	}
+}
+
+// vi: sw=4 ts=4 noet tw=80 cc=80 fdm=marker
diff --git a/src/chunklets/clex.h b/src/chunklets/clex.h
new file mode 100644
index 0000000..5d3a636
--- /dev/null
+++ b/src/chunklets/clex.h
@@ -0,0 +1,731 @@
+/*
+ * Copyright © Michael Smith <mikesmiffy128@gmail.com>
+ *
+ * Permission to use, copy, modify, and/or distribute this software for any
+ * purpose with or without fee is hereby granted, provided that the above
+ * copyright notice and this permission notice appear in all copies.
+ *
+ * THE SOFTWARE IS PROVIDED “AS IS” AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+ * REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+ * AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+ * INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+ * LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+ * OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+ * PERFORMANCE OF THIS SOFTWARE.
+ */
+
+/* WARNING: this library is incomplete and still subject to change! */
+
+#ifndef INC_CHUNKLETS_CLEX_H
+#define INC_CHUNKLETS_CLEX_H
+
+#ifdef __cplusplus
+#define _clex_bool bool
+#define _clex_restrict __restrict // XXX: is this portable enough?
+#else
+#define _clex_bool _Bool
+#define _clex_restrict restrict
+#endif
+
+#ifdef _WIN32
+#ifdef __cplusplus
+typedef wchar_t clex_os_char; // ugh
+#else
+typedef unsigned short clex_os_char;
+#endif
+#else
+typedef char clex_os_char;
+#endif
+
+#ifdef _WIN64
+typedef long long clex_size;
+#else
+typedef long clex_size;
+#endif
+
+#if defined(__GNUC__) || defined(__clang__)
+#define _clex_unreachable __builtin_unreachable()
+#elif defined(_MSC_VER)
+#define _clex_unreachable __assume(0)
+#else
+#ifdef __cplusplus
+[[noreturn]] static inline void _clex_invoke_ub(void) {}
+#else
+static inline _Noreturn void _clex_invoke_ub(void) {}
+#endif
+#define _clex_unreachable (_clex_invoke_ub())
+#endif
+
+/*
+ * Determines the amount of memory required to call clex() on a given file.
+ * That memory must be provided as a contiguous block, with at least 4 byte
+ * alignment.
+ *
+ * filesz is the number of bytes in the file, and namelen is the length of the
+ * name. The reason namelen is required is because the memory is reused to
+ * produce an error string when lexing fails.
+ *
+ * filesz is assumed to be non-zero, since empty C source files are not allowed,
+ * and common memory allocators do not accept zero sizes either. To gracefully
+ * handle an empty file if desired, check for a zero size and avoid attempting
+ * tokenisation altogether.
+ *
+ * If a null filename will be passed to clex(), which skips error message
+ * formatting, namelen can be set to 0.
+ *
+ * NOTE: This assumes that the file is reasonably-sized. On 64-bit architectures,
+ * the file size must be not exceed 4GiB *minus 2 bytes*, due to an internal
+ * implementation detail. On 32-bit architectures, the file has to be even
+ * smaller in order to fit the token data in memory; it is recommended to stay
+ * below ~256MiB.
+ */
+static inline clex_size clex_memreq(unsigned int filesz, unsigned int namelen) {
+	clex_size sz = ((clex_size)filesz + 2) * 5 + 5;
+	if (sz < namelen + 64) return namelen + 64; // make space for error messages
+	return sz;
+}
+
+/* The main structure returned by a call to clex() - see below. */
+struct clex {
+	/*
+	 * Check this first. It is a null pointer if lexing succeeded, or points to
+	 * a null-terminated error string on failure. See also clex() for a
+	 * description of how the error string is created.
+	 */
+	const char *err;
+	union {
+		/*
+		 * If err is not null, contains the length of the string, excluding the
+		 * null terminator.
+		 */
+		int errlen;
+		/*
+		 * If lexing was successful (err is null), contains the number of tokens
+		 * lexed. Use this to loop over toks and tokoffs.
+		 */
+		unsigned int ntoks;
+	};
+	//char pad[4];
+	/*
+	 * If lexing was successful (err is null), points to an array of token
+	 * offsets within the lexed file. On failure, the value is undefined.
+	 */
+	unsigned int *tokoffs;
+	/*
+	 * If lexing was successful (err is null), points to an array of basic token
+	 * types corresponding to each position in tokoffs. On failure, the value is
+	 * undefined.
+	 */
+	unsigned char *toks;
+};
+
+/*
+ * Lexes a C file into a list of tokens. See above comments on struct clex for
+ * return type information.
+ *
+ * buf and sz point to the contents of the file, which has to be fully loaded
+ * into memory prior to lexing.
+ *
+ * outmem is an opaque block of memory of at sufficient size calculated by
+ * clex_memreq(), and at least 4-byte alignment. It can be allocated by any
+ * means desired.
+ *
+ * filename is used purely for error reporting, though typically it would be the
+ * name of the file being lexed, of course. Normally the error string output
+ * will include the filename, row and column, and a brief description of the
+ * error. If this information is not needed, filename can be a null pointer
+ * instead, which disables this functionality.
+ */
+struct clex clex(const char *_clex_restrict buf, unsigned int sz,
+		void *_clex_restrict outmem, const char *_clex_restrict filename);
+
+/*
+ * These are the basic token types recognised by the lexer in its main pass.
+ * All the operators are grouped in such a way that they can be distinguished by
+ * looking at a single character.
+ *
+ * Identifiers and literals might not necessarily be totally valid; this is
+ * checked in more detail when parsing out specific values.
+ */
+enum clex_tok_type {
+	CLEX_TOK_IDENT, /* An identifier or keyword. */
+	CLEX_TOK_NUM, /* A numeric literal, which may or may not be totally valid */
+	CLEX_TOK_OP1, /* One of: + - * / = < > ! & | ^ ~ . , ( ) [ ] { } : ; ? # */
+	CLEX_TOK_OP2, /* One of: ++ -- -> && || ## :: */
+	CLEX_TOK_SHIFT, /* One of: >> << */
+	CLEX_TOK_OPEQ, /* One of: += -= *= /= &= |= ^= <= >= != == */
+	CLEX_TOK_SHIFTEQ, /* One of: >>= <<= */
+	CLEX_TOK_CHAR, /* A character literal. */
+	CLEX_TOK_STR, /* A string literal. */
+	CLEX_TOK_LNCOMM, /* A C++/C99-style one-line comment. */
+	CLEX_TOK_BLKCOMM, /* A classic C-style multi-line comment. */
+	CLEX_TOK_EOL /* A run of end-of-line characters, for parsing #directives. */
+};
+
+/* These are all the C operators, hopefully self-explanatory. */
+enum clex_op {
+	// -- op1 --
+	CLEX_OP_PLUS,		/* + */
+	CLEX_OP_MINUS,		/* - */
+	CLEX_OP_MULT,		/* * (could also be a deref) */ // XXX: bad name maybe?
+	CLEX_OP_DIV,		/* / */
+	CLEX_OP_ASSIGN,		/* = */
+	CLEX_OP_LT,			/* < */
+	CLEX_OP_GT,			/* > */
+	CLEX_OP_NOT,		/* ! */
+	CLEX_OP_BITAND,		/* & (could also be an address-of) */ // XXX: bad name?
+	CLEX_OP_BITOR,		/* | */
+	CLEX_OP_XOR,		/* ^ */
+	CLEX_OP_BITNOT,		/* ~ */
+	CLEX_OP_DOT,		/* . */
+	CLEX_OP_COMMA,		/* , */
+	CLEX_OP_LPAREN,		/* ( */
+	CLEX_OP_RPAREN,		/* ) */
+	CLEX_OP_LSQ,		/* [ */
+	CLEX_OP_RSQ,		/* ] */
+	CLEX_OP_LCURL,		/* { */
+	CLEX_OP_RCURL,		/* } */
+	CLEX_OP_COLON,		/* : */
+	CLEX_OP_SEMICOL,	/* ; */
+	CLEX_OP_QUESTION,	/* ? */
+	CLEX_OP_HASH,		/* # */
+	// -- op2 --
+	CLEX_OP_INC,		/* ++ */
+	CLEX_OP_DEC,		/* -- */
+	CLEX_OP_ARROW,		/* -> */
+	CLEX_OP_AND,		/* && */
+	CLEX_OP_OR,			/* || */
+	CLEX_OP_PASTE,		/* ## */
+	CLEX_OP_DCOLON,		/* :: C23 attribute vendors/namespaces */
+	// -- shift --
+	CLEX_OP_LSH,		/* << */
+	CLEX_OP_RSH,		/* >> */
+	// -- opeq --
+	CLEX_OP_PLUSEQ,		/* += */
+	CLEX_OP_MINUSEQ,	/* -= */
+	CLEX_OP_MULTEQ,		/* *= */
+	CLEX_OP_DIVEQ,		/* /= */
+	CLEX_OP_ANDEQ,		/* &= */
+	CLEX_OP_OREQ,		/* |= */
+	CLEX_OP_XOREQ,		/* ^= */
+	CLEX_OP_LTEQ,		/* <= */
+	CLEX_OP_GTEQ,		/* >= */
+	CLEX_OP_NOTEQ,		/* != */
+	CLEX_OP_EQ,			/* == */
+	// -- shifteq --
+	CLEX_OP_LSHEQ,		/* <<= */
+	CLEX_OP_RSHEQ		/* >>= */
+};
+
+/*
+ * Determines the exact operator from a token of type CLEX_TOK_OP1 at offset off
+ * by inspecting the first byte of the token.
+ *
+ * If the token is not of type CLEX_TOK_OP1, the behaviour is undefined.
+ */
+static inline enum clex_op clex_op1(const char *buf, unsigned int off) {
+	switch (buf[off]) {
+		case '+': return CLEX_OP_PLUS;
+		case '-': return CLEX_OP_MINUS;
+		case '*': return CLEX_OP_MULT;
+		case '/': return CLEX_OP_DIV;
+		case '=': return CLEX_OP_ASSIGN;
+		case '<': return CLEX_OP_LT;
+		case '>': return CLEX_OP_GT;
+		case '!': return CLEX_OP_NOT;
+		case '&': return CLEX_OP_BITAND;
+		case '|': return CLEX_OP_BITOR;
+		case '^': return CLEX_OP_XOR;
+		case '~': return CLEX_OP_BITNOT;
+		case '.': return CLEX_OP_DOT;
+		case ',': return CLEX_OP_COMMA;
+		case '(': return CLEX_OP_LPAREN;
+		case ')': return CLEX_OP_RPAREN;
+		case '[': return CLEX_OP_LSQ;
+		case ']': return CLEX_OP_RSQ;
+		case '{': return CLEX_OP_LCURL;
+		case '}': return CLEX_OP_RCURL;
+		case ':': return CLEX_OP_COLON;
+		case ';': return CLEX_OP_SEMICOL;
+		case '?': return CLEX_OP_QUESTION;
+		case '#': return CLEX_OP_HASH;
+	}
+	_clex_unreachable;
+}
+
+/*
+ * Determines the exact operator from a token of type CLEX_TOK_OP2 at offset off
+ * by inspecting the end of the token.
+ *
+ * If the token is not of type CLEX_TOK_OP2, the behaviour is undefined.
+ */
+static inline enum clex_op clex_op2(const char *buf, unsigned int off) {
+	for (;;) {
+		switch (buf[off + 1]) {
+			case '+': return CLEX_OP_INC;
+			case '-': return CLEX_OP_DEC;
+			case '>': return CLEX_OP_ARROW;
+			case '&': return CLEX_OP_AND;
+			case '|': return CLEX_OP_OR;
+			case '#': return CLEX_OP_PASTE;
+			case ':': return CLEX_OP_DCOLON;
+			// initial lex has validated backslash-newlines, so we can just
+			// ignore any of these characters here
+			case '\\': case ' ': case '\t': case '\r': case '\n': continue;
+		}
+		_clex_unreachable;
+	}
+}
+
+/*
+ * Determines the exact operator from a token of type CLEX_TOK_SHIFT at offset
+ * off by inspecting the first byte of the token.
+ *
+ * If the token is not of type CLEX_TOK_SHIFT, the behaviour is undefined.
+ */
+static inline enum clex_op clex_shift(const char *buf, unsigned int off) {
+	switch (buf[off]) {
+		case '>': return CLEX_OP_RSH;
+		case '<': return CLEX_OP_LSH;
+	}
+	_clex_unreachable;
+}
+
+/*
+ * Determines the exact operator from a token of type CLEX_TOK_OPEQ at offset
+ * off by inspecting the first byte of the token.
+ *
+ * If the token is not of type CLEX_TOK_OPEQ, the behaviour is undefined.
+ */
+static inline enum clex_op clex_opeq(const char *buf, unsigned int off) {
+	switch (buf[off]) {
+		case '+': return CLEX_OP_PLUSEQ;
+		case '-': return CLEX_OP_MINUSEQ;
+		case '*': return CLEX_OP_MULTEQ;
+		case '/': return CLEX_OP_DIVEQ;
+		case '&': return CLEX_OP_ANDEQ;
+		case '|': return CLEX_OP_OREQ;
+		case '^': return CLEX_OP_XOREQ;
+		case '<': return CLEX_OP_LTEQ;
+		case '>': return CLEX_OP_GTEQ;
+		case '!': return CLEX_OP_NOTEQ;
+		case '=': return CLEX_OP_EQ;
+	}
+	_clex_unreachable;
+}
+
+/*
+ * Determines the exact operator from a token of type CLEX_TOK_SHIFTEQ at offset
+ * off by inspecting the first byte of the token.
+ *
+ * If the token is not of type CLEX_TOK_SHIFTEQ, the behaviour is undefined.
+ */
+static inline enum clex_op clex_shifteq(const char *buf, unsigned int off) {
+	switch (buf[off]) {
+		case '<': return CLEX_OP_RSHEQ;
+		case '>': return CLEX_OP_LSHEQ;
+	}
+	_clex_unreachable;
+}
+
+/*
+ * Determines whether the lexed token at a given index is an operator (not
+ * counting the likes of sizeof, which will have to be handled as a special case
+ * of an identifier/keyword for syntax purposes).
+ */
+static inline _clex_bool clex_isop(const struct clex *c, unsigned int idx) {
+	switch (c->toks[idx]) {
+		case CLEX_TOK_IDENT: return 0;
+		case CLEX_TOK_NUM: return 0;
+		case CLEX_TOK_OP1: return 1;
+		case CLEX_TOK_OP2: return 1;
+		case CLEX_TOK_SHIFT: return 1;
+		case CLEX_TOK_OPEQ: return 1;
+		case CLEX_TOK_SHIFTEQ: return 1;
+		case CLEX_TOK_CHAR: return 0;
+		case CLEX_TOK_STR: return 0;
+		case CLEX_TOK_LNCOMM: return 0;
+		case CLEX_TOK_BLKCOMM: return 0;
+		case CLEX_TOK_EOL: return 0;
+	}
+	_clex_unreachable;
+}
+
+/* Returns true if a token is either a unary or binary plus operator. */
+static inline _clex_bool clex_isplus(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '+';
+}
+
+/* Returns true if a token is either a unary or binary minus operator. */
+static inline _clex_bool clex_isminus(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '-';
+}
+
+/* Returns true if a token is either a multiply or deference operator. */
+static inline _clex_bool clex_ismult(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '*';
+}
+
+/* Returns true a token is a division operator. */
+static inline _clex_bool clex_isdiv(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '/';
+}
+
+/* Returns true if a token is an assignment operator. */
+static inline _clex_bool clex_isassign(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '=';
+}
+
+/* Returns true if a token is a less-than operator. */
+static inline _clex_bool clex_islt(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '<';
+}
+
+/* Returns true if a token is a greater-than operator. */
+static inline _clex_bool clex_isgt(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '>';
+}
+
+/* Returns true if a token is a negation operator. */
+static inline _clex_bool clex_isnot(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '!';
+}
+
+/* Returns true if a token is either a bitwise and operator or an indirection. */
+static inline _clex_bool clex_isbitand(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '&';
+}
+
+/* Returns true if a token is a bitwise or operator. */
+static inline _clex_bool clex_isbitor(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '|';
+}
+
+/* Returns true if a token is a bitwise exclusive-or operator. */
+static inline _clex_bool clex_isxor(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '^';
+}
+
+/* Returns true if a token is a bitwise negation operator. */
+static inline _clex_bool clex_isbitnot(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '~';
+}
+
+/* Returns true if a token is a struct member dot. */
+static inline _clex_bool clex_isdot(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '.';
+}
+
+/* Returns true if a token is a comma. */
+static inline _clex_bool clex_iscomma(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == ',';
+}
+
+/* Returns true if a token is an opening/left parenthesis. */
+static inline _clex_bool clex_islparen(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '(';
+}
+
+/* Returns true if a token is a closing/right parenthesis. */
+static inline _clex_bool clex_isrparen(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == ')';
+}
+
+/* Returns true if a token is an opening/left square bracket. */
+static inline _clex_bool clex_islsq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '[';
+}
+
+/* Returns true if a token is a closing/right square bracket. */
+static inline _clex_bool clex_isrsq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == ']';
+}
+
+/* Returns true if a token is an opening/left curly brace. */
+static inline _clex_bool clex_islcurl(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '{';
+}
+
+/* Returns true if a token is a closing/right curly brace. */
+static inline _clex_bool clex_isrcurl(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '}';
+}
+
+/* Returns true if a token is a colon. */
+static inline _clex_bool clex_iscolon(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == ':';
+}
+
+/* Returns true if a token is a semicolon. */
+static inline _clex_bool clex_issemicol(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == ';';
+}
+
+/* Returns true if a token is a question mark. */
+static inline _clex_bool clex_isquestion(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '?';
+}
+
+/* Returns true if a token is a preprocessor hash. */
+static inline _clex_bool clex_ishash(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP1 && buf[c->tokoffs[idx]] == '#';
+}
+
+/* Returns true if a token is a unary increment operator. */
+static inline _clex_bool clex_isinc(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP2 && buf[c->tokoffs[idx] + 1] == '+';
+}
+
+/* Returns true if a token is a unary decrement operator. */
+static inline _clex_bool clex_isdec(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP2 && buf[c->tokoffs[idx] + 1] == '-';
+}
+
+/* Returns true if a token is an arrow for indirect struct member access. */
+static inline _clex_bool clex_isarrow(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP2 && buf[c->tokoffs[idx] + 1] == '>';
+}
+
+/* Returns true if a token is a conditional and operator. */
+static inline _clex_bool clex_isand(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP2 && buf[c->tokoffs[idx] + 1] == '&';
+}
+
+/* Returns true if a token is a conditional or operator. */
+static inline _clex_bool clex_isor(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP2 && buf[c->tokoffs[idx] + 1] == '|';
+}
+
+/* Returns true if a token is a token-pasting operator. */
+static inline _clex_bool clex_ispaste(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP2 && buf[c->tokoffs[idx] + 1] == '#';
+}
+
+/* Returns true if a token is a double colon for C23 attribute namespacing. */
+static inline _clex_bool clex_isdcol(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OP2 && buf[c->tokoffs[idx] + 1] == ':';
+}
+
+/* Returns true if a token is a left shift operator. */
+static inline _clex_bool clex_islsh(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_SHIFT && buf[c->tokoffs[idx]] == '<';
+}
+
+/* Returns true if a token is a right shift operator. */
+static inline _clex_bool clex_isrsh(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_SHIFT && buf[c->tokoffs[idx]] == '>';
+}
+
+/* Returns true if a token is a binary increment (plus-assignment) operator. */
+static inline _clex_bool clex_ispluseq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '+';
+}
+
+/* Returns true if a token is a binary decrement (plus-assignment) operator. */
+static inline _clex_bool clex_isminuseq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '-';
+}
+
+/* Returns true if a token is a multiply-assignment operator. */
+static inline _clex_bool clex_ismulteq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '*';
+}
+
+/* Returns true if a token is a divide-assignment operator. */
+static inline _clex_bool clex_isdiveq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '/';
+}
+
+/* Returns true if a token is a bitwise-and-assignment operator. */
+static inline _clex_bool clex_isandeq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '&';
+}
+
+/* Returns true if a token is a bitwise-or-assignment operator. */
+static inline _clex_bool clex_isoreq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '|';
+}
+
+/* Returns true if a token is a bitwise-xor-assignment operator. */
+static inline _clex_bool clex_isxoreq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '^';
+}
+
+/* Returns true if a token is a less-than-or-equal comparison operator. */
+static inline _clex_bool clex_islteq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '<';
+}
+
+/* Returns true if a token is a greater-than-or-equal comparison operator. */
+static inline _clex_bool clex_isgteq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '>';
+}
+
+/* Returns true if a token is an inequality comparison operator. */
+static inline _clex_bool clex_isnoteq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '!';
+}
+
+/* Returns true if a token is an equality comparison operator. */
+static inline _clex_bool clex_iseq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_OPEQ && buf[c->tokoffs[idx]] == '=';
+}
+
+/* Returns true if a token is a left-shift-assignment operator. */
+static inline _clex_bool clex_islshifteq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_SHIFTEQ && buf[c->tokoffs[idx]] == '<';
+}
+
+/* Returns true if a token is a right-shift-assignment operator. */
+static inline _clex_bool clex_isrshifteq(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	return c->toks[idx] == CLEX_TOK_SHIFTEQ && buf[c->tokoffs[idx]] == '>';
+}
+
+/*
+ * Determines the exact operator from a token of type CLEX_TOK_OP1,
+ * CLEX_TOK_OP2, CLEX_TOK_SHIFT, CLEX_TOK_OPEQ, or CLEX_TOK_SHIFTEQ.
+ *
+ * If the token is not of one of the above types, the behaviour is undefined.
+ * Call clex_isop() first to determine whether a token is of a valid type.
+ */
+static inline enum clex_op clex_op(const struct clex *c,
+		const char *_clex_restrict buf, unsigned int idx) {
+	unsigned int off = c->tokoffs[idx];
+	switch (c->toks[idx]) {
+		case CLEX_TOK_OP1: return clex_op1(buf, off);
+		case CLEX_TOK_OP2: return clex_op2(buf, off);
+		case CLEX_TOK_SHIFT: return clex_shift(buf, off);
+		case CLEX_TOK_OPEQ: return clex_opeq(buf, off);
+		case CLEX_TOK_SHIFTEQ: return clex_shifteq(buf, off);
+	}
+	_clex_unreachable;
+}
+
+enum clex_ident_validate_err {
+	// NOTE: do not reorder these values! clex_ident_errstr() relies on them
+	CLEX_IDENT_OK,
+	CLEX_IDENT_BADUCNHEX4, /* invalid \uXXXX UCN sequence */
+	CLEX_IDENT_BADCODEPOINT, /* invalid Unicode codepoint in UCN */
+	CLEX_IDENT_BADUCNHEX6, /* invalid \UXXXXXX UCN sequence */
+	CLEX_IDENT_UNEXPBACKSLASH /* unexpected backslash in identifier */
+};
+
+/*
+ * Double-checks that a TOK_IDENT token is syntactically valid (as the initial
+ * lexer pass does not account for bad UCN sequences or misplaced backslashes).
+ *
+ * Determines the extent of the token, i.e. how many bytes it occupies in the
+ * source, along with the length of the evaluated name, i.e. how many bytes are
+ * required to store the evaluated name as UTF-8 once line-continuations and
+ * UCNs are taken into account.
+ *
+ * This function may only be called on tokens of type TOK_IDENT; anything else
+ * produces undefined behaviour.
+ */
+struct clex_ident_validate_ret {
+	enum clex_ident_validate_err err; /* zero on success, nonzero on error */
+	// wonky union interweaving here because C++ doesn't do anonymous structs :(
+	union {
+		int ext; /* if !err: extent (source length) */
+		unsigned int err_off; /* if err: position of error in file */
+	};
+	int len; /* if !err: length of evaluated name (as utf-8). else undefined! */
+} clex_ident_validate(const struct clex *c, const char *_clex_restrict buf,
+		unsigned int idx);
+
+/*
+ * Evaluates the name of an identifier, taking into account line continuations
+ * and UCNs. The identifier has to have first been successfully validated using
+ * clex_ident_validate(). The buffer out must be large enough to hold the result
+ * which is indicated by the len member of the clex_ident_validate_ret struct.
+ */
+void clex_ident(const struct clex *c, const char *_clex_restrict buf,
+		unsigned int idx, char *_clex_restrict out);
+
+// NOTE: this is precalculated based on the implementation of clex_ident_errstr
+// and will need changed if the function changes!
+/*
+ * Determines the buffer size required to format an error message using
+ * clex_ident_errstr(), given the length of the filename.
+ *
+ * clex_ident_errstr() formats a string in the form "filename:line:col: message"
+ * so this macro can determine the worst-case memory requirement in advance.
+ *
+ * If a constant value is desired, e.g. for a stack buffer, simply use the known
+ * longest possible file path, for instance PATH_MAX on Unix-likes or MAX_PATH
+ * on Windows. The reason this is a macro is to allow it to produce an integer
+ * constant expression, so that a stack buffer can be used without creating a
+ * VLA.
+ */
+#define CLEX_IDENT_ERRSTR_MEMREQ(namelen) ((namelen) + 24 + 33)
+
+/*
+ * Formats an error message in the form "filename:line:col: message", given an
+ * error result from clex_ident_validate(). The resulting string is written to
+ * the buffer pointed to by out.
+ *
+ * out must point to a buffer of at least CLEX_IDENT_ERRSTR_MEMREQ(
+ * strlen(filename)) bytes. The resulting string is null-terminated for
+ * convenience, and the length of the string excluding the null terminator is
+ * returned.
+ *
+ * err has to be an error value from the clex_ident_validate_err enum; calling
+ * this function with a success result will produce undefined behaviour.
+ */
+int clex_ident_errstr(char *_clex_restrict out, const char *_clex_restrict buf,
+		enum clex_ident_validate_err err, unsigned int err_off,
+		const char *_clex_restrict filename);
+
+#undef _clex_unreachable
+#undef _clex_restrict
+#undef _clex_bool
+
+#endif
+
+// vi: sw=4 ts=4 noet tw=80 cc=80