27.6KB language-agnostic1 WebGPU syntax highlighter
Shu Ding at Vercel Labs
import { parse } from 'gpu-lexer'
const spans = await parse('source code')
// {
// type: 'plain' | 'comment' | 'string' | 'number' | 'keyword' | 'type' | 'function' | 'constant' | 'operator'
// start: number
// end: number
// }[]gpu-lexer splits source code into simple parts—words, whitespace, newlines, and symbols. Then a tiny WebGPU model combines local and whole-file context to label each part. It is designed for any language: instead of choosing a grammar, it guesses each part's type from the surrounding source, even when it never saw that language or syntax during training. Adjacent labels become the syntax spans returned to your code.
This is an experiment, not a grammar-equivalent highlighter (see the agreement chart below). The chart measures agreement with Shiki—not objective correctness—and results may vary on unseen languages or real-world code.
1 "Language-agnostic" means one shared tokenizer and classifier, not equal accuracy for every language. 83.02% is the share of held-out token labels that matched Shiki. Mixed-language code is supported too, including embedded <script> and <style> regions in HTML, Vue, and Svelte.
| agreement | verified languages (98) |
|---|---|
| 95–100% | angular-html, angular-ts, apex, jsx |
| 90–<95% | actionscript, actionscript-3, cython, dart, go, hcl, javascript, proto, protobuf, python, rust, solidity, svelte, swift, toml, tsx |
| 80–<90% | astro, c, cmake, coffeescript, cpp, csharp, css, docker, gdscript, glsl, haxe, hlsl, html, java, json, odin, php, plist, powershell, ruby, shell, shellscript, typescript, wgsl, xml, xsl |
| 70–<80% | bat, bicep, c3, fortran-fixed-form, haskell, kotlin, make, nginx, nim, nix, objective-cpp, r, rss, scala, vue, yaml, zig |
| 60–<70% | ada, d, diff, fortran-free-form, groovy, hack, less, lua, markdown, matlab, prisma, puppet, qml, scss, system-verilog, systemverilog |
| 50–<60% | asm, awk, emacs-lisp, gherkin, graphql, julia, mdx, objective-c, perl, shaderlab, sql, viml, wasm, webassembly |
| <50% | jinja, liquid, razor, vala, vb |
| training epoch | 4,679,585 tokens |
| model | 41,321 parameters |
Each language appears in one band; results with fewer held-out labels are less stable. Training tokens include context-only tokens and replay.
One browser run after one warm-up on September 16, 2026. The input was 10 concatenated copies of three.min.js (5.56M characters). MacBook Pro, Apple M4 Pro, 20-core GPU, 24GB, macOS 26.6.2, Chrome 152. Each engine ran in a dedicated worker; DOM rendering was excluded. gpu-lexer and Shiki returned token data, Starry Night returned a HAST tree, while Sugar High, Prism.js, and Highlight.js returned highlighted HTML. Sugar High 2.3.1, Prism.js 1.30.0, Highlight.js 11.12.0, Starry Night 3.11.0, and Shiki 4.4.3.
Minified and Brotli-compressed browser bundles. gpu-lexer was measured on September 16, 2026; comparison libraries on September 9, 2026. Major web includes javascript, typescript, css, html, json, and markdown. gpu-lexer uses the same bundle for every language. Starry Night totals include its Oniguruma WASM payload.
Shiki is the 100% normalization reference. Each library's token names are mapped to the same nine classes: plain, comment, string, number, keyword, type, function, constant, and operator. Scores compare non-whitespace source parts across 1,103 held-out files in the GitHub Innovation Graph top 25 for 2026-Q1, weighted by each language's pusher count. Unsupported languages score zero; corpus size does not affect the weights.
[ live demos ]
75 languagesreact.development.js0 byteshttps://unpkg.com/react@19.2.8/cjs/react.development.js—Experimental software. Highlighting is probabilistic, may differ from Shiki, and is not a parser or a substitute for compiler, linter, or security analysis.
Shu Ding at Vercel Labs.