The tokenization process was creating excessive allocations: StringBuilder.ToString() called for every token in TokenStore (including duplicates) For 100k tokens with 50% duplicates: 50k unnecessary ...
You can create a release to package software, along with release notes and links to binary files, for other people to use. Learn more about releases in our docs.