superbpe

Features and purpose

Train and use SuperBPE tokenizers for 20-33% token reduction across any project. Covers training, optimization, validation, and integration with any LLM framework. Use when you need efficient tokenization, want to reduce API costs, or maximize context windows.

Quick start

Follow the repository instructions for this skill and your agent. Review the files and dependencies before adding it to your workflow.