Open source

tokenizers v1: encode, decode and scaling, measured

Hugging Face has released version 1 of its tokenizers library, with measurements of encoding, decoding and scaling performance.