llm-echo 0.5 counts tool calls and reasoning tokens in its simulated usage
Simon Willison has updated llm-echo, a debugging plugin for his LLM tool that provides a fake model which simply repeats the prompts it receives. Version 0.5 makes its simulated token counts account for tool calls, system prompts and reasoning tokens, which helps when testing changes ahead of a new LLM release.