Stanford and UC Berkeley's TokenGate makes long-context prefill up to 11.3x faster
TokenGate is a learned token-pruning module that can be dropped into existing language models. Its authors report a 1.4x to 11.3x reduction in time to first token on long contexts and 1.5x to 3.4x lower peak memory, with no loss of accuracy.