LongCat-Flash-Lite

Overview

LongCat-Flash-Lite is a lightweight Mixture-of-Experts (MoE) model that explores a new scaling direction: N-gram embedding expansion. Instead of relying primarily on adding more experts, it allocates a large portion of total parameters into an N-gram embedding layer to improve local-context semantic capture, while keeping inference sparse via dynamic activation.

Technology Highlights

N-gram Embedding Layer

The N-gram embedding layer enhances the model’s ability to capture local context semantics. Using a hash function, the current token together with its preceding N-1 tokens is mapped into a single N-gram embedding vector, which is then fused with the token’s base embedding.

To reduce hash collisions, the design includes:

• Sub-table decomposition + linear projection: split a large embedding table into multiple sub-tables and project each separately
• Vocabulary size design: carefully select table sizes to lower collision probability
• Embedding amplification: scaling or normalization before output to keep the signal effective through residual paths

System Co-Design for Speed

Despite the large total parameter count, LongCat-Flash-Lite benefits from sparse activation and system-level optimizations to convert theoretical sparsity gains into real throughput.

• Parameter allocation: shift parameters into O(1) embedding lookup to reduce compute growth and expert communication overhead
• N-gram Cache + kernel fusion: GPU-managed N-gram ID caching and fused CUDA kernels to reduce I/O latency and improve utilization
• Speculative decoding collaboration: co-design with speculative decoding; draft model uses standard embeddings to avoid N-gram lookup overhead

Open Source & Resources

• Hugging Face: LongCat-Flash-Lite
• ModelScope: LongCat-Flash-Lite
• Technical Report (arXiv: 2601.21204)
• SGLang-FluentLLM (Inference Engine & Ops)
• Back to Models Overview