Introducing SPARSEUP by Linkup Research: a move towards more competitive open sparse models

Today, Linkup Research is releasing SPARSEUP, a new open-source sparse embedding model. At <150M parameters it is the strongest public vocabulary-based sparse retriever we know of on BEIR-13 (56+ nDCG@10), and in a controlled comparison with the same backbone and data, it lands close to DenseOn - while remaining behind LateOn.

This is the first contribution from the Linkup team to the research community, and it reflects our commitment to open science. SPARSEUP constitutes the first building block in a series of upcoming releases - this first iteration providing a strong, open-source, up-to-date sparse baseline. SPARSEUP is a simple SPLADE-style model, with a few modifications described below, coupled with a better backbone and training data - filling the missing sparse slot next to dense and late-interaction retrieval.

The TL;DR:

• Start from LateOn-unsupervised, whose token embeddings were trained for late interaction [1];
• Graft ModernBERT’s original MLM head on top;
• Fine-tune a slightly modified SPLADE-style encoder on LightOn’s fine-tuning mixture;
• SOTA among sparse encoders at this size on BEIR-13 (56+), close to DenseOn on decontaminated BEIR, and sub-millisecond on MS MARCO with Seismic at >97% recall vs exact search.

The model is released under Apache 2.0. Use it, have fun.

Genesis

Learned sparse embeddings are powerful alternatives to single dense vector retrieval. In addition to various efficiency aspects (storage + latency), sparse models actually constitute a whole different class of models, with different properties (for instance, a better ability to perform precise lexical matching on rare words [2]), strengths and weaknesses. Despite a lesser mainstream adoption, they remain attractive for those reasons, especially in production-oriented scenarios. The SPLADE series [3,4,5] set the standard for years, with relatively sparse (pun intended) follow-ups: OpenSearch’s neural sparse encoders [6], IBM Granite sparse [7], and LLM-based sparse models (for instance, LACONIC [8]). Separate from vocabulary-based SPLADE, some recent works have also explored the use of Sparse Autoencoders to avoid the reliance on a fixed vocabulary (Latent Terms [9], SAE-SPLADE [10], SPLARE [11]) and naturally unlock multilingual retrieval scenarios.

For various reasons, sparse embeddings are actually not as “popular” as single-vector dense retrieval. There are not so many available models (except the ones mentioned above), with even less being open source - which feels crazy when you compare to the huge number of dense embeddings available in the wild. Today’s “SOTA sparse” still lags comparable dense and late-interaction models, which keep getting newer backbones and better data. One of the reasons for their lesser adoption is the notoriously-known difficulty of training such models, which generally require very tight conditions for training to eventually converge. Also, multilinguality (or multimodality) is less natural when the output space is a tokenizer vocab, so they drifted out of the spotlight.

In the meantime, the success of late interaction has recently put sparse embeddings back on the map: late interaction engines are multi-step by nature, and the retrieval folks have naturally come up with efficient ways to perform candidate generation based on sparse vectors and inverted indices (from PLAID [12], which implicitly derives tf-idf like representations, to more explicit techniques like SMVE which represent stuff in a sparse high-dimensional space). The line between the two approaches is itself an active topic (for instance, ColBERTSaR [13] or SSR [14] blur the line between the two views). There has never been a better moment to work on sparse retrieval.

What motivates this work is LightOn’s recent release [1]: open data and a recipe that let you compare architectures, all other things equal. They shipped a dense model and a multi-vector one. We fill the missing slot with SPARSEUP - same backbone, same fine-tuning data - so the comparison is actually side by side. The goal is a strong open sparse baseline at the same table as dense and late interaction, not a final recipe. The model is cool and effective, but not without flaws (see the Examples section). It’s something to compare to, and to build on.

Sparse embeddings

Backbone

SPARSEUP uses a ModernBERT backbone (149M parameters). We start from LateOn-unsupervised, pre-trained on LightOn’s fine-tuning mixture;. Why not pre-train ourselves? Well, it’s a bit expensive, and given how difficult it can be to train sparse embeddings, it felt risky to start from there. In any case, models trained for other retrieval objectives transfer well to SPLADE anyway (e.g., CoCondenser for SPLADE++ [15]). It is also known that the embedding space of late interaction is actually quite suited for sparse retrieval (e.g., SPLATE [16]). We suspect a dedicated sparse pre-train would still help; we spent the compute on fine-tuning instead. Note that parallel experiments with either the bare ModernBERT or the DenseOn-unsupervised checkpoints led to less effective models overall.

SPARSEUP uses the same [Q] / [D] prefixes as LateOn (attended, not pooled). It pools every input as a single sparse vector in the BPE vocabulary space of the backbone (50k dims). The model relies on a standard SPLADE-like architecture (MLM ⇒ log(1 + ReLU) ⇒ max pooling), plus the three modifications introduced below. Note that LateOn-unsupervised has no MLM head attached anymore; there are different ways we could proceed to map hidden states to the vocabulary, but what ended up working well is simply branching back the MLM head from the initial checkpoint (ModernBERT). When inspecting top logits for a given token vector, we noticed that the MLM head still seems pretty much aligned with the LateOn checkpoint (it’s no surprise that hidden states did not drift that much actually). Predicted tokens look good, but the scale of those logits is off (which is ingredient #1 below). While we could also imagine further (re)adapting the head with MLM pre-training, starting from there proved to be good enough.

Training

We otherwise copy DenseOn and LateOn: contrastive training on the same mixture, 7 negatives sampled from a pool of 50, plus in-batch negatives. Training fits on a single H100 (no big batches needed). We use a small sparse regularization; it matters less though once #1 and #2 are in place. Temperature is t = 6: dots are unnormalized, so contrastive training needs a different range than cosine dense models, and 6 was a good trade-off.

Ingredients for better models

A vanilla SPLADE on this backbone was hard to train. BEIR nDCG@10 looked “decent,” but bags were huge and full of stuffing - stopwords and unrelated junk. We first tried the usual fixes, like pooled top-k pruning or things like DF-FLOPS [17]. We ended up realising the issue was more structural and could not be fixed with a mere “bandage”: MLM logits sitting in the wrong place for log(1 + ReLU), and a 50k-way expansion at every token. Three simple small changes fixed the effectiveness-efficiency trade-off and eventually became the released model.

Logit shifting (ingredient #1)

It is no surprise that an arbitrary MLM head is a poor fit for sparse embeddings - why would it be? Softmax is shift-invariant, so logits can sit far above zero at init, land in the saturated region of the log, and make training anywhere from difficult to impossible. DistilBERT-SPLADE was almost a lucky break: that head happened to sit in the sweet spot for log(1 + ReLU).

Here we saw a fat positive tail (weak dimensions in roughly (0, 15), content around 20–43), not a mean shifted up. Two consequences: (i) saturated term weights and tiny gradients through the log, (ii) extremely dense bags at init, which made the optimization even harder. We simply subtract a constant from the logits before the ReLU - log(1 + ReLU(x − θ)) - which sinks the operating point so support starts sparse (this was done previously, e.g., [18]). The shift is softmax-invariant, so the backbone’s MLM geometry is left alone.

We set logit_shift = 15, chosen by probing a sample of MS MARCO documents (10–20 all worked; it was not that critical). Training immediately became more stable and more effective. Related observations on ModernBERT sparse models use scalar normalization instead [19]; we tried combining both, with no extra gain. We also suspected LateOn’s anisotropy and poked at that; nothing conclusive. The offset is deceptively simple - a dirty hack - but simple recipes that work are the ones you keep.

Per-position top-k: towards more controlled expansion (ingredient #2)

The shift already gives a sparser start. Inspecting bags (one underrated feature of sparse models: you can read them) still showed a lot of stuffing, mostly from bad expansion. In vanilla SPLADE every token may expand to any of the 50k vocab dims. That is more freedom than the problem needs, and a wide door into a bad local minimum. Does a single token really need 100–200 expansions?

By restricting the space on which a token can perform expansion, the optimisation problem becomes again simpler (a few dims per token to optimize, less constraint on sparsity). We keep only each token’s k largest vocab dims before pooling. Unlike the pooled top-k we tried earlier, this does not cap the vector’s L0; it caps per-token fan-out. The bag is a union of small synonym neighborhoods, not a union of 50k-way MLM tails. This design implies a relevant expansion that is not already in the top-k may never recover. In practice: (i) if the MLM ranking is decent, that is rare; (ii) even then, surfacing it while fighting sparsity would be hard. Note that, interestingly, it also echoes what is done for sparse embeddings based on SAEs, which resort on TopK-SAE that map each token vector to a finite set of K latents. In our case, k=12 offers a good trade-off, and we sticked to that, but a better operational point might exist.

We did not push further “controlled expansion” ideas. The goal was still a fair architecture comparison, not a new sparse design.

The case for casing (ingredient #3)

SPARSEUP’s tokenizer is byte-level BPE (Ġ…), not the uncased WordPiece SPLADE was built on. That makes the output space clumsier for lexical bags than for dense or late interaction. Cased forms of the same word are different ids - everything is roughly duplicated [20] - and space-marked vs bare forms (Ġheat vs heat) duplicate it again. This is not great for efficiency.

For instance, The word heat has four variants in ModernBERT’s vocab: heat, Heat, Ġheat, ĠHeat. A passage about heat often lights up all four. That is extra “capacity,” and maybe a small effectiveness lever, but it hurts sparsity and makes the model dumber than it looks.

To avoid this, we rely on a simple folding, which restricts the output vocabulary (from 50k to around 34k). We build a map of {token ⇒ canonical token} (in this specific example, Ġheat). We fold after pooling. Tokenizer ids that share the same lowercase body are grouped - case and Ġ/bare variants, using whatever surface forms the vocab actually has. The canonical id is the best member of that group: space-marked lowercase (Ġheat) if it exists, then bare lowercase, then any leftover cased form. Every sibling is remapped onto it. Rule: same word, different surface form → keep the largest weight, write it on one id. So {heat: 1.2, Heat: 3.6, Ġheat: 3.4} becomes {Ġheat: 3.6}. Query and document both matching “heat” hit one dim instead of splitting mass across case and word-start forms. The map is applied at train and at inference, so there is no mismatch, and bags get much sparser. More complex techniques have recently been explored to switch vocabulary for learned sparse retrieval (typically in the ModernBERT case) [21], but our “dirty workaround” actually turned out to work quite well.

Results

Let’s look at numbers.

Comparison against competitive sparse embeddings

Main vocabulary-based sparse retrievers on BEIR-13 (nDCG@10), taken from the respective papers. LACONIC is a 1B LLM sparse model (impressive results actually, but a different size class). All others are encodersized (<150M). Bold is best among the small models.

SPARSEUP reaches 56.4 average nDCG@10 on BEIR-13 - the first sparse encoder under 150M we know of to go above 56. Unlike splade-v3 or opensearch-neural-sparse-encoding-doc-v3-gte, it is trained with contrastive learning only, not cross-encoder distillation (but the training mixture and hard negatives are also stronger in our setting). The OpenSearch model is a doc encoder (no learned query encoder) and is already very strong; distillation on SPARSEUP would likely add a bit more. The clear miss is DBPedia, a set where sparse models usually do well - possibly ModernBERT’s more aggressive BPE on entity names.

Comparison in a controlled setting - the (un)expected generalization story

Late interaction, single-vector dense, and learned sparse, all other things equal. BEIR-13 (nDCG@10). SPARSEUP is reported with Seismic (approximate); the LightOn numbers are exact search. Bold is best.

The more interesting comparison is not “SPARSEUP vs whatever sparse model is in a paper,” but vs DenseOn and LateOn: same backbone family, same pre-training and fine-tuning data. Late interaction beating single-vector dense, especially OOD, matches recent evidence (DenseOn vs LateOn on decontaminated BEIR; mDenseOn vs mLateOn on new languages).

Different findings from earlier years also tend to show that sparse embeddings generalize better compared to dense retrieval [22], due in part to their better ability to perform lexical match on rare/unknown terms. We were convinced that the same type of behavior would emerge, in this very controlled train/eval setting. We did not get a clean version of it.

On BEIR-13, SPARSEUP is 1.52 points behind DenseOn (56.36 vs 57.88, without MS MARCO). It wins the argument sets (ArguAna, Touché) and beats DenseOn on HotpotQA, where overlap is largely lexical. It lags on more semantic sets; the FiQA gap is the largest.

Decontaminated BEIR looks closer (−0.17 vs DenseOn), but that average is has to be interpreted carefully: for instance, decontaminated NQ and MS MARCO have 21 and 46 queries, so swings there are noisy (and the gaps are particularly large on those). It is not the clear “sparse overfits less” result we had in mind. In this controlled setup, DenseOn is the stronger of the two comparable encoders. This might be due to various things (including the BPE vocabulary, or most likely suboptimal training dynamic for SPARSEUP, etc.). Nonetheless, SPARSEUP is still a different and cool alternative, with different failure modes, that can run very fast (see next section).

Same three systems on decontaminated BEIR (nDCG@10). Treat NQ and MS MARCO as low-sample. Bold is best.

Good and fast

Some recent sparse models barely discuss efficiency - including the average number of non-zeros, which can explode without guardrails. We could buy about 1–2 BEIR points by inflating query and document L0. This release is meant to stay actually sparse.

On MS MARCO, SPARSEUP averages 47 non-zeros per query and 190 per document (SPLADE-v3: 25 and 170). Queries are a bit denser; documents sit in the same band. DenseOn, our main comparison, is a dense 768-d vector on every query and document (but runs on a completely different infrastructure). Also note that all our evaluations are performed using Seismic [23], which already gives us the approximate performance for efficient latency regimes - while results for single vector dense embeddings are often reported with exact search.

The plot below is the efficiency picture: recall vs exact search against per-query latency on MS MARCO, single-threaded. SPARSEUP sits in the same latency regime as SPLADE-v3: >97% recall in about 380µs! We purposely do not include latency comparisons against dense retrieval (or even late interaction), which can also be fast, but runs on different infra and generally hardware. These numbers should be taken in absolute: sub-ms on MS MARCO is fast, period!

Inspecting the model

A vocabulary-based sparse vector is readable. In production, that can help debug failures. For training it is a goldmine: you can see which terms fired, not just a score, and use those observations to fix some weird behavior. Our HF page exposes that (encode_to_dict, render, highlight).

Inspecting bags

We took a diverse set of queries coming from eval sets, and observed two standard patterns showing up.

• Term match plus useful expansion. Query tokens sit at the top, then real retrieval terms: salary / £ / footballer / FIFA; quarterback / NFL / Tampa; COVID / coronavirus / lineage / evolutionary. That’s what the model is supposed to do.
• Over-association, weak entities. For instance, buccaneers ⇒ Cuban/Caribbean/Columbus; phylogenetic ⇒ pedigree/familial. Almost all of this is bad expansion. The same bug is louder on numbers: digit queries explode into unrelated dims (see the two examples below). Many issues remain, which all come from wrong expansion: there’s still work to do!

“But these examples look dumb, I don’t want to use the model!” They do. That is the point. You can see the corner cases and steer the model - per-position top-k came from reading bags like these. Dense encoders probably fail too; you just cannot read the failure and assume they are smart.

Model “attribution”

Max-pooling gives a free source map: each output dim is tied to the input subtoken that produced the max. That is a crude importance signal that has been unexplored (could it be used for instance for token pruning in late interaction?). Two documents below: highlighted spans are positions that won at least one output dim; intensity is the largest weight they won.

Conclusion

Linkup Research releases SPARSEUP, the sparse companion of DenseOn and LateOn. It is the strongest public vocabulary-based sparse retriever we know of at this size (<150M), and it sits close to a comparable dense model while staying actually sparse. It is a different tool - not a drop-in winner on every BEIR set - and a baseline we hope others will beat.

The model is released under Apache 2.0; use it!

References

• [1] DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search, Raphaël Sourty, Antoine Chaffin, Paulo Roberto Moura Junior, Amélie Chatelain
• [2] Match Your Words! A Study of Lexical Matching in Neural Information Retrieval, Thibault Formal, Benjamin Piwowarski, Stéphane Clinchant
• [3] SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking, Thibault Formal, Benjamin Piwowarski, Stéphane Clinchant
• [4] SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval, Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant
• [5] SPLADE-v3: New baselines for SPLADE, Carlos Lassance, Hervé Déjean, Thibault Formal, Stéphane Clinchant
• [6] Exploring ℓ0 Sparsification for Inference-free Sparse Retrievers, Xinjie Shen, Zhichao Geng, Yang Yang
• [7] Granite Embedding Models, Parul Awasthy, Aashka Trivedi, Yulong Li, Mihaela Bornea, David Cox, Abraham Daniels, Martin Franz, Gabe Goodhart, Bhavani Iyer, Vishwajeet Kumar, Luis Lastras, Scott McCarley, Rudra Murthy, Vignesh P, Sara Rosenthal, Salim Roukos, Jaydeep Sen, Sukriti Sharma, Avirup Sil, Kate Soule, Arafat Sultan, Radu Florian
• [8] LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum, Zhichao Xu, Shengyao Zhuang, Crystina Zhang, Xueguang Ma, Yijun Tian, Maitrey Mehta, Jimmy Lin, Vivek Srikumar
• [9] Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies, Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato
• [10] From Tokens to Concepts: Leveraging SAE for SPLADE, Yuxuan Zong, Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski
• [11] Learning Retrieval Models with Sparse Autoencoders, Thibault Formal, Maxime Louis, Hervé Dejean, Stéphane Clinchant
• [12] PLAID: An Efficient Engine for Late Interaction Retrieval, Keshav Santhanam, Omar Khattab, Christopher Potts, Matei Zaharia
• [13] ColBERTSaR: Sparsified ColBERT Index via Product Quantization, Eugene Yang, Andrew Yates, Dawn Lawrie, James Mayfield, Saron Samuel, Rohan Jha
• [14] No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval, Lixuan Guo, Yifei Wang, Tiansheng Wen, Aosong Feng, Stefanie Jegelka, Chenyu You
• [15] From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective, Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant
• [16] SPLATE: Sparse Late Interaction Retrieval, Thibault Formal, Stéphane Clinchant, Hervé Déjean, Carlos Lassance
• [17] An Alternative to FLOPS Regularization to Effectively Productionize SPLADE-Doc, Aldo Porco, Dhruv Mehra, Igor Malioutov, Karthik Radhakrishnan, Moniba Keymanesh, Daniel Preoţiuc-Pietro, Sean MacAvaney, Pengxiang Cheng
• [18] Learning a Sparse Representation Model for Neural CLIR, Suraj Nair, Eugene Yang, Dawn J Lawrie, James Mayfield, Douglas W Oard
• [19] Rescaling MLM-Head for Neural Sparse Retrieval, Youngjoon Jang, Seongtae Hong, Jonah Turner, Heuiseok Lim
• [20] To Case or Not to Case: An Empirical Study in Learned Sparse Retrieval, Emmanouil Georgios Lionis, Jia-Huei Ju, Angelos Nalmpantis, Casper Thuis, Sean MacAvaney, Andrew Yates
• [21] Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps, Zhichao Geng, Yang Yang
• [22] Towards Effective and Efficient Sparse Neural Information Retrieval, Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant
• [23] Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations, Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, Rossano Venturini