Shrink Your Embeddings: Slashing Costs with MRL and BQL

Let's face it: vector embeddings are fantastic for many tasks, but if you've ever worked with large-scale vector search, you know the pain of watching your storage and compute costs skyrocket. But what if I told you there's a way to put your embeddings on a diet without sacrificing too much of their performance?

It's soon 2025, and I'm still seeing AI developers using chunky, fat float embeddings like it’s 2020. It's like using an expensive golden hammer when a simple toy hammer would give you the same results—you're burning compute and storage for minimal accuracy gain. 

The Fat Float Embedding Dilemma

Vector search is powering everything from recommendation systems to semantic search. And with that comes a tsunami of data and ballooning costs. So why aren't more developers jumping on the embedding compression bandwagon? My guess? They either need to learn these techniques or think it's too complex to implement. And if you're still on the fence, let me show you what you're missing out on.

The power of compact embedding representations

Two techniques made waves in the world of embeddings and vector search in 2024:

1. Matryoshka Representation Learning (MRL)

2. Binary Quantization Learning (BQL)

These aren't just fancy acronyms – they're the path to slimmer, more efficient embedding vector representations. Let's break them down.

Matryoshka Representation Learning (MRL)

Think of MRL like those Russian nesting dolls (yes, that's where the name comes from). Instead of one fixed-size embedding, you get a hierarchy. Want the best possible accuracy? Use all the dimensions. Need something lighter for a small percentage drop in accuracy? Just grab the first 100 or so. 

Key benefits:

• Flexibility in choosing the number of dimensions

• Linear reduction in computational complexity

• Maintains good performance with fewer dimensions than all

Binary Quantization Learning (BQL)

If MRL is about selectively downsizing dimensions, BQL is about compressing the dimension values. It turns your float vectors into binary—we're talking just 0s and 1s. 

Key benefits:

• Massive reduction in storage requirements (32x compared to float vectors)

• Significantly faster similarity computations using Hamming distance (20s faster)

• Enables efficient scaling of vector datasets

Combining MRL and BQL

Combine these two techniques, and you've got a powerhouse of efficiency. The two techniques are compatible; first, MRL is performed, and then BQL is used to reduce the precision. MRL allows flexibility in the number of dimensions, while BQL provides flexibility in the precision per dimension. Perfectly compatible. 

Storage savings: A concrete example

Let's put some numbers to this. Imagine you're storing 1 billion 1024-dimensional vectors:

That's right – you're looking at cost savings of up to 64x. 

Search performance gains

These compact representations aren't just easier on your storage footprint; they're a boost for similarity searchers, too:

• Hamming distance computations (used for binary vectors) are about 20 times faster than float dot products

• This speed-up allows for higher query throughput or reduced CPU costs

In other words, you're saving money and cutting down latency by 20x, or you need 20x less CPU to get the job done.  

Implementing in Vespa

Now, if you're using Vespa (and if you're not, you might want to consider it), you're in luck. Vespa supports both MRL and BQL in its native Hugging Face embedder. Here's a quick taste of what that looks like:

This little snippet creates a 64-dimensional binary embedding, combining MRL and BQL. Read more in this long blog post on MRL and BQL. 

Real-world impact

So, what does all this mean in the real world? By shrinking your fat float embeddings, you can:

1. Reduce storage costs dramatically

2. Increase query throughput and lower latency 

3. Enable new use cases that were previously too expensive

4. Implement more complex ranking pipelines within latency constraints

Yes, You're not just slashing costs and improving performance but opening up new possibilities to embed more data. 

Conclusion

Look, I get it. Changing your embedding strategy might seem like a hassle. But if you ask me, the cost reduction is worth it. By leveraging techniques like MRL and BQL, you're not just trimming the fat float but unlocking new use cases.

So go ahead and shrink those embeddings. Your systems (and your budget) will thank you. And hey, you might just be the one to show your team how to save a boatload of cash next year.