This is the engineering diary for BitFunnel, an experimental open-source information retrieval system.

The goal of this site is to explain how BitFunnel works. We will cover the math and science in order to lay down a theoretical foundation, but to understand the algorithm in practice, it helps to see real, working code. For this reason, we have begun a new open source project which is a first attempt at translating the core ideas of BitFunnel into a general purpose library that can be used outside of the Bing data centers.

We’ve only managed to bring over a small portion of the algorithm and it will be a while before the new system can actually ingest documents and process queries. Most days the code compiles on Windows, OSX, and Linux, but it is still in an incomplete state with a lot of rough edges and very little documentation.

We’re making this early code available in the spirit of open development and our plan is to continue bringing over pieces of code, while using this web site to document our engineering decisions and experiments.

If this is your first time visiting, check out the first couple of blog posts and browse our repositories.