Corpus File Format

Mon, Aug 29, 2016 One of the challenges in making BitFunnel relevant to the open source community is removing Bing-specific functionality that has deep dependencies on the internals of the rest of the Bing web crawling and index serving infrastructure. As I mentioned in my first post, we plan to start with an empty repository and bring over BitFunnel modules one by one. We are essentially bootstrapping the BitFunnel project, and this process will require a new test corpus, a set of performance benchmarks, and some system to help verify correctness. (read more...)

On the Road to Open Source

Mon, Aug 29, 2016 Today we are kicking off an effort to open source BitFunnel, a key part of Bing. BitFunnel is a library for high-performance full-text search over a chunk of the internet, spread across thousands of machines. It is based on a probabilistic algorithm that identifies and ranks documents according to queries involving keywords, phrases, and mathematical expressions. BitFunnel is the most significant accomplishment in my 35 years building software. I am proud of the system we created and want to share it with the world. (read more...)

Debugging unfamiliar code

Fri, Apr 15, 2016 This is a story about debugging a register allocation issue in a compiler. I recently joined the Bitfunnel project; there wasn’t much documentation at the time, so I started playing with the code by writing some simple examples that could later serve as documentation to anyone else coming in after me. The first thing I tried this for was NativeJIT, both because it’s self-contained and because it’s the first thing we open sourced. (read more...)