Photo by Mark Fletcher-Brown on Unsplash

The ML Hardware Startup Dilemma

The ML hardware startup industry has flourished over the last 5 years, and the reason is simple: ML programs are computationally intensive and, in many cases, traditional hardware such as a CPU isn’t the most suitable choice to run them. A CPU is like a Swiss Army knife — it can run various types of programs with decent performance, whereas for ML, you need hardware that’s great at running only a few types of programs— a german steak knife if you will. As it turns out, it’s actually relatively easy to build hardware that runs only a few types of programs, and for hardware experts, this is great news. Consequently, following inflection point of ML adoption, ML hardware startups flourished too.

I believe the last 5 years were a show-off-your-cool-new-hardware phase, in which startups just built the most powerful hardware possible and publicized exactly that. Effectively, they pushed the envelope on the best possible performance a.k.a. roofline performance numbers of the ML hardware. For a purebred engineer cofounder, this is an exciting phase, because the bigger those roofline numbers the more likely that a marquee investor will heed their startup. But these roofline numbers are only one piece of the puzzle and I’d argue, not the most important one. Running ML applications requires getting several pieces of the ML software stack right.

A bird’s eye view of the ML software stack

Now, these startups are transitioning into what I call an is-your-hardware-any-useful phase, in which the most important driver of success is going to be the usability of the product. One can show off all the performance numbers they want, but survival of the ML hardware will depend on

1. What percent of those roofline numbers are actually reproducible for real-world applications (not just benchmarks) and
2. How much effort is it for an average user to use and potentially benefit from the hardware

A successful transition from a cool technology to a useful product requires getting the unsexy aspects right — building tools and debugging capabilities, writing compilers that can interface with various frameworks, documentation, and more. Just as the german steak knife is only as good as the person using it, so too is the hardware only as good as the software, specifically the compiler, instructing it.

Unfortunately, there are still many hardware startups prioritizing just the hardware part, and software is usually an afterthought. Some don’t even get to it. Hardware development is fundamentally different than software development in ways mentioned here, so these startups often hope that a large company with virtually infinite resources will acquire them for their hardware magic sauce and take care of the software part. A few do succeed in this acquisition play, but owing to the sheer number of hardware startups in this space, most will need to realize that the real differentiator for hardware is the software.