Large Storage Pods for Everyone Else: The Evolution of 45 Drives

Over the last couple of months, everyone at 45 Drives has been totally immersed in work related to “launching” the new Storinator models that just went up on our site last Friday. I put quotes around “launch” because it’s not really a conventional product launch, because these new products have been built and sold for some time. In fact, just like the original large storage pods from Backblaze, they were built and used by clients of ours on a custom basis, and we’ve been quietly selling them in soft-launch for some time.

After our website went up late last week, we had some time to reflect on where we are, and we thought there were some interesting stories to tell about how large storage pods and the storage community created 45 Drives, and how we’ve been a conduit for the evolution of large storage pods.

I’m writing this because I believe that some will find this story relevant and interesting, and maybe inspire others to develop their ideas to reshape and refine their own areas of computing, just like our friends at Backblaze reshaped large storage.

Chapter One – How 45 Drives Came to Be

Back around 2002, my friend and now business partner Steve Lilley and I founded Protocase, 45 Drives’ parent company. Steve is a mechanical engineer who had worked his career in the electronics industry, eventually becoming VP Ops of a company that built electronics for marine navigation before working with me on various freelance technology projects. Protocase was born of the frustration that Steve, I and other colleagues experienced when we tried to buy custom electronic enclosures. The ‘engineering quantities’ that we needed weren’t of interest to the larger, more sophisticated players, and the small shops who would take our jobs would bump us to the back of the lineup if larger orders came in. The rest is history – we were correct assuming that others felt our pain, and we’ve built a wonderful niche business by building enclosures, sheet metal parts, and, most recently, machined parts, for scientists and engineers across North America and around the world.

One particularly fascinating customer was Backblaze. This Silicon Valley-based startup was in the process of refining and implementing their vision of reinventing storage, and was feeling the same pain that Steve and I felt. We were fortunate that they stumbled on us, beginning a long and productive relationship. Protocase’s design services and rapid mass-custom manufacturing allowed Tim Nufire and his staff to turn their ideas into physical reality. Through inspired design, hard work and multiple iterations, their thoughts were refined into the Large Storage Pod design that is the foundation of their business. (If you don’t know about them, check them out at www.backblaze.com, they sell unlimited backup for your computer for $5/mo. via an internet connection and a brilliant little utility that buzzes away in the background. It doesn’t slow your computer, and your data is always secure).

This story may have ended there, but the folks at Backblaze generously decided to share their view of what was missing in storage technology, why they developed their pod, and then open-source the design of the pod.  On top of that, they asked us if Protocase would be willing to sell enclosures to anyone who might wish to build a Backblaze Storage Pod. Saying yes brought us one step closer to 45 Drives.

Of course, in technology, nothing is ever as simple as it seems. We had a flurry of inquiries from potential self-builders, but less than 10% followed through and purchased. When we asked why, one answer was, “We can’t buy backplanes and other minor parts.” We then pondered whether we should get involved in specialty parts distribution. We took the leap, and found that this doubled the percentage of those who followed through and built pods. Both we and Backblaze were excited to see the beginnings of a real community.

But that still left almost 80% who were enthusiastic enough to communicate with us, but wouldn’t take the plunge, despite having compelling opportunities that these storage pods could address. Again we asked why, and the common thread was “I think this is the right direction, and worth the risk, but I’m a busy person and I just don’t have time to get into computer building projects.” We floated the idea of selling assembled pods, and got a great response. This was the genesis of 45 Drives.

Chapter Two – Backblaze, Volume Business Models, and the Storage Community

From the moment we began selling fully assembled storage pods, two things became clear to us:

1) The overall concept of the Backblaze storage pod really hit the mark

2) Backblaze’s needs were different from almost everyone else we’ve worked at 45 Drives

We saw right away that the pain felt by Backblaze in needing large storage was (and continues to be) shared by many others. However, over the years of working with them, we’ve also come to learn that Backblaze has needs that are distinct and highly specific. I think the driver is that fact that they focus on a high volume model. As of March 2015, they say they have 980 pods in service, with 44,000 hard drives, for a total of 150 Petabytes, and they have ambitions to exceed that in the future.

Here’s the math of volume: by the time they get to 10,000 pods, saving $100 per machine impacts their finances by $1,000,000. That’s enough money to justify spending a good bit of engineering time to value engineer their pod design.  And because they are doing a specific homogeneous task, they can also modify their software and processes to save money on hardware.

One Difference – Redundant Pods

One obvious example of Backblaze’s unique needs is their long-held strategy of addressing data security through redundancy of pods. Build the pods as inexpensively as you can, and design your software on a data centre level so that any one pod can fail, and others take up the slack. In addition to saving money, this lets them gain operational efficiencies by doing pod repair / rebuild as a scheduled batch process, rather than a much more expensive emergency basis.

Community Needs Diverge from Backblaze

Pretty much the rest of the community approaches things differently. Their technical strategies and economics mean single-machine reliability is important, and often critical. This is a key difference, which has major impacts on pod design choices. Here’s one small example: Backblaze designed the early pods to use a dual power supply. This was done to supply the huge peak amperage draw that occurs when spinning up 45 hard drives. One supply powered the motherboard and some storage drives, the other powered remaining drives. This worked well for Backblaze, but the rest of our customers pointed out that this was a problem. In addition to doubling the odds of a power supply failure event, it put those who used RAID arrays at great risk of data loss should the supply powering drives only fail during a write to a large array.   

The solution was simple: use a single supply. At the time, there was no suitable single non redundant supply on the market. However, there were suitable single redundant supplies with sufficient amperage. This had a cost impact of hundreds of dollars, so Backblaze was not interested. However, we developed a version for certain customers, and now, the vast majority of our customers choose redundant supplies, despite the added cost.

So it has become clear to us that Backblaze’s unique needs drive the development process in a direction that is different from the needs of most of the community that has been working with us. We must follow the wishes of our community, and our new Storinator designs are the result. 

We’ve reached a fork where 45 Drives is going in a different direction from Backblaze.  We still cooperate, and our Protocase division is actively working on new designs with them.

But our Storinator storage pods will be driven by our customer base now and in the future.

Read Doug’s Follow-Up Post: The Big Fork – Direct-Wired or Backplane?