What can you actually do to reduce the threat of hacks like xz?
This week, the entire software development ecosystem was thrown for a loop with the discovery of a hack that was both unprecedented in its depth and sophistication, and also… shockingly simple. The core vulnerability that was exploited was the humanity and fatigue of those maintaining a popular open source library. Maintainers are the most vital, and least renewable, resource in the open source community.
It will undoubtedly be a long time until we have a full and final diagnosis of all the things that went wrong, or that had to happen in just the wrong way, for such a wide-ranging and severe vulnerability to pop up. But truthfully, we already know that a fundamental cause of this situation is that a vitally important library was being maintained by a small team of under-resourced people tasked with an impossibly large set of responsibilities. And we know that’s the story because, well, that’s the story of a vast swath of the entire open source world.
That’s the thing we have to fix. Until we do, all the technological or process solutions in the world won’t help. At Fastly, we’re not (yet!) one of the trillion-dollar titans of the industry, and they can probably come up with bigger and more ambitious ways to solve the problem once and for all. But we all have to do our part, so we’re emphasizing exactly what we’re going to do to try to help reduce the risk of these kinds of exploits and vulnerabilities.
And the reason you’ll know we’re committed to doing that work is because we’ve been doing that work.
Giving where it counts
We talk to a lot of maintainers of open source projects, including many who are responsible for some of the most popular open source projects that have ever existed. And no matter what language they code in, what country they live in, what community they serve, what part of the stack they contribute to, they all come back to a few key points that we hear over and over:
The emotional and personal costs of maintaining significant open source projects are extremely high.
Most people who do this kind of maintenance work not only don’t work for one of the giant tech companies, but they’ve chosen to go without the kind of salary that comes with that kind of work so they can dedicate their time to their open source projects.
The negative comments, uncharitable attitudes, and unplanned late nights of open source wear people down; fatigue is never because of a valid bug report or a well-intentioned contributor who makes an honest mistake.
Kind words and occasional community efforts at organizing a campaign to support a project are meaningful and appreciated but don’t do much to undo the daily grind of thankless chores that can wear anyone down.
As a result, we took a look at ways that we could use our skills and our platform at Fastly to substantively improve the life of people who make open source, and the open internet, possible. Let’s say that again because it bears emphasis: the goal is to support the people who make open source possible. If we do that, then anything is possible.
That sounds nice as a generality, but what does it mean in practice? A few simple principles we’ve used in creating and supporting Fast Forward, our open source program at Fastly:
First, show up. We strive to talk to, connect with, and be in a community with open source maintainers. We build open communication channels and (non-transactional) relationships that all of us can count on when times get tough.
Pull requests. We contribute, in time and in code, to the projects that we rely on and support. We work to make sure everyone in our organization knows they are not only permitted to, but encouraged to, be good citizens of the open source communities they participate in.
And finally: A substantial, documented, economic commitment to structurally supporting open source. There’s an old saying, “Show me your budget, and I’ll tell you what you value.” and it applies here, too. At Fastly, our budget for direct platform support for open source projects is $50 million dollars — publicly committed. And that’s not the end of it, that’s the beginning.
What does this look like out in the real world? Let’s take a look at what Hannah Aubry, who leads our Fast Forward program, said in her keynote address at the recent Rust Nation conference.
As just one example, look at what Joel Marcey of the Rust Foundation said about Fastly’s support for their vital work: “It's partnerships like these that will contribute to the growth and sustainability of Rust into the future.” We want every open source project to feel that level of support and stability.
What’s next
All of this work is, of course, just a start. And it’s only one small part of the much larger effort we all need to put into supporting the open source ecosystem — and especially supporting the people who make the open source ecosystem thrive. We’ll have much more to talk about in the future, but right now, today, if you’re a maintainer of an open source project that’s worried about scaling the delivery or security of your infrastructure, get in touch and we’ll see if membership in our Fast Forward program can help you.
