DHH

Wouldn’t it be great if there really was just one secret you had to know, and all your professional or entrepreneurial dreams would come true? Then you wouldn’t have to bother trying what works for your or your domain. You’d just have to apply The Secret, and voila!

Needless to say, if there is such a unifying secret, I haven’t found or heard of it. And yet, I keep seeing this false hope powering one of the most common questions I get in interviews: What’s THE ONE THING that you’d tell your younger yourself/other entrepreneurs/new programmers?! This is followed by the breathless anticipation of whatever profound wisdom the questioner most wish they could glean from my life’s experiences.

The boring truth is that the big leaps are all the result of an interwoven tapestry of practices, each contributing a strand of progress or insight. If you pull just one, all you have is that thin, disconnected strand. It’s only together the colors come alive.

That’s why all the books Jason and I have written together really are just a collection of essays. Yes, there’s a theme, but no single, unlocking narrative. REWORK is 88 separate essays, It Doesn’t Have To Be Crazy At Work has 66.

As long as you’re stuck on a quest for that one super-power practice or North Star principle, you’re not going to make space in your brain for the fact that no individual secret is going to make the difference. Only compound wisdom will.

Apple keep insisting that only a “small number of customers have problems” with the MacBook keyboards. That’s bollocks. This is a huge issue, it’s getting worse not better, and Apple is missing the forest for the trees.

The fact is that many people simply do not contact Apple when their MacBook keyboards fail. They just live with an S key that stutters or a spacebar that intermittently gives double. Or they just start using an external keyboard. Apple never sees these cases, so it never counts in their statistics.

So here’s some anecdata for Apple. I sampled the people at Basecamp. Out of the 47 people using MacBooks at the company, a staggering 30% are dealing with keyboard issues right now!! And that’s just the people dealing with current keyboard issues. If you include all the people who used to have issues, but went through a repair or replacement process, the number would be even higher.

Worth noting here is that the 3rd generation membrane keyboard did nothing to fix the issues. Six out of thirteen – nearly half!! – of the 2018+ MacBooks we have at the company have a failed keyboard.

I backed up those figures with a Twitter poll that has over 7,000 respondents already. That’s a 63% failure rate!! But Apple is only seeing 11% of those, as the vast majority of customers are simply just living with their broken computer:

This is a disaster. A complete unmitigated disaster.

But as always, in a time of crisis, the event itself is less indicative of the health of a company than the response. Is Apple going to accept that they’re currently alienating and undermining decades of goodwill by shipping broken computers in mass quantities?

From 2:13am GMT March 13 / 9:13pm Central March 12 until around 4:10am GMT / 11:10pm Central, Basecamp 3 was mostly offline and Basecamp 2 unable to process file uploads and downloads, as our cloud storage provider had a severe, sustained outage.  We continued to have minor disruptions in service from 4:10am GMT / 11:10pm Central until everything was cleared at 6:53am GMT / 1:53am Central.

This is the second time in a week that I’m forced to write “I’m so sorry”. That’s incredibly painful. Both because it’s because we’re failing our customers for the second time in a week, but also because it’s showing us just how unprepared we’ve been as an organization to deal with these cloud challenges, despite our belief otherwise.

I’m not going to bother you with platitudes about “lessons to be learnt”, because I’ve already done that just a few days ago. This goes much deeper than just a few lessons. It has called into question our entire risk management and operational structure at Basecamp.

It’s also been a mighty fall. From reaching for 99.999% in uptime – the hallowed five nines! – we’re now scrambling for two of them. From riches of reliance to rags of shambles. To say this is humbling is an epic understatement.

We’re stopping all major product development at Basecamp for the moment, and dedicating all our attention to fixing these single points of failure that the recent cloud outages have revealed. We’re also going to pull back from our big migration to the cloud for a while, until we’re able to comfortably commit to a multi-region, multi-provider setup that’s more resilient against these outages.

I’m sorry. I’m really sorry (and ashamed).

Keep reading “Basecamp outage: When it rains, it pours”

From 4:30am GMT March 7 / 10:30pm Central March 6 until 1:02pm GMT / 7:02am Central, Basecamp 2 and the search feature in Basecamp 3 were mostly offline due to a catastrophic network failure with our cloud provider. Both our primary network link, our backup network link, and several additional ad-hoc network links between critical services needed to run Basecamp 2 were forced offline, as the cloud provider sought to deal with underlying network problems they were having.

Both Basecamp 2 and the search feature in Basecamp 3 are now fully back online.

But this was one of the worst outages we’ve had in the history of Basecamp. We’re incredibly sorry about just how long and broad of an interruption this caused, especially for our European customers of Basecamp 2. We’re so very sorry about this. We know this caused real and deep interruption to many people’s workflow from the early morning to the early afternoon on the main European timezones.  And of course to any other customers around the world, including the US, who were also affected.

We’ve learned some hard lessons about network availability, the limitations of redundant, and double redundant backup connections. We’ll be working diligently to change how we work with cloud providers in the future, and how we can insulate ourselves and our customers from any future incidents like this. While this incident may have been triggered by network issues outside of our immediate control, it’s always within our control how we architect our systems, how we prepare for disasters, and how we ensure something like this never has the power to inflict such a traumatic outage.

So I want to make absolutely clear that this is our failure. Even in this new world of cloud services, it’s still always our fault when Basecamp isn’t available. Whatever the underlying problem for an outage, there’s always something you could have done to prevent it. And our list in this case includes a number of both obvious and not-so-obvious steps we could have taken. We will now take them.

Once again, I’m deeply sorry for this terrible outage. We will work as diligently we can to ensure that this doesn’t happen to any version of Basecamp again, neither past or present. Thank you for understanding, thank you for your patience, and thank you for being a customer, even if you with all justification ran out of both understanding and patience during this utterly unacceptable outage.

Keep reading “Basecamp 2 and Basecamp 3 search outage report”

There are a lot of hoops to jump and obstacles to climb before starting a new business, but a lack of an all-burning passion for the pursuit shouldn’t be one of them. Yes, it’s easier to keep going when you like what you do, but it’s by no means a requirement to profess your love for the endeavor.

I know it often comes from a good place, this advice. That you should just wait until that magic idea comes along. Not be the fool rushing in. But this romantic idea that there’s the perfect opportunity just waiting out there for you to discover it is a mirage.

Most of business, most of the time, is pretty mundane! I’m not still working at Basecamp because, after nearly twenty years, I just spring out of bed every morning yearning to improve todos, events, messages, or project management in general. I like all those things, but the domain itself isn’t a burning bush of passion.

What working on Basecamp allows me to do is keep at the motions I most enjoy: Writing Ruby, sharing lessons and experiences, building a calm workplace, and being fair in dealings with customers. Those aren’t the only things I enjoy in life, but they’re definitely on the high list.

Thing is. I could have pursued all those things in a different domain than, say, helping businesses cope with growth and putting projects in order at Basecamp. In fact, I have! We’ve made quite a lot of applications at Basecamp over the years. Many related to a similar mission as Basecamp, but not all.

There are lots of reasons for why you’d want to start and run a business. Passion isn’t a permit you need to acquire before setting off.

Yesterday at 12:45pm central time, our ops team detected a dramatic spike in login requests to Basecamp. More than 30,000 login attempts were made in the hour that followed from a wide array of IP addresses. Our first line of defense was to block the offending addresses, but ultimately we needed to enable captcha to stop the attack.

After the attack was over, we diagnosed that 124 accounts had unauthorized access from the attack. We immediately reset the password for these accounts, logging out any intruders, and emailed the affected account holders with all the relevant information.

All of the unauthorized access was gained using the correct username and password for the account. It’s highly likely that these credentials were obtained from one of the big breaches, like those collected in combos like Collection #1, Anti Public, or Exploit.in. All the affected accounts showed as “owned” on haveibeenpwned.com.

Our preliminary investigation shows that none of the unauthorized access actually performed any actions within the accounts. It seemed like the attack focused on first validating which accounts were vulnerable, perhaps with a plan to later exploit these vulnerable accounts. Thankfully we were able to detect and stop the attack very quickly, and also ensure that any intruders were prevented further access.

Never the less, this is a serious reminder that you should never share the same password between multiple services. Particularly services such as Basecamp that may contain sensitive information. Here’s what we recommend you do to stay safe:

1.) Use a password manager to ensure you’re using different, secure passwords on every service you use. Then if one service is breached, you don’t have to worry about the rest. We use 1Password at Basecamp and recommend it.

2.) Subscribe to a breach notification service, like the one offered by haveibeenpwned.com. Then you’ll be alerted if your credentials are part of hack known to the public.

3.) Turn on two-factor authentication (2FA) wherever you can! We offer 2FA protection for Basecamp using Google Sign-In. Most services that deal with sensitive information offer 2FA these days. It’s especially important that you enable this for critical services, like your email address.

Our ops team will continue to monitor and fight any future attacks. They did an excellent job detecting and addressing this particular attack. But if someone has your username and password, and you don’t have 2FA protection, there are limits to how effective this protection can be.

Protecting yourself against attacks like this is important. Take the time to learn the basics, and take the steps outlined above to limit the risk.

Update: On January 31st, the mass-attack resumed in much greater strength than before. More than 5,000 IP addresses were used to test stolen credentials. 89 proven correct, but no content was accessed on these accounts, and we followed the same procedure of resetting all logins and writing the people affected. We’ve since beefed up our CAPTCHA protection across all applications and all clients, which has been effective at stopping the attack. CAPTCHA isn’t perfect, and some times it’s annoying, but it has provided effective protection against this wave of attack. We continue to work on shoring up defenses, but do follow the steps outlined above to protect yourself!

Since our company is itself our most important product, we keep tweaking, experimenting, and – hopefully! – improving the organizational software that makes it run.

Here’s an example of how we refine our process at Basecamp:

That change then got codified as an update to our How We Work guide in the Basecamp Employee Handbook.

We received over 700 applications to the opening for a data analyst at Basecamp last Fall. There were a lot of qualified candidates, but in the end it was Jane Yang who bubbled to the top, and this month we were thrilled to welcome her to the team.

It’s probably not a big secret that we don’t hire often at Basecamp. In fact, we have zero open positions at the moment, and we’re still happy with our self-imposed hiring freeze too. So when we do hire someone, it’s kinda a big deal for us.

That’s also why we’ve put such a dedicated effort into attracting a much broader pool of applicants than we did in the early years of the company. When people of different backgrounds and perspectives work together, we make better decisions and better software.

Jane’s background for the last several years has been in the non-profit sector, doing strategy & research at One Acre Fund and analytics at IFF. She worked for many years in Nairobi, Kenya, but has now returned to the US. Before that, she worked at Deloitte Consulting and graduated from Princeton.

This is a perfect background for us as we’re getting more serious about charity at Basecamp recently. In addition to matching employee donations, we’re also working on a project inspired by Patagonia’s 1% for the Planet pledge. Jane brings a lot of experience from the non-profit world to give us another boost on that account.

Because, as we wrote in the job opening, running the numbers at Basecamp, as Jane will be doing, is not about squeezing every metric until it squeals more, more, MORE. Data informs us, educates us, but it does not run us. We make decisions at Basecamp on a much broader base than just “shareholder value”.

We already have enough. We’re not chasing exponential growth. There’s no quest to dominate, capture, or own everything and everyone. Data analysis is here to help us get better, wiser, kinder. Not greedier, not more extractive.

With all that: Welcome Jane!

The web isn’t just another software platform. It’s the greatest software platform the world has ever seen. And yet even in its obvious glory, we’re still learning how to be grateful for all its constituent parts. Take View Source, for example.

I owe much of my career to View Source. It’s what got me started with web development in the first place. Going to sites that I liked, learning how they did what they did. Yes, I also bought a bunch of animal books from O’Reilly, and I read WIRED’s Webmonkey, and the web was full of tutorials even then. But it’s not the same. Seeing how something real is built puts the individual pieces of the puzzle together in a way that sample code or abstract lessons just don’t.

I’m clearly not alone in this story. Jason learned HTML the same way. Lots of people on the internet owe their formative steps to the marvelous wonder that is View Source.

Unfortunately View Source has been receding in recent years. Building stuff for the web has never been more complicated. And few of these new tools, frameworks, or techniques have seemed to prioritize making the web readable through View Source. That’s a real shame, because progress needn’t be the enemy of learning.

Keep reading “Paying tribute to the web with View Source”

During the dotcom boom back in the late 90s, I did a bunch of Photoshop-cut jobs. You know, where a designer throws a PSD file over the wall to an HTML monkey to slice and dice. It was miserable.

These mock designs almost always focused on pixel perfectness, which meant trying to bend and twist the web to make it so. Spacer pixels, remember those? We were trying to make the raw materials of the web, particularly HTML, then latter CSS, do things they didn’t want to do. Things they weren’t meant to do.

Then I got the pleasure of working with designers who actually knew HTML and CSS. It was a revelation. Not only would the designs feel like they were of the web, not merely put on the web, but they’d always be better. Less about what it looked like, more about what it worked like.

I attribute this in no small part to the fact that it was real. The feedback loop of working with the actual HTML/CSS, as it was destined to be deployed, gave designers the feedback from the real world to make it better. And the fact that designers had the power to do the work themselves meant that the feedback loop was shorter. It wasn’t make a change, ask someone else to implement the change, ponder its effectiveness, and then repeat. It was change, check, change, repeat.

For a while that felt like it was almost the norm. That web designers confined to the illusions of Photoshop mocks were becoming more rare. And that web designers were getting better at working with their materials.

But as The Great Divide points out, regression is lurking, because the industry is making it too hard to work directly with the web. The towering demands inherent in certain ways of working with JavaScript are rightfully scaring some designers off from implementing their ideas at all. That’s a travesty.

Keep reading “Designing for the web ought to mean making HTML and CSS”