DHH

One of the great tragedies of modern web development over the last five years or so has been the irrational exuberance for microservices. The idea that making a single great web application had simply become too hard, but if we broke that app up into many smaller apps, it’d all be much easier. Turned out, surprise-surprise, that it mostly wasn’t.

As Kelsey Hightower searingly put the fallacy: “We’re gonna break it up and somehow find the engineering discipline we never had in the first place”.

But it’s one of those hard lessons that nobody actually wants to hear. You don’t want to hear that the reason your monolith is a spaghetti monster is because you let it become that way, one commit at the time, due to weak habits, pressurized deadlines, or simply sheer lack of competence. No, what you want to hear is that none of that mess is your fault. That it was simply because of the oppressive monolithic architecture. And that, really, you’re just awesome, and if you take your dirty code and stick it into this new microservices tumbler, it’s going to come out sparking clean, smelling like fucking daffodils.

The great thing about such delusions is that they can keep you warm for quite a while. A yeah, sure, maybe the complexities of your new microservices monstrosity are plain as day right from the get go, but you can always excuse them with “it’s really going to pay off once we…” bullshit. And it’ll work! For a while. Because, who knows? Maybe this is better? But it’s not. And the day you have to really admit its not, you’re probably not even still there. On to the next thing.

Microservices as an architectural gold rush appealed to developers for the same reason TDD appeals to developers: it’s the pseudoscientific promise of a diet. The absolution of a new paradigm to wash away and forgive our sins. Who doesn’t want that?

Well, maybe you? Now after you’ve walked through the intellectual desert of a microservice approach to a problem that didn’t remotely warrant it (ie, almost all of them). Maybe now you’re ready to hear a different story. There’s a slot in your brain for a counterargument that just wasn’t there before.

So here’s the counterargument: Integrated systems are good. Integrated developers are good. Being able to wrap your mind around the whole application, and have developers who are able to make whole features, is good! The road to madness and despair lays in specialization and compartmentalization.

The galaxy brain takes it all in.

But of course, you cry, what if the system is too large to fit in my brain? Won’t it just swap and swap until I kernel failure? Yes, if you try to stick in a bloated beast of an application, sure.

So the work is to shrink the conceptual surface area of your application until it fits in a normal, but capable and competent, programmer’s brain. Using conceptual compression, sheer good code writing, a productive and succinct environment, using shortcuts and patterns. That’s the work.

But the payoff is glorious. Magnificent. SUBLIME. The magic of working on an integrated system together with integrated programmers is a line without limits, arbitrary boundaries, or sully gatekeepers.

Forget frontend or backend. The answer is all of it. At the same time. In the same mind.

This sounds impossible if you’ve cooked your noodle too long in the stew of modern astronautic abstractions. If you turn down the temperature, you’ll see that the web is actually much the same as it always was. Sure, a few expectations increased here, and a couple of breakthrough techniques appeared there, but fundamentally, it’s the same. What changed was us. And mostly not in ways for the better.

If your lived experience still haven’t hit the inevitable wall of defeat on the question of microservices, then be my guest, sit there with your folded arms and your smug pout. It’s ok. I get it. There’s not an open slot for this argument in your brain just yet. It’s ok. I’m patient! I’ll still be here in a couple of years when there’s room. And then I’ll send you a link to this article on twitter.

Peace. Love. Integration.

My name is David Heinemeier Hansson, and I’m the CTO and co-founder of Basecamp, a small internet company from Chicago that sells project-management and team-collaboration software.

When we launched our main service back in 2004, the internet provided a largely free, fair, and open marketplace. We could reach customers and provide them with our software without having to ask any technology company for permission or pay them for the privilege.

Today, this is practically no longer true. The internet has been colonized by a handful of big tech companies that wield their monopoly power without restraint. This power allow them to bully, extort, or, should they please, even destroy our business – unless we accept their often onerous, exploitive, and ever-changing terms and conditions.

Keep reading “Testimony before the House Antitrust Subcommittee”

Can you believe we used to willingly tell Google about every single visitor to basecamp.com by way of Google Analytics? Letting them collect every last byte of information possible through the spying eye of their tracking pixel. Ugh.

But 2020 isn’t 2010. Our naiveté around data, who captures it, and what they do with it has collectively been brought to shame. Most people now sit with basic understanding that using the internet leaves behind a data trail, and quite a few people have begun to question just how deep that trail should be, and who should have the right to follow it.

In this new world, it feels like an obligation to make sure we’re not aiding and abetting those who seek to exploit our data. Those who hoard every little clue in order to piece of together a puzzle that’ll ultimately reveal all our weakest points and moments, then sell that picture to the highest bidder.

The internet needs to know less about us, not more. Just because it’s possible to track someone doesn’t mean we should.

That’s the ethos we’re trying to live at Basecamp. It’s not a straight path. Two decades of just doing as you did takes a while to unwind. But we’re here for that work.

Keep reading “The last tracker was just removed from Basecamp.com”

We spend about $3 million every year to run all the versions of Basecamp and our legacy applications. That spend is spread across several on-premise data centers and cloud operations. It does not include the budget for our 7-person strong operations team, this is just the cost of connectivity, machines, power, and such.

There’s a lot of spend in that bucket. The biggest line item is the million dollars per year we spend storing 4.5 petabyte worth of files. We used to store these files ourselves, across three physical data centers for redundancy and availability, but the final math and operational hassle didn’t pan out. So now we’re just on S3 with a multi-region redundancy setup.

After that, it’s really a big mixed bag. We spend a lot of money on databases, which all run on MySQL. There’s ElasticSearch clusters that power our search. A swarm of Redis servers providing caching. There’s a Kafka pipeline and a Big Query backend for analytics. We have our own direct network connections between the data centers and the cloud.

Everything I’ve talked about so far is infrastructure we’d run and pay for regardless of our programming language or web framework. Whether we run on Python, PHP, Rust, Go, C++, or whatever, we’d still need databases, we’d still need search, we’d still need to store files.

So let’s talk about what we spend on our programming language and web framework. It’s about 15%. That’s the price for all our app and job servers. The machines that actually run Ruby on Rails. So against a $3 million budget, it’s about $450,000. That’s it.

Let’s imagine that there was some amazing technology that would let us do everything we’re doing with Ruby on Rails, but it was TWICE AS FAST! That would save us about ~$225,000 per year. We spend more money than that on the Xmas gift we give employees at Basecamp every year. And that’s if you could truly go twice as fast, and thus require half the machines, which is not an easy thing to do, despite what microbenchmarks might delude you into thinking.

Now imagine we found a true silver bullet. One where the compute spend could be reduced by an order of magnitude. So we’d save about $400,000/year, reducing everything we spend running our app and job servers to an unrealistically low $45,000/year. That reduction wouldn’t even pay for two developers at our average all-in cost at Basecamp!

Now let’s consider the cost of those savings. We spend more money on the 15-strong developer team at Basecamp than our entire operations budget! If we make that team just 15% less productive, it’ll cost us more than everything we spend to run Ruby and Rails at Basecamp!

Working with Ruby and Rails is a luxury, yes. Not every company pay their developers as well as we do at Basecamp, so maybe the rates would look a little different there. Maybe some companies are far more compute intensive to run their apps. But for most SaaS companies, they’re in exactly the same ballpark as we are. The slice of the total operations budget spent running the programming language and web framework that powers the app is a small minority of the overall cost.

For a company like Basecamp, you’d be mad to make your choice of programming language and web framework on anything but a determination of what’ll make your programmers the most motivated, happy, and productive. Whatever the cost, it’s worth it. It’s worth it on a pure cost/benefit, but, more importantly, it’s worth it in terms of human happiness and potential.

This is why we run Ruby. This is why we run Rails. It’s a complete bargain.

Apple’s stubborn four-year refusal to fix the terminally broken butterfly keyboard design led me to a crazy experiment last week: Giving Windows a try for the first time in twenty years.

Not really because I suddenly had some great curiosity about Windows, but because Apple’s infuriating failure to sell a reliable laptop reluctantly put me back in the market. So when I saw the praise heaped upon the Surface Laptop 3, and particularly its keyboard, I thought, fuck it, let’s give it a try!

The buying experience was great. There was nobody in the store, so with four sales people just standing around, I got immediate attention, and typed away a few quick sentences on the keyboard. It felt good. Nice travel, slim chassis, sleek design. SOLD!

The initial setup experience was another pleasant surprise. The Cortana-narrated process felt like someone from the Xbox team had done the design. Fresh, modern, fun, and reassuring. Apple could take some notes on that.

But ultimately we got to the meat of this experience, and unfortunately the first bite didn’t quite match the sizzle. The font rendering in Windows remains excruciatingly poor to my eyes. It just looks bad. It reminded me of my number one grief with Android back in the 5.0 or whenever days, before someone at Google decided to do font rendering right (these days it’s great!). Ugh.

I accept that this is a personal failure of sorts. The Windows font rendering does not prevent you from using the device. It’s not like you can’t read the text. It’s just that I don’t enjoy it, and I don’t want to. So that was strike one.

But hey, I didn’t pluck down close to $1800 (with taxes) for a Windows laptop just to be scared off by poor font rendering, right? No. So I persevered and started setting up my development environment.

See, the whole reason I thought Windows might be a suitable alternative for me was all the enthusiasm around Windows Linux Subsystem (WSL). Basically putting all the *nix tooling at your fingertips, like it is on OSX, in a way that doesn’t require crazy hoops.

But it’s just not there. The first version of WSL is marred with terrible file-system performance, and I got to feel that right away, when I spent eons checking out a git repository via GitHub for Windows. A 10-second operation on OSX took 5-6 minutes on Windows.

I initially thought that I had installed WSL2, which promises to be better in some ways (though worse in others), but to do so required me to essentially run an alpha version of Windows 10. Okay, that’s a little adventurous, but hey, whatever, this was an experiment after all. (Unfortunately WSL2 doesn’t do anything to speed up work happening across the Windows/Linux boundary, in fact, it just makes it worse! So you kinda have to stick with Linux tooling inside of Linux, Windows outside. Defeating much of the point for me!).

So anyway, here I am, hours into trying to setup this laptop to run *nix tooling with Windows applications, running on the bleeding edge of Windows, digging through all sorts of write-ups and tutorials, and I finally, sorta, kinda get it going. But it’s neither fast nor pleasant nor intuitive in any way. And it feels like my toes are so stubbed and bloody by the end of the walk that I almost forgot why I started on this journey in the first place.

I mean, one thing is the alpha-level of the software required to even pursue this. Something else is the bizarre gates that Microsoft erects along the way. Want to run Docker for Windows on your brand new Surface Laptop 3? Sorry, can’t do that without buying an upgrade to Windows Pro (the $1800 Surface Laptop 3 apparently wasn’t expensive enough to warrant that designation, so it ships with the Home edition. Okay, sheesh).

The default Edge browser that ships with Windows 10 is also just kinda terrible. I clocked a 38 on the Speedometer 2.0 test, compared to the 125 that my MacBook Pro 13 ran with Safari. (But hey, there’s another beta version of Edge, the one that now uses the Chrominum rendering engine, and that got it to a more respectable 68.)

Anyway, I started this experiment on a Monday. I kept going all the way through Friday. Using the laptop as I would any other computer for the internet, and my new hobby of dealing with the stubbed toes of setting up a *nix development environment, but when I got to Saturday I just… gave up. It’s clearly not that this couldn’t be done. You can absolutely setup a new Windows laptop today to do *nix style development. You can get your VS Code going, install a bunch of alpha software, and eventually you’ll get there.

But for me, this just wasn’t worth it. I kept looking for things I liked about Windows, and I kept realizing that I just fell back on rationalizations like “I guess this isn’t SO bad?”. The only thing I really liked was the hardware, and really, the key (ha!) thing there was that the keyboard just worked. It’s a good keyboard, but I don’t know if I’d go as far as “great”. (I still prefer travel, control, and feel of the freestanding Apple Magic Keyboard 2).

What this experiment taught me, though, was just how much I actually like OSX. How much satisfaction I derive from its font rendering. How lovely my code looks in TextMate 2. How easy it is to live that *nix developer life, while still using a computer where everything (well, except that fucking keyboard!) mostly just works.

So the Surface Laptop 3 is going back to Microsoft. Kudos to them for the 30-day no questions return policy, and double kudos for making it so easy to wipe the machine for return (again, another area where Apple could learn!).

Windows still clearly isn’t for me. And I wouldn’t recommend it to any of our developers at Basecamp. But I kinda do wish that more people actually do make the switch. Apple needs the competition. We need to feel like there are real alternatives that not only are technically possible, but a joy to use. We need Microsoft to keep improving, and having more frustrated Apple users cross over, point out the flaws, and iron out the kinks, well, that’s only going to help.

I would absolutely give Windows another try in a few years, but for now, I’m just feeling #blessed that 90% of my work happens on an iMac with that lovely scissor-keyed Magic Keyboard 2. It’s not a real solution for lots of people who work on the go, but if you do most of your development at a desk, I’d check it out. Or be brave, go with Windows, make it better, you pioneer, you. You’ll have my utter admiration!

Also, Apple, please just fix those fucking keyboards. Provide proper restitution for the people who bought your broken shit. Stop gaslighting us all with your nonsense that this is only affecting extremely few people. It’s not. The situation is an unmitigated disaster.

When it became clear to us last year that using SMS for two-factor authentication (2FA) was insecure, we kinda panicked. We’d spent a lot of time originally building that SMS-based 2FA login system for Basecamp, and the prospect of having to build an entirely new system compatible with proper authentication apps seemed daunting. Especially with major security liability hanging over our head.

So we went the easy route, and handed the 2FA authentication flow over to Google, using their Google Sign-In APIs. Now, that certainly gave us an immediate and secure solution. Nobody is disputing that Google knows security.

But requiring people to have a Google account to get a 2FA-protected Basecamp was an uncomfortable compromise. There are about a million good reasons for why you wouldn’t want Google to know everything about when you log into apps all over the internet. Google’s business is literally based on collecting as much data as possible, so it can use it all against you for ad targeting. That’s just not a regime we feel comfortable encouraging, let alone requiring.

So I’m thrilled to announce that we got our shit together and built our own, wonderful, and secure 2FA login protection for Basecamp. Google Sign-In still works, but it’s deprecated, and we’ll no longer be recommending it going forward.

Our new secure 2FA solution is built on the TOTP standard with backup codes as a fallback. So you can use any TOTP compatible authentication app, like Authy, 1Password, or Duo, and it works for all versions of Basecamp (here’s how to set it up in Basecamp 3 and Basecamp 2), as well as our legacy apps Highrise, Backpack, and Campfire.

Big kudos to Rosa Gutiérrez from our Security, Infrastructure & Performance team for putting our fears about doing our own TOTP-based 2FA system to shame. She led the project, did the work, and the final result is just great.

Finally, it feels good to have one additional area of the business free from Big Tech entanglement. We also dumped Google Analytics a few months back from Basecamp.com (relying on Clicky.com instead), and we’ll continue the work to untangle ourselves from Google and the rest of the industry behemoths. It’s a long slog, it’s unlikely ever to be fully complete, but every little bit helps.

Oh, and please, if you haven’t already, turn on 2FA to protect your Basecamp account. And if you aren’t already, use a password manager, like 1password. If you’re reusing a password on Basecamp, and you’re not protected by 2FA, you’re at a grave risk of having your account compromised. We work hard to protect everyone at Basecamp, but nothing will protect you online like using 2FA and a password manager everywhere you go.

When we launched Shape Up, we consciously didn’t want that book trapped behind a sleazy quid pro quo requirement for an email address. And now we’ve gone back to fix the mistake that was asking for one with Getting Real, our free ebook from 2006 about how to build a successful web application.

If you have a mailing list that’s worth signing up for, you don’t need to trick, cajole, or bribe people in other to get them on board. You only need to do that when you know that most people wouldn’t voluntarily join. That’s a pretty weirdly coercive play.

But that’s true of a lot of the industry BEST PRACTICES. There’s a whole cottage industry of bullshit around how CONTENT MARKETING is supposed to work. Ugh. Even just that word: CONTENT MARKETING. I can’t say without a slight gag.

If you have something to say, say it. If you have something to share, share it. Don’t invent things to say or to share just such that you can package up that pink slime as a golden nugget of truth to trade for someone’s contact information.

That’s the same insincere, manipulative logic behind influencer marketing. It’s all about disguising the sale with a thin, flimsy layer of purchased credibility. No wonder we’re all so skeptical and cynical these days. Because we have a million good A/B-optimized reasons to be.

Not everything needs to be tracked. Not everything needs to pay off. It’s perfectly fine to do things because it’s fun, feels good, is interesting, tickles your brain, or just helps someone out.

Enjoy Getting Real! Keep your email address in your pocket.

When Mike Davidson blew the lid off the invasive and appealing read receipts in a new personal email client called Superhuman, it brought about a full discussion of email tracking in general. At Basecamp, this lead to the conclusion that we wanted nothing to do with such tracking.

It wasn’t like we were doing anything as nefarious as those nasty Superhuman trackers, but still, we used the default settings in our mailing list software, which aggregates open rates, and had our own diagnostics tracker, to provide debugging insight for support.

But neither of those two use cases felt compelling enough to justify tracking everyone’s emails all the time. Reading an email shouldn’t leave a long data trail, regardless of whether that trail is used in fairly innocuous ways, like an aggregate open-rate calculation, or in its most devious, like spying on whether a personal email has been seen and from where.

So we killed the diagnostics tracking and turned off the mailing list tracking too. Now the only “tracking” that emails from Basecamp will do is to mark the message you’re seeing in your email as read within the application (and only if you’re a registered user). A feature in service of the recipient, not the sender, not us.

The tech industry has been so used to capturing whatever data it could for so long that it has almost forgotten to ask whether it should. But that question is finally being asked. And the answer is obvious: This gluttonous collection of data must stop.

So we keep taking the steps at Basecamp to examine our use of data, stop collecting it unless its strictly necessary in service of customers, and cut down all the ways we may be sharing it with others (dumping Google Analytics from our marketing pages is next!).

Privacy isn’t just the right thing to do, it’s also better business. Discerning customers are already demanding it, and everyone else will too soon enough.

The internet is hurting. It’s been colonized and exploited by a small cabal of tech companies. It’s taken most of us a while catch up to the gravity of the situation, but grave it is.

Yet we all sit with the power to ease the pain, even if we can’t cure it in an instant. You don’t have to go cold turkey on everything Big Tech. That’s almost impossible at the moment. Such is the stronghold. But every little bit helps.

So does the perspective that an alternative doesn’t have to give you 100% of what you were getting to be worth the switch. Yeah, so DuckDuckGo might not match Google on every search, but it’s more than good enough to be good enough most of the time.

The world is full of alternatives to the Big Tech offerings that give you 95% of the utility for 0% of the regret. But if you can’t even be bothered to give up 5% to help an alternative along, you also can’t be surprised when the alternatives are so few and far between.

You can heal the internet. One choice – one search! – at the time.

We ask a lot of job applicants at Basecamp. First, they have to make it through a long, detailed description of the opening. Then we request a dedicated cover letter that’s unique to Basecamp. And then there’s the polishing of a CV. It can easily take hours to apply to a job at Basecamp.

It used to be even worse too! For programmers, for example, we’d ask everyone upfront to submit code samples as part of their first application. Finding a good piece of code, either through open source, learning apps, or whatever, takes time. (Thankfully we don’t do that any more. You have to make it to the second round for us to talk code.)

So it’s not an unreasonable expectation to hope for some detailed feedback if the application isn’t successful. It’s entirely human to wish for feedback on “why didn’t I progress in process?”, “what could I have done better?”, “what else were y’all looking for?”.

And yet the math simply makes that impossible for us. A recent opening for a senior programmer drew over 400 applications (for customer support, we’ve seen over 1,000 applications per opening!). For that programmer role, someone spending just 20 minutes giving diligent feedback per application would keep them busy for almost 7 weeks, if they worked on that for 4 hours per day, 5 days a week!

That really sucks! It’s easily the worst part of hiring to reject hundreds of applicants who’ve put in a lot of their time with a generic “thank you for applying, but unfortunately we’ve moved forward with other candidates!”. Ugh.

So the least we can do is to be honest that this is the process. And, evidenced by the feedback from several applicants in the last hiring thaw, we clearly failed at that. So apologies to everyone who had reasonable expectations of some actionable feedback, and were left cold with a generic rejection.

Next time we’ll spell it out in the post. And applicants can make a more informed decision as to whether it’s worth their time given the risk that the sum of a response might well boil down to “thank you so much, but sorry!”.

Hiring is hard.