Our interview with Geoff Harcourt, CTO of CommonLit, about creating a culture where coding bootcamp grads don’t have to fear mistakes.
When I first heard Geoff’s voice over Zoom, I wasn’t expecting the wealth of coding knowledge he was about to share. He’d started his career as a product manager, and only learned to code later on. But, we quickly learned that Geoff’s late introduction to coding is part of what makes him a great CTO.
Geoff knows what it’s like to jump into a new software product as a new developer. During our call, he called it “parachuting.” Consumer-facing software can contain hundreds of thousands of lines of code and perform many different functions. For graduates of a three-month coding bootcamp, it can feel like plummeting into unfamiliar terrain.
As we wrote in Part I, Geoff’s team of 7 doesn’t include a single computer science major. Only one member of the team is a lifelong coder. But, CommonLit’s users need a product that works well and doesn’t have unexpected downtime. The team has to work fast. If they screw up, teachers and students (many from disadvantaged backgrounds) will be the ones to pay.
With that pressure, how does CommonLit ensure that shipping isn’t stressful?
Geoff is the first to admit that shipping several times a day wasn’t always easy.
“Three years ago deploys were scary,” he tells us, “they were done at night, and employees in India would batch up and deploy. That way we knew that when teachers arrived in the morning things would be working. But it still made deployments and hotfixes scary.”
His solution?
They invested a lot of effort upfront into writing tests and reorganizing. Testing became “like breathing.” A big part of code review was looking at tests to make sure they were doing the right thing.
Searching for a new Job? Send us your resume.
1. Avoiding Experimental Bias in Software Testing
In Physics, the Observer Effect refers to the impact observation has on experimental results. According to the Observer Effect, the act of measuring anything changes the results you get. Similarly, Confirmation Bias can lead scientists to unconsciously design experiments to get specific results.
The same problems play out in software testing. When testing a signing form, for example, you might enter data that are designed to pass, like a real email and birthdate. Then you might not catch a bug that causes old birthdays to crash the form.
At CommonLit, every piece of data that’s being tested is created by the test itself. The Test signs up as a teacher. Then it invites a student, logs in as the student, and logs back in as a teacher. While logged in as a teacher, the test creates a lesson and assigns it to the student. The test signs in again as the student and answers a bunch of multi-choice questions and an essay question.
If the test can complete this process successfully, Geoff’s team knows that the most important workflows are working.
2. Monitoring and Canary Testing Software
Geoff doesn’t trust anything to chance. He’s created a thorough system of testing that removes a lot of the stress from shipping code. For example, the team uses Percy to create visual snapshots in CI that take pictures of web browsers and flags snapshots if they differ by more than one pixel. By having several monitoring systems in place, Geoff ensures his team doesn’t live in fear of undiscovered bugs.
The team also does a second series of tests called “canary tests” every time they deploy. Each time, a hook gets fired that tells the build system to queue up some Cypress tests. Every public page and workflow gets visited and snapshotted by a real web browser. That way, the team will know quickly if the font moved or the color changed or a teacher can’t sign up anymore.
3. Single Server Cluster Test Deployments
Geoff says he got this idea from an Amazon Web Services blog post. For each new deployment, one server cluster gets it before all the rest. The team then runs tests against that particular server so that they get an extra level of verification.
4. Shared Verification and Teaching
There’s one theme Geoff returns to again and again. His team has “gotten very good at designing tests.” Everyone on the team works to do verification and it’s rare that someone checks new code and it doesn’t have tests.
That’s because for every new engineer, the team makes it a point to teach them how to do high-quality tests.
5. Flex Teams
At least one engineer at all times is not assigned to a project. That way, if something urgent comes up, Geoff isn’t asking someone to jump out of a task that requires deep work. The Flex Team isn’t just sitting around, however. They always have work to do. Often there’s key tech debt that’s not strictly on the roadmap but its stuff they can make progress on.
6. Housekeeping Day
“There are always things engineers know need to be fixed but the product team isn’t aware of the need like we need to update this React library,” says Geoff.
The CommonLit engineering team does two-week iterations. The first day of each iteration is Housekeeping Day. The goal of a housekeeping day is to fix things that need investment without taking a formal detour. Either the engineering or product teams will set priorities.
For example, the last Engineering Housekeeping Day was a big reorganization of how their JavaScript code was laid out. A Product Housekeeping Day, in contrast, would generally involve user interface fixes or little bugs.
7. The Principle of Least Surprise
The CommonLit engineering team follows the Principle of Least Surprise. It’s exactly like it sounds: they focus on consistency and clear, careful naming conventions so that the code isn’t surprising or confusing to someone who picks it up for the first time. If they want to do anything different, they do a design review or experiment first.
“Some people might say that’s a lot of orthodoxies and that’s a rigid code style” Geoff explains, “but I can parachute into a piece of code that I’ve never seen before and make a change and feel like I won’t bring the whole thing down. It lowers the cognitive load.”
Geoff Harcourt’s team currently has zero C.S. majors. Only one of his developers is a lifelong software engineer, and she didn’t major in Computer Science – she majored in Math. Most of his hires are mission-driven career-changers like himself. But, by creating a culture that teaches high-quality testing methods, he’s able to ensure his team ships code that competes with far larger organizations.
