Engineers around the globe are utilizing their tech skills to solve new problems brought about by COVID-19. We spoke with a local engineer at Upside Business Travel to get a pulse on how their team has adapted.
The Coronavirus pandemic is causing mass disruption for individuals and businesses alike. In the tech world, teams are using their coding expertise to solve previously nonexistent problems.
For example, one local DC startup that works in the travel industry, Upside Business Travel, has been impacted by the decrease in business related travel.
Their team quickly sought new revenue generating projects and pivoted to support these goals.
We spoke with Christopher Rung, a Senior Site Reliability Engineer at Upside, to hear about his experience. Christopher has worked at Upside since 2017, building AWS infrastructure with Terraform and Kubernetes.
He joined a team at Goal Financial that built an application to help small businesses apply for the Payroll Protection Program (PPP), established by the CARES Act, from the Small Business Administration (SBA).
The goal of the application was to enable businesses to quickly submit their application and receive much-needed money in a timely fashion.
We interviewed Christopher to learn more about how and why his team created the app and learned how it enabled hundreds of businesses to apply for a PPP loan. He told us about the problems they solved, the technologies they used, and how Upside’s culture allowed them to succeed.
The conversation below has been edited for length and content.
How did the team at Upside kick off this project?
My whole world changed April 2nd when I started a brand new, never-touched-before project with a team of eager, intelligent, and hardworking colleagues.
The team from Upside consisted of 2 infrastructure folks, 1 Product Manager, 1 Quality Engineer, 5 Full Stack Engineers, 1 Designer, 1 UX Writer, and 5 Customer Support/Marketing Specialists.
Since I’m on the infrastructure side, I was involved with the initial setup of the application and onboarding. There are many immediate needs that arise when you are starting from scratch. How should we communicate? Let’s set up a Slack workspace and sign up for G Suite. Which cloud hosting provider should host our service? Let’s use AWS. How do we track objectives to keep everyone sane? Let’s use Notion. Each of these services needs to be configured, and everyone needs an account to access them.
We worked 12 to 14-hour days, seven days a week. It was exhausting, but I woke up feeling excited each morning. We iterated quickly and made great progress with such a lean and nimble team. I learned a tremendous amount.
What problems did the team solve by building this tool?
We are facing the largest spike in unemployment since The Great Depression. People have bills to pay and need to collect their paychecks. The PPP grants 100% forgiveness for employers who use their loan money to keep their employees on the payroll.
Unfortunately, it has been a stressful and confusing nightmare for many to apply for their desperately needed loan. Banks are unprepared to handle the flood of applications and are disproportionately favoring their biggest clientele (e.g. The Lakers). The Mom-and-Pop shops that have had decades-long relationships with their banks are now being turned away, just as they need assistance the most. We want to help these people.
The tool that banks use to apply for a PPP loan, SBA’s E-Tran system, is largely written in COBOL, a language invented in the 60s. It suffices to say that COBOL wasn’t designed with modern scaling requirements in mind. There’s a great article in Forbes that discusses the rocky start of funding and the challenges that the banks are facing. After an employer fills out an application, the banks try to submit it, only to be met with timeouts and other errors from E-Tran.
Due to these challenges, many banks were only able to process a few loans per day. Congressional Bank, which serves the DMV area, is our lending partner and had a backlog of a few hundred applications that they were struggling to submit. With the tool that Goal Financial built, Congressional Bank was able to clear out their backlog and process all of them. Since the focus was on small companies and independent contractors, it felt great that our work was helping those who need it most.
I wasn’t involved in our application’s development, so I can’t take credit for it, but I understand that we have been successful due to a robust queuing and a retry system. We were seeing 80%+ error rates when requests were submitted, so we implemented things like queuing and retrying logic to deal with the incredibly high error rates we saw with our submissions to E-Tran.
There is an interesting parallel between solving the challenges of this legacy system and our work at Upside. The business travel world is also reliant on legacy systems. Airline and hotel suppliers book registrations using SABRE, a technology that was also built in the 60s and written largely in COBOL. Upside uses SABRE to fetch air, hotel, rail, and rental car availability, and we’ve become used to these types of challenges. Our team was just switching from one crazy old system to another.
What tech stack did you all use to build the system?
Most of the team had expertise in AWS, so we chose it as our cloud provider. We used Terraform to build our AWS infrastructure, which includes EKS (Elastic Kubernetes Service, AWS’ managed Kubernetes solution), networking, permissions, and databases. We used GitHub to host our code, CircleCI to build it, and host the artifacts on AWS’ ECR (Elastic Container Registry). Monitoring is handled by DataDog, and we have VictorOps set up to handle alerts.
This kind of experience—to build a production-ready system from scratch that will be used to make a meaningful difference in the world—doesn’t come often. I joined Upside after much of the groundwork was in place, so this was a great opportunity for me to fill knowledge gaps, especially since we leveraged much of what we built at Upside for our new work.
How did you balance creating a quality product, while also pushing something out quickly?
Since a big focus of this project was to iterate quickly, we built two environments: a development environment, used for testing; and a production environment, which hosts the public site. Typically, there is a third environment, staging, wherein QE tests the product before promoting it to production. While the engineers and designers cranked out new releases to our continuously deployed development environment, we relied on an incredible quality engineer to test these changes before promoting them to production.
One of the things I set up as soon as the infrastructure was in place was a “deployer” tool to allow the engineers to promote something from Dev to Prod independently. This improved the team’s velocity and freed up infrastructure time that would be spent on deployments.
Our team was also very good at being communicative on Slack. We kept statuses up to date and weren’t afraid to tell the team when we needed to take a break. It was very much appreciated to have an awesome project manager who had our back and was good about keeping the work-life balance reasonable, despite the intense timeframe.
Where does the project currently stand?
It was a crazy few-week sprint upfront on the infrastructure side. Largely thanks to the magic of Kubernetes, things have been running smoothly since then. All the product people and engineers are now iterating and improving upon the actual product.
Goal Financial was initially focused on providing people with an easy way to apply for their loan, but we also recently rolled out a tool that allows borrowers to track the forgiveness status of the loan. After submitting their application, employers want to know when they can expect their loan money to deposit into their account.
What aspects of Upside’s culture allowed your team to succeed?
The culture at Upside is very special. It’s a huge reason why I came on board and continue to stay on.
Many companies claim to be “one big happy family,” but talk is cheap. Especially during this pandemic, our leadership team has demonstrated extraordinary care and compassion for us. From the start, they have made sacrifices to make sure we can keep our amazing team together. We were one of the first companies I know that closed our office to keep everybody safe.
There’s an incredible level of respect and freedom that the company affords its employees.
As an example, employees get unlimited PTO. There’s the freedom to take a vacation and come back when we’re ready. Having this freedom makes it easier to work hard when necessary. Since we know the company has our back, we are more willing to push through a busy stretch.
Our People Ops team is fantastic at hiring people who are driven, self-motivated, and kind. They also help to foster a culture where co-workers can get along, be themselves, and truly enjoy working with one another. I am proud to call myself an Upsider.
