How we built two products with one engineering team
Welcome to TechByNimbus! You’ve discovered the engineering blog from the team that built Ads By Nimbus to support our core product Timehop.
We’re a small team in New York City that has been working on two very different products for several years, and this blog is dedicated to sharing that experience with the world. We believe we can contribute to the conversation by sharing our journey, highlighting fun features we’ve built, and telling our individual stories to the tech community here in NYC and beyond.
Timehop, founded in 2011, is a mobile app that showcases your digital memories every day from your social media connections and your phone’s local photos. Millions of people open Timehop every day to reconnect with their past, reminisce about those nostalgic moments, or take control of their digital history.
Nimbus, created in 2017, is an in-app programmatic mobile ad platform designed to do two things well: maximize revenue and deliver a polished ad experience to users. We built Nimbus in order to turn Timehop into a self-sustaining, profitable company without compromising the user experience while keeping it free for our users.
Let’s start with Timehop
You may have discovered Timehop when a friend of yours shared one of their memories on Instagram. You might have used a similar feature on Facebook called “On This Day.” Your phone may have shown you a notification about a memory in your Camera Roll from a trip you once took. Whatever the case may be, you’ve experienced the nostalgic journey that Timehop was designed to take you on, and we’re proud to have pioneered that experience for the digital age.
When you download Timehop, you can connect your social media accounts and local photos to the app, so we can retrieve the photos, videos, and posts you’ve made in the past and show them to you each day. We do this with a robust system of backend services each responsible for a discrete step in the process:
Memories Importer: Integrates with each API to fetch your content 7 days in advance
Daily Preparer: Analyzes & counts each user’s content to prepare the day for display
Push Notifier: Sends out push notifications at specific times each day to each user
API Gateway: Serves incoming requests for each user’s daily content
Media Proxy: Proxies requests for photos & videos, refreshing expired CDN urls
Import Scheduler: Queues users up for the Memory Importer to start the cycle over
This is an oversimplification of our architecture. There are microservices for each social integration, user management, back-office administration, and more. We also keep track of how many consecutive days each user opens the app - your “streak,” and we do this for millions of users who open the app every day. As of writing this, over 24 million users have downloaded and signed up for Timehop, connected over 55 million accounts, and viewed billions of photos, videos and posts.
Sound like a ton of data? It is. Does that get expensive? You betcha.
Experimenting with ads
Timehop raised money several times to fund our initial growth and scale up the team, but as users kept growing and their memories kept multiplying, we knew that wasn’t going to be enough. The team spent several years experimenting with ways to make this a viable business while keeping the app free to use. Your memories belong to you, and charging for it wasn’t going to be an option. So we made an attempt at integrating digital ads into the product, in the most native way possible.
Direct campaigns Timehop ran with Uber, SNHU, and Starbucks, featuring our mascot Abe
At best, results were mixed. Each campaign delivered as promised and most clients ran multiple campaigns overtime after seeing success advertising on Timehop. However, the level of effort it took to draft a proposal, negotiate an agreement, get everything signed, design the creative and execute each campaign was enormous. Engineering built a scalable, reusable system to manage these campaigns for flight dates, positioning, frequency capping, targeting, tracking, and all the usual bells and whistles, but could never truly get humans out of the loop. Abe, our lovable time-traveling dinosaur, was having a great time, but we were not.
In parallel, we started experimenting with displaying programmatic ads using client-side mobile SDKs from well-known vendors. Google and Facebook, the two largest players in digital ads, each had its own solution for running programmatic ads in mobile apps, as did other smaller vendors. We cycled through each of them in an attempt to further monetize Timehop, but repeatedly came up against deal-breakers we couldn’t live with.
a. Ads that took over the entire screen
b. Ads that didn’t handle user gestures
c. Ads that played audio unexpectedly
Users interact with Timehop by tapping or swiping through their memories. You can tap on either edge of the screen to advance or go back, or you can swipe anywhere on the screen in either direction. Most of the SDKs we tried displayed their interstitial ads with a gigantic takeover of the screen (a) with an animation that interrupted the user’s flow through their day. This was a non-starter for us, and we ruled out using those vendors.
Other SDKs that could render the ads in a more native way simply didn’t handle these gestures elegantly. Because ads like to be clicked, as demonstrated by their underlying code that triggers a click on ontouchstart or ontouchtap events on mobile devices. The slightest tap could trigger a click on certain ads (b) interrupting the user experience and taking them somewhere they didn’t want to be.
Finally, many ads, whether they contain video or not, came with an audio track included (c) that often did not respect the hardware mute switch or the volume setting. These were the worst offenders. Reminiscing through your most memorable moments when you wake up in the morning should never be accompanied by an annoying marketing jingle that jars you awake, and we would never want to put out a product that could risk that.
Other problems lingered. CPMs were too low, reporting was often delayed, and we couldn’t have both static display and video ads in the same ad unit. Nothing out there existed that met all of our needs. We were not in control of our own revenue, and therefore couldn’t control Timehop’s destiny.
Enter Nimbus
We decided to take a leap of faith and build our own ad platform. We figured if nothing out there existed that works for us, we would simply create what we need in-house. Did we have the resources for such an undertaking? Not entirely. But we did have a robust backend team that knew its way around AWS, so we directed those resources to build a server-side solution to what we originally thought was a client-side problem.
The challenge was: how to do this without hurting existing revenue while also maintaining velocity on Timehop. We started out with a proof of concept, hacked together just before Thanksgiving, to see if this path was practical and viable. We rolled this out to 10% of our users at a time using an in-house feature-flagging system, and monitored revenue judiciously each step of the way. In a matter of days and with minimal effort, we had our answer: CPMs jumped dramatically, fill rates started to climb higher, and the quality of the ads we were showing were objectively better integrated into Timehop. This was truly the way forward.
Building Nimbus in-house gave us the ability to connect directly to demand sources, cutting out third party vendors from the transaction. We were now able to conduct our own header-bidding auction and choose the highest bid for that ad impression, which inevitably led to higher CPMs and better transparency for Timehop as the publisher. We were also able to conduct the auction by including video and static display ads competing for the same ad placement, which helped raise the price ceiling on each ad, while also maintaining an overall high fill rate.
One key aspect to this approach was that we were able to see all of this data in real-time. By putting together an impression tracking system backed by ElasticSearch, we could see the results of each bid request, the winner of every auction, and the data behind each impression and aggregate the results on the fly. This empowered us to make decisions in real-time and to rapidly change our strategy to optimize for yield without having to contact any outside party for SDK updates or having to wait on an app release cycle to see the consequences of our choices.
We quickly did away with direct ad campaigns and the overhead associated with them and went fully onto programmatic ads, doubling down on Nimbus and the lessons learned from our new approach. Our internal success with Nimbus powering all the programmatic ads on Timehop rapidly drove the company into sustained profitability and operational stability. We quickly realized there are other companies developing mobile apps that are looking for the same things, and are struggling to find it, which led us to the inevitable decision to turn Nimbus into a SaaS product for third parties to integrate with.
In the months since that decision, we’ve made great strides towards maturing Nimbus into a full-fledged, world class, in-app unified auction advertising platform. We have integrated with dozens of demand partners, including SSPs, DSPs, and exchanges. We’ve developed a web-based dashboard for revenue reporting, demand placement configuration, discrepancy reporting, ad blocking and more. We’ve gone global across six regions with our cloud-based hosting provider, optimizing for latency around the world for asset delivery and impression tracking. And we’ve built two mobile SDKs - for iOS and Android - that we implement within Timehop and support for other third party publishers who choose to adopt it.
One Team, Two Products
Resources are limited, and the new enterprise occupied a majority of our team’s resources for the better part of 2019. While we made some excellent strides on Nimbus, we were admittedly unable to make similar progress on Timehop and the roadmap we envisioned for it. Spinning up a brand new business inside an established company reminded us that there were so many lessons to be learned, and we continue to learn new ones each and every day.
For instance, while the software development lifecycle for a mobile consumer product like Timehop was very familiar to us, we had a lot to discover about how different the process is for a service business in which other engineering teams are your customers. And while the known universe of features, bugs, and technical debt that exists in the Timehop product is more or less familiar to us, Nimbus’ server-side applications and mobile SDKs spent a lot of time in their growth and discovery phase, in which the amount of unknown unknowns is impossible to measure.
Our processes had to change, and our team’s structure along with it. We recognized our backend team, already de-facto split up between the products, would now be formally allocated 2-to-1 to Nimbus and Timehop respectively. The mobile team would split their time between the two, usually alternating their focus between them with each sprint. DevOps and QA, each functioning as a team of one, would focus on both. And we would hire a full-time full-stack engineer who would go on to build the public facing marketing website, dashboard, and this blog. Meanwhile, we would have to move task management out of Github Issues and into monday.com, to facilitate smoother interaction with the non-technical stakeholder son the team.
Sprint planning would also evolve to consider the needs and priorities of both products, not only within their own roadmaps, but how the roadmap items from each product interact with one another. For instance, we already had a process at Timehop for addressing user-reported bug tickets and feature requests. For that process to translate to a client-service relationship, we had to rethink how we triage inbound requests. Feature requests went through an ICE scoring process before they ever hit engineering. Account Managers and Solutions Engineers were hired to manage client relationships and field inbound questions while evangelizing our documentation and tutorials.
Engineers have to balance staying in a productive and focused zone to do their work with being available to field urgent and important issues, which can slow down velocity and impact delivering on time. Estimates are notoriously difficult in software engineering, and the added cost of context switching and on-demand client servicing have made defining and committing to deadlines an even greater challenge.
After a couple of years and a number of projects going over their allotted timelines, we’ve started making some changes. We’ve learned to slow down and be more judicious about how much we plan and how many things we choose to work on in parallel. We’ve supplemented the usual standups and sprint planning meetings with a more high-touch schedule of kickoffs, technical planning meetings, build reviews and post-mortems, regardless of the outcome. We’ve brought on Techgrok to help us get better at hiring, onboarding, writing for this blog, speaking at conferences, and hosting events in our office to better connect with the New York engineering community. And we continue to evaluate whether any of this is working well and remain open to changing whatever isn’t.
The Plan Moving Forward
We’ve set a number of internal product goals for both Timehop and Nimbus, along with a series of engineering goals across the company. One of those was to launch and regularly contribute to this blog with interesting technical posts, project spotlights, upcoming team events, and company culture insights. The sign on the door says Timehop, and that’s who we continue to be each and every day. We chose “TechbyNimbus” for this blog as a play on Ads by Nimbus, and to acknowledge where the company is growing the most these days, and where the source of much (but not all) of our content will come from.
Some posts will be deep dives into our tech stack, while others will tell the story of a project’s life-cycle and what problems we’ve solved for our customers. We aim to share the most relatable parts of what we do, how we do it, and why we think it’s a valuable story to write about. We believe the insights we share will shine a light on some unique experiences we’ve had here, and in due time, we hope to look back on them fondly - in true Timehop fashion.
