A story of how one CTO took the company to the brinks of its existence and another almost managed to save the ship
I have been sitting on this post for some time now — I wasn’t sure if this is a story that others would find interesting or useful. But after speaking with Jeff Meyerson on the Software Engineering Daily podcast I realised there might be a lot of lessons other engineers and would-be founders can learn from what transpired at Yubl and avoid making the same mistakes.
Serverless Startup with Yan Cui_http://traffic.libsyn.com/sedaily/ServerlessBurningMonk.mp3Podcast: Play in new window | Download After raising $18…_softwareengineeringdaily.com
On a side note, one of my submissions to Hacker News about things we learnt about AWS Lambda at Yubl also got some traction a while back. As you can see from the comments some readers tried to extrapolate a causality between Yubl shutting down to our decision to use AWS Lambda.
Anyone who was there knows that was not the case, and although I had previously touched on the context around Yubl’s closure it’s a story that needs to be told in its entirety for one to understand the burning shipwreck it was and how close we came to saving it. Had one of the investors followed through with the investment he verbally agreed on, things could have turned out very differently and those involved will get the accolade they deserve for turning things around in a short time.
But alas, there was no fairy tale ending. Instead we have a cautionary tale of how one CTO took a well-funded startup to the brinks of its existence, and for another CTO to (almost) turn things around through ruthless, decisive hiring & firing. A tale that I will tell with characters from beloved TV series Game of Thrones in order to protect the people involved.
Disclaimer : I joined Yubl in April 2016 so I wasn’t present during the reign of the first CTO, the stories from that era are first-hand accounts by those who were there and lived to tell the tale.
Prologue (~Aug 2013)
A group of fresh university graduates were literally put into a room to think up ideas for a new startup. After some market research and brainstorming, the idea of a new social networking app is born — an app where users can interact with your post via interactive buttons that are part of the post rather than via meta-actions (liking, retweeting, etc.).
The idea is pitched to private investors, (non-trivial amount of) investments are raised, and soon a commercial entity is formed around this idea.
The app will be known as YUBL, or, YoUr social BubbLe.
The Littlefinger Era (2014 to ~Aug 2015)
Petyr “Littlefinger” Baelish joined the company as consulting CTO on the strength of his remarkable CV. On paper, Petyr Baelish looks the perfect fit for this young, ambitious and well-funded startup looking to take on the established giants in the field — the Facebook, Instagram and Twitter.
It’s no small feat.
Luckily we now have a fancy office in the heart of London and Petyr Baelish has assembled an elite marketing team to help us take on the big boys! The only problem is that, we don’t have a product yet…
The engineering team would be situated far away (at one point the engineering team was in a different building!) from the “product people” so they can focus on doing what their do best. Petyr Baelish would act as the bridge between the teams and provide regular status updates to the investors to let them know that everything is on track.
Requirements are passed down to the engineering team in the form of a 300 page document, lovingly crafted by the same eager young minds that had conceived the very idea the company is founded on.
Meanwhile, the engineering team scrambles to build more and more features.
- We need a timeline feature similar to Twitter. Check.
- Users should be able to follow/unfollow other users. Check.
- Users can search for other users by first name and/or last name. Check.
- We need a Suggested Users feature to recommend people for you to follow. Check.
- We need a Nearby Users feature to show you nearby users for you to follow. Check.
and the list goes on, and on.
Amidst all the excitement, we also secured contracts with various outsourcers:
- a company to moderate contents on our platform (for the paltry sum of £10k/month!)
- a company to provide us with contract PHP developers to develop a CRM/CMS system
- long term contract with MixPanel at £3k/month
- …
18 months flew by.
No one outside the engineering team had seen the product yet. Despite all the reports from Petyr Baelish that we’re making good progress and are on track, the investors grew weary.
And yet, all the engineers are also singing the same tune. If Petyr Baelish and his engineers are to be believed, everything is rosy, right?
One of the lead investors, Jeor Mormont (the original Lord Commander) did manage to find the truth he seek from the diligent and honest QA engineer Podrick Payne.
From what Jeor and Podrick told me, this is how that conversation went.
Jeor: “So tell me, how are we really doing?”
Podrick: “We are f**ked”
Podrick: “No, that’s not true. We are really really f**ked!!”
Podrick: “The app is untestable, nothing works! And the devs are not remotely interested in fixing bugs, when I bring bugs to the server team they just blame the client, when I bring them to the client team the client team just blame the server, nobody wants to find the actual problem.”
It sounded unbelievable to me, I have never ever worked at a place where developers would just keep passing the buck and not take any ownership of their work! Yet, that was a recurring theme when I spoke to folks about what happened during _Petyr Baelish’_s reign — distrust, incompetence, arrogance and downright disrespect and hostility towards one another.
Two of the Node.js engineers would forbid the other server developers to look at the code they’re working on. It would later transpire that they spent months building their own framework for doing pub-sub via RabbitMQ. What’s worse, the system they conjured together was so complicated they had no way of debugging and making sense of it.
How bad was it? Well, see for yourself…
Here, the handler for a RabbitMQ message would decide which of the modules named 1, 2, 3, 4, and so on would process the message by giving each a chance to run a predicate function against the message. The reason the modules have to be named this way is to enforce precedence.
To call it the worst, over-engineered, ill-conceived piece of crap would be an insult to crap. I’ve seen bad code in my time, but this takes the cake.
Try figuring out what happens when the handler receives a message from RabbitMQ…
The feature in question, however, is crucial for retention — when you post a piece of new content it should be distributed to your followers by background workers. The QA team had a detailed spec of how it should work and a suite of test cases, the system failed most of the test cases.
We reimplemented the feature from the ground up after consulting with the product team to understand how the feature is supposed to work. The rework took just over 2 weeks with most of the work carried out by one developer. The reimplemented feature passed all of QA team’s test cases.
The feature was rewritten using services AWS offers out of the box, was far simpler, and was covered by a suite of automated tests that were created with the help of the QA and product team.
The culture of a startup stems from the founders and the management team, and judging by the evidences on offer one must question the leadership of both Aerys Targaryen (the CEO at the time) and Petyr Baelish, specifically at their decisions to bring in engineers with such destructive qualities into the company. These bad apples were brought in at a great expense too, all of them were on pretty high day rates even by today’s market standard.
Angered by what he heard, Jeor took immediate actions. Within days both Petyr Baelish and Aerys Targaryen are gone.
Petyr Baelish was dismissed in a spectacular and public style, and Aerys Targaryen was gently pushed aside as Jeor Mormont took over the reigns as CEO.
_Baelish’_s close allies soon fled the company, and with _Baelish’_s removal there is now a power vacuum that needs to be filled.
In comes Edmure Tully as the new head of engineering.
We would later discover some questionable hires by Petyr Baelish that brought forth questions about his integrity:
- Pycelle, a tester turned contract Scrum Master after a crash course (well, technically he is certified) was on the payroll for 2 months on a staggering £900/day. When quizzed about his role by Daenerys he openly admitted that he was doing nothing. Well, at least he’s honest.
- Jaqen H’ghar, a server engineer who allegedly wrote the “whole” backend system in C++ although he was never in the office and no one has seen any evidence that this backend system exists in any form anywhere. What we had, was an invoice for £60k. In the email thread that Petyr Baelish shared (to prove that Jagen was in fact under Yubl’s employment) was a conversation between the two men where Baelish was coaching Jagen on how to make his invoices seem more believable. I don’t believe Jagen was ever paid that £60k.
The Edmure Era (~Sep 2015 to Jan 2016)
If _Petyr Baelish’_s reign was marred by questionable dealings and a lack of visible results, then Edmure’s reign is the personification of desperation and ineptitude, with devastating consequences.
Everything I have heard about Edmure paints the picture of a nice guy, a good engineer, and someone who genuinely wants to be everyone’s friend. Whilst he makes a perfectly fine teammate, he lacks a crucial skill required of leaders — knowing when to say no.
From my experience, all the best leaders I have worked with are able to hold both the carrot and the stick and use the appropriate tool to motivate/push his team to get the job done. Those are also the qualities that I look for in the leadership team in any company I join.
Sir Alex Ferguson is perhaps the most famous example of a leader that is able to effortlessly combine both approaches and apply them depending on the player and the situation.
To Edmure’s defence, this is his first time in the hot seat, and thanks to previous regime’s failures he is put under immense pressure to deliver immediately. The investors have already spent millions and seen nothing in return, they want to see results right away.
Given where the engineering team was at, and the upheaval that has just happened, there was no way it could be done and the reasonable thing to do would be to explain the situation to Jeor Mormont and ask to revise the plan accordingly. Jeor is a perfectly reasonable man and a successful businessman, and he would have understood had Edmure communicated clearly the absolute mess they were in at the time.
Unfortunately for him, and for Yubl as a whole, he did the one thing he shouldn’t have — he said Yes.
His plan? Hire more developers.
It must have been like an early xmas present for recruiters and contractors alike. Here is a startup with a good idea and a lot of money, desperate, willing to pay above market rates to bring people in by the bucket load…
At the peak of this hiring craze, Yubl had 15 mobile developers for each mobile platform — that’s a total of 30 mobile developers! All of them expensive contractors.
But wait! Why would Edmure do that? We know bringing more people on board a software project late on is only going to slow things down, and we have known that since 1975 thanks to the Mythical Man-Month. Ironically, when later confronted by Daenerys, Edmure would use the Mythical Man-Month as argument to support his decision to bring in an army of developers to help get the project over the line…
After firing one CTO and spending a lot of money bringing in all these contractors, Jeor expected to see progress. Yet, good news were hard to come by and after a continued lack of progress (especially given the investment and support) it was clear that things had to change, again.
In comes Daenerys Targaryen as the new CTO.
The Daenerys Era (Jan 2016 to Nov 2016)
In his own words, Daenerys is a man (I know I know, Daenerys is a woman in the TV show) who does not tolerate bullsh*ters.
In his first few days on the job, Daenerys interviewed everyone in the tech team and fired most of the newly hired contractors. In addition, a few of the bad apples who were either bullsh*ters or disruptive forces were also removed.
After just a few days the size of the technology team was down by nearly half, and the reduced communication overhead allowed those remaining to better focus on getting the app ready for launch.
Aside from trimming out the fat in the staffing, Daenerys was also acting as the Bad Idea Terminator. He discussed with the product team and trimmed down the launch feature set to a more manageable level. At the time, the essential features in the app were still barely functioning but the engineers were spending a lot of time working on new features such as VOIP.
The Bad Idea Terminator_Melissa Perri presents how to destroy bad product ideas before they go to market while achieving the business goals and…_www.infoq.com
In a minor miracle, Daenerys managed to launch the app within 7 weeks of taking over as CTO. After more than 2 years sitting on a shelf, most people had lost hope. The CEO even offered Daenerys the option to start over if he thought the ship could not be salvaged.
But things still aren’t great.
The app is still buggy.
The mobile platforms are not in feature parity.
The backend infrastructure is not going to scale.
We are down to 2 server developers (Edmure had resigned at this point, after a failed attempt to get Daenerys fired). There are still a few questionable characters in the mix due to contractual constraints with their agencies.
Daenerys needed to surgically reconstruct the server team and he needed people who are doers, people he can trust.
I was introduced to Daenerys through a mutual friend, I had the skillset he needs and he can offer me the new challenge that I’m after. Soon after I joined a few other people Daenerys knew and trusted from his previous company also followed suit.
What followed was an intense 6 months of transforming the company from both a product and engineering point of view. I am immensely proud of the work me and my team accomplished in such a short time, our efforts did not deserve the bitterly disappointing end we received.
I have discussed our work during the Software Engineering Daily podcast and at various talks, and I have shared many of our learnings on my blog.
The folks at AWS were very impressed with our work and were in the process of writing up a case study on us when the company was abruptly shut down.
Yubl's road to Serverless architecture_theburningmonk Server Architect Developer @ Yubl. I like C#, F#, Erlang, Go, Dart, Elm, PostSharp and occasionally dabs…_theburningmonk.com
By the end, we had attracted closed to 1 million installed users, and had around 200k MAU. The VC funds we were speaking to said we were much further along than other social networks after 6 months, but they won’t step in without seeing 12 months of live data.
Epilogue (Nov 2016 — present)
Since Yubl closed down, we have all gone our separate ways. My biggest disappointment is that the server team (the one that got together in the Daenerys era) had been broken up. In the short time we worked together we had developed a great rapport and we worked so effectively as a team.
We tried to stay together as a team but none of the avenues we pursued panned out. After speaking to quite a few companies — by now I have developed a much better sense of what a good start up looks like and the sort of environment I would like to work in — I decided to go back to the mobile games industry and joined Space Ape Games.
Space Ape Games is a really weird company in all the good ways — it defies many norms amongst tech companies and does many good things where other companies do badly. During my interview process I was really impressed by my conversation with the founders — they have a clear vision for the company and clearly defined the company’s culture, every recruit is hired for that fit. The company’s mission statement is clearly communicated regularly, and there is great transparency and honesty in the way the founders communicate with the rest of the company.
It’s no surprise that, Supercell decided to acquire Space Ape Games after speaking to over a hundred game studios all around the world. The two companies bear many resemblance in how they’re structured and the way they operate with small, autonomous game teams.
Supercell acquires 62% of London's Space Ape Games for $55.8 million_Clash of Clans maker Supercell has acquired 62 percent of London-based Space Ape Games for $55.8 million. The deal is…_venturebeat.com
I’m currently working on a secret new IP which I can’t talk about just yet, but hopefully it’ll continue to progress through our creative funnel and become publicly playable in the near future!