Scalability issues for dummies

Every once in a while I get people asking me what’s taking me so long to open my startup Inkzee to the public. They also ask me what exactly have I been doing as the web seems exactly the same. I normally answer that things aren’t easy, that it takes time, specially if you are alone, like I am. After a while I end up explaining my problems with scalability and that’s the point where people just can’t follow you. I’m going to explain here what are scalability problems and how deep the repercussions are for a small company.

Most web applications, like Inkzee, Facebook, Twitter, … are made of 2 parts. What we, the tech nerds, call frontend and backend. The frontend is the part of the application that’s exposed to the users, that is, the user interface (UI), the emails, the information that is shown. All that UI is a mix of different programming codes, let it be PHP, javascript, html, etc. The frontend is in charge of drawing the UI on the user’s screen and to display all the information the user is expecting from the application. But this information has to come from somewhere, well, that’s the backend.

The backend are all the programs and software applications that run behind the scenes and that are in charge of generating, maintaining and delivering the information the frontend displays to the user. The backend can be very homogeneous or very heterogeneous, but it’s normally comprised of 2 parts, the database (where the information and data is stored) and the software that deals with that database, does the data crunching and connects this to the frontend.

Now, some web applications have a barebone backend, very simple and light weighted. Normally some software that gets what the user inputs on the interface and stores it in the database and viceversa, retrieves it from database and shows it to the user. Other web applications have an extremely complex backend (i.e Twitter, Facebook, …). These not only manage the data retrieval, but have to do really complex operations with the data. Not only complex, but very expensive operations in terms of computational power. For example, each time a user uploads a picture to Facebook follows this path:

• The picture is stored in a specific hard drive. The backend has to determine which hard drive corresponds to that user (yes, there are multiple hard drives and each one is assigned to a bunch of users so the load is distributed).
• Once stored, the picture is sent to a processing queue where it will be turned into a thumbnail by an image processing software. This process is expensive as it has to analyze the picture and reduce it to a smaller representation if the image but still maintaining part of its quality.
• After processing it, the backend stores the newly created thumbnail in the database and stores, both the picture and the thumbnail in an intermediate “database” in memory for faster access (cache). This is because it’s faster to retrieve data from memory than from a hard drive.

This is an approximation of what a picture does when you upload it to a social network. I’m pretty sure it goes through a lot more processes though. So, supposing 1% of a social network’s users are uploading pics at any single moment, imagine uploading ~20 photos per user, 2.5 million users at the same time (Facebook has around 250 million users currently). Trust me when I tell you, that’s a lot of data crunching.

The problem

The best user interfaces (frontend) are designed so that all that complexity that goes behind the scenes is never showed to the end user. The problem is that the frontend depends gravely on the backend. If the backend is slow, the frontend won’t be able to have the info the user is requesting or expecting and it will seem SLOW to the end user. Not only slow, but in many cases inefficient or just not available to use at all (meet the Twitter Fail whale :P).

So, now, what will cause the backend to be slow? Ohhhhhh don’t get me started!! There are so many reasons why the backend might be slow or broken! But, most of them are triggered by growth. That is, as the web application is being used by more and more users, the backend will start to fall apart. That’s what, in the tech world is known as scalability problems. That is, the backend can’t scale at the same speed the users pour into the application. The problem is that it’s not only a problem of more users, but having users that interact more heavily with the site. For example you might have 100,000 active users but never had experience big scalability problems. Suddenly you release a feature that allows your users to share pictures more easily… BAM!! Your backend goes down in 10 minutes. Why!! Why?!! you might scream while you watch your servers go down in flames. After all you have the same amount of users, so what happened? Well, most probably your backend system that handles picture sharing was designed and tested only with few users. Now it chokes with the big deal.

The REAL problem

Once you have scalability problems, the next logical step is to find where the bottleneck is and why is it happening. This, which might seem very easy, isn’t at all. It’s like looking for a needle in a haystack. Big backends are normally VERY complex with many parts coded in different programming languages by different persons. Not only that, but sometimes problems arise in different parts of the backend. So after a couple of really stressful hours you find the bottlenecks and think of a solution to fix them. Ahh my friend, then you realize it’s not as easy to fix as you thought. First of all, you have no clue if the fixes your team has come up with are good enough. Why? Because you’re stepping into unexplored territory. Few persons have had to tackle a similar problem and even less people have dealt with your data and systems. So even if you find someone else with the same problem, the solution might be slightly different depending on what systems you use for your backend or which architecture you have. This is the point where you realize that developers aren’t engineers, but craftsmen and that fixing these problems isn’t exactly a science but black voodoo magic.

So, here you are, with a bunch of possible fixes to a problem but with no clue if they will really work or it will just be a patch that will need extra fixes in 2 weeks. Normally you try to benchmark the solutions, but that’s not an easy task, specially because you have no real load to test it against except in your production servers and no, you don’t want to fuck the productions servers more than they are.

Finally, after some black magic and some simple testes you cross your fingers and try the fix on the production servers. After several hours of monitoring the backend for new “leaks”, you scream of happiness as the patch seems to work. Then you start to realize that the patch won’t hold on forever and that you need some extreme solution to the problem.

You sit down with your tech team (our on your own as it’s my case 😦 ) and you start drafting a new solution. Suddenly you realize that the best fix implies changing the way your backend works. And by change I mean, you need to redevelop a big chunk of your backend to fix the problem. This implies a couple of things, you’ll need to invest a lot of time and resources, you’ll loose the stability your backend had (prior to the incident), you’ll walk into a new unexplored territory for your team and worst of all, you can’t just unplug your production servers and change the backend, you need to do it so both backends coexist for a while until you switch all of your servers from using the old one to the new one.

Now, the REAL problem is that this change, this new redesign grinds the whole company to a halt. All resources, let it be people or money are invested in redesigning efforts so nothing new can be done. Most outsiders just don’t understand the depth of this change and will bash the company for not doing new things, for not releasing new features, for not fixing old bugs, etc. Not only that, investors will start to get anxious and will demand things to start moving. So, the outside world only sees that you’ve stalled, while the inside teams are suffering the pressure. Not only that, developers inside the company will get extremely frustrated by the pace of things. They won’t be able to add new features and even when fixing bugs they’ll need to fix them twice, one in the old backend, one in the new backend.

So, in the end, you realize the shit hit the fan and you got all of it. It’s hard, very hard to be there. If you haven’t experienced it you have no idea how hard it is. Not only as a developer but as a founder, CEO, or executive position you’ll feel the pain. You won’t be able to publicize your site cause more stress might accelerate the old backend problems, you can’t give users new features because you have no resources, you will try to explain the problem to investors but they won’t understand a clue of what you’re talking about… “backend what?”. Current customers will be pissed at you because the site is running slow and you are doing nothing to fix it. So, in the end, everything freezes until the new backend is in place.

How long does this takes? Depends. Depends on the size of the redesign, the size of the tech team, the skills of the team and specially, the skills of the management. During this phase, management must execute impeccably. Sadly, this is not the case in most places and so priorities are changed, mistakes are made and the redesign gets delayed over and over again.

It takes a very good leadership to make it through this period. Someone that knows where their priorities lie and that is able to foresee the future and the importance of the task ahead. Needless to say that such figure is lacking in most companies. That’s the reason it took so long for Twitter to pull their act together, to speed up Facebook, etc.

I am there, I am suffering the redesign phase (twice now). It’s hard, it’s lonely, it’s discouraging and frustrating, but it needs to be done. I just wrote this post so that outsiders can get a glimpse of what is it to be there and how it affects the whole company, not just the tech department. Scalability problems aren’t something you can discard as being ONLY technical, it’s roots might be technical but its effects will shake the whole company.

Let there be light 🙂

Would you buy a cool Twitter name?

Leo Borj asks a very interesting question, should we start registering potentially valuable Twitter names, like if they were domains? This is a very common practice with domain names known as cyber squatting. You register a domain name with the idea of selling it in a future for a higher price to a 3rd party. You could also call it, name speculation. The domain squatting is a quite profitable market, albeit it sometimes touches some ethic boundaries. Anyway, the real question is, is it worth doing the same with Twitter names?

In my opinion, it might be worth trying, although there is a fundamental flaw. With domain names, it gets down to who registers it first (except in cases where you can probe that someone else is using a registered mark with nasty intentions and even though, it’s hard to get it back if you aren’t a multimillionaire corporation). In Twitter that law doesn’t holds, or better of, doesn’t holds always. Instead you have the following general condition:

We reserve the right to modify or terminate the Twitter.com service for any reason, without notice at any time.

As you can see, if you play the squatting game, you have a very high risk of having your account suspended if Twitter deems so.

Another question that arises me is, would people follow an account that isn’t from a real person nor a company nor a brand? I mean, would you follow a Twitter user named: @sexy_toys ? Personally I wouldn’t, but hey, I’m sure some people would if that account follows them first (the problem with this is that after the recent spam limits imposed by Twitter you have a fair chance of having the account suspended).

And finally, would you sell your Twitter account, either directly to 3rd persons or indirectly selling ad Tweets? I’m sure this is going to hit hard, I do think it would work if you play it well. Some basic ground rules I would like to see:

• Don’t spam your followers. Some recommendations are ok, but don’t transform the account into a damn billboard
• Don’t advertise things that no one cares about or that don’t have nothing to do with your Twitter audience.
• Don’t advertise or recommend something you don’t like or haven’t tested. It’s bad for your followers and it’s bad for your reputation.

After all, I’m a true believer of what Dave Winer sometime said: “because perfectly targeted advertising is just information.”

What are your opinions on the matter?

Twitter as a marketing tool

It’s been twice in a week that I’ve talked with Marketing and Communication people about using Twitter for their companies and the answer has always been NO. It really strikes me, specially because both companies are part of the Tech Industry, one of them is even a well known startup.

The response I always get about Twitter is: “We don’t use it because it’s only for early adopters and geeks“. While this has been the case until very recently, Twitter is gaining critical mass at an amazing pace. Just to throw some numbers, Twitter has a daily reach of approximately 0.1% according to Alexa. That means that 0.1% of the Internet users access Twitter each day. But, that’s not the only thing, according to Biz Stone the Twitter API gets 10x that traffic, bumping Twitter’s daily reach to a very nice 1.1%. Right now it has broken the early adopter barrier and you can start seeing a more general use of it by online users. That’s the reason I think it’s the best moment to start using Twitter as a Marketing tool.

Why is Twitter a Marketing tool? Because it allows persons and/or companies the following:

a) Give a human face to the company
b) Keep your customers up to date with the latest news of the company
c) Listen to your customers feedback
d) Interact one to one with your customers
e) Keep key influencers (tech gurus, bloggers, journalists, etc.) updated about your company
f) Brand tracking

Who should use Twitter as a Marketing tool? Not everybody of course. Twitter users have a tech profile, although this profile is being widened by other online non-tech users day by day. This is something that many old school marketers don’t grasp. Everyday the digital breach between the online and offline worlds is diminishing. Currently you can see baby boomers buying food, books or flights through the Internet. You can see them chating, emailing, commenting on forums or writing blogs. Some of them are real social network junkies. Marketers have to understand that the profile of an Internet user is becoming much more general than it used to be.

So, again, who should use Twitter as a Marketing tool? I would say that it should be used by anyone with a company that is related to the tech industry. That would be websites, blogs, newspapers, software vendors, hardware vendors, musicians, … hell, everyone that can make business through the Internet. Right now that is mostly everyone!

Why are people going to adopt Twitter? It’s very easy, Twitter is Free and Simple. The sophistication level needed to use Twitter is close to 0 for any single Internet user. It’s ease of use is one of its strongest points. Also, the need to be connected and to know what is happening in other people’s lives is something inherent to humans. Humans are curious by nature, Twitter lets you be curious about other people at a never seen scale. The best proof of this are social networks like Facebook. What’s the real value of a social network? Simple, stay connected with your friends, or in plain English, know what your friends are doing. In that sense Twitter is much simpler and faster than any social network out there. In my humble opinion, it’s just a matter of time before all the social network late adopters take over Twitter.

Finally, why do I say it’s now the best moment to start using Twitter? Well, Twitter, as what happens with blogs lets you build a big audience. And the key concept here is “build”. You can’t start using Twitter and expect a relevant audience of 10.000 users following your account. Why do I say relevant? There are many Twitter users which think that if they follow many people they’ll get followed in exchange and while it’s true that this happens (byproduct of our society, people feel bad if they are being followed but they don’t follow you back), these type of users aren’t really listening. That’s why, even though you can do that, you won’t build a relevant user base. Your messages won’t be read by the users you want. The funny part of this is that the people that follow thousands of users just to get more followers, are usually so called “Marketing experts“. That’s pretty ironic if you ask me, as they should know better. So trust me when I say that building a reliable audience on Twitter takes time and good, thoughtful messages. That’s why it’s important to start now, so that when Twitter goes mainstream your company has already a good number of followers. As they say, the more number of follower you have, the faster you’ll get new ones.

Any experience with Twitter and your company? Please share it with us! Are you a Twitter user already? Follow my Twitter user http://twitter.com/abarrera

UPDATE: I found a very good hands on example of using Twitter as a Marketing tool.

Because every word counts: Twitter experiences

Recently I started using Twitter. I must confess I wasn’t very fond of it. I just didn’t understand what use I could get out of it. Even though I’m still not a great fan of the service, I have to admit that it gives me some value. Many people try to describe Twitter, and most of them end up saying that it’s like a chat (irc, icq, etc.). My own definition would be that “Twitter is a slow motion chat where you get to decide who talks in it“. The key and really interesting part is the decision of who talks in the chat. For me that’s a huge difference between irc and Twitter.

From a business perspective I use Facebook to see what key people in my industry are doing. I can monitor which events they are going to, with whom they are talking, what posted items they are sharing. Again, the good thing about Facebook is that I choose who I want to be friends with. Nevertheless, one of the differences between Facebook and Twitter is that, for Facebook I always need the friendship to be approved, while on Twitter (except for protected accounts which are rare) I can follow whoever I want.

As for the quality of the information I must say that it’s just different. If you want to write about something and it’s long enough (be it more than 2 phrases) you’ll probably write it down in your blog. But if it’s just a link you want to share or an idea about something, you don’t have a tool to share it to a wide audience. Granted that you could write it as a blog post, but you risk burning your readers with a high frequency of posts with very few content. So, that’s where Twitter gets into action. It allows you to post your short musings to a different kind of audience. Getting back to the quality of the information, the good part of it is that you get to choose high profile twitters that you think might say or share interesting things. For example, Martin Varsarsky, Jeremiha Owyang or Mike Butcher are good examples of that. Again, if you don’t like someones content you can always “unfollow” them with no repercussion.

Finally, while reading a book from Ricardo Semler (Angel, thank you so much for the recommendation), I read a very good quote from Mark Twain: “I’m sorry I wrote a long letter, I didn’t have time to write a shorter one“. It holds an awful big truth, it’s harder to write small but meaningful texts than big cluttered ones. So that got me thinking about Twitter and its repercussions on heavy users. How will a 140 character restriction will transform there way of writing and even thinking? I suppose this is something we won’t see at first, but in the long row. I know that I’ve changed the way I listen to people. I’m so used to crawl hundreds of blog posts a day that I look for the essence of things and only if I like the essence, then I’ll read the whole post. This way of working is transcending into my offline life. Now I always find myself telling people to cut the crap and to get to the bottom line (I must say that people in general and in Spain specifically talk, way too much and say way too little).

I also think that, in the same way bloggers evolve and the way they write posts change with time (for better I hope), the same principle applies for Twitter. At first users just write about there life, and then they start to shift away from that and into a more information rich environment (this doesn’t applies to everybody though).

In conclusion, Twitter covers a different niche than blogs or Facebook dies and it targets a different audience. That being said, I recommend people that consider themselves information junkies to give it a try if you haven’t. You can follow me on my Twitter account and hopefully I’ll start changing what I write there. Twitter should read: “What are you thinking?” instead of “What are you doing?”.

Twittories and the art of writing

Some days ago, our good friend Duncan Riley wrote about Twittories. It’s a website where many people collaborate to write a short story. The catch is that they have to do it with a twitter message. This means, each user can only submit one message with a maximum length of 140 characters. While reading the story I was reminded of a game we used to play when we were little, I know it has another name in English but can’t remember now. In Spanish it was called the broken phone. Anyway, although it’s a fun idea and a cool experiment, there is no way you can read a story written in that fashion. The central plot is so broken it’s impossible to follow. But I must say it got me thinking about the quality of most articles. Before creating my current startup, I developed a prototype for what was going to be my first try in the entrepreneur world. I designed a semantic algorithm that was able to detect and highlight structural problems in texts. In other words, a style corrector. While I was testing it, I realized how badly written where most news articles. One of the most common problem was the use of overly large phrases in terms of words. And when I talk about large phrases I’m talking about 50 or more words per phrase without a full stop. So, back to the twittories, I realized that, even though the plot might be messed up, the 140 characters restriction could be a great thumb rule for many writers (including myself of course). Maybe we could enforce some rules like that on our blog platforms. Now that TypePad is also open source, there is no excuse 😉

What are your thoughts about this?