Inside Facebook

For those interested in building scalable systems, Eugene Lutuchy, lead engineer on Facebook Chat, has posted details on many of the key engineering decisions his team made designing the Chat back-end infrastructure.

Letuchy writes, “When your feature’s userbase will go from 0 to 70 million practically overnight, scalability has to be baked in from the start.” That’s an understatement! While we’ve gotten several reports of Facebook Chat breaking under Firefox 3.0 RC1, on the whole Facebook Chat’s rollout has been very, very solid.

Some highlights from the report:

• The most resource-intensive operation performed in a chat system is not sending messages. It is rather keeping each online user aware of the online-idle-offline states of their friends, so that conversations can begin.
• Another challenge is ensuring the timely delivery of the messages themselves. The method we chose to get text from one user to another involves loading an iframe on each Facebook page, and having that iframe’s Javascript make an HTTP GET request over a persistent connection that doesn’t return until the server has data for the client. The request gets reestablished if it’s interrupted or times out. This isn’t by any means a new technique: it’s a variation of Comet, specifically XHR long polling, and/or BOSH.
• Having a large-number of long-running concurrent requests makes the Apache part of the standard LAMP stack a dubious implementation choice.
• For Facebook Chat, we rolled our own subsystem for logging chat messages (in C++) as well as an epoll-driven web server (in Erlang) that holds online users’ conversations in-memory and serves the long-polled HTTP requests. Both subsystems are clustered and partitioned for reliability and efficient failover. Why Erlang? In short, because the problem domain fits Erlang like a glove. Erlang is a functional concurrency-oriented language with extremely low-weight user-space “processes”, share-nothing message-passing semantics, built-in distribution, and a “crash and recover” philosophy proven by two decades of deployment on large soft-realtime production systems.
• Having Thrift available freed us to split up the problem of building a chat system and use the best available tool to approach each sub-problem.

Asked about the press’s speculation today that Microsoft - freshly spurned from its attempts to acquire Yahoo outright and rumored to now be trying to cherry pick Yahoo’s search and search marketing teams - also wants to acquire Facebook, Facebook CEO Mark Zuckerberg responded,

You can tell, from our history and what we’ve done, that we really wanted to keep the company independent, by focusing on building and focusing on the long-term.

While everybody has a price for almost everything in life, I believe that as long as Zuckerberg is in control, Facebook will remain independent. Do you agree?

In a note to developers tonight, the Facebook Platform team announced that it has turned on www.beta.facebook.com a day in advance of tomorrow night’s weekly push in order to give developers a chance to bang around on the release candidate.

This could very well be the dev server for the upcoming profile redesign sandbox. In the very least, it’s a nice gesture by the team to increase the chances of finding bugs that could adversely affect developers before the update is pushed.

While some developers have speculated that the upcoming Facebook profile redesign may include in-line commenting and content creation inside the new Feed, Facebook has confirmed to me that this is not part of the upcoming release. All content creation will happen via Facebook’s new “Publisher” (think souped up Wall attachments), not within the Feed itself.

However, like others, I think adding FriendFeed-style commenting (or better yet, a fully-featured Publisher) inside the Feed would facilitate even more constructive communication on the Facebook profile. Many in the tech community have already turned to FriendFeed over blog commenting in many cases because the conversations are more centralized in one place and more private than the open web. Supporting in-line content creation would also be better for application developers because it would allow more scenarios for richer interaction.

If Facebook were to add in-line commenting and content creation to the Feed, would you use it?

The MySpace Developer Platform team announced this week that they are adding privacy controls for users to choose whether or not applications can access their public and private photos and videos.

Although some developers will be upset that privacy settings were previously not a part of the photo/video API, this is the right long term solution. Photo and video access is a powerful part of the MySpace Platform, and in order for users to trust applications (especially with private photos), explicit per-application permissions makes sense.

Here’s what users will see when adding apps. Developers are also able to gain permission to photos/videos later by using the opensocial.requestPermission call.

The Bebo Platform team announced over the weekend that it has launched Facebook-compatible support for pre-load SNQL. This basically means that developers can make their apps run a lot faster on the Bebo Platform by specifying beforehand what platform data the application requires instead of making lots of queries as you go.

While we haven’t heard any major news out of the Bebo Platform lately, Bebo has told me they are working on new engagement metrics for applications. These metrics (like time spent, etc.) will reward developers that create the most engaging applications.

With over 4,500 games from over 1,700 developers, Kongregate has been helping Flash game developers and players meet for over two year - and it’s about to extend its reach into Facebook. Starting this week, Kongregate will launch the first of several Kongregate games ported to the Facebook Platform.

Inside Social Games sat down with Kongregate CEO Jim Greer yesterday to learn more about Kongregate’s plans for bringing its games to Facebook and other social networks.

>> Read the full interview at Inside Social Games

After Facebook at least temporarily removed itself from the Google Friend Connect program this week (a program it never officially joined as far as I can tell), some folks in the 280/101 corridor have been complaining that Facebook is not currently more open to sharing user data.

Ultimately, with most questions of balancing consumer and corporate interests, I am a strong believer in the power of the free market to do what is best for the customer. If you don’t like the way Facebook is handling your user data, you should start a campaign. If Facebook doesn’t listen, leave. If enough users leave, Facebook will have to change its approach, simple as that.

The problem is, nobody outside Silicon Valley cares about making their Facebook data portable right now. There are two groups of 50 users each on all of Facebook campaigning on this issue. The DataPortability.org Cause has 92 members. It’s not that this is not an important discussion, it’s just that almost no Facebook users care enough to make a stink about it - they’re all too busy playing Scrabulous.

It’s been a very interesting week in the world of social platforms and data sharing: after years of talk in the data portability community, industry behemoths Facebook, MySpace, and Google all announced major data, identity, and privacy portability initiatives within the span of 3 business days.

• Thur May 8: MySpace announces its “Data Availability” initiative, whereby it will share public user profile data with partner sites like Yahoo, eBay, and Twitter.
• Fri May 9: Facebook announces its “Facebook Connect” initiative, which will let third party developers access Facebook user profile and friend data.
• Mon May 12: Google announces its “Friend Connect” initiative, which will let users do much of the same thing through an OAuth/OpenID/OpenSocial system administered by Google.

None of these press events, however, included any actual product announcements. It’s something Marc Andreessen called a case of “strategitis” - making big strategy announcements before developers or users have any products to test out.

Today, Facebook upped the ante significantly by announcing that it is banning Google Friend Connect from the Facebook Platform for violating the Platform terms of service. Facebook’s Charlie Cheever writes,

All Facebook Platform developers agree to the Developer Terms of Service, which strictly limit the collection, use, and redistribution of user information. We have technology and a team to ensure applications abide by those policies.

We’re excited that our industry partners are taking greater steps toward openness and enabling users to share their information around the web. We hope, though, that we can collectively find a model that allows users to share data while protecting the privacy of our users’ data and ensuring that the user is always in control.

In the past, when we found applications passing user data to another party (for instance, to ad networks for the purpose of targeting), we suspended those applications and worked with those developers to ensure they respect user privacy. Now that Google has launched Friend Connect, we’ve had a chance to evaluate the technology. We’ve found that it redistributes user information from Facebook to other developers without users’ knowledge, which doesn’t respect the privacy standards our users have come to expect and is a violation of our Terms of Service. Just as we’ve been forced to do for other applications that redistribute data in a way users might not expect or understand, we’ve had to suspend Friend Connect’s access to Facebook user information until it comes into compliance. We’ve reached out to Google several times about this issue, and hope to work with them to enable users to share their data exactly when and where they choose.

We think MySpace’s Data Availability, Google Friend Connect, and Facebook Connect can be part of a great movement in the industry to give users a better and safer experience online, while respecting user privacy. We look forward to working with our developer community and everyone else in the industry to help all of our users take their information, and their privacy, with them wherever they go.

The amount of innovation in this space right now is good for everyone, but ultimately, Facebook, Google, MySpace, and other social networking platforms have major challenges ahead to provide the kind of portability users want without violating user privacy or cutting platforms out of the loop. We’ll stay on top of this story as it unfolds.

Google hosted the OpenSocial Summit today at its home offices in Mountain View to discuss the state of the OpenSocial API and the roadmap for version 0.8.  Amongst the many presentations by containers and app developers, Hi5 shared some very interesting stats on the Hi5 Platform.

These are some of the first detailed stats on the Hi5 Platform to be released - Hi5 doesn’t publish installs in the app gallery.

Hi5 Platform Totals

• 617 applications
• 1 million total daily installs
• 14 apps have been installed on more than 1 million profiles
• 6.5 million total daily canvas page views
• 9 apps with more than 10 million total canvas page views so far

Amongst Active Users

• 3.7 apps on average
• 52% have at least one app
• Max apps installed by any one user is 23

Shindig Stats

• >15 billion requests served
• 10k requests/second
• 35 servers total