Archive for the ‘Perl’ Category

Friendster Architecture

July 11, 2007

Friendster is one of the largest social network sites on the web. it emphasizes genuine friendships and the discovery of new people through friends.


Information Sources

Friendster – Scaling for 1 Billion Queries per day



What’s Inside?

Dual x86-64 AMD Opterons with 8 GB of RAM
Faster disk (SAN)
Optimized indexes
Traditional 3-tier architecture with hardware load balancer in front of the databases
Clusters based on types: ad, app, photo, monitoring, DNS, gallery search DB, profile DB, user infor DB, IM status cache, message DB, testimonial DB, friend DB, graph servers, gallery search, object cache.

Lessons Learned

No persistent database connections.
Removed all sorts.
Optimized indexes
Don’t go after the biggest problems first
Optimize without downtime
Split load
Moved sorting query types into the application and added LIMITS.
Reduced ranges
Range on primary key
Benchmark -> Make Change -> Benchmark -> Make Change (Cycle of Improvement)
Stabilize: always have a plan to rollback
Work with a team
Assess: Define the issues
A key design goal for the new system was to move away from maintaining session state toward a stateless architecture that would clean up after each request
Rather than buy big, centralized boxes, [our philosophy] was about buying a lot of thin, cheap boxes. If one fails, you roll over to another box. Architecture

July 10, 2007

Mixi is a fast growing social networking site in Japan. They provide services like: diary, community, message, review, and photo album. Having a lot in common with LiveJournal they also developed many of the same approaches. Their write up on how they scaled their system is easily one of the best out there.


Information Sources – scaling out with open source



What’s Inside?

They grew to approximately 4 million users in two years and add over 15,000 new users/day.
Ranks 35th on Alexa and 3rd in Japan.
More than 100 MySQL servers
Add more than 10 servers/month
Use non-persistent connections.
Diary traffic is 85% read and 15% write.
Message traffic is is 75% read and 25% write.
Ran into replication performance problems so they had to split the database.
Considered splitting vertically by user or splitting horizontally by table type.
The ended up partitioning by table type and user. So all the messages for a group of users would be assigned to a particular database. Partitioning key is used to decide in which database data should be stored.
For caching they use memcached with 39 machines x 2 GB memory.
Stores more than 8 TB of images with about 23 GB added per day.
MySQL is only used to store metadata about the images, not the images themselves.
Images are either frequently accessed or rarely accessed.
Frequently accessed images are cached using Squid on multiple machines.
Rarely accessed images are served from the file system. There’s no profit in caching them.

Lessons Learned

When using dynamic partitioning it’s difficult to pick keys and algorithms for where data should be stored.

Once you partition data you can no longer do joins and you have to open a lot of connections to different databases to merge the data back together.

It’s hard to add new hosts and rearrange data when you partition. For example, let’s say your partitioning algorithm stores all the messages for users 1-N on host 1. Now let’s say host 1 becomes overburdened and you want to repartition users across more hosts. This is very difficult to do.

By using distributed memory caching they rarely hit the DB and there average page load time is about .02 seconds. This reduces the problems associated with partitioning.

You will often have to develop strategies based on the type of content. For example, image will be treated differently than short text posts.

Social networking sites are very time oriented, so it might be useful to partition data by time as well as user and type.