Why Friendster Went Down

The data center power outage that brought down Friendster didn’t just leave them powerless. It also left their data center soaked in fire suppressant.

“Regrettably, the maintenance did not go as planned and we suffered a catastrophic UPS failure at 8:22 am Pacific Standard Time,” said Mark Waddington, President of Quality Technology Services, in an incident report for customers. “The UPS failed to stand in and smoothly transfer power from the utility to the temporary generators due to a voltage regulator problem with the temporary generators. The failure resulted in the triggering of the FM200 (fire suppression) system in the enclosed battery room and the subsequent EPO as part of our life safety system.”

FM200 is a popular fire suppression system that uses a chemical “clean agent” rather than water. The EPO (Emergency Power Off) button instantly cuts power in the data center when a situation presents a risk to worker safety or equipment.

The Santa Clara facility was back on generator power within two hours, but Friendster remained offline for more than 23 hours over three days. While it has been eclipsed in the U.S. by MySpace and Facebook, Friendster has seen strong growth in international markets (particularly the Philippines) and says it has 85 million users.

When Friendster came back online, many of its users found large chunks of their friend lists missing, triggering rumors that the site had been hacked.

Friendster’s spotty data caching caused slow updates in years past, and contributed to its decline in the face of MySpace and Facebook. In the face of disaster, that problem returns to hound Friendster.

As a result, the memory caching of data was affected. Since Friendster has 85 million users, the cache is taking a considerable amount of time to build. Once the cache has been completely built, all friends will re-appear.

Notes to self: do not host sites at “Quality” Technology Services, and do not concentrate social graph on Friendster. Notes to Friendster: make data cache less volatile, find a better data center, and keep customers informed during a crisis through a corporate blog in a separate data center.

(Via Jim Ayson.)

Like This Post: Share This Post: Tweet This Post Share on Facebook Stumble This Post Post to Reddit