[Tech] The Crash of 2009

Kaelon · November 22, 2009, 08:28 PM

Serale, Aelyrians,

On Friday afternoon, November 20, 2009, at approximately 6:30 pm Eastern Time, the Aelyria Server began to exhibit strange a significant slow-down. The site had to be rebooted several times by our data center engineers, until it was discovered that a firewall problem was initially preventing a number of our users from connecting. The problems continued to escalate until at about 8:45 pm Eastern Time, the Aelyria Server's database crashed. One of the engineers had attempted to perform an optimization of our database while it was still live and dozens of users were connected and posting; this caused a corruption in the database that led to a general system crash.

My immediate concern was repairing our database and recovering all lost data. At that time, I disabled public access to Aelyria (such as it was) in order to perform a repair and optimization of our 5 Gb database; this process lasted over 12 hours, and was ultimately unsuccessful because one of our tables, a key index of posts, had been truncated. Thankfully, we had a daily backup from November 20th, 1 am, and I was able to restore our site using this backup.

By now, you may have noticed that any post made between 1 am and 8:45 pm on November 20th was lost. Given our volume, this was several hundred posts. Thankfully, no other content was jeopardized whatsoever.

The Aelyria Cloud.

In light of the catastrophic failure of our site -- which had been caused in large part by an incompetent engineer -- I decided to kill two birds with one stone, and I moved Aelyria away from our current dedicated colocation provider to the Cloud. I describe the Aelyria Cloud in a new blog posting, but here's a snippet:

Quote:

Originally Posted by Kaelon

Since 1996, when we launched on the Internet on our own hardware, Aelyria has relied upon its own physical servers to power its site. This has required us to have a “behind the scenes” team that we contract to help ensure that our servers are working properly. Following the November 21 Crash, I have moved us to the Cloud – a vast network of millions of machines that power such sites as Facebook and Twitter. One immediate benefit: instead of paying for an entire set of dedicated hardware that we may or may not need, we only pay for the resources that our site uses – nothing more, nothing less. Another longer term benefit: our network will grow with us, and as Aelyria continues to expand, we will not need to undergo migrations or maintenance to have added hardware resources at our disposal.

So, while on the one hand, this crash took Aelyria offline for approximately 48 hours and cost us about half a day's worth of posts, it also opened up the opportunity to move us away from our old dedicated colocation servers and take us in a direction that will (a) save us money, and (b) improve performance, redundancy, and reliability for everyone.

I will be explaining what the Cloud means at greater length in the days to come. In the meantime, however, please feel free to check-in and rejoin your fellow Aelyrians. To those of you who were following us on Facebook and Twitter -- thank you for your positive thoughts and support! We hope that our updates on our server status kept you in the loop on our march to recovery.

KNOWN ISSUES:

All posts made after 1 am Eastern Time on 20-Nov-2009 were lost.
User CP and User Profiles appear to be returning 500 Server Errors. This is being investigated. (Corrected)
The Attachments system is currently not working properly; this will be corrected ASAP. (Corrected as of 11/23 @ 9:30 am ET)
The front-page of our site has been temporarily replaced by a splash page. The old front page has been taken offline due to a bug discovered during migration.

Juan

Aderyn te tuin Lothain · November 22, 2009, 08:32 PM

User Control Panel and Character Profiles (Not the CIR ones) are also providing error messages now.

Cheers, Ady/Matt

Kaelon · November 22, 2009, 08:33 PM

Quote:

Originally Posted by Aderyn te tuin Lothain

User Control Panel and Character Profiles (Not the CIR ones) are also providing error messages now.

Cheers, Ady/Matt

Thanks. I will look into this right away.

Tonguez · November 22, 2009, 08:33 PM

see every cloud has a silver lining

Kaelon · November 22, 2009, 08:38 PM

Quote:

Originally Posted by Aderyn te tuin Lothain

User Control Panel and Character Profiles (Not the CIR ones) are also providing error messages now.

Cheers, Ady/Matt

This (and other related Internal Server Error 500's) has now been corrected. It was being caused by a misset memory value, which has now been increased. We have allocated additional memory resources to the Aelyria site from the Cloud.

Aderyn te tuin Lothain · November 22, 2009, 08:41 PM

Nifty

Oh, and the weird... thing, on the front page >_> Seriously ugly. Some things just have to be said - and that one's even more awkward that Ba'tu's strange CGI Baijani image of old on the front page.

Cheers, Ady/Matt

Archalen Andares · November 22, 2009, 08:44 PM

Great to have the site back up. One question on my mind; Due to the nature of this catastrophe and the sheer number of posts lost (not to mention the whole of a weekend being lost, a time where many of us post for the week) are there plans or will there be, to compensate those loyal and dedicated supporters of Aelyria who likely made the switch to a newer generation of server possible (ie. Companions)?

Alexis Sapientia · November 22, 2009, 08:47 PM

As I said in chat:

Interdictor 1, Milo 0.

Kaelon · November 22, 2009, 08:59 PM

Quote:

Originally Posted by Archalen Andares

Great to have the site back up. One question on my mind; Due to the nature of this catastrophe and the sheer number of posts lost (not to mention the whole of a weekend being lost, a time where many of us post for the week) are there plans or will there be, to compensate those loyal and dedicated supporters of Aelyria who likely made the switch to a newer generation of server possible (ie. Companions)?

Yes. All Companions will be compensated 2 full days of subscription time. This will be made effective sometime tomorrow. This is our long-standing policy, and I am glad to honor it always.

Kaelon · November 22, 2009, 09:04 PM

Quote:

Originally Posted by Aderyn te tuin Lothain

Nifty

Oh, and the weird... thing, on the front page >_> Seriously ugly. Some things just have to be said - and that one's even more awkward that Ba'tu's strange CGI Baijani image of old on the front page.

Cheers, Ady/Matt

Yes, well, I didn't want to spend too much time throwing something up on the front page of the site, and I have a handful of old commissioned artpieces from the mid 1990s.

It will be gone soon enough!

Dzrg · November 22, 2009, 09:44 PM

Old man Murphy scores again.

Tis good to have Aelyria back, apparently in better shape for the long run than before the crash.

Thanks to those who fixed it.

Kavene Darklight · November 22, 2009, 09:51 PM

Well, you should have all figured out by now... that I caused the crash.

I am sorry,

I finally got around to posting on all my active threads, all of them,

and the system was so shocked that it crashed!

(ill hopefully get those posts back up soon)

Kour'el Kha'Serith · November 22, 2009, 10:10 PM

Aelyria is Currently Propagating.

I feel old now -_- So should anyone who gets that lovely reference!

Serion D'Rinishad · November 22, 2009, 10:33 PM

Haha...dont remind me Kour >.>

Kaelon · November 22, 2009, 10:37 PM

One tip for those of you who are trying to find lost posts/threads -- use Google Searches for Aelyria. Google apparently indexed our site with 11,000 pages the very day of the crash, and I was able to recover a couple of my Wiki articles that way. Just run a search with "site:aelyria.com" included it in and hopefully Google can pull up cached versions of pages you can use to re-post!