Back from the dead

Posted: Sun Feb 27, 2005 7:25 pm

There have been significant load and stability issues with the site in the past few days, amplified by highest traffic levels ever and the strain placed on the servers. In a domino effect the servers were not operational most of the time. To make a long story short, There were 2 main issues:

- Bug in shared memory caching that hangs the servers
This is fixed with writing my own content caching library. I tried several implementations and settled on building on top of Linux's magical tmpfs. Load now is at an order of magnitude lower than was before the new influx of traffic.

- Hardware reaching limit on size and load
Our new dual xeon database server is now up and running. BTHub tracker DNS is now using its own separate database server. Our additional 6th web server is also up.

NOTE: Yesterday night (Feb. 27) starting at 3am, there was network issues with the new database so we switched back to our old database in read-only mode. Site registrations and posts up to 3pm the same day were discarded, as we rolled back to the state of the new database before it was down. Everything should now be back to normal.

Anyways our new setup should last for a while, although I've said this many times before. Rolling Eyes