PDA

View Full Version : Site reliability improvements



J
06-05-2012, 00:43
Good evening everyone.

This evening we have completed a full sync and replication system to a mirrored server for this site. That means that, at random, you will be looking at either our new server in Louisville, CO... or the old server in Pennsylvania. Should one of these servers fail, the other will instantly pick up and serve requests for all users. After this happens... when the other server comes back up it will catch back up with the online server, then begin serving requests once again. Better yet, when we need to do maintenance in the future, we can route traffic to one server, upgrade the other, reverse traffic and do the remaining server... most upgrades can be done with no to minimal server downtime. (forum software updates are the exception, they have to run everywhere)

Additionally, this set up allows us to have constant replica backups to 2 places, but it also allows us to have daily snapshot backups to two additional offsite servers. There is very little chance of another huge data loss happening under these circumstances (data in 4 locations, probably not all going bad at once).

Please note that this is still a new system in these regards, please PM me if you feel that there is something odd going on with the site/account.

Also, in recognition of the season, I am now sporting the purple color, to root on the Rockies in the quest to not be laughed at in October.

jerrymrc
06-05-2012, 05:09
It is alive. Went to bed, woke up and it was still here. [Beer]

hghclsswhitetrsh
06-05-2012, 06:22
Awesome, sounds sophisticated. Thanks for all the hard work!

00tec
06-05-2012, 06:23
So you took care of the server eating zombies?

I see the time is now accurate as well

spleify
06-05-2012, 06:32
Thanks for all of the hard work on this.

Like Jayock said if anyone notices anything different or funky, small as it may be, please contact Jayock and let him know and give him as much detail with the issues you are experiencing so he might be able to take that intel and fix it

theGinsue
06-05-2012, 07:16
I HUGE round of applause for our site server admins foxtrot and jayock.

Jayock has been a tremendous new asset to the team.

ChadAmberg
06-05-2012, 07:23
OK, I want to know the technical stuff, Load Balancer or DNS round robin?

J
06-05-2012, 07:28
DNS RR. I have a load balancer for local stuff, for HA and performance hungry services.

This is over WAN, and a single load balancer at one server location kind of defeats the geographic redundancy IMO. Maybe someday we can have a pair of F5 balancers spread out and do it all the fancy way.

spleify
06-05-2012, 07:42
Yes, great work purple team!!

J
06-05-2012, 07:43
Purple power!

StagLefty
06-05-2012, 08:54
Are our permits valid while on the Pennsylvania server ?
Thanks for all the hard work guys !! [Beer]

RMD
06-05-2012, 09:03
Excellent. It must feel good to know the plan has been implemented!

blacklabel
06-05-2012, 09:05
Great work guys!

Hotchef181818
06-05-2012, 10:55
Not sure if it was site wide or what happened but an hour ago I got a permanent ban message, obviously that has changed now ?? Any thoughts ?

J
06-05-2012, 10:56
Yep. Coffee spilled on the server. Not really. Sorry about this guys, not really banned. Help spread the word.

More details: http://ww2.ar-15.co/forums/showthread.php?t=53152

Musashi
06-05-2012, 10:56
Nice work! Lovin that the site is back up and redundant.

68Charger
06-05-2012, 11:15
Thanks for all the hard work!

Glad you mentioned off-site backups, a live replica of data can just mean you have 2 copies of crap when SHTF.

I keep at least a week's worth of daily backups of a critical database, in case we need to go back to an old state.

kwando
06-26-2012, 09:36
Did we loose some data during this? I had some info in my PM box and now all my PMs are dated back to 10/11.

J
06-26-2012, 09:59
Not during this. But the reliability improvements were done in response to:

http://www.ar-15.co/forums/showthread.php?t=49268

You must not have been around for a while to not notice this.

aix42day
11-13-2012, 09:41
Great job guys, really like the new interface and speed! Thanks for gutting it out on all the issues from the last ISP! Another techie wanna know: servers metal or virtualized?

J
11-13-2012, 09:51
Metal, Dedicated server. We are now no longer mirrored, but that is in the works. We do have triple redundant backups, and I have a temporary place we can move the site in the event the new server catches fire or something of the sort.

Mirroring will be put back in place shortly.

aix42day
11-13-2012, 11:15
Cool, sounds like you have planned to really square away the configuration. As long as you're using Linux and MySQL and not Windoze 8-). Yeah, I'm a Linux/UNIX guy trapped in a MSFT world...

J
11-13-2012, 11:41
100% powered by Windows Vista!! [ROFL2][ROFL2][ROFL2][ROFL2]











Ok....
Ubuntu Server 12.04 LTS x86_64 running MySQL 5.5.28, Apache 2.2.22 and PHP 5.3.10