Any weird stuff going on on the forums?

[Ridge]

Twilight is loookin a little cracked out in that one.

She spent all day working on the servers with Derpy.

[J]

Some mildly bad news...

It is bad news, but not like it would have been before.

We completely lost the number 2 mirror, total disk corruption. So, from 10:15a until 3:30p yesterday, we did in fact lose approx. 50% of the data during that time frame. Anything that happened on the primary mirror is ok, and anything on either mirror before 10:15a is good. The second mirror was completely removed from the pool at 3:30p. So only half of the data in that time frame is gone, and any other times before and after will all be here.

The #2 mirror, may it rest in peace, did a wonderful job of running the site for the past 2 weeks while the primary was down. So I think we had a big win there by keeping the site up for the past 2 weeks when the hosting provider of the primary had various issues.

We have crazy backups set in place now. But rolling back would reset everything until 3AM yesterday, which is worse. This plan mitigated the impact but didn't eliminate it.

There were various causes of the hard crash of mirror #2, mostly related to being overwhelmed by taking over for primary while it was down. None of them were avoidable, and considering the alternative, I like where we ended up.

Sorry to be the bearer of bad news, but it is much better than it could have been. Unlike last time, if you notice a PM or a post missing, you will have to re-post it.

Thanks and sorry--

--J

[sellersm]

Thanks for the good work! I know what PITA this stuff is...

[ghettodub]

Some mildly bad news...

It is bad news, but not like it would have been before.

We completely lost the number 2 mirror, total disk corruption. So, from 10:15a until 3:30p yesterday, we did in fact lose approx. 50% of the data during that time frame. Anything that happened on the primary mirror is ok, and anything on either mirror before 10:15a is good. The second mirror was completely removed from the pool at 3:30p. So only half of the data in that time frame is gone, and any other times before and after will all be here.

The #2 mirror, may it rest in peace, did a wonderful job of running the site for the past 2 weeks while the primary was down. So I think we had a big win there by keeping the site up for the past 2 weeks when the hosting provider of the primary had various issues.

We have crazy backups set in place now. But rolling back would reset everything until 3AM yesterday, which is worse. This plan mitigated the impact but didn't eliminate it.

There were various causes of the hard crash of mirror #2, mostly related to being overwhelmed by taking over for primary while it was down. None of them were avoidable, and considering the alternative, I like where we ended up.

Sorry to be the bearer of bad news, but it is much better than it could have been. Unlike last time, if you notice a PM or a post missing, you will have to re-post it.

Thanks and sorry--

--J

Thanks for the info, I was wondering where a post I put up went.

Thanks for your work, also

[StagLefty]

Well at least Stuving can't get on his repost bandwagon
Thanks for all your hard work