CenturyLink - Anyone else having fun? :D

[rondog]

I work for CTL in the NOC center, and it's been causing havoc and ulcers for a lot of people - but not me, fortunately, this is a hardware issue, not a fiber cut. Apparently a card (circuit pack) went haywire somewhere in the US and it affected comms to all nodes nationwide of the same manufacturer. NOC had no visibility to that network, so pinpointing the source has been a nightmare. Glad I'm not a field tech anymore!

[CS1983]

I work for CTL in the NOC center, and it's been causing havoc and ulcers for a lot of people - but not me, fortunately, this is a hardware issue, not a fiber cut. Apparently a card (circuit pack) went haywire somewhere in the US and it affected comms to all nodes nationwide of the same manufacturer. NOC had no visibility to that network, so pinpointing the source has been a nightmare. Glad I'm not a field tech anymore!

This aligns with what we were hearing on the bridge.

Per what we heard: Essentially your core architecture was engaging in behavior similar to a broadcast storm (but not a broadcast storm), spiking CPU usage on devices, failing over to another set of devices, rinse repeat.

Sounds like your vendor for hardware ain't sure why it did this either.

Any word on speculation of a virus geared toward the hardware itself to cause this behavior or just a pure hardware malady? Based on what I was hearing, it sounds distributed to your entire core, which makes a pure hardware issue highly odd.