View Full Version : CenturyLink - Anyone else having fun? :D
https://www.cnet.com/news/centurylink-customers-get-belated-coal-in-their-stockings-with-hours-long-nationwide-outage/
Users of the internet, phone and TV provider's services have been hit by a nationwide outage, in places including Idaho (https://www.idahopress.com/news/local/nationwide-centurylink-outage-reported-affecting-customers-in-boise/article_871e0091-a257-5e18-b23a-88988db6907d.html), New Mexico (https://www.krqe.com/news/new-mexico/albuquerque-officials-report-verizon-outage-affected-911-call-center/1676481817) and Minnesota (https://kbjr6.com/iron-range/2018/12/27/911-outage-reported-for-st-louis-county/). It's unclear what's caused the outage, or why it's stretched to now more than 12 hours. DownDetector reported initial signs of the outage around 4 a.m. ET Thursday (https://downdetector.com/status/centurylink).
More at link.
On a bridge right now related to this.
CenturyLink is pulling people off of vacation, vendors on horn, etc. Affecting CL services across the nation.
Wish I could repeat some of the invectives going back and forth between two respective executives on different sides of the issue, but I won't.
Our VPN over CL was horribly sluggish today, so everyone wound up using the Zayo route instead... guessing they were saturating what wasn’t offline, heavily.
Figured it was just me having issues. Internet kind of coming and going throughout the day.
theGinsue
12-27-2018, 19:09
Our VPN over CL was horribly sluggish today, so everyone wound up using the Zayo route instead... guessing they were saturating what wasn’t offline, heavily.
Figured it was just me having issues. Internet kind of coming and going throughout the day.
I thought it was me too! I downloaded a 1.2Gb file earlier @ an average of 30Kb/s. About once every ten minutes I'd get a 1 second spike up to 1.7Mb/s. To say it took a horrendous amount of time is an understatement.
I thought it was me too! I downloaded a 1.2Gb file earlier @ an average of 30Kb/s. About once every ten minutes I'd get a 1 second spike up to 1.7Mb/s. To say it took a horrendous amount of time is an understatement.
Yeah, my porn downloads were horrible too.
The element we work with is seeing certain geographic regions down which aren't down for us, including their SDPs at locations I can ssh into on the other side of the SDP. Riddle me that, batman.
Just remoted in to my centurylink box (from my comcast box) and it was fine.
-John
hurley842002
12-27-2018, 19:38
Fine in Littleton.
gnihcraes
12-27-2018, 19:42
Questionable in Lakewood. Slow at times.
Sent from my SM-T813 using Tapatalk
SideShow Bob
12-27-2018, 19:56
So that?s what it was ! I was trying to get a new all in one printer hooked up and going. It was throwing me fits to get the drivers downloadedto my MAC and tablets wirelessly set up.
And of course the magenta ink cartridge squirting ink all over was a hoot....... Not !
The element we work with is seeing certain geographic regions down which aren't down for us, including their SDPs at locations I can ssh into on the other side of the SDP. Riddle me that, batman.
Probably something to do with the flux capacitor I bet.
I think it's their monitoring toold lagging. Once we said we could get into our stuff they finally tried actually connecting to theirs, rather than just look at a status tool. Imagine that.
theGinsue
12-27-2018, 21:58
Yeah, my porn downloads were horrible too.
Brown Chicken, Brown Cow.
I think it's their monitoring toold lagging. Once we said we could get into our stuff they finally tried actually connecting to theirs, rather than just look at a status tool. Imagine that.
Do you mean to tell me that SolarWinds isn't actually the end-all/be-all for troubleshooting a network? Say it isn't so! Heads will roll I tell you...heads will roll! :)
I've never used SolarWinds. We use another product. I'm actually kinda worried since that's all I ever see in other job ads.
theGinsue
12-27-2018, 22:07
Meh. If I can learn SolarWinds, then you can learn it easily.
One of my guys used it in the AF. He said he hates it and much prefers what we use. It's just another tool. Any tool can be learned, I suppose.
gnihcraes
12-27-2018, 23:31
Not a fan of solarwinds, by the time you figure out what might be the problem, it doesn't exist anymore. Maybe ours is poorly configured, but never have liked using it.
Sent from my SM-T813 using Tapatalk
Well looks like I am having CL fun. Cancelled the service Nov 29th, but of course they did not. So now I have to find all the passwords and account info. Stupid asses
Grant H.
12-27-2018, 23:53
SolarWinds has it's place, but it has to be configured/managed VERY well, or it blows donkey balls.
I never got Spiceworks to a place that I was willing to trust it to any of the networks that I look at.
thedave1164
12-28-2018, 07:41
Most places don’t ever get SolarWinds configured right, most of it has to do with the SW salesman selling them the basic package, omitting the fact that in order to really get what the salesman said you would get, you have to buy all these modules/clients or what ever they are calling it this week.
So a lot of these places go back to relying on their old school ping tools, and SW just sits there......
Does SW manage/monitor via snmp polling or what?
Tool config seems to be a problem regardless of the vendor. Our Network Defense guys have certain tools that, in combo with their conspiratorial mindset, causes them to freak out over nothing. We have had it in the past where our Network Monitoring Tools team hasn't updated our tool and so we have to scramble, only to find out nothing is wrong and they didn't do something right (if at all) when a change happens.
whitewalrus
12-28-2018, 12:54
Never had a problem with their service, but I have had plenty of fun over the years with their billing [hahhah-no]
Tools usually aren't the problem. It's how people define them and configure them that's usually the problem. I've seen very intelligent tools after the monkey-in-the-middle rips all of the intelligence out of it.
...only to have a new tool brought in to resolve the shortcomings of the previous and get morphed right back into the same garbage you had before. Repeat.
Well, that certainly explains a lot. I thought it was just us.
hurley842002
12-28-2018, 16:18
Fine in Littleton.May have spoke too soon, it started getting really slow shortly after I posted it was fine.
.455_Hunter
12-28-2018, 16:39
Our entire Corp infrastructure was hit hard yesterday (HQ in Albuquerque). With bulk of the workforce out of the office, the OWA was down, and the remote VPN was overloaded.
I work for CTL in the NOC center, and it's been causing havoc and ulcers for a lot of people - but not me, fortunately, this is a hardware issue, not a fiber cut. Apparently a card (circuit pack) went haywire somewhere in the US and it affected comms to all nodes nationwide of the same manufacturer. NOC had no visibility to that network, so pinpointing the source has been a nightmare. Glad I'm not a field tech anymore!
OtterbatHellcat
12-28-2018, 16:44
Super irritating lag here the last few days. Recurring periods of ten minutes of channel, mixed with two minutes of buffering.
Great during a movie OR anything live.
I work for CTL in the NOC center, and it's been causing havoc and ulcers for a lot of people - but not me, fortunately, this is a hardware issue, not a fiber cut. Apparently a card (circuit pack) went haywire somewhere in the US and it affected comms to all nodes nationwide of the same manufacturer. NOC had no visibility to that network, so pinpointing the source has been a nightmare. Glad I'm not a field tech anymore!
This aligns with what we were hearing on the bridge.
Per what we heard: Essentially your core architecture was engaging in behavior similar to a broadcast storm (but not a broadcast storm), spiking CPU usage on devices, failing over to another set of devices, rinse repeat.
Sounds like your vendor for hardware ain't sure why it did this either.
Any word on speculation of a virus geared toward the hardware itself to cause this behavior or just a pure hardware malady? Based on what I was hearing, it sounds distributed to your entire core, which makes a pure hardware issue highly odd.
Bailey Guns
12-28-2018, 17:04
Still no 911 service in many areas up here including where we are.
ChickNorris
12-28-2018, 20:20
FYI. In the CTL acquisition of L3, they failed to appreciate & retain the L3 talent & this is the fallout.
Next thing you know CTL's BGP is going to be routing all of us through China, AGAIN.
Poit
Delfuego
12-28-2018, 21:49
I have been kinda laughing about this. Ever since Centurylink bought Qwest it's been downhill and fast. They started treating businesses like home users. They made no improvements for years. They fired almost anybody that was good at their job and earned a decent paycheck. We moved everyone away from Centurylink to other providers. I hope these new ISP monopolies are torn apart or burned to the ground. To bad nobody in Washington can even power on a computer or read an email without and intern to help them...
Was just reading an article this morning that CenturyLink was getting rid of some jobs. Sounds like they've got this all figured out. [hahhah-no]
Still no 911 service in many areas up here including where we are.
fcc is opening an investigation for that, apparently yakima was down for a long time
OtterbatHellcat
12-29-2018, 02:58
Still observing super high quality interruptions and lack of service here. Periodic failure of internet and television.
Many of our Corporate offices US wide were still being affected late today last I saw an email.
Thankfully, I work from home and have comcast. Client in AZ I was working remote with was having some occasional serious lag on our webex, to the point of dropping a few times, unsure if they have CL, I didn't think to ask at the time as I just wanted to get done.
Would depend on whomever the local exchange carrier (LEC) is, too, and if they are leasing out CL backbone, I'd think.
FYI. In the CTL acquisition of L3, they failed to appreciate & retain the L3 talent & this is the fallout.
Next thing you know CTL's BGP is going to be routing all of us through China, AGAIN.
Poit
This has been my thought. We were calling people in off of vacation as well on Thursday trying to troubleshoot this. And I've been concerned ever since CL bought L3, because CL is a shit ISP and now they own the company that runs the backbone of the WWW.
ChickNorris
12-29-2018, 10:37
Would depend on whomever the local exchange carrier (LEC) is, too, and if they are leasing out CL backbone, I'd think.
The architecture has changed some.
ChickNorris
12-29-2018, 10:46
fcc is opening an investigation for that, apparently yakima was down for a long time
Properly routing 911 has always been irksome for IP. Yep, edge cases galore.
Naarrrrf
ChickNorris
12-29-2018, 20:18
Um, i know what you're thinking...get your IPV6 off my IPV4, riiiight?
FYI. In the CTL acquisition of L3, they failed to appreciate & retain the L3 talent & this is the fallout.
Next thing you know CTL's BGP is going to be routing all of us through China, AGAIN.
Poit
Actually, just the opposite. CTL's people are the ones being herded out, and L3's people are taking over everywhere. It's crazy.
ChickNorris
12-30-2018, 06:08
Actually, just the opposite. CTL's people are the ones being herded out, and L3's people are taking over everywhere. It's crazy.
Um, retain L3 talent. : )
Um, retain L3 talent. : )
You don't want to know my opinions of L3.
ScottR65
12-30-2018, 13:45
You don't want to know my opinions of L3.
I?ll take that bait!
Sent from my iPhone using Tapatalk Pro
ChickNorris
12-30-2018, 21:01
You don't want to know my opinions of L3.
Why, was I unclear? I doubt it differs much if at all, from mine.
I said they failed to retain 'the talent' . Can't speak to them there 'people who are taking over everywhere' now. Following?
& I promise, my little opinion doesn't count. ZORT
OtterbatHellcat
12-30-2018, 21:58
Stabby, stab, stab.
I love that.
Report: Huge CenturyLink outage caused by bad networking card in Colorado (https://www.geekwire.com/2018/report-huge-centurylink-outage-caused-bad-networking-card-colorado/)
A widespread outage on CenturyLink’s cloud network that lasted almost two days and hampered emergency phone service in Washington and other states was caused by a faulty network management card, according to a customer notice.
Brian Krebs, a veteran security journalist, posted a copy of a notice sent to CenturyLink’s “core customers” to his Twitter feed Saturday that blamed a card at its data center in Colorado for “propagating invalid frame packets across devices,” causing a series of issues that forced the company to reboot much of its networking equipment. It took CenturyLink more than two days from when it first identified the issues to sound the all-clear on Saturday morning, a period during which 911 services in several states including Washington were down or spotty.
Al Gore would have prevented this from happening! Gore in 2020!!
https://media.giphy.com/media/l0HlGmbxkwkICKykM/giphy.gif
Powered by vBulletin® Version 4.2.3 Copyright © 2025 vBulletin Solutions, Inc. All rights reserved.