why was fongo down last night around 3am?

[killerjay]

Last night at around 3am my connection to fongo was down, the light was blinking red and i had no phone. I tried to go to this forum and see if there was an announcement for maintenance or something but the entire site was down as well. i went to bed at 4am and it was still down. all is good now but im wondering if that was on my end or yours? what happened?

[tbrummell]

The logical question to ask is, were you able to go to any other site at the time of the outage? That will give you a good starting clue s to whether it is your connection, or FPL.

[tbrummell]

Looks like it was FPL:

Lost Registration:

Code: Select all

[2012-08-30 02:50:42] NOTICE[29504] chan_sip.c: Peer 'FPL' is now UNREACHABLE!  Last qualify: 38

Some 425 attempts later...

Code: Select all

[2012-08-30 05:33:14] VERBOSE[29504] dnsmgr.c:        > doing dnsmgr_lookup for 'voip.freephoneline.ca'
[2012-08-30 05:33:14] NOTICE[29504] chan_sip.c:    -- Registration for '1613729XXXX@voip.freephoneline.ca' timed out, trying again (Attempt #425)

Finally regained Registration:

Code: Select all

[2012-08-30 05:33:34] NOTICE[29504] chan_sip.c: Peer 'FPL' is now Reachable. (34ms / 2000ms)

Maybe the powers that be didn't even notice, or maybe they didn't want us to notice!

[killerjay]

hmm, funny how theres no official comments... try to keep failures on the down low...

[dibsmft]

Perhaps, the server was doing nightly backups and there was a long time-out. It is not unusual for daily backup to be done in the early morning when the load is at it minimum but in Canada with so many time zones it is difficult to decide exactly what is early morning.

[TheHardy]

hmm, funny how theres no official comments... try to keep failures on the down low...

While FONGO as a whole could do a better job with stickies and official notifications, the forums are a voluntary thing that only a few employees participate in.

Also, for the vast majority of issues, if they resolve themselves, or if not very many people complain about them, I am sure they do NOT publicize them.

To be completely fair, as FONGO is backboned on Fibernetics, and they have their OWN techs, I am positive that there are a lot of 'behind-the-scenes' minor issues (and a few larger or major ones) that go unreported/unpublicized DAILY --- they are simply found and corrected without any fanfare or announcement. This is the way that all major operations are....

Heck, I am the fleet maint guy at my company --- major truck issues requiring an outside mechanic or serious failures are reported to management, but I add oil, change wiper blades, tighten loose bolts all the time ... all without telling anyone. If a driver has put in a "punch sheet" for a defect, I will reply to that, but other than that, anything under 1/2hr labour generally is undocumented except in my maint logs ....

I don't think anyone is trying to HIDE anything, but doing a PR campaign on every small cut, blister and wart is not what FONGO forums are all about either!

[tbrummell]

Hate to burst your bubble, but 2.5 hrs of outage is not a small cut or blister in the telecom wold, that's some major down time. A few seconds to a minute of down time is a small cut or blister. Achieving "5 9's" of uptime with a 2 1/2 hour outage just isn't possible. It wouldn't cut it for a business account either, but since this is just the wife's phone, I let it slide.

[TheHardy]

I was not trying to minimize anyone's outtage, and I have also been asking Fongo to be more transparent on what is going on, to the point of asking for a STATUS page (or at least message/baner) that lists current issues. Of course, this would also need the tech side of things and back office (both Fongo and Fibernetics) to actually update what is going on, and that may be the catch right there.

As for the 2.5 hours, I really have not seen too many other complaints about that timeframe from other users, so perhaps the outtage was not widespread, and may have been DNS related or locallized rather than a network issue?

[tbrummell]

As for the 2.5 hours, I really have not seen too many other complaints about that timeframe from other users, so perhaps the outtage was not widespread, and may have been DNS related or locallized rather than a network issue?

Unless you sit and stare at your ATA, or try and use your phone during that time frame, you wouldn't have noticed. Now, if you run a server of some sort, you can at least go back and check logs like I did. I used to have a script checking for me, but during the last extended service interruptions as it was emailing me every 5 minutes. I never enabled it after that.

[tbrummell]

It is still kind of funny that this whole thread continues and not a peep out of FPL staff. Hmmmm

[Jake]

It is still kind of funny that this whole thread continues and not a peep out of FPL staff. Hmmmm

Are you really that surprised

[tbrummell]

It is still kind of funny that this whole thread continues and not a peep out of FPL staff. Hmmmm

Are you really that surprised

No, no, I guess not.

[TheHardy]

Being that it is a voluntary participation forum for the Fongo staff and Steve is kinda outta town, that leaves less ever-probing eyes .... anyone taken to PM'ing Mike to bring his attention to this thread?

[FONGO_mike]

In a case such as this, we are not receiving any reports of outages form people calling in. If there was an outage, it is looking as though it was a VERY small pocket of individuals who may have been affected. In this particular case, i have not personally seen any reports of anyone loosing connection for two and a half hours other then what i have read here.

[Askarii]

Our connection is still down (since 7 am this morning, it is now 11h30 am)

I am at an internet cafe at the moment, since I have no means to contact freephoneline support.
I restarted the router, this gave nothing.

I see the light "Broadband connection" but the phone line 1 and internet lights are off. i have a Thompson router which was shipped with my freephoneline bundle.

[Jake]

Our connection is still down (since 7 am this morning, it is now 11h30 am)

I am at an internet cafe at the moment, since I have no means to contact freephoneline support.
I restarted the router, this gave nothing.

I see the light "Broadband connection" but the phone line 1 and internet lights are off. i have a Thompson router which was shipped with my freephoneline bundle.

I had the same problem. I shut down the modem for about 20 minutes and powered it back up - but that didn't help. Mine came back online at around 9am EST this morning.

[FONGO_mike]

At this time, we know that a modem reboot is required. Unplug the power cord from the modem for 30 seconds to 1 minute, then plug it back in.

[watermark]

Registration timeout is happening now, PDT 3:04pm. No phone works.

-- Got SIP response 480 "Temporarily Unavailable" back from 208.65.240.165

Checked again. Registration was back to normal at 3:07. SIP registration is unstable recently.

[CSCO]

It was down last night and it is down NOW. I woke up to a reg failed message on both of my FPL lines, 4 other providers I have were unaffected.


[4:16PM MST] It just came back. To be honest, none of my other providers have those mini outages. I have multiple ways of monitoring this and ping is usually alive when this occurs.

I'm not sure how tech at the fongo are monitoring servers, but since they say they don't see those outages, I suspect they are using ping. This won't show outages where ping is good but SIP registration fails. They should use TCP handshake or actual sip registration, if not doing so already.

[FONGO_steve]

Registration timeout is happening now, PDT 3:04pm. No phone works.

-- Got SIP response 480 "Temporarily Unavailable" back from 208.65.240.165

Are you still timing out? Although I'm not in the office for a few weeks, I was in contact with the office most of the afternoon today and did not hear of any issues at this time. I also haven't noted any other comments in the forums about problems this afternoon either.

[Jake]

Mine just went down for about 30 minutes, but it is back again now.

[watermark]

Registration is fine now. The down time was from 14:53 to 15:03 in PDT (GMT - 7) according to my server log.

Registration timeout is happening now, PDT 3:04pm. No phone works.

-- Got SIP response 480 "Temporarily Unavailable" back from 208.65.240.165

Are you still timing out? Although I'm not in the office for a few weeks, I was in contact with the office most of the afternoon today and did not hear of any issues at this time. I also haven't noted any other comments in the forums about problems this afternoon either.

[FONGO_steve]

We have posted an announcement about maintenance that will be happening overnight tonight for teams to do some work regarding the downtime experienced today.

At this time I can't comment any further on the work being done because I'm about 2000 miles away from the office with fairly limited contact

[watermark]

The following is the recent down time in PDT (GMT - 7):

[Aug 29 23:51:03] ~ [Aug 30 02:34:30]
[Sep 1 01:02:27] ~ [Sep 1 01:03:47]
[Sep 5 23:19:54] ~ [Sep 5 23:29:20]
[Sep 6 05:13:40] ~ [Sep 6 06:39:04]
[Sep 6 14:55:10] ~ [Sep 6 15:03:05]

[watermark]

Registration times out again since "[Sep 6 15:55:41]" and is still going on.

It was back to normal on 16:04 PDT.