APRIL 26 2010 - SERVICE INTERRUPTIONS

[FONGO_kris]

APRIL 26 2010 - SERVICE INTERRUPTIONS EXPERIENCEDS

THIS ISSUE HAS BEEN REOLVED, THOUGH IS STILL CURRENTLY UNDER INVESTIGATION

Please be advised, a recent disk failure caused a major outage that affected all areas. We are working on implementing a backup server to keep service semi-operable until the full replacement is installed.

ETA: Before end-of-day Monday April 26th 2010.

The backups may still cause intermittent issues in some areas but is expected to be fully resolved within 4 hours when the replacement is live.

We are working in disaster mode here to get this resolved as fast as possible. Please hold tight and check back to this post for updates as I get them. We here at Freephoneline.ca sincerely apologize for the troubles and delays caused by this. It was something we wern't able to expect as it was a sudden and abrupt failure.

[UPDATE]: We were able to get most areas of service restored; to those areas still affected we've been able to restore FollowMe.

[FONGO_kris]

[UPDATE - THE GIST - 20:43PM]

We've repartitioned one of the redundant disks from the storage array and put it in with a new file system.

Space shouldn't be an issue, since the storage / swap space as roughly doubled in size.

Our SIP Server is now commencing rebuilding itself completely, but the main point is that service is restored and should be working (if not better) than before.

Please note that there is an on-going investigation into why this happened, and how to prevent it in the future.

[bluevenom]

Hi,
Im am still not able to make calls....
Will it be working soon?

[kpsingh]

Dear Admin,

After this upgrade It's taking around 20-25 seconds of delay in call connection when I am calling numbers of other Networks (e.g Rogers , Bell etc ). But if I call other Freephone Number or my voicemail then its normal timing. Is there any impact on Dial Plan after this upgrade ? or its delay in SIP server...

Thanks

[bluevenom]

I am experiencing the same issue.

[FONGO_kris]

I may add that this was not an upgrade but an equipment failure, that was resolved in part by the quick efforts of our amazing network engineers.

The server is still rebuilding and recompiling itself, we will continue to investigate and test to see where this failure came from.

[leper1983]

So is everything supposed to be working again.. because it seems its not working for moi

[Liptonbrisk]

Dear Admin,

After this upgrade It's taking around 20-25 seconds of delay in call connection when I am calling numbers of other Networks (e.g Rogers , Bell etc ).

I'm having the exact same issue. And calls from my Telus cellphone to my freephoneline number are now going straight to voicemail.

[FONGO_kris]

Thank you all for your input, we will be updating you continuously today (April 27th, 2010) of any developments regarding this issue. For any further updates, please Private Message, or Email Steve or I directly, as we don't want to clutter the board up too much with the same issue

Please also note that though you may email us today, we may not get back to you ASAP since most of our time will be dedicated to finding the root of this troublesome weed.

[et750]

Kris, Just one suggestion. I believe the confusion is in the title which states "[RESOLVED]", which is causing people to continue to post and possibly send you PMs. You might consider changing it to [Under Investigation].

Thanks,

[bluevenom]

All seems to have been returned to normal. The 20 sec delay is gone, all calls working as usual.
Great work on getting everything up and running so quickly. Hopefully there will be no more failures, but its understandable.

Great work guys.

[FONGO_steve]

All seems to have been returned to normal. The 20 sec delay is gone, all calls working as usual.
Great work on getting everything up and running so quickly. Hopefully there will be no more failures, but its understandable.

Great work guys.

Thanks - we try our best.
There's a lot of investigation going on here today around this incident still, so some planning and future-proofing will certainly be done. We're always trying to bring you the most reliable service we can!

[abhanwa]

I am still experiencing an issue when is this likely to be resolved ??

is there an ETA on restoring the services, ?

are there no high availability or redundency built into the infrastructure ?


WOW

[FONGO_kris]

When you say "an issue" what exactly are you experiencing.

In fact we have 2 redundant servers to help maintain everything until we are able to implement a brand spankin' new, permanent system.

[sternem]

I still have a long delay before an outgoing call rings. 15 seconds or so. Definitely longer than before.

[FONGO_kris]

Delay before calls is most likely your dial plan, make sure you're using the one on our website (If it's a Linksys/Sipura device). If you are, paste it exactly as it appears here.

[sternem]

Hi Kris,

I have a slightly modified version:

(911<:@gw0>|<#8,:>x.<:@gw0>|<#7,:>x.<:@gw1>|<#6,:>x.<:@gw2>|[2-9]xxxxxxxxx|1xxxxxxxxxx|110xxxxxxxxxxxx|98*|[6-7]x*xxxxxxxxxxx)

I'm pretty sure the "[2-9]xxxxxxxxx" I have should be causing the correct match without delay right? The delays started coincidentally since Tues morning.

[FONGO_kris]

Does:

(911<:@gw0>|<#8,:>x.<:@gw0>|<#7,:>x.<:@gw1>|<#6,:>x.<:@gw2>|[2-9]xxxxxxxxx|1xxxxxxxxxxS0|110xxxxxxxxxxxx|98*|[6-7]x*xxxxxxxxxxx)

work for you?

On a side note, nice job on the gateways, wish I'd thought of that...

Keep in mind the '.' means "unlimited amount of numbers" after the initial 'x', which means about 15+ seconds and the ',' is a pause. Make sure you keep track of where these are located. Add an "S0" after any numbers you want dialed immediately (ie. <411:18004664411S0> would dial Google411 immediately after hitting 411, no pause).

What sort of PSTN gateway are you using? (assuming since you have the <:@gw0>)

[abhanwa]

Until yesterday everything was working great but now the ATA is failing to register

I am using voip.freephoneline.ca for porxy and OB proxy, I was told when I called support the issue is being worked on ?

any update ?

[FONGO_kris]

The original issue was resolved despite what anyone will tell you. This was the original hard disk failure, which was replaced by a disk with twice the space on it, thus twice the swap space etc....

The issues we're experiencing now appears to be with the authentication database which was linked to it, which yes, is being worked on feverishly.

There are a few steps involved, backing up the data for a worst-case scenario, reformatting a partition of the disk for the database, then expanding it to it's full size once it's restored, now there are dozens of other steps that happen here but those are really the ones that are effecting you.

We have many people on this who've not caught a wink since Mondays incident, so it's safe to say that there are many eyes on it and that it will be resolved as soon as humanly possible.

[sternem]

Does:

(911<:@gw0>|<#8,:>x.<:@gw0>|<#7,:>x.<:@gw1>|<#6,:>x.<:@gw2>|[2-9]xxxxxxxxx|1xxxxxxxxxxS0|110xxxxxxxxxxxx|98*|[6-7]x*xxxxxxxxxxx)

work for you?

On a side note, nice job on the gateways, wish I'd thought of that...
...
What sort of PSTN gateway are you using? (assuming since you have the <:@gw0>)

So I tried "S0" where you stated, tried dialing with a 1905xxxxxxx, but it still took 12 seconds. Then I added an S0 after a few other rules I had and the last one took. Then I tried the simple 1905xxxxxxx and it was quick again. I took out all the "S0" entries (basically back to what I had) and things remained quick. I can't explain it, but it seems like things are back to normal .

As for the PSTN, that's still a Bell line. I'm guessing that even after I do my number transfer, 911 could still be handled by Bell (though I'd have no paid services through them) and thus can get me connected faster to emergency services?

(As for the delay in number port, I haven't had a chance to dial in during 9-5 )

[FONGO_steve]

As for the PSTN, that's still a Bell line. I'm guessing that even after I do my number transfer, 911 could still be handled by Bell (though I'd have no paid services through them) and thus can get me connected faster to emergency services?

Since they will likely physically disconnect your loop from their DSLAM, I'm going to say that likely won't work. Anyone want to confirm?

[Liptonbrisk]

*67 isn't working properly.

[FONGO_kris]

Yes, this interruption seems to have reset quite a few things, we're working on getting this back up and running. Currently no ETA.

[MikeD]

*67 isn't working properly.

*67 hasn't work for me in over about a month - when was it last working for you that you know of?