[Resolved] "power outage at one of our primary data centers"

Have a question or problem with your Fongo application? This forum is the place to get help from both staff and fellow community members.
Fongo recommends Fongo Home Phone for a fully supported Home Phone system for only $4.95/mo
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: "power outage at one of our primary data centers"

Post by Liptonbrisk »

Erft wrote: 05/28/2026 Thank-you, Liptonbrisk. Although I am currently swamped and don't have time at the moment, I am sure that Method A will work, and if I need to investigate further, you have directed me to Method B, before, and it was a crucial part of solving the puzzle
Part of the Method A is a prerequisite for Method B. That is, the default admin password must be changed via Method A before IVR code 7932# will function.

Anyway, good luck!
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: "power outage at one of our primary data centers"

Post by Liptonbrisk »

Visit https://status.fongo.com/incidents/tctwp1OWzOCe first for information.

"The outage is mostly resolved. There are still a few services out. It actually wasn’t Fongo’s servers that went down but a data centre upstream from us. Some users may still be experiencing some intermittent issues."

If you want to troubleshoot, try rebooting your SIP client (Fongo Mobile users force close and restart the app; Fongo Home Phone and Freephoneline users power cycle or turn off and then on your ATA, IP Phones, etc.).

Otherwise, wait for https://status.fongo.com/incidents/tctwp1OWzOCe to indicate the problem is resolved. You can also visit https://status.fongo.com/.
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
astanlofter
Just Passing Thru
Posts: 16
Joined: 11/23/2017
SIP Device Name: Thomson TG784

Re: "power outage at one of our primary data centers"

Post by astanlofter »

So for that entire hour after the ATA registers successfully, until the registration expires (1 hour), logging in at https://www.freephoneline.ca/showSipSettings will indicate, incorrectly, that your ATA is "connected" (registered). After 3600 seconds pass, the registration expiry times out, and your online account's web portal will finally indicate, correctly, that the ATA is "disconnected" (isn't registered).
Just conducted an experiment: disconnected my ATA for 1h 25min, the status at that link remains "SIP Status: connected"

Also, the way I understand the protocol, from ATA's box perspective, when the connection is lost, it marks the registrar as unavailable and when the connection is restored, it will redo the entire registration sequence immediately (will send a fresh REGISTER request to the SIP proxy/registrar) without waiting for 1h expiration. Otherwise it will not be receiving incoming calls. But I only speculate, didn't verify...

PS. My service got restored (by just reconnecting the cable). But the incident on their status page is still open.
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: "power outage at one of our primary data centers"

Post by Liptonbrisk »

astanlofter wrote: 05/28/2026
Just conducted an experiment: disconnected my ATA for 1h 25min, the status at that link remains "SIP Status: connected"
That is not what I observed previously when I reported a SIP status issue involving Obihai devices to Fongo admin in 2023. At that time, after the next registration interval passed without a new registration attempt, the portal eventually showed "disconnected" (as expected). In this case, the portal status appears not to have updated yet. In other words, that's not normal and may be more indicative of problems related to the data centre outage than anything else.

In any event, your observation supports the broader point that the portal is not a real-time indicator of whether the ATA is currently reachable. SIP registration is soft state (state that expires unless it is periodically refreshed), so the server can retain a stored registration binding after the ATA has gone offline. That is, the portal may continue to display stale registration information, which is why the ATA's current status is the better indicator.
Also, the way I understand the protocol, from ATA's box perspective, when the connection is lost, it marks the registrar as unavailable
That is not how standard SIP works; it is not a rule in RFC 3261. The ATA does not automatically know that the registrar is unavailable the instant the network path drops. It learns that something is wrong when it sends a SIP transaction, such as a scheduled REGISTER refresh or a keep-alive packet, and that transaction fails or times out. If the ATA is idle during the outage, it has no new SIP-level evidence that the registrar is unavailable, so it will not immediately treat the registrar as down.

The ATA may detect the failure later, but it does not necessarily mark the registrar unavailable at the moment the connection is lost. Moreover, during a network disconnection, the ATA cannot send a REGISTER to refresh the binding because no communication path exists to exchange SIP messages with the remote server.

and when the connection is restored, it will redo the entire registration sequence immediately (will send a fresh REGISTER request to the SIP proxy/registrar) without waiting for 1h expiration
RFC 3261 does not require immediate re-registration the moment connectivity returns. It says the User Agent (UA) refreshes its bindings before they expire, and it also says that if a REGISTER request times out, the client should not immediately re-attempt registration to the same registrar, but should wait a reasonable interval. That is not the same as instant re-registration on link restoration. With Freephoneline, keep-alive timing and failed-registration retry timing are separate settings. A missed keep-alive does not by itself establish that an immediate fresh REGISTER will be sent, because RFC 3261 does not require it and Freephoneline’s own interval settings treat those two mechanisms independently.

In other words, RFC 3261 does not mandate that a SIP User Agent be link-state aware. Tying the network interface's physical status (cable plugged/unplugged) to the SIP stack to force an immediate REGISTER request would be a custom firmware design choice by the manufacturer to improve user experience. It is not a SIP requirement. The Thomson TG784 is an integrated router/ATA, and while it is possible that some firmware version could implement link-up-triggered re-registration as an internal recovery mechanism, I don’t see authoritative documentation or public analysis showing that the TG784 does that. Frankly, I would be surprised.

Otherwise it will not be receiving incoming calls.
Registration is required for incoming calls, but failed incoming calls do not by themselves prove that the registrar has already removed the ATA’s registration. They only show that inbound service was not functioning properly at that time. In other words, “incoming calls fail” and “portal still says connected” can both be true at the same time.

On Obihai devices, X_KeepAliveMsgType="options" or "notify" sends active SIP keep-alive probes. According to Obihai’s documentation, a missing response to those probes can trigger proxy failover, but only if proxy redundancy is enabled. Without redundancy configured, the documentation does not establish that a failed OPTIONS or NOTIFY keep-alive forces immediate re-registration. The device continues to rely on its normal REGISTER refresh and retry timers. Any re-registration behaviour observed in that scenario would be a recovery choice made by the endpoint or SIP client, instead of a requirement of the SIP protocol.

PS. My service got restored (by just reconnecting the cable). But the incident on their status page is still open.
That shows service resumed after connectivity was restored, but by itself it does not establish the exact SIP sequence that occurred afterwards. Restoring the cable allowed service to resume, but it does not follow that the ATA sent an immediate fresh REGISTER as soon as the link came back. A packet capture may confirm the sequence, and ATA logs, if available or obtainable, might also show whether a new REGISTER was sent when the cable was reconnected.

Your service could have been restored because reconnecting the cable allowed the ATA’s next scheduled registration request or retry timer to successfully transmit. That may have re-established the NAT mapping in your router and refreshed the SIP binding without requiring an out-of-cycle re-registration, which is consistent with the service resuming once connectivity returned.
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: "power outage at one of our primary data centers"

Post by Liptonbrisk »

Liptonbrisk wrote: 05/28/2026
astanlofter wrote: 05/28/2026
Just conducted an experiment: disconnected my ATA for 1h 25min, the status at that link remains "SIP Status: connected"

That is not what I observed previously when I reported a SIP status issue involving Obihai devices to Fongo admin in 2023. At that time, once the next registration interval passed without a new registration attempt, the portal eventually showed "disconnected" (as expected). In this case, the portal status appears not to have updated yet. In other words, that's not normal and may be more indicative of problems related to the data centre outage than anything else.
Once https://status.fongo.com/incidents/tctwp1OWzOCe is reported as being resolved, I'll test again to see if there's a bug to report.
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: "power outage at one of our primary data centers"

Post by Liptonbrisk »

Liptonbrisk wrote: 05/28/2026 In this case, the portal status appears not to have updated yet.
Liptonbrisk wrote: 05/28/2026
Once https://status.fongo.com/incidents/tctwp1OWzOCe is reported as being resolved, I'll test again to see if there's a bug to report.
Yeah, there's something the matter with SIP Status reporting on the FPL user portal at the time of this post. One of my OBi2182 IP Phones is registered and accepting incoming calls perfectly fine while SIP status at https://www.freephoneline.ca/showSipSettings shows "disconnected". SIP status is not being updated in FPL's web portal at all right now for this account.

Everything isn't fixed yet.

I just checked another FPL account with another OBi2182 that's registered and working fine. I just turned it on. SIP status indicates "disconnected".

Registered on voip.freephoneline.ca

I'll try the other servers . . .

voip2.freephonline.ca:5060 registered, but SIP status shows "disconnected"

Trying voip4.freephone.ca:6060 . . .
Registered, but SIP Status indicates "disconnected"


SIP status updates aren't working using all Freephoneline servers as I type this post (two FPL accounts tested).
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
Erft
Active Poster
Posts: 133
Joined: 05/14/2018
SIP Device Name: Cisco SPA122
Firmware Version: 1.3.3 (015)
ISP Name: Teksavvy, cable
Computer OS: Windows 10
Router: SPA122, LAN, 2 FXS

Calls not logging after datacentre and phone service outage.

Post by Erft »

I'm able to call out, and receive calls, but they're not being logged. The first phone call I made today was at 4.38pm, and it's currently 10.28pm and that call, and the ones I made afterwards, still aren't in the call logs. The last call logged is from 12.30pm yesterday (27 May 2026).
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: Calls not logging after datacentre and phone service outage.

Post by Liptonbrisk »

Erft wrote: 05/28/2026 I'm able to call out, and receive calls, but they're not being logged. The first phone call I made today was at 4.38pm, and it's currently 10.28pm and that call, and the ones I made afterwards, still aren't in the call logs. The last call logged is from 12.30pm yesterday (27 May 2026).
The broken SIP Status reporting further confirms the problems caused by the datacenter power outage.

1. The servers that handle the actual call signalling, routing, and (soft-state) registrations are functioning. That is why your SPA122 can still negotiate outbound calls and successfully receive inbound routes.

2. The Fongo (Freephoneline) web portal relies on a backend database to display your SIP status and record Call Detail Records (CDRs) to populate your call logs. The link between the active SIP proxies and this logging database is currently disrupted (severed, perhaps) or heavily backlogged. Effectively, the web-facing reporting mechanisms aren't working

There's nothing wrong with your equipment or configuration. The call logs (along with the SIP status indicator) likely will not update properly until the data centre outage is resolved and databases are re-synced.
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
Erft
Active Poster
Posts: 133
Joined: 05/14/2018
SIP Device Name: Cisco SPA122
Firmware Version: 1.3.3 (015)
ISP Name: Teksavvy, cable
Computer OS: Windows 10
Router: SPA122, LAN, 2 FXS

Re: Calls not logging after datacentre and phone service outage.

Post by Erft »

It sounds like you believe the logs will update after awhile. That's reassuring. Thanks.
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: Calls not logging after datacentre and phone service outage.

Post by Liptonbrisk »

Erft wrote: 05/28/2026 It sounds like you believe the logs will update after awhile. That's reassuring.
Fongo Mobile users are also affected. I'm pretty sure Fongo Home Phone customers are as well.

They have to fix this database issue. Customers need access to those records to help monitor World Credits, and customers need to provide call log information to support staff in the event of a problem.

Fongo needs to work with 3rd parties to resolve this problem.
Thanks.
You're welcome.
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: Calls not logging after datacentre and phone service outage.

Post by Liptonbrisk »

Liptonbrisk wrote: 05/29/2026
Fongo Mobile users are also affected. I'm pretty sure Fongo Home Phone customers are as well.

They have to fix this database issue. Customers need access to those records to help monitor World Credits, and customers need to provide call log information to support staff in the event of a problem.

Fongo needs to work with 3rd parties to resolve this problem.

"We are having some database replication issues which may show stale data. We are working with [3rd parties] to resolve this." -- Fongo admin
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.
User avatar
Liptonbrisk
Technical Support
Posts: 3675
Joined: 04/26/2010
SIP Device Name: Obihai 202/2182, Groundwire
Firmware Version: various
ISP Name: no CGNAT
Computer OS: Windows 11 Pro (25H2)
Router: Asuswrt-Merlin & others

Re: [call logs for all services and FPL SIP status affected] "power outage at one of our primary data centers"

Post by Liptonbrisk »

The outage is now marked as resolved at https://status.fongo.com/incidents/tctwp1OWzOCe.

Call logs (when calls are answered, including by voicemail systems) and SIP status (in the expected manner I explained in this thread) are now being updated.
Please do not send me emails; I do not work for nor represent Freephoneline or Fongo. Post questions on the forums so that others may learn from responses or assist you. Thank you. If you have an issue with your account or have a billing issue, submit a ticket here: https://support.fongo.com/hc/requests/new. Visit http://status.fongo.com/ to check FPL/Fongo service status. Freephoneline setup guides can be found at http://forum.fongo.com/viewforum.php?f=15.