System Status

On 11/20/2025, between 16:15 PST and 19:05 PST, customers in our DFW02 \(Dallas, TX\) PoP may have experienced increased latency and routing instability. This incident was triggered by an anomalous configuration on a customer device, which subsequently created routing noise across our upstream transit providers. We have resolved the issue by isolating the specific origin from our network and are actively working with the customer to clean up any invalid routes being created by their equipment. Here is a detailed breakdown of what happened, why it happened, and the steps we took to resolve it. _All times listed below are in PST._ ### What Happened At 16:15, we received reports of degradation in inbound route availability into our DFW02 PoP. Our team immediately began investigating the source of the instability. Initial monitoring data pointed to a specific customer server broadcasting bizarre routing updates. At 16:27, engineers began to apply filtering to the rogue server in order to reduce the instability. While this reduced the severity of the impact, it did not fully eliminate the routing issues. By 17:06, monitoring indicated that the instability was isolated to routes received via Arelion, one of our upstream transit providers. To protect the broader network, we removed Arelion from our DFW02 transit blend. Traffic was automatically routed through other providers. ### The Recurrence Following the removal of Arelion, network performance appeared to return to normal levels. However, at 18:20, customer reports indicated that routing instability had started to resurface. The incident was immediately escalated and more team members were brought in to assist. At 18:42, we were able to confirm that the root cause remained on the originally identified customer server. The mitigation measures originally put in place proved to be insufficient to prevent the erratic behavior, which was leaking through despite the access control policies in place. We immediately isolated the server from the rest of the network. ### Resolution By 19:05, all isolation measures were in place. All bad routes were removed, and stability was restored to the PoP. We have since verified that all systems are operating at full capacity. ### Timeline of Events \(PST\) * **16:15:** Engineering receives alerts regarding inbound routing instability at DFW02. * **16:27:** Source identified as a customer origin. Initial filtering applied. * **17:06:** Secondary instability detected via upstream provider Arelion. Arelion is temporarily removed from the transit blend. * **17:30:** Monitoring data indicates temporary stabilization. * **18:20:** Renewed reports of routing degradation. Incident escalated for deeper analysis. * **18:42:** Analysis confirms the original customer server is bypassing initial dampening. * **19:05:** Strict network isolation applied to the affected server. Incident resolved. * **20:50:** Network verified stable and running at full capacity. ### Next Steps Over the next few days, we will be working to update our detection logic to make sure that single origin anomalies are isolated more aggressively in the future, preventing them from impacting all customers. This was the worst outage in our DFW02 PoP yet. We’ve had small hiccups, but this big of an outage was unacceptable. Our systems are architected to be resilient to failure. This has proven to be wrong. In the coming days, our engineering team will be working to roll out changes to prevent these issues from ever happening again. We have high standards for our network, and today we fell short of them. On behalf of the Soraxus team, we would like to sincerely apologize for the issues caused today. -Ryan & Tavas

We are currently experiencing routing issues in our DFW02 PoP.

This incident has been resolved.

We are aware of the outage in our dfw2 PoP. This outage appears to be related to the on-going botnet attacks targeting major T1 carriers. Both Arelion and NTT (our two primary carriers in that PoP) appear to be fully down. Our team is looking into solutions as we speak. We appreciate your patience.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We've identified this to be an upstream issue. We are working with our provider to get it resolved.

This incident has been resolved.

Our team has identified the issue and is actively working to resolve it.

We're currently experiencing an outage with our Tenantos control panel (manage.soraxus.com). Our team is currently investigating the cause of the outage. All customer services and WireGuard instances are unaffected.