RAPID CITY, S.D. – A significant disruption in the Amazon Web Services (AWS) US-EAST-1 cloud computing region caused widespread internet service failures and elevated error rates Monday, impacting thousands of websites and applications globally, with potential ripple effects for tech-reliant businesses in South Dakota.
This outage is effecting Adobe, Venmo, Apple Pay, Snapchat and others.
The outage, which began early Monday morning, Pacific Daylight Time, affected multiple AWS services including DynamoDB, SQS, Amazon Connect, and new EC2 instance launches. Amazon’s cloud computing division, the largest provider of its kind worldwide, reported that the initial source of the problem was a “potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region,” later specifying it was “related to DNS resolution.”
The US-EAST-1 region, located in Northern Virginia, is one of AWS’s most crucial and heavily used data centers. Its issues quickly escalated, affecting “global services or features that rely on US-EAST-1 endpoints,” such as Identity and Access Management (IAM) updates and DynamoDB Global tables, effectively reaching customers across the globe and in South Dakota.
Updates from the AWS Health Dashboard indicated a cascading series of issues throughout the morning, with engineers working through recovery and mitigation steps.
- 12:11 AM PDT: AWS first reports “increased error rates and latencies” for multiple services in the US-EAST-1 Region.
- 2:01 AM PDT: Potential root cause identified as related to DNS resolution of the DynamoDB API endpoint.
- 3:35 AM PDT: AWS reports the “underlying DNS issue has been fully mitigated,” and most service operations are recovering, but EC2 instance launches were still experiencing high error rates.
- 7:14 AM PDT: The company confirms “significant API errors and connectivity issues” across multiple services.
- 9:13 AM PDT: AWS takes additional mitigation steps, focusing on an internal subsystem responsible for monitoring network load balancers, noting “connectivity and API recovery” is now occurring.
The disruption underscores the reliance of local economies, including those in Rapid City, on a small number of centralized cloud providers. Companies across South Dakota that rely on AWS for website hosting, data storage, or customer relationship management (CRM) tools may have experienced service degradation or temporary outages that hindered operations and customer interaction throughout the morning.
While services were showing signs of recovery throughout the day, the incident serves as a fresh reminder of the need for robust redundancy planning for any organization leveraging cloud infrastructure. The US-EAST-1 region has been the focal point of several previous AWS outages over the years, leading industry experts to frequently highlight the risk of centralizing critical operations in a single geographical area.