EVM Endpoint Degraded / Not Responding

Incident Report for Flow

Postmortem

Postmortem: EVM Endpoint Incident - December 5, 2025

Introduction

On December 5, 2025, the Flow EVM JSON-RPC endpoint experienced five hours of unresponsiveness, followed by an additional five hours of degraded performance characterized by very high response times. During this period, the chain itself remained fully operational and continued producing blocks normally but EVM based apps and wallets had trouble interacting with the chain. This postmortem summarizes what happened and the steps we are taking to prevent similar issues in the future.

Summary

A load test earlier that morning revealed an inherent limitation in the current EVM Gateway architecture: gateways that are simultaneously ingesting blocks with a high number of transactions and serving RPC traffic can fall behind the chain. Once behind, they become unable to serve RPC requests efficiently.

This caused the public EVM endpoint to be non-responsive for several hours, followed by degraded performance until additional gateway nodes were added. Horizontal scaling ultimately restored service to normal levels.

Action Items to Prevent Future Incidents

1. Gateway Performance Investigation

Deep-dive into the root contention between block ingestion and RPC serving.
Reproduce the issue in controlled environments to isolate performance bottlenecks.
Implement architectural or performance improvements to ensure gateways remain responsive under heavy load.

2. Stronger Detection & Alerting

Add detailed alerting for RPC latency and slow endpoint behavior for the public endpoints.

Posted Dec 10, 2025 - 23:12 UTC

Resolved

Gateway performance has fully recovered and Flow EVM endpoint is now operating normally.

Flow mainnet remained healthy throughout and we will continue monitoring to ensure everything stays stable.

Thanks for your patience.

Posted Dec 06, 2025 - 06:43 UTC

Update

All EVM gateway nodes have now fully caught up and are serving traffic normally. However, we are observing a high volume of clients attempting to backfill data, which is causing elevated load on the nodes. As a result, response times may remain higher than usual while the system processes this backlog.

Our team is actively monitoring node performance and working to mitigate saturation. We will provide another update as conditions improve.

Posted Dec 06, 2025 - 02:23 UTC

Monitoring

We’re seeing temporary degraded performance on the EVM gateway due to elevated traffic during a scheduled mainnet load test earlier in the day.

Flow mainnet is operating normally, but the indexing lag on the gateways is causing reduced responsiveness for Flow EVM apps.

We are working on scaling the gateway capacity and bringing indexing back to real time now. We’ll share another update once everything is fully back to normal

Posted Dec 06, 2025 - 01:23 UTC

Investigating

We are currently aware of an issue where the EVM Endpoint is not responding or is responding significantly slower than expected. This may affect applications relying on EVM RPC calls and related functionality.

Current Action:
We are actively investigating the root cause and working to restore normal performance as quickly as possible.

Next Update:
We will provide an update as soon as more information becomes available.

Thank you for your patience.

Posted Dec 05, 2025 - 22:15 UTC

This incident affected: Flow mainnet Access APIs (EVM Gateway).