Flow Testnet Network Issue

Yesterday 2/11, Flow Testnet experienced an outage during routine maintenance.

What Happened

The Flow team was performing maintenance to ensure testnet consensus nodes had enough disk space. During this work, a disk space reclamation process was run.

There was a flaw in this process when applied to consensus nodes that caused the DKG file to be deleted. Once a majority of the nodes were updated, they were no longer able to finalize blocks. As a result, finalization halted and the network stopped progressing.

Please note: Mainnet was not impacted.

Recovery

Possible recovery paths were evaluated. The safest and least disruptive way to restore the network was to perform a full network upgrade on testnet.

The upgrade restored consensus and returned the network to a healthy state.

EVM Follow Up

After the upgrade, EVM nodes encountered a separate issue and were not able to smoothly transition to the new network state. Additional remediation was required to bring EVM services back into sync.

Next Steps

• Updating and testing the disk space reclamation process, adding more safety checks to ensure critical consensus state is preserved, along with clearer documentation.
• Improving the EVM gateway to make it more robust during network upgrades and transitions.
• Reviewing alerting and operational safeguards to surface issues more quickly.