📅 Incident Summary
- Date: May 23, 2025
- Duration: 6:00 AM – 10:00 AM Pacific Time
- Impact: Transaction execution on Flow Testnet halted.
- Status: Resolved
🧨 Root Cause
A specific transaction submitted to Flow Testnet triggered an unhandled edge case in the Cadence resulting in a complete halt in transaction processing.
✅ Resolution & Fixes
Several targeted Cadence code fixes were implemented promptly to mitigate the issue:
- 🛠️ Parser fixes to handle the edge case.
- 🛠️ Error reporting improvements for better diagnostics.
- 🛡️ Defensive parsing logic added via Cadence PR #3974
These fixes were deployed through an Height Coordinated Upgrade (HCU) on testnet and then subsequently an HCU mainnet on the same day.
🧪 Prevention & Follow-Up Actions
To avoid similar incidents in the future, the following preventive measures are being implemented:
1. Enhanced Fuzz Testing
- Setup regularly run of the
cadence-fuzzer, a tool designed to generate and test random Cadence programs as part of CI/CD (Cadence issue #3985)
2. Consider Pre-execution Simulation (long-term)
- Introduce simulation of transactions in a controlled environment (e.g., Access Node) before execution on core nodes to catch anomalies early. However, this is a long term strategy and needs to be investigated further.