On February 14, 2025 the Chia blockchain experienced an issue where many farmers were suddenly unable to form blocks. Fortunately during this time there were still a portion of the network that was able to continue farming to keep the chain moving forward but it did impact users in noticeable ways.
Although the network never actually halted and transactions continued to be made, the situation could have been worse. The impact was mitigated due to existing client diversity, some foresight, and a bit of luck.
While we wait for the official post mortem, let’s dive into what we know so far, the impact, and the resolution.
The Cause
The primary trigger for this issue seems to be an invalid spend bundle in the mempool, created by an enterprising developer learning Chialisp and innocently playing with singletons. This spend bundle was somehow replicating itself and filling the mempool with a chain of invalid transactions.
As a refresher, the mempool can be thought of as a staging area for transactions with full nodes constantly syncing mempool items with each other. Each full node may have a slightly different view of the mempool at any given time. When a farmer finds a valid proof of space (“wins a block”), the full node that it is connected to will propose a new block filled with valid transactions selected from the mempool. One important decentralizing feature of Chia is that the official pooling protocol allows farmers that run their own full nodes to propose their own blocks.
In recent versions of the Chia reference software (2.2.0 to 2.5.0), upon winning a block the process of checking the validity of these invalid transactions in the mempool would take a very long time — much longer than the ~30 second time limit for responding to a signage point. Thus farmers that won a block didn’t end up proposing a new block in time, thus missing out on that block reward and not moving the chain forward.
Unaffected Farmers
Not all farmers were impacted by this bug and some were able to continue making blocks. One of the underlying bug appears to be related to Singleton Fast Forward introduced in version 2.2.0 therefore farmers running full node software prior to and including version 2.1.4 (released Jan 2024), which includes Chia Gigahorse v2.1.4.giga32 (released March 2024) were unaffected. Farmers on 2.5.1 release candidates were also unaffected since a mitigating fix was already in place, tested, and ready to be released even before the issue arose.
Based on data from the day prior to the network issue, only 5.2% of clients were using unaffected versions accounting for an estimated 500 PiB of netspace. If it were just this proportion alone farming, block production would have slowed significantly until the next difficulty reset. Luckily there were two other groups of farmers that continued making blocks.
Farmers on h9 (formerly HPool) and NoSSD appeared to be unaffected, presumably because their pools use a centralized custom full node implementation that ignores low-fee transactions (< 1000 mojos) or otherwise ignores these invalid transactions in the mempool. At the time, these two pools accounted for ~40% of netspace, so it would have been expected for block production to slow to about half the normal rate immediately.
The Impact to the Network
The Chia Dashboard provides a glimpse into the impact on the network before and after this issue.
The mempool cost chart shows the rapid growth to the (common) cost limit of 110B. The two spikes down are likely artifacts of the dashboard’s backend full node restarting and clearing its local mempool.
The netspace graph shows a steady drop with netspace bottoming out near 12 EiB before recovering to around 16 EiB, implying ~2 EiB of netspace aren’t paying close enough attention to upgrade their full node. Netspace estimates are calculated over a moving window so the point-in-time farmable space would have instantly dropped down to as low as 10 EiB by my estimates.
The Nakamoto Coefficient (NC) has also dropped from 8 to 3. This measure is also calculated over a window of time so the point-in-time NC would have instantly dropped to 1.
Network difficulty was held elevated for almost twice as long as usual with extended time between blocks. Once difficulty readjusted, block times went back to normal (or even a bit faster with farmers gradually upgrading nodes).
The Impact to the Users
Farmers, users, and pools felt impacts as well.
- Pending transactions without a fee were likely to stay pending.
- A full mempool necessitated a minimum fee (of 5 mojos per cost) on new transactions to bump one of the zero fee transactions.
- Some fee estimates in wallets and websites were not high enough depending on their view of the mempool.
- Pools were initially rewarding farmers for partials even if they couldn’t form blocks (dead weight). Some pools eventually stopped crediting partials from affected farmer software versions.
- Farmers that were able to form blocks saw higher rewards after the difficulty adjusted lower due to lower netspace.
The Resolution
Within a few hours of discovery, Chia Network pushed out 2.5.1 which includes a mitigation to the issue, but didn’t solve the underlying bug(s).
The mitigation is to time-out transaction validation if it takes too long so affected farmers can continue forming blocks. This feature was actually tested and added proactively into 2.5.1 release candidate versions as a control against long validation times with 100% block fill. This bit of luck and foresight meant that the 2.5.1 release could be confidently expedited to address the immediate issue. The steady uptake of this release is reflected in the rise in estimated netspace.
The longer term solution is to fix the mempool logic to properly purge these invalid transactions from local mempools and not gossip them further to the rest of the network. Version 2.5.2 was released on February 19 implementing this.
It will be interesting to see how quickly farmers upgrade to this version to resume proper transaction inclusion in blocks but I suspect the invalid transactions will continue to stick around older local mempools until enough nodes upgrade to 2.5.2 to build “herd immunity” on the network and/or enough purposeful big spendbundles are broadcasted to bump out all the invalid transactions from older mempools before they can re-propagate.
One side effect of this required upgrade for farmers is that it also pushes through the acceptance of the CHIP-36 keccak256 CLVM operator soft fork. Adding this operator isn’t a controversial fork in my opinion but it does raise a question of how farmers can be empowered to easily signal their desire in similar situations in the future.
Conclusion
As a result of this issue, the network didn’t halt but block times and network security were degraded. Users were disrupted by unexpected transaction fee requirements, and farmers missed out on potential block rewards.
A similar issue had happened two months into mainnet with a negative coin spend in the mempool, halting the chain for 45 minutes. At that time there was only one implementation of the full node. The saving grace this time was client diversity. The existence of multiple reference node versions and large swaths of netspace using custom software meant that a single bug didn’t bring down the whole network.
There’s still room to improve for client diversity especially with an upcoming new plot format that leaves the role for centralized pools unclear. I’m sure there are lessons to take away to improve release testing practices and have more scrutiny placed on mempool logic.
In the end, a critical mempool bug was identified, a bug bounty was awarded, consensus was not affected, and the Chia blockchain continues to make blocks.
More to watch
FoMo! also covered this topic and is worth a watch
