Stacks resumes block production after mempool outage, warns of occasional outages

After nearly a week of struggling to resolve the issue that affected the network, Layer 2 platform Stacks on Bitcoin officially announced the resumption of block production on May 24. The issue stems from a logical error in the stacks-node mempool synchronization mechanism, which has been quietly present since 2020 but only became apparent when the network processed large amounts of data in recent blocks.

According to the announcement from the development team, the error caused some nodes to return inappropriate responses during the execution of RPC calls - an essential tool for synchronizing the mempool. When other nodes received incorrect responses, they could not synchronize properly and fell into an unresponsive state, which directly affected the block production process.

While the bug has been identified and a patch has been released as update 3.1.0.0.11, the Stacks team warned that block production could be “irregularly degraded” until all miners and signers on the network have completed their node upgrades.

“We recommend that all node operators, especially those participating in block production, update to the latest release as soon as possible to ensure stable network operation,” the Stacks development team shared via the X platform (Twitter).

The bug was initially discovered by the team on April 18, when the block production delays were first reported. The team initially thought it was a minor issue that could be resolved with a “simple patch” and did not require any action from operators. However, the problem recurred after the Bitcoin network fork at block 897442, forcing the development team to conduct a deeper investigation from May 19.

By May 23, after up to 70% of the signers had restored the chain state to the previous version, the Stacks network was able to operate again. However, just four days later, a similar problem occurred, showing the seriousness and complexity of the bug that had existed silently for many years.

The release of this new patch is expected to help resolve the problem once and for all, while ensuring the long-term stability of the Layer 2 network, which is highly appreciated for its ability to expand smart contract applications on the Bitcoin platform.

However, the problem also poses a great lesson about thorough testing and the ability to monitor unusual behaviors in the distributed node system - especially in the context of blockchain networks operating at higher scale and speed.