The Bitcoin Mempool: Private Mempools

In the last Mempool article, I went through the dynamics of transaction propagation when different nodes on the network are running different mempool relay policies. In this piece I’ll be looking at the dynamics of private mempools, and the implications that has for the utility of the public mempool, mining incentives, and the health of the Bitcoin network overall. 

At the heart of the purpose of the mempool is facilitating the aligned incentives of two different parties, miners and transacting users. Users want to transact, and are willing to pay miners’ transaction fees in order to do so. Miners want to make money, and transaction fees are an additional source of revenue in addition to the new coin subsidy in each block, as well as a necessary primary revenue source to cultivate in the long term as the subsidy dwindles. 

Bitcoin is a system secured by incentives. This core dynamic is what drives the security of the system, you have a customer(s) and a provider, and the two of them attempting to fulfill their wants and needs is what ensures the blockchain continues ticking forward with a sufficient amount of thermodynamic security. 

Attempts to introduce friction into this facilitation mechanism does not ultimately do anything at all to change the incentives of these two parties. A user who wants to make a certain kind of transaction is still going to want to make that transaction, and pay for it. A miner who is willing to accept those kinds of transactions is still going to want to accept them, and collect the fee by including them in a block. 

If the transaction is valid, then these two parties are still going to have their unmet wants and needs, and are still going to be strongly motivated to meet them in some form or fashion. 

Miner API

Individual end users are not necessarily capitalized enough or competent enough in order to route around friction artificially introduced between both ends of a coincidence of wants, but miners most definitely are. As the old adage goes, “if you build it, they will come.”

The preferential situation for miners is obviously to acquire fee paying transactions in-band through the public mempool. It requires the lowest overhead possible for them, simply running a standard Bitcoin client out of the box, it is a very resilient propagation mechanism that ensures a very high degree of reliability in getting miners the highest fee paying transactions, and they don’t have to do anything. Just download the client and run it. 

However, in a very hostile environment such as a network wide effort to filter consensus valid transactions during their propagation across the network, that traditional assumption can be drawn into question. 

In such a scenario miners have every incentive to set up out-of-band mechanisms for accepting transactions that are not properly being relayed across the network. Marathon’s Slipstream API for non-standard transactions is not the only example of this. There is in fact a long standing precedent from almost ten years ago that was widely implemented by many mining pools, and still exists to this day. Transaction accelerators. 

We now live in a world of Full-RBF, where any transaction, regardless of using the historical “opt-in” flag, can be fee-bumped. Any node who has upgraded to Full-RBF will relay any transaction that is spending an unconfirmed output already pending in the mempool as long as it is paying a higher fee. This has not always been the case. Historically only transactions that were originally made with a flag to opt-in to RBF use could be replaced and expected to propagate across the network. 

Transaction accelerators were created by miners in order to facilitate this behavior for transactions that did not opt-in to RBF use. 

Third Party APIs

While the overhead is not exorbitantly high for a miner or pool to create their own transaction submission API, it isn’t free. It still does require at least one developer and time to go through the design and release cycle of any piece of software. The curve isn’t particularly exaggerated, but it still does favor larger miners over smaller ones in terms of how much resources they will have to devote to such an endeavor. 

Mempool.space has proven that it is a viable endeavour for a third party unrelated to miners to create such an API, allowing miners to simply connect to their service rather than expend the effort to create one themselves from scratch. This does have its issues though, such a third party is not going to build and operate such a service for free. They will want their cut. 

There are two ways that this dynamic can go, either these services wind up requiring a higher cost in order to allow both the miners and service providers to earn revenue, or miners will have to share a smaller cut of the revenue in order for such services to remain competitive with directly miner operated ones. This means miners using a third party submission API rather than their own will earn less revenue than the miners operating their own API. 

Private Order Flow

Either of the above possibilities introduces serious problems when it comes to the overall system incentives, reliability of end-user software, and potentially even the security model of second layer systems that rely on the use of pre-signed transactions and a reactive security model in order to keep user funds safe. 

When transactions are submitted to a private API, they are not visible to network participants until they are actually confirmed in a block. The entire queue of unconfirmed transactions making use of these systems is opaque. This could be made public by the operators of these APIs, but not in a trustless fashion. There is no way to prove or guarantee that operators are not withholding information. 

Withholding transactions from public view could distort fee estimates that users make, and even open the door to the possibility of manipulating those feerates by stuffing blocks with their own transactions. Transactions used in the operation of second layer systems could be withheld from public view until confirmation, which can delay users ability to react to transactions they must respond to in order to guarantee the security of their funds. 

Lastly, just the existence of such APIs if the demand or need for them is high enough is a massive centralization pressure. Having to handle connecting to each individual API to submit a transaction is a hassle, poor UX, and potential back end complexity. This tends to reinforce the use of the largest API(s) and ignoring the tailend, which creates a feedback loop. 

The API operators with the largest hashrate will have the quickest and most reliable confirmations, guaranteeing only those largest miners reliably earn this extra revenue, giving them more capital to grow larger, etc. 

Parallel Mempools

On the other end of the spectrum is the possibility of creating totally independent public relay networks. While this does replicate the current openness of the existing public mempool, and avoids the worst of the centralizing pressures of central APIs, it still is not ideal. 

Having multiple mempools introducing complexity for miners, for end users, and for end user applications. Users now need to keep track of all the independent mempools, especially ones used for systems they interact with that are not propagated over the primary relay network, in order to have a view of unconfirmed transactions. 

If Lightning (or some other Layer 2) were to start making use of a parallel mempool, tracking it would be critical for any user of Lightning (or that other Layer 2). It would also be necessary to track all of the parallel relay networks in order to have an accurate view of the other unconfirmed transactions you are bidding against for inclusion in the next block. Tracking only a subset of them would lead to potentially large margins of error in any users fee estimation. 

You Just Make Things Worse

Trying to prevent transactions with willing fee paying users without addressing them at the consensus level is just not possible. Bitcoin is an engine driven by incentives, and when the incentives of multiple parties align they will be facilitated in one form or another. 

Trying to pretend that is not the case, and that things can be stopped, disincentivized, or otherwise delayed is a fool’s errand. Not only that, but trying at any serious scale comes with very serious negative consequences, in addition to being doomed to fail. 

Bitcoin’s consensus rules are the framework in which incentives are played out. The only thing that can trump incentives is changing that framework. It is literally what informs and shapes the incentives in the first place. 

Trying to interfere with those incentives at any other layer is a fool’s errand, and can do nothing but exacerbate the negative outcomes driven by incentives, i.e. centralization.