Difference between revisions of "Techniques to reduce transaction fees"
(→Change avoidance: Added anecdote of a bitcoin casino using change avoidance)
(→References: Add to category scalability)
|Line 332:||Line 332:|
Revision as of 19:53, 15 February 2019
This page provides a list of currently-available techniques that may allow spenders to reduce the amount of transaction fee they pay. Not all techniques will apply to all situations, and some techniques require trading off other benefits for lower fees.
When you reduce the fee you pay, you almost always reduce the fees other users have to pay also. For high-frequency spenders, this effect can be large and can provide significant additional second order savings. For example, If an organization is creating 10% of all transactions and begins making transactions 50% more efficiently, an additional 5% of block space is freed up for all users of Bitcoin. This would be expected to lower fees by some amount (likely more than 5%), providing a second order of savings on top of the first-order savings of 50%.
Note, this page only describes techniques that apply to payment-oriented transactions. Data carrier transactions (e.g. OP_RETURN or OpenTimestamps), colored coin protocols, and other proposed uses of Bitcoin transactions may benefit from some of the following techniques, but the recommendations are not aimed at those use cases.
- 1 Efficiency improvements
- 2 Intelligent fee selection
- 3 Offchain payments
- 4 Potential future improvements
- 5 References
Creating transactions that are smaller in size (weight), or which accomplish more for a given size, provide a more efficient way of using scarce block space and so pay less total fee to achieve a feerate that is equivalent to less efficient payments. This section describes several techniques for producing more efficient transactions.
Technically offchain payments such as those made in payment channels are a type of extremely efficient payment and so belong to this category, but they've been given a separate category because of the distinctive way they achieve their high efficiency.
Compressed public keys
Universally useful but already widely deployed
The original Bitcoin software released in 2009 used 65-byte uncompressed public keys to identify the owner of a set of bitcoins. In 2012, Bitcoin protocol developer Pieter Wuille implemented a change to the program now known as Bitcoin Core that allowed it to use 33-byte compressed public keys instead. As Bitcoin Core users adopted this feature and other wallets upgraded to use it as well, this reduced the size of a typical transaction on the network (with one input and two outputs) from about 258 bytes to about 226 bytes, a 14% savings.
The change was fully backwards compatible and did not change security in any way, but it did require users wanting to access the space savings to generate new Bitcoin addresses.
Since 2012, many wallets have adopted compressed public keys—but still some wallets continue to use the less efficient uncompressed public keys. These wallets could achieve a significant savings in transaction fees for the same priority by switching to compressed public keys. If all wallets adopted it, this would effectively lead to a small increase in the available block space:
- Compressed public keys — Bitcoin.org Developer Guide
- Bitcoin Core 0.6.0 release notes
- BIP143 - Requires all pubkeys in segregated witness witnesses be compressed pubkeys
Very useful for high-frequency spenders (e.g. organizations); moderately useful for lower-frequency spenders (e.g. individuals)
Every Bitcoin transaction must reference the funds being spent and provide proof that the transaction was authorized by the owner of those funds. To spend a single collection of funds takes a minimum of 79 vbytes under normal circumstances. This same amount of block space is used no matter how many recipients are paid in that transaction. For example, consider the following two scenarios:
- Alice pays Bob in one transaction and then pays Charlie in a second transaction. Each transaction contains a minimum of 79 vbytes related to the funds being spent—a total of 158 vbytes.
- Alice uses a single transaction to pay both Bob and Charlie. The single transaction only needs one set of 79 vbytes related to the funds being spent—a savings of 50% for this field.
This type of savings increases as more payments are added to a single transaction until the cost per payment is just barely more than the cost of adding the vbytes directly related to that payment in the transaction. The plot below shows the cost per payment for a native segwit P2WPKH transaction paying other P2WPKH outputs:
We see the cost drop by more than 50% after five payments are added, with the savings increasing up to about 70% for segwit transactions. Not shown is that the savings increase up to about 80% for legacy transactions because these transactions start off less efficient than segwit transactions. That's a major benefit and one that's easily obtainable by high-frequency spenders, such as organizations.
Many wallets support batching payments. In graphical wallets, there's often a button that allows you add additional recipients to a transaction (see image below). In command-line and RPC wallets, there's often a call such as
sendmany that lets you pay multiple recipients.
Note that there are other parts of a transaction that stay a constant size (or nearly so) when payments are added, so the benefits stack up faster than a fixed cost of just 79 vbytes might suggest. The section below about change avoidance addresses how one of these cases can itself be eliminated as a cost.
- Saving up to 80% on Bitcoin transaction fees by batching payments
- SendMany RPC - Bitcoin.org Developer Reference
Transactions that spend bitcoins secured by segregated witness (segwit) use less space in a block than equivalent non-segwit (legacy) transactions, allowing segwit transactions to pay less total fee to achieve the same feerate as legacy transactions. The amount of savings varies depending on the details of your transaction, but here are a few common transaction types an an example:
|Type||Legacy vbytes|| P2SH-wrapped
Note that the multisig examples above use the same security as the equivalent legacy P2SH multisig. Segwit optionally allows access to a multisig form that is more secure on one dimension but it requires an extra 12 vbytes per output, which would reduce efficiency somewhat.
To access these savings, you must use a wallet that supports generating P2SH-wrapped segwit addresses (addresses that start with a "3", although not all addresses that start with a 3 are segwit-enabled). When you spend bitcoins received to these P2SH-wrapped segwit addresses, your transactions will automatically consume less block space, allowing your wallet to pay proportionally less fee.
- Segregated Witness
- Segregated Witness Wallet Development Guide - BitcoinCore.org
- List of wallets, libraries, and services that support segwit - BitcoinCore.org
Universally useful. Complete usage requires native segwit adoption by the people sending you bitcoins, but you may be able to use it for your change outputs immediately
The P2SH-wrapped segwit described above is backwards compatible with the P2SH address format supported by older wallets, but a new and non-backwards compatible format is available that saves additional space. The following examples and savings are compared to the size of the P2SH-wrapped examples above:
| Native segwit
To access these savings, you must use a wallet that supports generating native segwit addresses, called bech32 addresses (addresses that start with a "bc1"). When you spend bitcoins received to these native segwit addresses, your transactions will automatically consume less block space than even P2SH-wrapped segwit addresses, allowing your wallet to pay proportionally less fee.
Once a wallet supports native segwit, it can begin using it immediately for any change outputs it generates back to itself without waiting for anyone else to begin using native segwit. In early 2018, when native segwit adoption is low, this may make it easier to identify which output is change and so reduce your privacy. However, once native segwit adoption increases just slightly, this is not expected to adversely affect privacy.
Useful for high-frequency recipients (e.g. organizations), especially those who already effectively use transaction batching
When a Bitcoin transaction references the funds it wants to spend, it's required to spend all of those funds. For example, if you received 5 BTC in a previous transaction and now you want to spend 2 BTC, Bitcoin requires that you spend all 5 BTC. For this reason, almost all Bitcoin transactions currently pay some bitcoins back to the spender. For example, you'd return the remaining 3 BTC from the previous example back to yourself. This return of money to yourself is called change by analogy to the change a store clerk hands you when you (for example) buy a $2 item with a $5 bill.
A typical change output adds about 32 vbytes to the size of a transaction. If the spender can pay with exact change—that is, doesn't need to refund any change to them self—they can eliminate the change output and generate transactions that are up to 23% smaller. In addition, a change output will later be spent at a typical cost of 69 vbytes or more, but when paying with exact change, this future cost is also avoided.
To use change avoidance requires having previously received a payment or set of payments that's close to the size of the amount being spent (including the transaction fee). This can be rare in the case of individual user wallets that don't receive many payments to choose from and can't significantly vary the amount of their spending transactions, but for organization wallets that receive many payments and already use payment batching to combine multiple outgoing payments, change avoidance can be an easily-obtainable efficiency improvement. The number of ways n inputs choose m outputs combines grows exponentially, so if the organization has a decent number of outputs well under the value being paid then this method is very practical.
The spender doesn't need to match the inputs and outputs of their transaction exactly to Bitcoin's full 10 nanobitcoin precision, but can instead overpay or underpay fees slightly by including inputs that are (respectively) slightly more or slightly less than the desired amount. Even when paying slightly more fees than desired, this can result in savings if the slight increase in fees is still less than would normally be paid for the extra 32 vbytes (or so) for a change output and the typical mininum of 69 vbytes for later spending that output. When paying slightly lower fees, confirmation of the transaction may be delayed, but current-generation (2018) fee estimates are still inaccurate enough that small differences in feerates may not strongly correlate to delays.
For example in a notable bitcoin casino implementing this technique, when a player withdraws money they are given two options of Instant Send or Queued Send. The latter option puts their withdrawal in a queue where it may or may not be included when changeless withdrawal transactions are calculated. A large amount of people prefer the queued send, as they can save money from the savings being passed onto them. The feature has a nice UX where the customer can see it's waiting, and can cancel it if it's taking too long.
- An Evaluation of Coin Selection Strategies - Mark Erhardt
- Bitcoin Core pull request #10,637 - A proposed implementation of Erhardt's "branch and bound" strategy
- Bitcoin-Dev mailing list post - With good coin selection, I am able to avoid change about ~75% of the time in my simulations..."
Useful for all wallets but may require users sacrifice some privacy
The amount of fee a transaction pays is proportional to its size in vbytes, and one the main contributors to size is the number of inputs the transaction spends. Each input is a reference to the funds the transaction wants to spend, and when a wallet contains only low-value inputs, it can't create a comparatively higher output paying a recipient without adding many of those inputs to the transaction. Each input adds a minimum of 41 vbytes to the transaction and almost always 69 or more vbytes, so any strategy that reduces the number of inputs is worth considering.
Given that fees vary over time, one method that can reduce overall fees is input consolidation—combining a set of smaller inputs into a single larger input by spending them from yourself to yourself during a period of time when fees are lower than normal. For example, if your usual target feerate during normal-priced fee periods is 100 nanobitcoins per vbyte and the current feerate is 10 nanobitcoins per vbyte, you could save up to almost 90% on fees by consolidating now before you next need to spend some of those inputs.
Consolidation does come with the overhead of creating the consolidation transaction, which is equal to the cost of spending one input to one output, which is about 110 vbytes for P2WPKH inputs/outputs. Also, if you combine inputs that were originally sent to addresses unconnected to each other, you may reduce your privacy in some cases by making that connection in your consolidation transaction (although it's believed that few people currently manage to spend their inputs in a manner that preserves this element of privacy).
Intelligent fee selection
When it comes to fees, sending a Bitcoin transaction is similar to mailing a package: you can pay a high fee for fast high-priority service or a lower fee for slower low-priority service. This section describes several techniques for taking advantage of the more affordable low-priority service.
Dynamic fee estimation
Universally useful. Already widely deployed but still being improved as research and development continues
Early Bitcoin wallets often defaulted to paying a fixed fee on every transaction—enough to incentivize a miner to include it but not enough to compete against other transactions seeking confirmation in the block space market. As blocks have filled, this has changed, and as of early 2018 all widely-used wallets use dynamic fee estimation to select a fee based on the condition of the current fee market.
However, some fee estimation tools may be better than others, achieving confirmation by the desired time even when paying lower fees. Although data from multiple wallets and fee estimation services can be compared and there have been some attempts to compare fee estimation between different wallets, there is no known survey of fee estimation quality across a large number of popular wallets as of early 2018.
In addition, also as of early 2018, some techniques have recently been described that could significantly improve fee estimation, such as factoring current mempool data into confirmation-based fee estimates.
- Transaction fees
- The Challenges of Bitcoin Transaction Fee Estimation - Jameson Lopp
- High level description Bitcoin Core's fee estimation algorithm - Alex Morcos
Very useful for non-urgent transactions. Not useful for urgent transactions
Spenders who can patiently wait for their transactions to confirm can take advantage of variations in the feerate necessary to achive confirmation. For example, sometimes several Bitcoin blocks are produced in quick succession, raising the effective supply of block space by several multiples; other times, demand drops off, such as fees on Sunday being on average 20% lower than fees on Friday in Q4 2017. Looking at data from the widely-used fee estimator included in Bitcoin Core, we can see fee savings of 90% or more possible, with around 50% being easily obtainable by many patient spenders.
In the data plotted above, the Bitcoin Core fee estimator suggests that for this sample period the following savings are available:
|Wait||Conservative savings||Economical savings*|
|24 hours (1 day)||39%||55%|
|48 hours (2 days)||47%||55%|
|72 hours (3 days)||52%||55%|
|96 hours (4 days)||92%||92%|
* Economical savings have a lower probability of being achieved and are recommended for use with the transaction replacability described in a later section.
Although waiting up to four days is impractical for many use cases, there are also many cases where it can be practical. A few examples: moving funds between one's own wallets, consolidating many smaller inputs into one larger input that can be more efficiently spent, or payments or remittances to friends or family who trust the spender and so don't need fast confirmation.
Opt-in transaction replacement
Universally useful, but may cause UI issues in recipient wallets
Although not strictly a method for reducing fees by itself, opt-in transaction replacement allows a wallet to update previously-sent transactions with new versions that pay higher fees and, possibly, make other changes to the transaction. The method is also called opt-in Replace-By-Fee (RBF). This technique allows wallets to initially pay lower fees in case there's a sudden increase in the supply of block space, a sudden decrease in demand for that space, or another situation that increases the chance of low-fee transactions being confirmed. If none of those things happens, the spender can then increase ("bump") the transaction's fee to increase its probability of confirming.
This often allows wallets that support transaction replacement to pay lower fees than wallets that don't support replacement.
Transaction replacement can appear odd in some recipient wallets. Wallets such as Bitcoin Core (pictured below) show each replacement as a separate payment in the list of transactions. When one version of the replacements is confirmed, it is shown as a normal transaction; the other versions are then shown with an X icon to indicate that they are conflicted (cannot occur together in the same valid block chain). This helps communicate the status of all affected payments to the recipient, but it may not be entirely clear what's happening to users who aren't familar with replacements. Other wallets may not show transactions opting in to replacement at all until one version of the transaction has been confirmed. There's no clear community consensus on the correct way to handle this situation using user interfaces, documentation, or both.
Transaction replacement can be advantageously combined with payment batching (described previously). Instead of waiting, for example, 30 minutes to batch all outgoing payments, the spender can batch the first 10 minutes of payments with a low feerate. If that doesn't get confirmed within 10 minutes, the spender can replace that transaction with a slightly higher feerate transaction that also includes then next 10 minutes of payments. If that again doesn't confirm, another update can also include the third 10 minutes of transactions at the original intended feerate. This allows the recipients of the first 10 minutes of payments to receive a notification that the payment has been sent up to 20 minutes earlier than with normal batching, and it also gives those early payments a chance to confirm at a lower-than-expected feerate. Updating a batched transaction with more payments can be done as many times as necessary up to a relay limit on transaction size of 100 kilobytes.
Currently, transaction replacement does have one significant downside: it tends to reduce privacy. When the fee on a transaction is increased, either additional inputs must be added or the value of the change output must be decreased. In either case, this makes the change output easier to identify among the different outputs being paid by a transaction. Value-blinding techniques such as confidential transactions could improve this situation, but there are no near-term plans to add such a feature to Bitcoin as of early 2018. If transaction replacement is always combined with change avoidance, it could avoid this privacy issue.
- Transaction replacement
- Opt-in Replace-By-Fee (RBF) FAQ - BitcoinCore.org
- BIP125 - Opt-in Full Replace-by-Fee Signaling
Pre-computed fee bumping
Pre-computed fee bumping is an idea to create and sign multiple replacements for a transaction at the time the initial transaction is created. The initial transaction version could be broadcast immediately, and each of the replacements would pay successively higher fees. This idea could be combined with Bitcoin's existing nLockTime feature to allow the replacement versions to forbid being included in a block earlier than a specified time or block height, which would allow the replacements to be trustlessly transmitted to third parties (even miners themselves).
For example, Alice would tell her wallet that she wanted to pay Bob within the next 10 blocks and also indicate what is the highest fee she’s willing to pay. Alice's wallet would then use its existing fee estimation feature to create an initial version of the transaction to Bob that paid the lowest expected fee for a transaction to confirm within 10 blocks. At the same time, Alice's wallet would also create possibly 9 other versions of the transaction, the first one using nLockTime to ensure it can’t be added until two blocks from now; the second timelocked until three blocks from now; etc…
Each of these subsequent transactions would pay a slightly higher fee than the original transaction (up to Alice’s indicated max fee) to increase the incentive for miners to mine that transaction.
Because all versions of the transaction would be signed at the time Alice sent the initial transaction, she would only need to unlock her wallet once. Also, because all subsequent versions of the transaction would use nLockTime, Alice could trustlessly distribute copies of these transactions to other people for later broadcast in case she went offline.
In short, the software would automatically offer Alice’s maximum fee if it had to, but it’d pay a lower fee if it could get away with it—ensuring Alice got close to the best price possible without any extra effort on her part.
- Bitcoin Core IRC meeting summary for 2017-03-02 - See "Precomputed fee bumping" bullet point
Not every Bitcoin payment needs to be added to the Bitcoin block chain. For example, imagine Alice pays 1 BTC to Bob, and then Bob pays 1 BTC to Charlie. Instead of adding both of these payments to the block chain, we could more efficiently just add a 1 BTC payment from Alice to Charlie.
An early description using this specific optimization called it transaction cut-through, but the concept of not every payment being recorded on the block chain is more widely known today as off-chain payments.
Up to 99% of all Bitcoin payments are estimated to be offchain payments, with most of them likely being buy and sell orders on Bitcoin exchanges. Other third-parties may also help facilitate offchain payments for their users, for example tipping services. These services can be extraordinarily efficient because they don't carry Bitcoin's burden of maintaining consensus among a large set of users—but this efficiency comes at a cost of generally requiring that users trust the service operators not to steal funds. These are unenforceable offchain payments.
However, there also exist enforceable offchain payments where no trust in a specific third-party is required—only trust in Bitcoin continuing to operate without transaction censorship.
This section currently focuses on enforceable offchain payments, but it may later be expanded to cover concepts for unenforcable offchain payments that use cryptography and other security solutions to provide more secure and more private solutions than simple trusted third parties.
Very useful for small-size and moderate-size payments, but not yet widely deployed as of early 2018
Payment channels are a type of enforceable offchain payment where bitcoins are deposited into a smart contract between two or more parties, allowing the involved peers to make trustless payments to each other. Combined with hashlocks, payments can be securely routed across a network of peers, as in the Lightning Network, allowing Alice to pay Charlie by routing a payment through their mutual peer Bob.
Opening a payment channel requires a confirmed transaction, incurring the cost of the fees for that transaction, although those fees are identical to sending a regular transaction. Closing a channel also requires a confirmed transaction, adding that transaction's fees to the cost of the payment channel, although those fees too are similar to the costs of the recipient of the funds re-spending them to a new recipient. This means a payment channel's fee cost is very similar to making onchain payments.
Outside of the channel open and channel close transactions, a payment channel can support an unlimited number of payments without paying any further transaction fees. This can make them extraordinarily efficient—for example, imagine making a million in-channel payments for the cost of just two transaction fees. Note: your channel peers and other channel nodes you route payments through may require their own fee for using their services; these are not related to the transaction fees that are the topic of this article.
As of early 2018, no payment channel implementation is widely used, but several cooperating implementations of a first version of Lightning Network are available.
Potential future improvements
Some of the following improvements and proposed improvements may become available to Bitcoin users. Users who adopt the improvements may be able to further lower their fees.
- Efficiency improvements
- Public key and signature aggregation: the ability to combine multiple public keys and signatures into a single public key and signature, allowing multisig transactions to be as efficient to spend as single-signature transactions are today. It may also be possible to combine signatures from multiple inputs in a transaction. It is hopped that this will be possible with variant of Schnorr signatures. Requires a soft fork.
- Merkelized Abstract Syntax Trees (MAST): a method for committing to spending scripts that allows the partial omission of unused conditions, allowing complex scripts to be much more efficient, even more so if used with taproot. Combines well with signature aggregation (described above) in certain use cases. Requires a soft fork.
- CoinJoin with signature aggregation: CoinJoin is a technology for Bitcoin that enhances privacy without reducing security. A group of patient spenders can enter a coinjoin together and obfuscate which of them paid which recipient and which change outputs (if any) they received back. Combined with signature aggregation (described in a previous bullet point), this enhanced privacy mode would be cheaper than each spender making a separate less-private transaction—and it would still be just as secure. Requires the signature aggregation soft fork described previously.
- Intelligent fee selection
- Offchain payments
- Channel factories: (enforceable offchain payments) a potentially more efficient way to open payment channels such as those used by Lightning Network. Combined with signature aggregation (described above) this could reduce the cost of opening a payment channel by up to 96%. Does not require any consensus changes.
- Chaumian banks: (unenforceable offchain payments) a trusted third party who controls bitcoins for a pool of Bitcoin users (custodial wallet, AKA "Bitcoin bank") but who can't know which users own which bitcoins. This is accomplished using Chaum blinded signatures. Combined with hardware security modules, it could be made extremely difficult for banks to steal depositor funds. Does not require any protocol changes.
- Strong federations (sidechains) (unenforceable offchain payments) a set of trusted third parties controls bitcoins for a pool of Bitcoin users, but uses multisig to prevent an individual or small number of those trusted third parties from stealing depositor funds. Uses a public block chain (but not Bitcoin's block chain) to allow any user to audit all transfers into, out of, or within the system. Hardware security modules (HSMs) are used to further reduce trust in the third parties. Although a block chain is used, this qualifies as an offchain payment solution from the perspective of Bitcoin. As of early 2018, free software federated sidechain software provides support for confidential transactions that give them many of the same benefits of the Chaumian banks described in a previous bullet point, but federated sidechain software can (and does) support many other features also.
- Bitcoin Core 0.6.0 release notes, Bitcoin.org, 2012-03-20, retrieved 2018-01-27
- Transaction Merging (bip125 relaxation), Rhavar, bitcoin-dev mailing list, 2018-01-22
- Fee Estimation, P2SH.info, retrieved 2018-01-23
- The Challenges of Bitcoin Transaction Fee Estimation, Jameson Lopp, BitGo block, 2017-05-10, retrieved 2018-01-23
- Optimizing fee estimation via the mempool state (slides, PDF), Karl-Johan Alm, Scaling Bitcoin 2017 (Stanford), 2017-11-05, retrieved 2018-01-23
- Optimizing fee estimation via the mempool state (presentation video), Karl-Johan Alm, Scaling Bitcoin 2017 (Stanford), 2017-11-05, retrieved 2018-01-23
- Transaction_fees#The_market_for_block_space, Bitcoin Wiki, retrieved 2018-01-27
- Transaction cut-through, Gregory Maxwell, 2013-08-23, retrieved 2018-01-25
- Pure off-chain is a weak form of layer 2, Adam Back, 19 June 2015
- What is a Bitcoin Merkelized Abstract Syntax Tree?, David A. Harding, BitcoinTechTalk.com, 2017-10-12, retrieved 2018-01-27
- Taproot: Privacy preserving switchable scripting, Gregory Maxwell, Bitcoin-Dev mailing list, 2018-01-23
- Elliptic curve Schnorr-based signatures in Bitcoin, Pieter Wuille, Scaling Bitcoin 2016 (Milan), 2016-10-09, retrieved 2018-01-27
- Scalable Funding of Bitcoin Micropayment Channel Networks; Conrad Burchert, Christian Decker, and Roger Wattenhofer; retrieved 2018-01-27
- Chaumian E-Cash For Custodial Bitcoin Wallets And Services To Scale Bitcoin, nopara73, Medium.com, 2017-12-22, retrieved 2018-01-27
- Fidelity-bonded banks: decentralized, auditable, private, off-chain payments, Peter Todd, 2013-02-23, retrieved 2018-01-27
- Strong Federations: An Interoperable Blockchain Solution to Centralized Third Party Risks; Johnny Dilley, Andrew Poelstra, Jonathan Wilkins, Marta Piekarska, Ben Gorlick, and Mark Friedenbach; retrieved 2018-01-28
- Confidential Transactions, ElementsProject.org, retrieved 2018-01-28
- Elements (features), ElementsProject.org, retrieved 2018-01-28