Mitigating Lost Transactions on EOSIO
Overview
The EOSIO ecosystem has recently seen a resurgence of collaboration and development.
For the past several months, developers and node operators on EOSIO networks across the world have been hard at work building protocol improvements for the Mandel network upgrade. These include several improvements to subjective billing (SB) and lost transactions.
In the last article, we looked at subjective billing, the setting that allows nodes to limit failed transactions. We outlined its history and explored the edge cases that can cause confusion and frustration among affected users. This article will outline what network participants can do to minimize these issues.
Reducing Subjective Billing Issues
Many members of the EOSIO community are looking forward to Mandel’s transaction lifecycle improvements, but EOSIO developers and operators are already well positioned to prevent some of SB’s most frustrating headaches.
The existing SB feature includes several nodeos options that operators can use to reduce SB for average users. One option disables SB entirely. Another disables it for specific accounts. A third option is supposed to disable SB specifically for API or P2P endpoints. Note that this third option is buggy, and operators should avoid it until they implement the new Mandel 3.1 binaries on their nodes.
Other configuration options can help nodes process transactions more efficiently by modifying chain state storage locations.
Smart contract developers can also take steps to reduce SB. On-chain billing, or “objective billing,” is visible to users through block explorers, while subjective billing is not. To avoid undetectable transaction failures, it’s important to avoid SB. Developers can detect conditions that would lead to transaction failure, and reject them as early as possible to minimize SB. Alternatively, they can design contracts to fail as early as possible to reduce resource use for failed transactions.
Even users can take action to avoid subjective billing issues. By buying extra account CPU resources, using third-party resource providers, or connecting to reliable API endpoints, users can reduce their chance of losing transactions.
Together, these steps can help mitigate issues with subjective billing and lost transactions until the adoption of Mandel brings fixes and powerful new tools to optimize the user experience.
Quick reference for fixing subjective billing issues and lost transactions:
- Users can buy extra CPU resources or use an external resource provider to reduce the chance of their transaction being rejected due to excessive subjective billing. Careful selection of API endpoints can also prevent transactions from being overbilled.
- Node operators can determine which accounts are unlikely to send too many failed transactions and can disable subjective billing on those accounts using:
disable-subjective-account-billing = <accountname>
An example could apply to Greymass Fuel. A node operator might decide whether or not Greymass has enough rate-limiting or Sybil resistance to be “trusted.” The operator could disable subjective billing for Greymass Fuel using:
disable-subjective-account-billing = greymassfuel - Due to known issues with the implementation in nodeos 2.x, node operators using the disable_subjective_api_billing and disable_subjective_p2p_billing optional parameters should set both to the same Boolean value, both true or both false. Nodes can fix this bug by upgrading their node to Mandel 3.1.
- Alternatively, operators can use the disable-subjective-billing = true option to turn off subjective billing entirely. This option may be deprecated, or discouraged and marked for potential deletion, in a future release, because it duplicates the previously mentioned disable-subjective-api-billing and disable-subjective-p2p-billing options.
- Node operators, especially those on cloud nodes, can reduce disk reads and resource usage using the nodeos options, database-map-mode = heap or database-map-mode = locked. They can also mount the chain state directory as a temporary file system with the Linux option, tmpfs, potentially speeding up transaction execution.
- Smart contract developers can preemptively detect conditions that cause transaction failures and reject these transactions as early as possible. This does not eliminate subjective billing, but reduces its magnitude and its impact on users.
- Developers may also be able to structure their contracts so that the most likely failures happen early before incurring significant billing.
In-Depth Reference – Fixes in Detail
User strategies for preventing subjective billing issues
Node configuration adjustments are most effective at addressing subjective billing issues. However, users can make several minor changes that potentially reduce their chance of experiencing issues.
The first thing users can do is purchase extra CPU resources. This extra CPU will cover any SB balance a transaction may encounter on its propagation path to the active block producer. Users can’t see their SB balance on any nodes and can’t tell if they need extra resources to cover the SB. Holding extra CPU helps ensure any SB doesn’t prevent the node from passing a user’s transaction across the network.
Another helpful practice is using resource providers like Greymass Fuel. These tools separate the burden of resource management from the user experience. A resource provider acts as the first signer of a transaction, paying the transaction’s resource costs and charging a fee to the user. Because of the popularity of these tools, node operators often disable SB for these accounts, giving users an easy solution for avoiding SB issues.
More experienced users can change the options on their preferred wallet to ensure they are connecting to a reliable API endpoint. There is no official list of these endpoints. Still, users could infer API performance from the data at Aloha EOS’s Block Producer Benchmarks. Users can use this tool and select API endpoints from block producers (BPs) with a consistently low execution time, which are likely to have high-performance API nodes.
As node operators improve their node configurations, and Mandel 3.1 introduces improvements to SB, the network may be able to mitigate many issues. In the meantime, these strategies can help improve the experience of EOSIO users.
Subjective billing considerations for smart contract developers
Smart contract developers can reduce issues with SB by making special development considerations.
Functions like assert can cause an entire transaction to fail, incurring subjective billing. The more of a transaction that has already executed before failure, the more SB is applied to the transaction’s first signing account.
By implementing early detection of common failure conditions, developers can exit a failed transaction earlier and reduce the SB applied.
Alternatively, dApps can incorporate more pre-transaction checks to avoid sending transactions with invalid parameters.
Developers should note that any smart contract function that fails, aside from initial signature validation, will be subjectively billed.
Subjective billing begins to apply after the signature verification stage of a transaction (see A-1 for detail). Because of this, smart contract developers may reduce subjective billing by reordering the functions within an action or the actions within a transaction.
An example could be ordering check functions earlier in a transaction if they are more likely to fail, as long as this doesn’t compromise the security logic of the smart contract. These functions should be evaluated as early as possible, ideally before complex computations that use lots of resources.
Configuration options for node operators
Nodeos is a transaction validator application that ships with EOSIO. Node operators use it to operate nodes on the chain. It includes optional parameters that enable, disable, or adjust features. Node operators can use these options to optimize nodeos for their server’s architecture (see A-3).
Disabling subjective billing for specific accounts
One nodeos option allows nodes to ignore subjective billing for individual accounts.
disable-subjective-account-billing = ACCTNAME
The disable-subjective-account-billing option will turn off SB for the account with the name ACCTNAME. Node operators can use this option to whitelist accounts that they expect to have a low risk of resource overutilization.
As an example, if a node operator decides that Greymass Fuel is a reliable resource provider with transaction rate-limiting tools in place, they may choose to activate the option:
disable-subjective-account-billing = greymassfuel
With this option enabled, the node disables the SB tool for the greymassfuel account. This option ensures that Greymass Fuel can provide a fallback option for accounts with problems related to subjective billing.
Disabling subjective billing
Some node operators, like those running highly performant API nodes or private chains, may be able to turn off SB entirely. Two sets of configuration options accomplish this, with slightly different behaviour. Operators should avoid the second variant until a future Mandel update brings bug fixes.
Variant 1:
disable-subjective-billing = <BOOLEAN>
If set to 1 (true), the disable-subjective-billing parameter will turn off SB entirely for that node. Operators can use this option on nodes that don’t see failed transactions clogging up their bandwidth. The default value is true.
Variant 2:
disable_subjective_api_billing = <BOOLEAN>
or
disable_subjective_p2p_billing = <BOOLEAN>
The disable_subjective_api_billing and disable_subjective_p2p_billing options are designed to individually turn off SB for a node’s API endpoint and P2P endpoint, respectively.
It is important to note that these options exhibit unexpected behaviour when one is “true” and the other is “false.” Mandel 3.1 fixes these issues.
In one of the bugs, an endpoint with SB disabled doesn’t check an account’s existing SB balance, but if a transaction fails, the endpoint still adds SB. The P2P endpoint and API endpoint share the same subjective billing ledger, so the P2P endpoint sees all the SB applied by the API endpoint and vice versa. This shared ledger can cause an endpoint to reject transactions from accounts to which the other endpoint unintentionally charged SB.
The other bug with these options is more perplexing. It only occurs when the node has disable_subjective_api_billing set to true and disable-api-persisted-trx set to false.
The disable_subjective_api_billing option is intended to ignore subjective billing. Instead, if a transaction is successful, the node checks the account’s SB on the next block.
Because of these behaviours, operators should set the two options to the same value until implementing Mandel 3.1 on their nodes. Operators can also use the disable-subjective-billing option, although it is an older practice and may be deprecated to avoid redundancy.
Cloud node considerations
Cloud nodes allow small teams to operate nimbly and reliably. Many node operators prefer cloud-based nodes for their uptime and efficiency. They can benefit from the same security and networking as the internet’s top companies by leaving high-level server operations to dedicated services. However, with suboptimal configuration, cloud nodes can experience unexpected transaction failures and subjective or objective over-billing.
Specifically, these failures involve the memory architecture and limited disk I/O lanes that most cloud providers impose. These can create execution time issues for transactions that access slower disk memory rather than the server’s RAM.
A few nodeos options can help with the performance issues caused by this.
Option 1:
database-map-mode = heap or database-map-mode = locked
Node operators may be able to reduce disk I/O by using the optional configuration parameters database-map-mode = heap or database-map-mode = locked. These options change the way servers store the chain state.
In heap mode, nodes preload the chain state into swappable memory. The state is stored preferentially in RAM, with overruns of physical RAM pushed to on-disk swap memory. Swap performance alone is insufficient for effective node operation. However, with enough physical RAM, nodes using this option will speed up database calls to the most frequently accessed data. On the WAX blockchain, a test found that nodes could function efficiently with as low as half the state stored in physical RAM.
In locked mode, nodes store the entire state in a reserved RAM partition without swap memory. This option requires physical RAM at least as large as the state size.
The heap and locked modes require a minimum amount of server RAM, so nodes with limited RAM should not use these options. With the impending release of Trust EVM and other EOS Network Foundation (ENF) initiatives, these operators may consider boosting their resources to accommodate increased usage.
Option 2:
tmpfs <DIRECTORY/TO/CHAIN/state>
Nodes can use the Linux command tmpfs to create a shared partition split between physical and swap memory. Operators can use this tmpfs partition to store the chain state and build it from a snapshot (see A-4).
Once a node builds the state from a snapshot, the tmpfs folder removes the need to rebuild the state each time nodeos restarts. The node only needs to rebuild the state if the machine is powered down or the tmpfs folder is unmounted.
The tmpfs folder may be able to reduce the number of disk reads. Reduced disk reads lower the chance that a transaction will use all of its available CPU time waiting to receive data from the disk. This disk latency is a frequent cause of transaction failures.
These tools should help node operators prevent subjective billing issues. The adoption of Mandel will bring a host of additional features and improvements.
Mandel Improvements
The above steps are best practices for helping node operators, developers, and users avoid lost transactions.
EOSIO core network developers expect protocol improvements to make these steps less necessary. The next iteration of the EOSIO software, Mandel 3.1, will implement several transaction lifecycle improvements. The next article will delve deeper, but we’ve briefly summarized four major feature additions and updates here:
- Transaction Retry monitors the network and automatically retries transactions that don’t make it onto a finalized block.
- Transaction Resource Cost Estimation allows wallets and apps to send test transactions to estimate resource use without applying the transactions to the chain.
- Transaction Finality Status tracks incoming transactions with a transaction pool. It keeps track of each transaction’s status until it has achieved finality for a set number of blocks.
- Subjective Billing Improvements address SB features. One new parameter allows for adjusting SB decay time. Another allows nodes to drop an account’s remaining queued transactions when the account has sent more than its allowance of failed transactions within a single block.
Appendices: Technical Details and Example Usage
A-1: Transaction processing workflow of nodes
A more detailed description of the steps a node takes when processing a transaction, and its subjective billing implications, follows:
An account initially signs a transaction, potentially with one or more cosigners, such as a smart contract or resource provider (1).
When a transaction reaches a node, the node first verifies the signatures of all the transaction’s signers (2).
If signature validation fails, the node drops the transaction without billing the account (3).
If the signature verification succeeds, the transaction moves on (4).
Next, the node calculates the account’s remaining resource allowance. Nodes with SB disabled proceed without checking or applying SB (5).
Nodes with SB enabled (6) check their SB tables (7) for the account associated with the transaction’s first signer.
If the account doesn’t have enough resources for its subjective bill, the transaction will be dropped without being billed (8).
If the account has enough resources, the transaction proceeds to the next step (9).
Next, the node executes the transaction (10). It tracks how much CPU time it uses while executing the transaction.
If a part of the transaction fails, the node drops the transaction. It subjectively bills the account for the resources used before transaction failure (11).
If a transaction executes successfully (12), it moves to the next stage.
The node next adds the account’s SB balance to resource billing, as calculated. It then compares this to the account’s available resources (13).
If the account runs out of resources, the node drops the transaction and subjectively bills the account for resource usage (14).
If the account has enough resources, the node sends the transaction over its P2P protocol to other nodes in the network (15).
Or, if the node is the active block producer, it adds the transaction and calculated billing to a block (16).
A-2: Advanced Mitigation for Smart Contract Developers
An additional consideration for smart contract developers involves the use of assert() and check() functions, which trigger transaction failure, and return statements, which do not. Return statements can ensure billing is visible instead of subjective. However, this practice should be used carefully, as users or smart contracts often expect incorrect parameters to lead to transaction failure, and don’t bother to check the return statement. They may see a successful transaction without checking if the return statement reported a success or a failure, and trigger actions that should not be triggered.
This practice is contrary to other common smart contract practices, so it should only be used in situations where no other SB mitigation techniques are effective.
To read more about the motivation behind this practice in the context of the three-strikes rule and subjective billing, see its corresponding GitHub issue.
A-3: Recommended configuration settings for API node operators
This is a recommended configuration example for API nodes. Operators add these options to the nodeos config.ini file. Other recommended configuration settings can be found at EOS Nation’s main article on the subject, at https://github.com/EOS-Nation/bpconfig/.
wasm-runtime = eos-vm-jit
chain-state-db-size-mb = 32768
reversible-blocks-db-size-mb = 2048
http-max-response-time-ms = 300
read-mode = head
database-map-mode = heap
p2p-accept-transactions = false
disable-api-persisted-trx = true
http-validate-host = false
p2p-max-nodes-per-host = 2
agent-name = INSERT NAME OF OPERATOR HERE
max-clients = 0
net-threads = 5
http-threads = 8
verbose-http-errors = true
abi-serializer-max-time-ms = 2000
http-server-address = 0.0.0.0:8888
enable-account-queries = true
plugin = eosio::http_plugin
plugin = eosio::chain_api_plugin
tmpfs Usage Example:
A-4: tmpfs Usage Example
The following example outlines the creation of a tmpfs folder in the filesystem table, which is then mounted and used to store the chain state, after it is generated from a snapshot. The example is adapted from an article by EOSphere regarding running a nodeos instance on the WAX blockchain. It can be found here: https://medium.com/eosphere/wax-technical-how-to-11-43695f583e89
Creating and Mounting a tmpfs Partition
This example creates a tmpfs partition using zfs (“zettabyte file system”), which is a file system like Windows’s ntfs or Apple’s apfs file system, but with the ability to manage physical volumes as well as logical volumes. Zfs is not strictly necessary, but is often preferred to improve data integrity and performance. It changes a value in the file system table, fstab.
> sudo nano /etc/fstab
tmpfs <DIRECTORY/TO/state> tmpfs rw,nodev,nosuid,size=129G,x-systemd.after=zfs-mount.service 0
> sudo mount -a
**check that it was mounted**
> df -h
Setting Up a Swap File:
The EOSphere team uses a swap file rather than a partition in order to easily adjust swap size. Using SSDs, they haven’t experienced issues with using a swap file. The following example configures a swap file as 128GB using the following functions:
***Turn off existing swap, which would probably be a partition***
> sudo swapoff -a
> sudo fallocate -l 128G /swap.img
> sudo chmod 600 /swap.img
> sudo mkswap /swap.img
> sudo swapon /swap.img
***Configure fstab and comment out the old swap statement***
> sudo nano /etc/fstab
/swap.img none swap sw 0 0
Starting nodeos from a Snapshot
Next, set the database size and load a snapshot into nodeos. The nodeos instance stores the snapshot state in virtual memory, and will need to rebuild it each time the host machine is shut off or the tmpfs partition is unmounted. The ~/eosdata field is an arbitrary name that is user-defined, and if the operator is using a custom directory name they must ensure every instance of “/eosdata” is replaced with the custom directory name.
Startup takes longer than usual, and even longer if lots of swap storage is used. In the case of 40GB of RAM and 128GB database size, with 81.4GB of memory utilization, EOSphere found a rebuild took about 30 minutes. The state will not have to be rebuilt for restarts of the nodeos software, and is available until machine restart or tmpfs unmount.
***Configure State Database Size***
> cd ~/eosdata
> nano config.ini
chain-state-db-size-mb = 131072
How to download a snapshot:
Snapshots are available in many locations, for example at https://snapshots.eosnation.io/. Operators should save the snapshot to ~/eosdata/snapshots/snapshot.bin.
Start from a Snapshot:
nodeos –data-dir ~/eosdata –config-dir ~/eosdata –snapshot ~/eosdata/snapshots/snapshot.bin
API+ Working Group Education Series
This series of articles will outline the lifecycle of an EOSIO transaction, highlighting the subjective billing feature and the lost transactions it can cause. First, we learned all about subjective billing. Second, we will outline best practices for current smart contract developers and node operators to avoid problems. Last, we will take a look at a few new features in Mandel that will help further mitigate these issues.
