> ## Documentation Index
> Fetch the complete documentation index at: https://docs.story.foundation/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> Common problems and solutions when running Story nodes

Welcome to Story node troubleshooting! This section covers common problems and solutions when running Story nodes.

### Node Setup

<Accordion title="What are the hardware requirements?">
  See the [system specs](/network/operating-a-node/node-setup-mainnet)
</Accordion>

<Accordion title="What's the max expected TPS?">
  \~700
</Accordion>

<Accordion title="Is it fully EVM-compatible? Is there any customization already being made on the IP blockchain? Or are there any coming customization to be applied?">
  Yes, it's EVM-compatible. Story's execution client is a fork of Geth with our custom precompiles, which enhance the IP graph's performance while maintaining strict EVM compatibility. Other Ethereum execution clients, such as RETH and Erigon, can be supported later.
</Accordion>

<Accordion title="Which is your consensus mechanism?">
  Our consensus mechanism is CometBFT
</Accordion>

<Accordion title="Batches support? Limit on batch request?">
  Batch RPCs are supported - for Geth there is a 1K limit and on the consensus side there is 10 request limit
</Accordion>

<Accordion title="WS connections? (if yes, how do they work)">
  Yes, WS is enabled on the execution client, and is recommended for subscription use-cases. It is open on port 8546
</Accordion>

<Accordion title="How many different paths does node serves (several path with diff methods RPC)?">
  Please see Geth's latest JSON-RPC documentation for a full comprehensive list [here](https://ethereum.org/en/developers/docs/apis/json-rpc/#web3_clientversion). In the future, we may add more.
</Accordion>

<Accordion title="Caching rules for RPC method?">
  We recommend employing standard in-memory caching with a 1-10 min TTL based on the RPC method
</Accordion>

<Accordion title="What is the best method to get latest block and check node is healthy and in sync?">
  Use `eth_syncing` RPC call on the execution client to check if the node is sync and `eth_blockNumber` for getting the latest block
</Accordion>

<Accordion title="What are the heaviest RPC methods? How much time does it take to respond to request with such method?">
  `eth_call` / `eth_getLogs` / `eth_getBlockByNumber` \
  We are still running latency tests to get a sense of response times.
</Accordion>

<Accordion title="Is archive node provisioning a requirement? If yes how big?">
  No, not at the moment.
</Accordion>

<Accordion title="Are there snapshots available for full / archive?">
  Not yet, but we are working on it.
</Accordion>

<br />

### Common Issues

<Accordion title="Database Initialization Failure">
  **Error:**

  ```bash theme={null}
  ERRO !! Fatal error occurred, app died️ unexpectedly !! err="create db: failed to initialize database:
  ```

  **Solution:**

  1. Save your validator state:

  ```bash theme={null}
  cp $HOME/.story/story/data/priv_validator_state.json $HOME/.story/story/priv_validator_state.json.backup
  ```

  > 🚧 Be very careful with this file, especially if your validator is already signing blocks.

  * Check your the database backend type, your node must support the same as you are using the snapshot:

  ```bash theme={null}
  cat $HOME/.story/story/config/story.toml
  ```

  Default is `app-db-backend = "goleveldb"`. The fallback is the `db_backend` value set in CometBFT's `config.toml`.

  ```bash theme={null}
  cat $HOME/.story/story/config/config.toml
  ```
</Accordion>

<Accordion title="High Gas Fees">
  **Problem:** Need to adjust gas fees on RPC node

  **Solution:**
  Add the `--rpc.txfee` flag to your geth startup command:

  ```bash theme={null}
  sudo tee /etc/systemd/system/story-geth.service > /dev/null <<EOF
  [Unit]
  Description=Story-Geth Node
  After=network.target

  [Service]
  User=$USER
  Type=simple
  WorkingDirectory=$HOME/.story/geth
  ExecStart=$(which geth) --story --syncmode full --rpc.txfee 2
  Restart=on-failure
  LimitNOFILE=65535

  [Install]
  WantedBy=multi-user.target
  EOF
  ```
</Accordion>

<Accordion title="Failed to send PacketPing">
  **Error:**

  ```bash theme={null}
  ERRO Failed to send PacketPing module=p2p peer=19fa6dd52e72e4e85bbb873b705282cf73217a6b@158.220.80.96:40128 err="write tcp 139.59.139.135:26656->158.220.80.96:40128: write: broken pipe"
  ```

  Solution:

  * If the node is synchronized, you can ignore this error. Your client may be a little behind.
  * If the node stops, you should restart the services.
</Accordion>

<Accordion title="Cosmovisor: failed to read upgrade info">
  An error occurs when starting the cosmovisor:

  ```bash theme={null}
  panic: failed to read upgrade info from disk unexpected end of JSON input
  ```

  Solution:

  * You must ensure that the installed cosmovisor version must be at least [v1.7.0.](https://docs.cosmos.network/main/build/tooling/cosmovisor)
  * Then check your info file (edit version `v0.13.0` in your case):

  ```bash theme={null}
  cat $HOME/.story/story/cosmovisor/upgrades/v0.13.0/upgrade-info.json
  ```

  If you don\`t have create new one:

  ```bash theme={null}
  echo '{"name":"v0.13.0","time":"0001-01-01T00:00:00Z","height":858000}' > $HOME/.story/story/cosmovisor/upgrades/v0.13.0/upgrade-info.json
  ```

  Find out more about automatic updates with cosmovisor [here](/network/operating-a-node/node-setup-mainnet#custom-automation).
</Accordion>

<Accordion title="IPC endpoint closed">
  Error:

  ```bash theme={null}
  INFO HTTP server stopped
  INFO IPC endpoint closed
  ```

  Solution:

  * It looks like port 8551 stopping, the background process running `iptables` blocking ip and port and access posix.
  * For solution try uninstall `ufw posix` and `iptables`:

  ```bash theme={null}
  iptables -I INPUT -s localhost -j ACCEPT
  ```
</Accordion>

<Accordion title="Found signature from the same key">
  Error:

  ```bash theme={null}
  panic: Faile to consensus  state: found signature from the same key
  ```

  Solution:

  * The validator has been double signed. It is currently not possible to restore the validator after it has been double signed.
  * To avoid such situations, see this post on how to correctly [migrate a validator to another machine](/network/become-a-validator#migrating-a-validator-to-another-machine).
</Accordion>

<Accordion title="Failed to validate create flags: missing required flag(s): moniker">
  Error:

  ```bash theme={null}
  4-11-26 08:42:20.302 ERRO !! Fatal error occurred, app died️ unexpectedly !! err="failed to validate create flags: missing required flag(s): moniker" stacktrace="[errors.go:39 flags.go:173 validator.go:168 validator.go:384 command.go:985 command.go:1117 command.go:1041 command.go:1034 cmd.go:34 main.go:10 proc.go:271 asm_amd64.s:1695]"
  ```

  Solution:

  * You missed flag `--moniker`.
  * The command to create a new validator should look like this:

  ```bash theme={null}
  ./story validator create --stake ${AMOUNT_TO_STAKE_IN_WEI} --moniker ${VALIDATOR_NAME}
  ```

  See more options [here](/network/become-a-validator#validator-creation).
</Accordion>

<Accordion title="Error adding vote">
  Error:

  ```bash theme={null}
  ERRO failed to process message msg_type= *consensus.VoteMessage err:" error adding vote"
  ```

  Solution:

  * It looks like your node is down. To get started, check the current versions of the binaries [here](/network/operating-a-node/node-setup-mainnet).
  * If you have up-to-date binary - try updating peers, this usually happens when a node loses p2p communication:

  ```bash theme={null}
  PEERS="..."
  sed -i -e "/^\[p2p\]/,/^\[/{s/^[[:space:]]*persistent_peers *=.*/persistent_peers = \"$PEERS\"/}" $HOME/.story/story/config/config.toml
  ```
</Accordion>

<Accordion title="Error signing vote">
  Error:

  ```bash theme={null}
  ERRO failed signing vote module=consensus height=403750 round=0 vote="Vote{23:B12C6AE31E8E 403750/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) FA591EB1E540 000000000000 000000000000 @ 2024-11-08T16:58:10.375918193Z}" err="error signing vote: height regression. Got 403750, last height 420344"
  ```

  Solution:

  * Looks like you have a problem with your `priv_validator_state` of validator.
    > 🚧 Be very careful with this file, especially if your validator is already signing blocks.
  * You can make a copy of your state with a command:

  ```bash theme={null}
  cp $HOME/.story/story/data/priv_validator_state.json $HOME/.story/story/priv_validator_state.json.backup
  ```

  Check your validator state:

  ```bash theme={null}
  cat $HOME/.story/story/data/priv_validator_state.json
  ```

  * If you get this error, you can reset your state (🚧 ONLY IF YOUR VALIDATOR HAS NOT YET SIGNET BLOCKS).
  * Stop node.

  ```bash theme={null}
  sudo tee $HOME/.story/story/data/priv_validator_state.json > /dev/null <<EOF
  {
    "height": "0",
    "round": 0,
    "step": 0
  }
  EOF
  ```

  * Start node.
</Accordion>

<Accordion title="Unknown flag: --home">
  Error:

  ```bash theme={null}
  ERRO !! Fatal error occurred, app died️ unexpectedly !! err="unknown flag: --home"
  ```

  Solution:

  * It looks like a misconfiguration. You must try to remove the `--home` flag from the startup command.
  * Your systemd to run might look like this:
</Accordion>

<Accordion title="Failed to register the Ethereum service">
  Error:

  ```bash theme={null}
  Fatal: Failed to register the Ethereum service: incompatible state scheme, stored: path, provided: hash
  ```

  Solution:

  * You have problems with the state of validator or a corrupted database.
  * Try using a snapshot.
    > 🚧 Be very careful with this file, especially if your validator is already signing blocks.
  * We have described how to reset your state [here](/network/more/troubleshooting#error-signing-vote).

  ## Failed to reconnect to peer

  Error:

  ```bash theme={null}
  24-09-25 06:38:45.235 ERRO Failed to reconnect to peer. Beginning exponential backoff module=p2p addr=e0600fa5f2129e647ef30a942aac1695201ff135@65.109.115.98:26656 elapsed=2m29.598884906s
  ```

  Solution:

  * If the node is synchronized and not far behind, you can ignore this error.
  * If the node is lagging or has stopped completely, try updating peers, this usually happens when a node loses p2p communication:

  ```bash theme={null}
  PEERS="..."
  sed -i -e "/^\[p2p\]/,/^\[/{s/^[[:space:]]*persistent_peers =./persistent_peers = \"$PEERS\"/}" $HOME/.story/story/config/config.toml
  ```
</Accordion>

<Accordion title="Processing finalized payload halted while evm syncing">
  Warn:

  ```bash theme={null}
  WARN Processing finalized payload halted while evm syncing (will retry) payload_height=...
  ```

  Solution:

  * It just means that story-geth is syncing, you can ignore this warn.
  * However, if it takes a long time, we recommend that you stop the processes one at a time and start them again later in the following order:

  ```bash theme={null}
  sudo systemctl stop story-geth story
  sudo systemctl daemon-reload
  sudo systemctl start story-geth
  sudo systemctl enable story-geth

  sudo systemctl daemon-reload
  sudo systemctl start story
  sudo systemctl enable story
  ```
</Accordion>

<Accordion title="Upgrade handler is missing">
  Error:

  ```bash theme={null}
  ERRO error in proxyAppConn.FinalizeBlock      module=consensus err="module manager preblocker: wrong app version 0, upgrade handler is missing for upgrade plan"
  ```

  Solution:

  * Looks like you missed an update.
  * To get started, check the current versions of the binaries [here](/network/operating-a-node/node-setup-mainnet).
</Accordion>

<Accordion title="Home directory contains unexpected file">
  Error:

  ```bash theme={null}
  ERRO !! Fatal error occurred, app died️ unexpectedly !! err="home directory contains unexpected file(s), use --force to initialize anyway"
  ```

  Solution:

  * This means that you have already initialized the node.
  * `$HOME/.story/story` directory created, and there are files in it. Delete it, or try with it.
</Accordion>

<Accordion title="Err='create comet node: create node">
  Error:

  ```bash theme={null}
  ERRO !! Fatal error occurred, app died️ unexpectedly ! err="create comet node: create node
  ```

  Solution:

  * It appears that your node is using incorrect versions.
  * Check the current versions of the binaries [here](/network/operating-a-node/node-setup-mainnet).
  * And most likely you need to perform a rollback binary to current versions.
</Accordion>

<Accordion title="WAL does not contain">
  Error:

  ```bash theme={null}
  ERRO catchup replay: WAL does not contain
  ```

  Solution:

  * Looks like an `AppHash` issue.
  * To get started, upgrade to the current versions of the binaries [here](/network/operating-a-node/node-setup-mainnet).
  * If your versions are newer than the current ones, perform a rollback.
</Accordion>

<Accordion title="Err='load engine JWT file: read jwt file">
  Error:

  ```bash theme={null}
  ERRO !! Fatal error occurred, app died️ unexpectedly !! err="load engine JWT file: read jwt file: open /root/.story/geth/odyssey/geth/jwtsecret: no such file or directory
  ```

  Solution:

  * It seems your node can't get `jwtsecret`.
  * Check your `WorkingDirectory` in your `geth-service` , by default `WorkingDirectory=$HOME/.story/geth`.
  * Check all paths, you can get your `jwtsecret`with command (for odyssey network):

  ```bash theme={null}
  cat .story/geth/odyssey/geth/jwtsecret
  ```
</Accordion>

<Accordion title="Couldn't connect to any seeds">
  Error:

  ```bash theme={null}
  ERRO Couldn't connect to any seeds module=p2p
  ```

  Solution:

  * If the node is synchronized and not far behind, you can ignore this error.
  * If the node is lagging or has stopped completely, try updating seeds/peers, it usually happens when a node loses p2p communication (we recommend that you stop the node and delete the addrbook).

  ```bash theme={null}
  rm -rf $HOME/.story/story/config/addrbook.json
  SEEDS="..."
  PEERS="..."
  sed -i -e "/^\[p2p\]/,/^\[/{s/^[[:space:]]*seeds *=.*/seeds = \"$SEEDS\"/}" \
         -e "/^\[p2p\]/,/^\[/{s/^[[:space:]]*persistent_peers *=.*/persistent_peers = \"$PEERS\"/}" $HOME/.story/story/config/config.toml
  ```
</Accordion>

<Accordion title="Processing finalized payload failed err='rpc forkchoice updated">
  Warn:

  ```bash theme={null}
  WRN Processing finalized payload; evm syncing
  WRN Processing finalized payload failed: evm fork choice update (will retry) status="" err="rpc forkchoice updated v3: beacon syncer reorging"
  ```

  Solution:

  * Everything is fine, it just means that `story-geth` is syncing, which takes some time.
  * If the node is not far behind, you can ignore this warning.

  ## Dial tcp 127.0.0.1:9090

  Warn:

  ```bash theme={null}
  WRN error getting latest block error:"rpc error: dial tcp 127.0.0.1:9090"
  ```

  Solution:

  * The logs show a connection failure on port `9090`.
  * Check the listening ports:

  ```bash theme={null}
  sudo ss -tulpn  | grep LISTEN
  ```

  * If other node uses `9090`, then modify it to another.
  * Normally, this WARNING should not affect the performance of your node.
</Accordion>

<Accordion title="Wrong AppHash">
  Error:

  ```bash theme={null}
  ERRO Error in validation module=blocksync err="wrong Block[dot]Header[dot]AppHash  Expected [...]
  ```

  Solution:

  * `Wrong AppHash` type logs means the story node version you are using is wrong.
  * Upgrade to the current versions of the binaries [here](/network/operating-a-node/node-setup-mainnet).
  * If your versions are newer than the current ones, perform a rollback.
</Accordion>

<Accordion title="Connection failed sendRoutine / Stopping peer">
  Error:

  ```bash theme={null}
  ERRO Connection failed @ sendRoutine module=p2p peer=...
  ERRO Stopping peer for error module=p2p peer=...
  ```

  Solution:

  * If the node is synchronized and not far behind, you can ignore this error.
  * If the node is lagging or has stopped completely, try updating peers, this usually happens when a node loses p2p communication:

  ```bash theme={null}
  PEERS="..."
  sed -i -e "/^\[p2p\]/,/^\[/{s/^[[:space:]]*persistent_peers =./persistent_peers = \"$PEERS\"/}" $HOME/.story/story/config/config.toml
  ```
</Accordion>

<Accordion title="Moniker must be valid non-empty">
  Error:

  ```bash theme={null}
  ERRO !! Fatal error occurred, app died️ unexpectedly ! err="create comet node: create node: info.Moniker must be valid non-empty
  ```

  Solution:

  * Looks like a problem with your node moniker.
  * Be sure to use `""` when executing init:

  ```bash theme={null}
  story init --network "..." --moniker "..."
  ```

  * Go to config, find the moniker and put it inside `""` only:

  ```bash theme={null}
  sudo nano ~/.story/story/config/config.toml
  ```
</Accordion>

<Accordion title="Invalid address (26656)">
  Error:

  ```bash theme={null}
  Fatal error occurred, app died️ unexpectedly ! err="create comet node: create node: invalid address (26656):
  ```

  Solution:

  * The logs report a connection failure on port `26656`.
  * Check the listening ports:

  ```bash theme={null}
  sudo ss -tulpn  | grep LISTEN
  ```

  * If another node is using `26656`, change it to another and keep the default `26656` for story in the `P2P configuration` options in `config`:

  ```bash theme={null}
  sudo nano ~/.story/story/config/config.toml
  ```
</Accordion>

<Accordion title="Eth_coinbase does not exist">
  Warn:

  ```bash theme={null}
  WARN Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!
  Served eth_coinbase eth_coinbase does not exist
  ```

  Solution:

  * This error indicates that the network has stopped.
</Accordion>

<Accordion title="Verifying proposal failed">
  Warn:

  ```bash theme={null}
  WARN Verifying proposal failed: push new payload to evm (will retry) status="" err="new payload: rpc new payload v3: Post \"http://localhost:8551\": round trip: dial tcp 127.0.0.1:8551: connect: connection refused" stacktrace="[errors.go:39 jwt.go:41 client.go:259 client.go:180 client.go:724 client.go:590 http.go:229 http.go:173 client.go:351 engineclient.go:101 msg_server.go:183 proposal_server.go:34 helpers.go:30 proposal_server.go:33 tx.pb.go:299 msg_service_router.go:175 tx.pb.go:301 msg_service_router.go:198 prouter.go:74 abci.go:520 cmt_abci.go:40 abci.go:85 local_client.go:164 app_conn.go:89 execution.go:166 state.go:1381 state.go:1338 state.go:2055 state.go:910 state.go:836 asm_amd64.s:1695]"
  WARN Verifying proposal
  ```

  Solution:

  * It looks like port 8551 stopping, the background process running `iptables` blocking ip and port and access posix.
  * For solution try uninstall `ufw posix` and `iptables`:

  ```bash theme={null}
  iptables -I INPUT -s localhost -j ACCEPT
  ```
</Accordion>