Software Upgrade Guide

Overview #

Learn how to manage chain upgrades for your full and validator nodes.

Upgrade Categories #

There are 3 different categories for upgrades:

Planned or Unplanned Upgrades: Chain upgrades can be scheduled at a given height through an upgrade proposal plan.
Breaking or Non-breaking Upgrades: Upgrades can be API or State Machine breaking, which affects backwards compatibility. To address this, the application state or genesis file would need to be migrated in preparation for the upgrade.
Data Reset Upgrades: Some upgrades will need a full data reset in order to clean the state. This can sometimes occur in the case of a rollback or hard fork.

Additionally, validators can choose how to manage the upgrade according to their preferred option:

Automatic or Manual Upgrades: Validator can run the cosmovisor process to automatically perform the upgrade or do it manually.

Planned and Forks Upgrades #

Planned Upgrades #

Planned upgrades are coordinated scheduled upgrades that use the upgrade module logic. This facilitates smoothly upgrading ICPlaza to a new (breaking) software version as it automatically handles the state migration for the new release.

Governance Proposal #

Governance Proposals are a mechanism for coordinating an upgrade at a given height or time using an [SoftwareProposal].

::: tip All governance proposals, including software upgrades, need to wait for the voting period to conclude before the upgrade can be executed. Consider this duration when submitting a software upgrade proposal. :::

If the proposal passes, the upgrade Plan, which targets a specific upgrade logic to migrate the state, is persisted to the blockchain state and scheduled at the given upgrade height. The upgrade can be delayed or expedited by updating the Plan.Height in a new proposal.

Hard Forks #

A special type of planned upgrades are hard forks. Hard Forks, as opposed to [Governance Proposal}(#governance-proposal), don’t require waiting for the full voting period. This makes them ideal for coordinating security vulnerabilities and patches.

The upgrade (fork) block height is set in the BeginBlock of the application (i.e before the transactions are processed for the block). Once the blockchain reaches that height, it automatically schedules an upgrade Plan for the same height and then triggers the upgrade process. After upgrading, the block operations (BeginBlock, transaction processing and state Commit) continue normally.

::: tip In order to execute an upgrade hard fork, a patch version needs to first be released with the BeginBlock upgrade scheduling logic. After a +2/3 of the validators upgrade to the new patch version, their nodes will automatically halt and upgrade the binary. :::

Unplanned Upgrades #

Unplanned upgrades are upgrades where all the validators need to gracefully halt and shut down their nodes at exactly the same point in the process. This can be done by setting the --halt-height flag when running the ICPlazad start command.

If there are breaking changes during an unplanned upgrade (see below), validators will need to migrate the state and genesis before restarting their nodes.

::: tip The main consideration with unplanned upgrades is that the genesis state needs to be exported and the blockchain data needs to be reset. This mainly affects infrastructure providers, tools and clients like block explorers and clients, which have to use archival nodes to serve queries for the pre-upgrade heights. :::

Breaking and Non-Breaking Upgrades #

Upgrades can be categorized as breaking or non-breaking according to the Semantic versioning (Semver) of the corresponding software release version(i.e vX.Y.Z):

Major version (X): backward incompatible API and state machine breaking changes.
Minor version (Y): new backward compatible features. These can be also be state machine breaking.
Patch version (Z)#: backwards compatible bug fixes, small refactors and improvements.

Major Versions #

If the new version you are upgrading to has breaking changes, you will have to:

Migrate genesis JSON
Migrate application state
Restart node

This needs to be done to prevent double signing or halting the chain during consensus.

To upgrade the genesis file, you can either fetch it from a trusted source or export it locally using the ICPlazad export command.

Minor Versions #

If the new version you are upgrading to has breaking changes, you will have to:

Migrate the state (if applicable)
Restart node

Patch Versions #

In order to update a patch:

Stop Node
Download new release binary manually
Restart node

Data Reset Upgrades #

Data Reset upgrades require node operators to fully reset the blockchain state and restart their nodes from a clean state, but using the same validator keys.

Automatic or Manual Upgrades #

With every new software release, we strongly recommend full nodes and validator operators to perform a software upgrade.

You can upgrade your node by either:

automatically bumping the software version and restart the node once the upgrade occurs, or
download the new binary and perform a manual upgrade

Follow the links in the options above to learn how to upgrade your node according to your preferred option.

Rollback #

Learn how to rollback the chain version in the case of an unsuccessful chain upgrade. {synopsis}

In order to restore a previous chain version, the following data must be recovered by validators:

the database that contains the state of the previous chain (in ~/.ICPlazad/data by default)
the priv_validator_state.json file of the validator (also in ~/.ICPlazad/data by default)

If validators don’t possess their database data, another validator should share a copy of the database. Validators will be able to download a copy of the data and verify it before starting their node. If validators don’t have the backup priv_validator_state.json file, then those validators will not have double-sign protection on their first block.

Restoring State Procedure #

First, stop your node.
Then, copy the contents of your backup data directory back to the $ICPlaza_HOME/data directory (which, by default, should be ~/.ICPlazad/data).

# Assumes backup is stored in "backup" directory
rm -rf ~/.ICPlazad/data
mv backup/.ICPlazad/data ~/.ICPlazad/data

Next, install the previous version of ICPlaza.

# from ICPlaza directory
git checkout <prev_version>
make install
## verify version
ICPlazad version --long

Finally, start the node.

ICPlazad start

Manual Upgrades #

Learn how to manually upgrade your node.

1. Upgrade the ICPlaza version #

Before upgrading the ICPlaza version. Stop your instance of ICPlazad using Ctrl/Cmd+C.

Next, upgrade the software to the desired release version. Check the ICPlaza releases page for details on each release.

::: warning Ensure that the version installed matches the one needed for the network you are running (mainnet or testnet). :::

cd ICPlaza
git fetch --all && git checkout <new_version>
make install

::: tip If you have issues at this step, please check that you have the latest stable version of Golang installed. :::

Verify that you’ve successfully installed ICPlaza on your system by using the version command:

$ ICPlazad version --long

name: ICPlaza
server_name: ICPlazad
version: 3.0.0
commit: fe9df43332800a74a163c014c69e62765d8206e3
build_tags: netgo,ledger
go: go version go1.18 darwin/amd64
...

::: tip If the software version does not match, then please check your $PATH to ensure the correct ICPlazad is running. :::

2. Replace Genesis file #

::: tip You can find the latest genesis.json file for mainnet or testnet in the following repositories:

Mainnet: github.com/tharsis/mainnet
Testnet: github.com/tharsis/testnets :::

Save the new genesis as new_genesis.json. Then, replace the old genesis.json located in your config/ directory with new_genesis.json:

cd $HOME/.ICPlazad/config
cp -f genesis.json new_genesis.json
mv new_genesis.json genesis.json

::: tip We recommend using sha256sum to check the hash of the downloaded genesis against the expected genesis.

cd ~/.ICPlazad/config
echo "<expected_hash>  genesis.json" | sha256sum -c

:::

3. Data Reset #

Remove the outdated files and reset the data:

rm $HOME/.ICPlazad/config/addrbook.json
ICPlazad tendermint unsafe-reset-all --home $HOME/.ICPlazad

Your node is now in a pristine state while keeping the original priv_validator.json and config.toml. If you had any sentry nodes or full nodes setup before, your node will still try to connect to them, but may fail if they haven’t also been upgraded.

::: danger 🚨 IMPORTANT 🚨

Make sure that every node has a unique priv_validator.json. DO NOT copy the priv_validator.json from an old node to multiple new nodes. Running two nodes with the same priv_validator.json will cause you to double sign. :::

4. Restart Node #

To restart your node once the new genesis has been updated, use the start command:

ICPlazad start

Live Upgrade Tutorial #

This document demonstrates how a live upgrade can be performed on-chain through a governance process.

Start the network and trigger upgrade

# start a ICPlaza application full-node
$ ICPlazad start

# set up the cli config
$ ICPlazad config trust-node true
$ ICPlazad config chain-id testing

# create an upgrade governance proposal
$ ICPlazad tx gov submit-proposal software-upgrade <plan-name> \
--title <proposal-title> --description <proposal-description> \
--from <name-or-key> --upgrade-height <desired-upgrade-height> --deposit 10000000stake

# once the proposal passes you can query the pending plan
$ ICPlazad query upgrade plan

Performing an upgrade
Assuming the proposal passes the chain will stop at given upgrade height.
You can stop and start the original binary all you want, but it will refuse to run after the upgrade height.
We need a new binary with the upgrade handler installed. The logs should look something like:
```
E[2019-11-05|12:44:18.913] UPGRADE "<plan-name>" NEEDED at height: <desired-upgrade-height>:       module=main
E[2019-11-05|12:44:18.914] CONSENSUS FAILURE!!!
...
```
Note that the process will hang indefinitely (doesn’t exit to avoid restart loops). So, you must manually kill the process and replace it with a new binary. Do so now with Ctrl+C or killall ICPlazad.
In ICPlaza/app/app.go, after upgrade.Keeper is initialized and set in the app, set the the corresponding upgrade Handler with the correct <plan-name>:
```
    app.upgradeKeeper.SetUpgradeHandler("<plan-name>", func(ctx sdk.Context, plan upgrade.Plan) {
        // custom logic after the network upgrade has been executed
    })
```
Note that we panic on any error - this would cause the upgrade to fail if the migration could not be run, and no node would advance - allowing a manual recovery. If we ignored the errors, then we would proceed with an incomplete upgrade and have a very difficult time every recovering the proper state.
Now, compile the new binary and run the upgraded code to complete the upgrade:
```
# create a new binary of ICPlaza with the added upgrade handler
$ make install

# Restart the chain using the new binary. You should see the chain resume from
# the upgrade height:
# `I[2019-11-05|12:48:15.184] applying upgrade <plan-name> at height: <desired-upgrade-height>      module=main`
$ ICPlazad start

# verify there is no pending plan
$ ICPlazad query upgrade plan

# verify you can query the block header of the completed upgrade
$ ICPlazad query upgrade applied <plan-name>
```

← Upgrading Your Node

Basic Concepts →

PreviousUpgrading Your Node NextBasic Concepts

Last updated 1 year ago

On this page