> For the complete documentation index, see [llms.txt](https://ictech.gitbook.io/icplaza-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ictech.gitbook.io/icplaza-docs/chain-dev/software-upgrade-guide.md).

# Software Upgrade Guide

#### On this page

* [<mark style="color:blue;">Overview</mark>](#overview)
  * [<mark style="color:blue;">Upgrade Categories</mark>](#upgrade-categories)
  * [<mark style="color:blue;">Planned and Forks Upgrades</mark>](#planned-and-forks-upgrades)
  * [<mark style="color:blue;">Planned Upgrades</mark>](#planned-upgrades)
  * [<mark style="color:blue;">Unplanned Upgrades</mark>](#unplanned-upgrades)
  * [<mark style="color:blue;">Breaking and Non-Breaking Upgrades</mark>](#breaking-and-non-breaking-upgrades)
  * [<mark style="color:blue;">Data Reset Upgrades</mark>](#data-reset-upgrades)
  * [<mark style="color:blue;">Automatic or Manual Upgrades</mark>](#automatic-or-manual-upgrades)
* [<mark style="color:blue;">Rollback</mark>](#rollback)
  * [<mark style="color:blue;">Restoring State Procedure</mark>](#restoring-state-procedure)
* [<mark style="color:blue;">Manual Upgrades</mark>](#manual-upgrades)
  * [<mark style="color:blue;">1. Upgrade the ICPlaza version</mark>](#1-upgrade-the-icplaza-version)
  * [<mark style="color:blue;">2. Replace Genesis file</mark>](#2-replace-genesis-file)
  * [<mark style="color:blue;">3. Data Reset</mark>](#3-data-reset)
  * [<mark style="color:blue;">4. Restart Node</mark>](#4-restart-node)
* [<mark style="color:blue;">Live Upgrade Tutorial</mark>](#live-upgrade-tutorial)

### Overview <mark style="color:blue;">#</mark> <a href="#overview" id="overview"></a>

Learn how to manage chain upgrades for your full and validator nodes.

#### Upgrade Categories <mark style="color:blue;">#</mark> <a href="#upgrade-categories" id="upgrade-categories"></a>

There are 3 different categories for upgrades:

* **Planned or Unplanned Upgrades**: Chain upgrades can be scheduled at a given height through an upgrade proposal plan.
* **Breaking or Non-breaking Upgrades**: Upgrades can be API or State Machine breaking, which affects backwards compatibility. To address this, the application state or genesis file would need to be migrated in preparation for the upgrade.
* **Data Reset Upgrades**: Some upgrades will need a full data reset in order to clean the state. This can sometimes occur in the case of a rollback or hard fork.

Additionally, validators can choose how to manage the upgrade according to their preferred option:

* **Automatic or Manual Upgrades**: Validator can run the `cosmovisor` process to automatically perform the upgrade or do it manually.

#### Planned and Forks Upgrades <mark style="color:blue;">#</mark> <a href="#planned-and-forks-upgrades" id="planned-and-forks-upgrades"></a>

#### Planned Upgrades <mark style="color:blue;">#</mark> <a href="#planned-upgrades" id="planned-upgrades"></a>

Planned upgrades are coordinated scheduled upgrades that use the upgrade module logic. This facilitates smoothly upgrading ICPlaza to a new (breaking) software version as it automatically handles the state migration for the new release.

**Governance Proposal&#x20;**<mark style="color:blue;">**#**</mark>

Governance Proposals are a mechanism for coordinating an upgrade at a given height or time using an \[`SoftwareProposal`].

::: tip All governance proposals, including software upgrades, need to wait for the voting period to conclude before the upgrade can be executed. Consider this duration when submitting a software upgrade proposal. :::

If the proposal passes, the upgrade `Plan`, which targets a specific upgrade logic to migrate the state, is persisted to the blockchain state and scheduled at the given upgrade height. The upgrade can be delayed or expedited by updating the `Plan.Height` in a new proposal.

**Hard Forks&#x20;**<mark style="color:blue;">**#**</mark>

A special type of planned upgrades are hard forks. Hard Forks, as opposed to \[Governance Proposal}(#governance-proposal), don’t require waiting for the full voting period. This makes them ideal for coordinating security vulnerabilities and patches.

The upgrade (fork) block height is set in the `BeginBlock` of the application (i.e before the transactions are processed for the block). Once the blockchain reaches that height, it automatically schedules an upgrade `Plan` for the same height and then triggers the upgrade process. After upgrading, the block operations (`BeginBlock`, transaction processing and state `Commit`) continue normally.

::: tip In order to execute an upgrade hard fork, a [<mark style="color:blue;">patch version</mark>](#breaking-and-non-breaking-upgrades) needs to first be released with the `BeginBlock` upgrade scheduling logic. After a +2/3 of the validators upgrade to the new patch version, their nodes will automatically halt and upgrade the binary. :::

#### Unplanned Upgrades <mark style="color:blue;">#</mark> <a href="#unplanned-upgrades" id="unplanned-upgrades"></a>

Unplanned upgrades are upgrades where all the validators need to gracefully halt and shut down their nodes at exactly the same point in the process. This can be done by setting the `--halt-height` flag when running the `ICPlazad start` command.

If there are breaking changes during an unplanned upgrade (see below), validators will need to migrate the state and genesis before restarting their nodes.

::: tip The main consideration with unplanned upgrades is that the genesis state needs to be exported and the blockchain data needs to be reset. This mainly affects infrastructure providers, tools and clients like block explorers and clients, which have to use archival nodes to serve queries for the pre-upgrade heights. :::

#### Breaking and Non-Breaking Upgrades <mark style="color:blue;">#</mark> <a href="#breaking-and-non-breaking-upgrades" id="breaking-and-non-breaking-upgrades"></a>

Upgrades can be categorized as breaking or non-breaking according to the Semantic versioning ([<mark style="color:blue;">Semver</mark>](https://semver.org/)) of the corresponding software release version(*i.e* `vX.Y.Z`):

* **Major version (`X`)**: backward incompatible API and state machine breaking changes.
* **Minor version (`Y`)**: new backward compatible features. These can be also be state machine breaking.
* **Patch version (`Z`)**<mark style="color:blue;">#</mark>: backwards compatible bug fixes, small refactors and improvements.

**Major Versions&#x20;**<mark style="color:blue;">**#**</mark>

If the new version you are upgrading to has breaking changes, you will have to:

1. Migrate genesis JSON
2. Migrate application state
3. Restart node

This needs to be done to prevent [<mark style="color:blue;">double signing or halting the chain during consensus</mark>.](https://github.com/tendermint/tendermint/blob/v0.34.x/spec/consensus/signing.md)

To upgrade the genesis file, you can either fetch it from a trusted source or export it locally using the `ICPlazad export` command.

**Minor Versions&#x20;**<mark style="color:blue;">**#**</mark>

If the new version you are upgrading to has breaking changes, you will have to:

1. Migrate the state (if applicable)
2. Restart node

**Patch Versions&#x20;**<mark style="color:blue;">**#**</mark>

In order to update a patch:

1. Stop Node
2. Download new release binary manually
3. Restart node

#### Data Reset Upgrades <mark style="color:blue;">#</mark> <a href="#data-reset-upgrades" id="data-reset-upgrades"></a>

Data Reset upgrades require node operators to fully reset the blockchain state and restart their nodes from a clean state, but using the same validator keys.

#### Automatic or Manual Upgrades <mark style="color:blue;">#</mark> <a href="#automatic-or-manual-upgrades" id="automatic-or-manual-upgrades"></a>

With every new software release, we strongly recommend full nodes and validator operators to perform a software upgrade.

You can upgrade your node by either:

* automatically bumping the software version and restart the node once the upgrade occurs, or
* download the new binary and perform a manual upgrade

Follow the links in the options above to learn how to upgrade your node according to your preferred option.

### Rollback <mark style="color:blue;">#</mark> <a href="#rollback" id="rollback"></a>

Learn how to rollback the chain version in the case of an unsuccessful chain upgrade. {synopsis}

In order to restore a previous chain version, the following data must be recovered by validators:

* the database that contains the state of the previous chain (in `~/.ICPlazad/data` by default)
* the `priv_validator_state.json` file of the validator (also in `~/.ICPlazad/data` by default)

If validators don’t possess their database data, another validator should share a copy of the database. Validators will be able to download a copy of the data and verify it before starting their node. If validators don’t have the backup `priv_validator_state.json` file, then those validators will not have double-sign protection on their first block.

#### Restoring State Procedure <mark style="color:blue;">#</mark> <a href="#restoring-state-procedure" id="restoring-state-procedure"></a>

1. First, stop your node.
2. Then, copy the contents of your backup data directory back to the `$ICPlaza_HOME/data` directory (which, by default, should be `~/.ICPlazad/data`).

```bash
# Assumes backup is stored in "backup" directory
rm -rf ~/.ICPlazad/data
mv backup/.ICPlazad/data ~/.ICPlazad/data
```

3. Next, install the previous version of ICPlaza.

```bash
# from ICPlaza directory
git checkout <prev_version>
make install
## verify version
ICPlazad version --long
```

4. Finally, start the node.

```bash
ICPlazad start
```

### Manual Upgrades <mark style="color:blue;">#</mark> <a href="#manual-upgrades" id="manual-upgrades"></a>

Learn how to manually upgrade your node.

#### 1. Upgrade the ICPlaza version <mark style="color:blue;">#</mark> <a href="#id-1-upgrade-the-icplaza-version" id="id-1-upgrade-the-icplaza-version"></a>

Before upgrading the ICPlaza version. Stop your instance of `ICPlazad` using `Ctrl/Cmd+C`.

Next, upgrade the software to the desired release version. Check the ICPlaza releases page for details on each release.

::: warning Ensure that the version installed matches the one needed for the network you are running (mainnet or testnet). :::

```bash
cd ICPlaza
git fetch --all && git checkout <new_version>
make install
```

::: tip If you have issues at this step, please check that you have the latest stable version of [<mark style="color:blue;">Golang</mark>](https://go.dev/dl/) installed. :::

Verify that you’ve successfully installed ICPlaza on your system by using the `version` command:

```bash
$ ICPlazad version --long

name: ICPlaza
server_name: ICPlazad
version: 3.0.0
commit: fe9df43332800a74a163c014c69e62765d8206e3
build_tags: netgo,ledger
go: go version go1.18 darwin/amd64
...
```

::: tip If the software version does not match, then please check your `$PATH` to ensure the correct `ICPlazad` is running. :::

#### 2. Replace Genesis file <mark style="color:blue;">#</mark> <a href="#id-2-replace-genesis-file" id="id-2-replace-genesis-file"></a>

::: tip You can find the latest `genesis.json` file for mainnet or testnet in the following repositories:

* **Mainnet**: github.com/tharsis/mainnet
* **Testnet**: github.com/tharsis/testnets :::

Save the new genesis as `new_genesis.json`. Then, replace the old `genesis.json` located in your `config/` directory with `new_genesis.json`:

```bash
cd $HOME/.ICPlazad/config
cp -f genesis.json new_genesis.json
mv new_genesis.json genesis.json
```

::: tip We recommend using `sha256sum` to check the hash of the downloaded genesis against the expected genesis.

```bash
cd ~/.ICPlazad/config
echo "<expected_hash>  genesis.json" | sha256sum -c
```

:::

#### 3. Data Reset <mark style="color:blue;">#</mark> <a href="#id-3-data-reset" id="id-3-data-reset"></a>

Remove the outdated files and reset the data:

```bash
rm $HOME/.ICPlazad/config/addrbook.json
ICPlazad tendermint unsafe-reset-all --home $HOME/.ICPlazad
```

Your node is now in a pristine state while keeping the original `priv_validator.json` and `config.toml`. If you had any sentry nodes or full nodes setup before, your node will still try to connect to them, but may fail if they haven’t also been upgraded.

::: danger 🚨 **IMPORTANT** 🚨

Make sure that every node has a unique `priv_validator.json`. **DO NOT** copy the `priv_validator.json` from an old node to multiple new nodes. Running two nodes with the same `priv_validator.json` will cause you to [<mark style="color:blue;">double sign</mark>](https://github.com/tendermint/tendermint/blob/v0.34.x/spec/consensus/signing.md). :::

#### 4. Restart Node <mark style="color:blue;">#</mark> <a href="#id-4-restart-node" id="id-4-restart-node"></a>

To restart your node once the new genesis has been updated, use the `start` command:

```bash
ICPlazad start
```

### Live Upgrade Tutorial <mark style="color:blue;">#</mark> <a href="#live-upgrade-tutorial" id="live-upgrade-tutorial"></a>

This document demonstrates how a live upgrade can be performed on-chain through a governance process.

1. Start the network and trigger upgrade

   ```bash
   # start a ICPlaza application full-node
   $ ICPlazad start

   # set up the cli config
   $ ICPlazad config trust-node true
   $ ICPlazad config chain-id testing

   # create an upgrade governance proposal
   $ ICPlazad tx gov submit-proposal software-upgrade <plan-name> \
   --title <proposal-title> --description <proposal-description> \
   --from <name-or-key> --upgrade-height <desired-upgrade-height> --deposit 10000000stake

   # once the proposal passes you can query the pending plan
   $ ICPlazad query upgrade plan
   ```
2. Performing an upgrade

   Assuming the proposal passes the chain will stop at given upgrade height.

   You can stop and start the original binary all you want, but **it will refuse to run after the upgrade height**.

   We need a new binary with the upgrade handler installed. The logs should look something like:

   ```bash
   E[2019-11-05|12:44:18.913] UPGRADE "<plan-name>" NEEDED at height: <desired-upgrade-height>:       module=main
   E[2019-11-05|12:44:18.914] CONSENSUS FAILURE!!!
   ...
   ```

   Note that the process will hang indefinitely (doesn’t exit to avoid restart loops). So, you must manually kill the process and replace it with a new binary. Do so now with `Ctrl+C` or `killall ICPlazad`.

   In `ICPlaza/app/app.go`, after `upgrade.Keeper` is initialized and set in the app, set the the corresponding upgrade `Handler` with the correct `<plan-name>`:

   ```go
       app.upgradeKeeper.SetUpgradeHandler("<plan-name>", func(ctx sdk.Context, plan upgrade.Plan) {
           // custom logic after the network upgrade has been executed
       })
   ```

   Note that we panic on any error - this would cause the upgrade to fail if the migration could not be run, and no node would advance - allowing a manual recovery. If we ignored the errors, then we would proceed with an incomplete upgrade and have a very difficult time every recovering the proper state.

   Now, compile the new binary and run the upgraded code to complete the upgrade:

   ```bash
   # create a new binary of ICPlaza with the added upgrade handler
   $ make install

   # Restart the chain using the new binary. You should see the chain resume from
   # the upgrade height:
   # `I[2019-11-05|12:48:15.184] applying upgrade <plan-name> at height: <desired-upgrade-height>      module=main`
   $ ICPlazad start

   # verify there is no pending plan
   $ ICPlazad query upgrade plan

   # verify you can query the block header of the completed upgrade
   $ ICPlazad query upgrade applied <plan-name>
   ```

[<mark style="color:blue;">← Upgrading Your Node</mark>](/icplaza-docs/chain-dev/upgrading-your-node.md)

[<mark style="color:blue;">Basic Concepts →</mark>](/icplaza-docs/chain-dev/basic-concepts.md)