Replication Group Member Recovery

This page provides recovery instructions based on the specific issue that you are experiencing.

Failover Recovery Overview

If the primary (write) database fails unexpectedly, such as in the case of power failure, network failure, etc., Porta will automatically failover to the backup database and continue operations. Here is a high-level overview of the failover process:

Primary failure

Upon detection of a failure, the remaining databases will elect a new primary database.

Failover to backup

The backup database is now given the MEMBER_ROLE of PRIMARY and the ability to write data. Porta detects and establishes this new connection, and operations proceed as usual.

Failed machine begins recovery

If the failed machine regains connectivity, it will attempt to rejoin the group and begin recovering data that was missed during the failure. During this period, it will have a MEMBER_STATE of RECOVERING.

Note

Depending on the amount of data that was missed, this process may take some time. In most cases, recovery will only take a minute or two.

Failed Machine is fully operational again

When the failed machine has fully recovered, it will have a MEMBER_STATE of ONLINE and will be fully operational again. However, the MEMBER_ROLE of PRIMARY will remain with the backup machine until it leaves group, such as via failure, shutdown, or other means.

Manual Recovery Steps

There are some cases in which manual intervention will be required after a failure or shutdown. Below are some common issues and their recovery steps.

All/Most Databases Are Down and Won’t Come Online Correctly

When all databases are down and won’t come online correctly, as in the case of a site power failure, we need to manually bootstrap the group. See: Bootstrap the Replication Group

Single Database Failing to Rejoin Group

1. Check the status of all members in the group
   • Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database.
   • In the porta-database folder, double click view-ALL-group-repl-status.bat
2. If the other machines have MEMBER_STATE of ONLINE and there is a member with MEMBER_ROLE of PRIMARY, then we can try simply restarting replication on the machine that is failing to rejoin the group: Restart Replication
3. If the other machines have MEMBER_STATE of ONLINE and there is NOT a member with MEMBER_ROLE of PRIMARY, then bootstrap a machine that is not ONLINE: Bootstrap Replication
   • After bootstrapping the machine:
     • On another machine, Restart Replication to ensure it joins the group
     • On the last machine, Restart Replication to ensure it joins the group
4. If one of the above steps does not work, then we can try resetting the database container that is failing to rejoin the group:
   • Reset the Database
   • ! WARNING: Resetting will wipe the database data on the machine !

Every Machine Only Registers Itself in the Group

If each machine only sees itself in the replication group with a MEMBER_STATE of OFFLINE, this likely means that there was no existing group to join. This can happen if the machines are not able to communicate with each other, or if the group replication process was not bootstrapped on any of the machines.

To fix this, we need to bootstrap the replication group: Bootstrap the Replication Group

A Machine is Listed as UNREACHABLE

If the other machines are listed as ONLINE, then restart replication on the machine that is listed as UNREACHABLE: Restart Replication

If the restart fails to fix the issue, then we need to Bootstrap the Replication Group

A Joining Member Creates Its Own Group

If a joining member creates its own group, this likely means that the group_replication_bootstrap_group configuration was not set to OFF on the joining member. We will need to turn this off in the configuration file and restart replication on the machine: Turn Off Default Bootstrap.

Note: It is possible that creating its own group may prevent it form joining the existing group, even after turning bootstrap off. If this happens, we may need to Bootstrap the Replication Group. If the machine is unable to join the group after bootstrapping, then we may need to Reset the Database.

Misc

For more troubleshooting details and error messages, see the Group Replication Troubleshooting document.

---

Common Replication Recovery Actions

Stop Replication

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click STOP-repl.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.

Start Replication

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click START-repl.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.

Restart Replication

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click restart-repl.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.

Bootstrap Replication

For bootstrapping a single member.

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click bootstrap.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.
8. If prompted for migration, enter n.
9. If prompted for seeding, enter n.

Bootstrap the Replication Group

For bootstrapping the replication group on all machines, such as when a primary member does not exist.

1. On the main machine, Stop Replication
2. On the backup machine, Stop Replication
3. On the arbiter machine, Stop Replication
4. On the main machine, Bootstrap Replication
5. On the backup machine, Start Replication
6. On the arbiter machine, Start Replication

All members should be running correctly and ONLINE.

Reset Database

Danger

RESTTING THE PORTA DATABASE WILL WIPE ALL DATA ON THE MACHINE!

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click create-backup.bat to create a backup of the database.
3. In the actions folder, navigate to the caution folder.
4. In the caution folder, double click reset.bat
5. Enter or confirm the current Windows user.
6. Enter or confirm the default WSL user.
7. Enter or confirm the current machine type.
8. Enter or confirm the current machine IP address.
9. Enter or confirm additional machine information.
10. Let the process run until finished.

Turn Off Default Bootstrap

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions, then caution
2. In the caution folder, double click set-bootstrap-OFF.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.
8. Navigate up to the actions folder and Restart Replication to rejoin the existing group.

Turn On Default Bootstrap

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions, then caution
2. In the caution folder, double click set-bootstrap-ON.bat.
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.
8. Navigate up to the actions folder and Restart Replication.