Replication Group Member Recovery

This page provides recovery instructions based on the specific issue that you are experiencing.

All/Most Databases Are Down and Won’t Come Online Correctly

When all databases are down and won’t come online correctly, as in the case of a site power failure, we need to manually bootstrap the group. See: Bootstrap the Replication Group

Single Database Failing to Rejoin Group

1. Check the status of all members in the group
   • Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database.
   • In the porta-database folder, double click view-ALL-group-repl-status.bat
2. If the other machines have MEMBER_STATE of ONLINE and there is a member with MEMBER_ROLE of PRIMARY, then we can try simply restarting replication on the machine that is failing to rejoin the group: Restart Replication
3. If the other machines have MEMBER_STATE of ONLINE and there is NOT a member with MEMBER_ROLE of PRIMARY, then bootstrap a machine that is not ONLINE: Bootstrap Replication
   • After bootstrapping the machine:
     • On another machine, Restart Replication to ensure it joins the group
     • On the last machine, Restart Replication to ensure it joins the group
4. If one of the above steps does not work, then we can try resetting the database container that is failing to rejoin the group:
   • Reset the Database
   • ! WARNING: Resetting will wipe the database data on the machine !

Every Machine Only Registers Itself in the Group

If each machine only sees itself in the replication group with a MEMBER_STATE of OFFLINE, this likely means that there was no existing group to join. This can happen if the machines are not able to communicate with each other, or if the group replication process was not bootstrapped on any of the machines.

To fix this, we need to bootstrap the replication group: Bootstrap the Replication Group

A Machine is Listed as UNREACHABLE

If the other machines are listed as ONLINE, then restart replication on the machine that is listed as UNREACHABLE: Restart Replication

If the restart fails to fix the issue, then we need to Bootstrap the Replication Group

A Joining Member Creates Its Own Group

If a joining member creates its own group, this likely means that the group_replication_bootstrap_group configuration was not set to OFF on the joining member. We will need to turn this off in the configuration file and restart replication on the machine: Turn Off Default Bootstrap.

Note: It is possible that creating its own group may prevent it form joining the existing group, even after turning bootstrap off. If this happens, we may need to Bootstrap the Replication Group. If the machine is unable to join the group after bootstrapping, then we may need to Reset the Database.

Misc

For more troubleshooting details and error messages, see the Group Replication Troubleshooting document.

---

Common Replication Recovery Actions

Stop Replication

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click STOP-repl.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.

Start Replication

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click START-repl.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.

Restart Replication

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click restart-repl.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.

Bootstrap Replication

For bootstrapping a single member.

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click bootstrap.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.
8. If prompted for migration, enter n.
9. If prompted for seeding, enter n.

Bootstrap the Replication Group

For bootstrapping the replication group on all machines, such as when a primary member does not exist.

1. On the main machine, Stop Replication
2. On the backup machine, Stop Replication
3. On the arbiter machine, Stop Replication
4. On the main machine, Bootstrap Replication
5. On the backup machine, Start Replication
6. On the arbiter machine, Start Replication

All members should be running correctly and ONLINE.

Reset Database

!! WARNING: THIS WILL WIPE DATA ON THE MACHINE !!

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions
2. In the actions folder, double click create-backup.bat to create a backup of the database.
3. In the actions folder, navigate to the caution folder.
4. In the caution folder, double click reset.bat
5. Enter or confirm the current Windows user.
6. Enter or confirm the default WSL user.
7. Enter or confirm the current machine type.
8. Enter or confirm the current machine IP address.
9. Enter or confirm additional machine information.
10. Let the process run until finished.

Turn Off Default Bootstrap

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions, then caution
2. In the caution folder, double click set-bootstrap-OFF.bat
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.
8. Navigate up to the actions folder and Restart Replication to rejoin the existing group.

Turn On Default Bootstrap

1. Navigate to the desktop’s porta-onprem-bundle folder, then to porta-helpers, then porta-database, then actions, then caution
2. In the caution folder, double click set-bootstrap-ON.bat.
3. Enter or confirm the current Windows user.
4. Enter or confirm the default WSL user.
5. Enter or confirm the current machine type.
6. Enter or confirm the current machine IP address.
7. Let the process run until finished.
8. Navigate up to the actions folder and Restart Replication.