// to display images directly on GitHub
ifdef::env-github[]
:encoding: UTF-8
:lang: en
:doctype: book
:toc: left
:imagesdir: ../images
endif::[]

////

    This file is part of the PacketFence project.

    See PacketFence_Clustering_Guide.asciidoc
    for authors, copyright and license information.

////

//== Troubleshooting a cluster

=== Checking the MariaDB sync

In order to check the MariaDB sync, you can look at the status of the `wsrep` status values inside MariaDB.

----
MariaDB> show status like 'wsrep%';
----

Important variables:

  * `wsrep_cluster_status`: Display whether or not the node is part of a primary view or not. A healthy cluster should always show as primary
  * `wsrep_incoming_addresses`: The current members of the cluster. All the nodes of your cluster should be listed there.
  * `wsrep_last_committed`: Sequence number of the most recently committed transaction. You can identify the most advanced node with this value.
  * `wsrep_local_state_comment`: Current sync state of the cluster. A healthy state is 'Synced'. Refer to the Galera cluster documentation for the meaning of the other values this can have.

In order for the cluster to be considered healthy, all nodes must be listed under `wsrep_incoming_addresses` and `wsrep_local_state_comment` must be `Synced`. Otherwise look in the MariaDB log ([filename]`/usr/local/pf/logs/mariadb.log`)

=== Automatic clustering resolution service: galera-autofix

Since v10, the <<_the_galera_autofix_service,`galera-autofix`>> service will try to resolve automatically issues on Galera clusters.

On a cluster which has an issue, once all nodes are back online, you should wait:

* around 10 minutes when at least one of the nodes of the cluster is able to offer database service
* around 20 minutes when there is no database service available

If your cluster still has issues after that time, you can try to resolve the issue by looking at sections below.

=== Manual resolution

If `galera-autofix` service is not able to solve your issue, you can try to solve it manually.

First, you need to perform checks. Then you will be able to identify in which situation you are:
Cluster offers database service without all nodes or None of the nodes is offering database service.

==== Database checks

NOTE: MariaDB root password has been defined during configurator.

===== Ensure MariaDB is running on each node

Ensure you can connect to MariaDB (through UNIX socket) on *each* node using:

[source,bash]
----
mysql -u root -p
----

===== Check cluster integrity

Once connected to MariaDB, run these commands on *each* node:

[source,sql]
----
# Each node in the cluster should provide the same value
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_state_uuid';

# Each node in the cluster should provide the same value
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_conf_id';

# Expected value: number of configured nodes in the cluster
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size';

# Expected value: Primary
SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status';
----

If all checks returned expected values, the cluster is up and has integrity.

===== Check individual node status

Once connected to MariaDB, run these commands on *each* node:

[source,sql]
----
# Expected value: ON
SHOW GLOBAL STATUS LIKE 'wsrep_ready';

# Expected value: ON
SHOW GLOBAL STATUS LIKE 'wsrep_connected';

# Expected values when node is part of the primary component: Joining, Waiting on SST, Joined, Synced or Donor
SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
----

If all checks returned expected values, individual nodes are in working order.

==== PacketFence checks

===== Check PacketFence application can connect to the database

In order to emulate how PacketFence connects to the database, you can run following command:

.For PacketFence versions 11.0.0 and later
[source,bash]
----
mysql -u $(perl -I/usr/local/pf/lib_perl/lib/perl5 -I/usr/local/pf/lib -Mpf::db -e 'print $pf::db::DB_Config->{user}') -p$(perl -I/usr/local/pf/lib_perl/lib/perl5 -I/usr/local/pf/lib -Mpf::db -e 'print $pf::db::DB_Config->{pass}') -h $(perl -I/usr/local/pf/lib_perl/lib/perl5 -I/usr/local/pf/lib -Mpf::db -e 'print $pf::db::DB_Config->{host}') pf
----

.For PacketFence versions prior to 11.0.0
[source,bash]
----
mysql -u $(perl -I/usr/local/pf/lib -Mpf::db -e 'print $pf::db::DB_Config->{user}') -p$(perl -I/usr/local/pf/lib -Mpf::db -e 'print $pf::db::DB_Config->{pass}') -h $(perl -I/usr/local/pf/lib -Mpf::db -e 'print $pf::db::DB_Config->{host}') pf
----

If you got a prompt, it means PacketFence must be able to connect to the database.

===== Perform a query using PacketFence codebase

To perform a small query to the database using PacketFence codebase, you can run:

[source,bash]
----
/usr/local/pf/bin/pfcmd checkup
----

If the command doesn't return any database error, PacketFence is able to perform reads on database.

==== Cluster offers database service without all nodes

When at least one of the nodes of the cluster is able to offer database
service, you can apply the following commands on a broken node to rejoin it to
the cluster:

[source,bash]
----
systemctl stop packetfence-mariadb
systemctl stop packetfence-galera-autofix
rm -fr /var/lib/mysql/*
systemctl start packetfence-mariadb
systemctl start packetfence-galera-autofix
----

This action will not cause service disruption on current cluster.

After all nodes have joined back cluster, you should verify <<_checking_the_mariadb_sync,MariaDB sync>>.

WARNING: After stopping the `packetfence-mariadb` service, be sure there is no more `mysql` process running.

==== None of the nodes is offering database service

When there is no more database service in your cluster, you need to do a full recovery.

You must identify the node you wish to keep the data from and start it with the
`--force-new-cluster` option.

Find the node which has the highest `seqno` value in
[filename]`/var/lib/mysql/grastate.dat`.

If the `seqno` value is `-1`, you need to start MariaDB manually with
`--wsrep-recover` to update the `seqno` value using the commands below:

[source,bash]
----
systemctl stop packetfence-galera-autofix
systemctl stop packetfence-mariadb
mysqld_safe --defaults-file=/usr/local/pf/var/conf/mariadb.conf --wsrep-recover
----

In MariaDB log file under [filename]`/usr/local/pf/logs` or in output of `journalctl -u packetfence-mariadb`, you should
find a line like this:

----
[Note] WSREP: Recovered position: 220dcdcb-1629-11e4-add3-aec059ad3734:1122
----

The recovered position is a pair `<cluster state UUID>:<sequence
number>`. The node with the highest sequence number in its
recovered position is the most up-to-date, and should be chosen as
bootstrap candidate.

Once you have identified the most up-to-date node, run
following commands on it:

[source,bash]
----
systemctl stop packetfence-mariadb
systemctl stop packetfence-galera-autofix
/usr/local/pf/sbin/pf-mariadb --force-new-cluster
----

WARNING: After stopping the `packetfence-mariadb` service, be sure there is no more `mysql` process running.

==== On each of the discarded servers

First, stop `packetfence-mariadb` and `packetfence-galera-autofix` on all
the servers you want to discard data from.

[source,bash]
----
systemctl stop packetfence-mariadb
systemctl stop packetfence-galera-autofix
----

On each of the servers you want to discard the data from, you must destroy all
the data in `/var/lib/mysql` and start `packetfence-mariadb` so it resyncs its
data from scratch.

[source,bash]
----
rm -fr /var/lib/mysql/*
systemctl start packetfence-mariadb
----

You should then see `/var/lib/mysql` be populated again with the data and once
MariaDB becomes available again on the server, it means the sync has
completed. In case of issues, look in the MariaDB log file
(`/usr/local/pf/logs/mariadb.log`)

WARNING: After stopping the `packetfence-mariadb` service, be sure there is no more `mysql` process running.

==== On the node started with --force-new-cluster

If you were performing a full recovery, you should now break the
`--force-new-cluster` command and start `packetfence-mariadb`
normally using:

[source,bash]
----
systemctl start packetfence-mariadb
----

==== On all servers

When your database offers service again, you can restart
`packetfence-galera-autofix` service using:

[source,bash]
----
systemctl start packetfence-galera-autofix
----

Be sure to verify <<_checking_the_mariadb_sync,MariaDB sync>>.
