// to display images directly on GitHub
ifdef::env-github[]
:encoding: UTF-8
:lang: en
:doctype: book
:toc: left
:imagesdir: ../images
endif::[]

////

    This file is part of the PacketFence project.

    See PacketFence_Clustering_Guide.asciidoc
    for authors, copyright and license information.

////

//== Cluster Setup

=== Setup on all servers

Following actions need to be run on *all cluster members*.

==== Interfaces names

You need to make sure the interfaces names will be the same on *all
servers*. You can follow these guides to have predictable network interface
names on distribution you used:

* link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/consistent-network-interface-device-naming_configuring-and-managing-networking[RHEL-based systems]
* link:https://wiki.debian.org/NetworkInterfaceNames[Debian-based systems]

==== sysctl.conf

You will need to configure each server so the services can bind on IP
addresses they don't currently have configured. This allows faster failover of
the services.

You also need to disable IPv6.

On *all your servers*, add following lines in [filename]`/etc/sysctl.conf`:

----
net.ipv4.ip_nonlocal_bind = 1
net.ipv6.conf.all.disable_ipv6 = 1
----

and run:

[source,bash]
----
sysctl -p
reboot
----

NOTE: If you plan to use Postfix to send emails, you need to set `inet_protocols = ipv4` in [filename]`/etc/postfix/main.cf` to be able to use it.

==== Installation of PacketFence

Before starting cluster setup, you need to install
PacketFence on each cluster member by following instructions in
<<PacketFence_Installation_Guide.asciidoc#_installation,PacketFence
Installation Guide>>.

==== Install the database replication tools

NOTE: In this example, the database stack uses the native PacketFence https://mariadb.com/kb/en/library/galera-cluster/[MariaDB Galera cluster] integration. Although other MySQL based clustering stacks are supported, they aren't covered in this guide. If you use an external database or want to use another clustering stack for the database, you can ignore this section and jump to Step 2 directly.

CAUTION: Galera cluster is only supported in 3 nodes cluster and more (with an odd number of servers).

First, you will need to install, *on each servers*, Mariabackup for the synchronization to work correctly.

.On RHEL-based systems
[source,bash]
----
yum install MariaDB-backup socat --enablerepo=packetfence
----

.On Debian-based systems (for PacketFence versions 11.0.0 and later)
[source,bash]
----
apt-get install lsb-release wget gnupg2 ; \
apt-get update ; \
apt-get -y install mariadb-backup
----

.On Debian-based systems (for PacketFence versions prior to 11.0.0)
[source,bash]
----
apt-get install lsb-release wget gnupg2 ; \
apt-get update ; \
apt-get -y install mariadb-backup-10.2
----

For the next steps, you want to make sure that you didn't configure anything
in [filename]`/usr/local/pf/conf/cluster.conf`. If you already did, comment all the
configuration in the file and do a configreload ([command]`/usr/local/pf/bin/pfcmd configreload hard`).

=== Setup on the first server of your cluster


First, *on the first server*, ensure `packetfence-mariadb` is running and make sure it was able to start in 'standalone' mode.

[source,bash]
----
systemctl status packetfence-mariadb
----

Then, you will need to create a user for the database replication that
PacketFence will use. You can use any username/password combination. After
creating the user, keep its information close-by for usage in the
configuration.

WARNING: *aMuchMoreSecurePassword* is only for example purpose, you need to define your own password. This user should have a password that contains only alphanumeric characters (letters, numbers and importantly, *no spaces*).

[source,bash]
----
mysql -u root

CREATE USER 'pfcluster'@'%' IDENTIFIED BY 'aMuchMoreSecurePassword';
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT, SUPER ON *.* TO 'pfcluster'@'%';

CREATE USER 'pfcluster'@'localhost' IDENTIFIED BY 'aMuchMoreSecurePassword';
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT, SUPER ON *.* TO 'pfcluster'@'localhost';

FLUSH PRIVILEGES;

----

=== Basic PacketFence configuration

==== First server

Now, on *the first server* of your cluster, you should go through the configurator, until last step. You should leave the services **stopped** at the end of the configurator.

NOTE: When configuring the network interfaces, ensure that you mark the management interface as *high-availability*. Otherwise, you will not be able to perform the database synchronization.

Then restart PacketFence's mariadb on *the first server*:

[source,bash]
----
systemctl restart packetfence-mariadb
----

==== Other servers (optional)

On the *other servers* of your cluster, configure only the network interfaces
(step 1) without going past that section in the configurator. If the other
servers already have the right IP addresses configured on their interfaces,
you can ignore this step.

This step is only necessary to configure IP addresses on
interfaces (at OS level). PacketFence configuration of interfaces will be done later.

==== Current state

At this point, for a VLAN enforcement configuration for example, the network
interfaces of your servers must be configured, and you must be able to see,
for each server:

|===
|In `/etc/sysconfig/network-scripts/`|
|One Management Interface|ifcfg-*YourFirstInterfaceName*

|One Secondary Interface|ifcfg-*YourSecondInterfaceName*

|One Registration Interface|ifcfg-*YourSecondInterfaceName*.*YourRegistrationVLANID*

|One Isolation Interface|ifcfg-*YourSecondInterfaceName*.*YourIsolationVLANID*
|===


=== Create the new cluster

==== PacketFence Configuration Modification (first server only)

In order for PacketFence to communicate properly with your MariaDB cluster, you need to change the following.
This change only needs to be done on the first server of the cluster. It will be synchronized later.

Add the following to the bottom of [filename]`/usr/local/pf/conf/pf.conf` :

----
[database]
host=100.64.0.1
port=6033

[active_active]
# Change these 2 values by the credentials you've set when configuring MariaDB above
galera_replication_username=pfcluster
galera_replication_password=aMuchMoreSecurePassword

[webservices]
# Change these 2 values by the credentials you want
user=packet
pass=anotherMoreSecurePassword

[advanced]
configurator=disabled

[services]
galera-autofix=disabled
----

Then, add the following to the bottom of [filename]`/usr/local/pf/conf/pfconfig.conf` :

----
[mysql]
host=100.64.0.1
port=6033
----

Now, restart `packetfence-config` and reload the configuration. You will see errors related to a cache write issue but you can safely ignore it for now. These appear because `packetfence-config` cannot connect to the database yet.

[source,bash]
----
systemctl restart packetfence-config
/usr/local/pf/bin/pfcmd configreload hard
----

==== Configure cluster.conf (first server only)

In order to create a new cluster, you need to configure [filename]`/usr/local/pf/conf/cluster.conf` *on the first server* of your cluster.

You will need to configure it with your server hostname. Use : [command]`hostname` command (without any arguments) to get it.

In the case of this example it will be `pf1.example.com`.

The `CLUSTER` section represents the virtual IP addresses of your cluster that will be shared by your servers.

In this example, eth0 is the management interface, eth1.2 is the registration interface and eth1.3 is the isolation interface.

Create a configuration similar to this :

----
[CLUSTER]
management_ip=192.168.1.10

[CLUSTER interface eth0]
ip=192.168.1.10

[CLUSTER interface eth1.2]
ip=192.168.2.10

[CLUSTER interface eth1.3]
ip=192.168.3.10

[pf1.example.com]
management_ip=192.168.1.5

[pf1.example.com interface eth0]
ip=192.168.1.5

[pf1.example.com interface eth1.2]
ip=192.168.2.5

[pf1.example.com interface eth1.3]
ip=192.168.3.5

[pf2.example.com]
management_ip=192.168.1.6

[pf2.example.com interface eth0]
ip=192.168.1.6

[pf2.example.com interface eth1.2]
ip=192.168.2.6

[pf2.example.com interface eth1.3]
ip=192.168.3.6

[pf3.example.com]
management_ip=192.168.1.7

[pf3.example.com interface eth0]
ip=192.168.1.7

[pf3.example.com interface eth1.2]
ip=192.168.2.7

[pf3.example.com interface eth1.3]
ip=192.168.3.7
----

Once this configuration is done, reload the configuration and perform a checkup:

[source,bash]
----
/usr/local/pf/bin/pfcmd configreload hard
/usr/local/pf/bin/pfcmd checkup
----

The reload and the checkup will complain about the unavailability of the
database, which you can safely ignore for now. Most important is that you
don't see any cluster configuration related errors during the checkup.


==== Database setup

===== Second and third servers

Make sure you stopped MariaDB on the two others servers:

[source,bash]
----
systemctl stop packetfence-mariadb
----

===== First server

Start MariaDB forcing it to create a new cluster using configuration defined in [filename]`/usr/local/pf/conf/cluster.conf`:

[source,bash]
----
systemctl stop packetfence-mariadb
/usr/local/pf/bin/pfcmd generatemariadbconfig
systemctl set-environment MARIADB_ARGS=--force-new-cluster
systemctl start packetfence-mariadb
----

Then, restart PacketFence to apply all your changes:

[source,bash]
----
/usr/local/pf/bin/pfcmd service pf restart
----

===== Expected state on first server

If no error is found in the previous configuration, the previous restart of
PacketFence should have started: `keepalived` and `radiusd-loadbalancer` along
with the other services. If you have set up a mail server on your first
server, you should have receive a mail from `keepalived` to inform you that
your first server got Virtual IP (VIP) adresses.

You should now have service using the first server on the IP addresses defined in the `CLUSTER` sections.

NOTE: You can check the status of the services using [command]`/usr/local/pf/bin/pfcmd service pf status`

NOTE: You can check with [command]`ip -br a`, on the first server, you need to find the *VIP* on the first ethernet interface. On the others server, be sure to have the `interface.VLANID` interfaces with the good IPs.

==== Enable PacketFence clustering services at boot (all servers)

Make sure the PacketFence clustering services will be started at boot by running the following command on *all of your servers*:

[source,bash]
----
systemctl set-default packetfence-cluster
----

=== Integrating the two other nodes

WARNING: If you reboot any of the nodes you're joining, you will need to stop all the PacketFence services (`/usr/local/pf/bin/pfcmd service pf stop`) and restart the steps from here.

WARNING: If you reboot the management node (first server), you will need to stop `packetfence-mariadb` (`systemctl stop packetfence-mariadb`) and start it with the new cluster option so the servers can join (`systemctl set-environment MARIADB_ARGS=--force-new-cluster && systemctl restart packetfence-mariadb`)

Now, you will need to integrate your *two other nodes* in your cluster.

==== Stop iptables (all servers)

On *all your servers*, make sure that `packetfence-iptables` is stopped:

[source,bash]
----
systemctl stop packetfence-iptables
----

==== Sync the PacketFence configuration across the cluster (second and third servers)

Do (and make sure it completes without any errors):

[source,bash]
----
 /usr/local/pf/bin/cluster/sync --from=192.168.1.5 --api-user=packet --api-password=anotherMoreSecurePassword
----

NOTE: Space before last command is on purpose to avoid record of password in shell history

Where :

* '192.168.1.5' is the management IP of the *first server* node
* 'packet' is the webservices username you have configured on the *first server* node during <<_packetfence_configuration_modification_first_server_only>>
* 'anotherMoreSecurePassword' is the webservices password you have configured on the *first server* node during <<_packetfence_configuration_modification_first_server_only>>

Then, reload the configuration and start the webservices on second and third servers:

[source,bash]
----
systemctl restart packetfence-config
/usr/local/pf/bin/pfcmd configreload
/usr/local/pf/bin/pfcmd service proxysql restart
/usr/local/pf/bin/pfcmd service httpd.webservices restart
----

==== MariaDB sync (second and third servers)

Ensure `packetfence-mariadb` is still stopped on the two servers that will be joined:

[source,bash]
----
systemctl stop packetfence-mariadb
----

Now, flush any MariaDB data you have on the two servers and restart `packetfence-mariadb` so that the servers join the cluster.

WARNING: If you have any data in MariaDB on these nodes, this will destroy it.

[source,bash]
----
rm -fr /var/lib/mysql/*
systemctl restart packetfence-mariadb
----

If you see following message when running [command]`systemctl status packetfence-mariadb`, your nodes have successfully joined cluster:

----
INFO: Successful clustered connection to the DB
----

To be sure your cluster is correctly setup, take a look at <<_checking_the_mariadb_sync>> section.

In case you have some issues, ensure your MariaDB instance running with `--force-new-cluster` is still running on the first server, if its not, start it again.

==== Starting the first server normally

Once all servers are synced, go *on the first server* that should still be running with the `--force-new-cluster` option and run:

[source,bash]
----
systemctl stop packetfence-mariadb
systemctl unset-environment MARIADB_ARGS
systemctl start packetfence-mariadb
----

Now, restart `packetfence-iptables`:

[source,bash]
----
systemctl restart packetfence-iptables
----

===== Enabling galera-autofix service (first server)

Before starting services on all servers, `galera-autofix` service need to be re-enabled and configuration synced across cluster:

[source,bash]
----
curl -X PATCH -d '{"galera-autofix":"enabled"}' localhost:22224/api/v1/config/base/services ; echo
----

==== Wrapping up

Now restart PacketFence *on all servers*:

[source,bash]
----
/usr/local/pf/bin/pfcmd service pf updatesystemd
/usr/local/pf/bin/pfcmd service pf restart
----

You should now reboot *each server one by one* waiting for the one you
rebooted to come back online before proceeding to the next one:

[source,bash]
----
reboot
----

After each reboot, ensure the database sync is fine by performing the checks outlined in
<<_checking_the_mariadb_sync>> section.

=== Additional steps

==== Securing the cluster: Keepalived secret

NOTE: It is highly recommended to modify the keepalived shared secret in your cluster to prevent attacks.

From the PacketFence web administration interface (using virtual IP address of
your cluster), go in _Configuration -> System Configuration -> Cluster_ and
change the `Shared KEY`.

Make sure you restart `keepalived` on *all your
servers* using:

[source,bash]
----
/usr/local/pf/bin/pfcmd service keepalived restart
----

If you already use VRRP protocol on your network, you can also change the default `Virtual Router ID` and enable `VRRP Unicast`.

==== Domain join

Next, make sure to join domains through _Configuration -> Policies And Access Control -> Domains -> Active Directory Domains_ on *each node*.

