From fb6a28550a5049b6d07e34149c6d2f3af7afc0b2 Mon Sep 17 00:00:00 2001 From: mrpa Date: Thu, 18 Feb 2021 11:20:13 +0100 Subject: Added in the High Availability section into the Getting Started Manual. Change-Id: Ib0920f6e1e089a72ede58a63fd9e3f1088b0c384 Signed-off-by: mrpa --- .../doc/advanced_configurations.xml | 626 +++++++++++++++++++++ .../doc/images/high_av_setup.png | Bin 0 -> 436204 bytes 2 files changed, 626 insertions(+) create mode 100644 doc/book-enea-nfv-access-getting-started/doc/images/high_av_setup.png diff --git a/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml b/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml index 1b04c40..74ff1dd 100644 --- a/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml +++ b/doc/book-enea-nfv-access-getting-started/doc/advanced_configurations.xml @@ -420,4 +420,630 @@ node0.1048576kB = 3 + +
+ Installing the Enea uCPE Manager in High Availability Mode + + The following describes the setup needed for running the Enea uCPE + Manager in High Availabilty (HA) mode, with a MariaDB database cluster. + The desired setup is depicted in the following diagram: + +
+ The High Availability setup + + + + + + +
+ +
+ Requirements for High Availability + + The following hardware is needed for deploying the base configuration: + + + + Machines running the Enea uCPE Manager and MariaDB: + + + + 4 CPU cores + + + + 12 - 16 GB memory + + + + 256 - 512 GB hard disk + + + + + + Machines running only MariaDB: + + + + 2 CPU cores + + + + 8 GB memory + + + + 256 - 512 GB hard disk + + + + + + The Enea uCPE Manager machines should run CentOS 7, this is + the only currently supported version. + + + + All machines should be on the same subnet. For geographically + distributed servers, a VPN can be used. + + + + All VCPE devices will typically connect to the external IP + (WAN) address (exported by the Big-IP firewall). + + + + WAN traffic will be HTTPS, whereas internal communication will + be through HTTP. + + + + External clients (browsers using the GUI as well as clients + using the REST API) will connect to the external (WAN) + address. + + +
+ +
+ Firewall Rules + + The following firewall configuration is needed: + + + + Disable SELINUX on all database servers by + editing /etc/sysconfig/selinux and changing the + following: + + SELINUX=disabled +SELINUXTYPE=targeted + + + + Reboot the server: + + [root@localhost ~]# sudo shutdown -r now + + + + The following ports should be opened in the local firewall (not + Big-IP), for each Enea uCPE Manager machine: + + + Ports for Enea uCPE Manager Machines + + + + + + + Port (Protocol) + + Usage + + + + + + 80 (TCP) + + HTTP (used by Big-IP firewall) + + + + 443 (TCP) + + HTTPS + + + + 54327 (UDP) + + Cluster multicasting (Hazelcast) + + + + 5701 - 5708 (TCP) + + Hazelcast communications + + + + 4334 (TCP) + + NETCONF call-home + + + + 7000 - 7009 (TCP) + + Reverse SSH connection pool + + + +
+ + For each MariaDB machine, the following firewall configuration is needed: + + + Ports for MariaDB Machines + + + + + + + Port (Protocol) + + Usage + + + + + + 3306 (TCP) + + Client connections + + + + 4567 (UDP/TCP) + + Galera cluster replication with multicasting + + + + 4568 (TCP) + + Incremental state transfer + + + + 4444 (TCP) + + State snapshot transfer + + + +
+ + The following ports should be accessible externally and translated + to the Virtual IP side as shown below (by the Big-IP firewall): + + + Ports for Virtual IP + + + + + + + External Port (Protocol) + + Usage + + Local Port (Protocol) + + + + + + 443 (TCP) + + HTTPS to/back HTTP + + 80 (TCP) + + + + 4334 (TCP) + + NETCONF call-home + + 4334 (TCP) + + + + 7000 - 7009 (TCP) + + Reverse SSH connection pool + + 7000 - 7009 (TCP) + + + +
+
+ +
+ Installing High Availability + + The Enea uCPE Manager can be installed in High Availability mode with + a MariaDB database cluster by performing the following steps. The mandatory + Java configuration is also detailed. + +
+ Installing and configuring the MariaDB cluster + + Install the latest MariaDB packages on all servers. + + + The setup was tested using MariaDB 10.5.8, built for CentOS + 7. + + + How to install MariaDB + + + + Make sure the following packages are installed: + + MariaDB-compat-10.5.8-1.el7.centos.x86_64 +MariaDB-common-10.5.8-1.el7.centos.x86_64 +MariaDB-server-10.5.8-1.el7.centos.x86_64 +MariaDB-client-10.5.8-1.el7.centos.x86_64 +galera-4-26.4.6-1.el7.centos.x86_64 + + These provide the MariaDB server, client and the Galera + wsrep provider library. + + + + Copy the wsrep template: + + [root@localhost ~]# cp /usr/share/mysql/wsrep.cnf /etc/my.cnf.d + + + + Change the following configuration: + + # Full path to wsrep provider library or 'none' +wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so + +# Provider specific configuration options +#wsrep_provider_options= + +# Logical cluster name. Should be the same for all nodes. +wsrep_cluster_name="ucpemanager" + +# Group communication system handle +wsrep_cluster_address="gcomm://192.168.10.11,192.168.10.12,..,192.168.10.16" + +# Human-readable node name (non-unique). Hostname by default. +wsrep_node_name=Node1 +# current node's name. set node name for each server in the cluster + +# Base replication <address|hostname>[:port] of the node. +# The values supplied will be used as defaults for state transfer receiving, +# listening ports and so on. Default: address of the first network interface. +wsrep_node_address=192.168.10.11 +#current node's interface IP . must be set for each node in the cluster + + + Steps 2 and 3 must be performed for each MariaDB node in + the cluster. + + + + + Bootstrap the first node in the cluster (identified by + Node1 for example), by running: + + [root@localhost ~]# galera_new_cluster + + This script passes the + --wsrep-new-cluster to + mysqld which tells the node that there is no + pre-existing cluster to connect to. The node will create a new + UUID to identify the new cluster. + + + Do not execute this script when connecting to an existing + cluster. It will create a new UUID to identify the cluster + again, and the node won't reconnect to the old cluster. + + + + + Go to Node1 and start the service: + + [root@localhost ~]# systemctl start mariadb + + Subsequently, start the service on the other servers. + + + + Verify that the nodes have entered the cluster: + + [root@localhost ~]# mysql --host=localhost --user=root -p +MariaDB [(none)]> show status like 'wsrep_cluster_conf_%'; ++-----------------------+-------+ +| Variable_name | Value | ++-----------------------+-------+ +| wsrep_cluster_conf_id | 3 | ++-----------------------+-------+ +1 row in set (0.001 sec) + + + + Run the initial configuration script (only once, on one of + the machines in the cluster): + + [root@localhost ~]# mysql_secure_installation + +Switch to unix_socket authentication [Y/n] Y +Enabled successfully! +Reloading privilege tables.. + ... Success! +… +Change the root password? [Y/n] Y +New password: +Re-enter new password: +Password updated successfully! +Reloading privilege tables.. + ... Success! +… +Remove anonymous users? [Y/n] Y + ... Success! +… +Disallow root login remotely? [Y/n] Y + ... Success! +… +Remove test database and access to it? [Y/n] Y (optional) + - Dropping test database... + ... Success! + - Removing privileges on test database... + ... Success! +Reload privilege tables now? [Y/n] Y + ... Success! + +Cleaning up... + +All done! If you've completed all of the above steps, your MariaDB +installation should now be secure. + +Thanks for using MariaDB! + + + + Create the initial database and grant access to it: + + [root@localhost application]# mysql --host=localhost --user=root -p +MariaDB [(none)]> CREATE DATABASE ucpemanager CHARACTER SET='utf8' \ +COLLATE='utf8_bin'; +Query OK, 1 row affected (0.004 sec) + +MariaDB [(none)]> GRANT ALL PRIVILEGES ON ucpemanager.* \ +TO 'enea'@'%' IDENTIFIED BY 'somepassword' WITH GRANT OPTION; + + +
+ +
+ Installing the Java SDK + + The following steps describe the installation of Java 11 SDK on + the CentOS 7 machines that will run the Enea uCPE Manager + installation: + + + + Install the following packages: + + java-11-openjdk-devel-11.0.10.0.9-0.el7_9.x86_64 +java-11-openjdk-11.0.10.0.9-0.el7_9.x86_64 + + + + Check that java points to the current JRE: + + root@localhost ~]# java -version +openjdk version "11.0.10" 2021-01-19 LTS +OpenJDK Runtime Environment 18.9 (build 11.0.10+9-LTS) +OpenJDK 64-Bit Server VM 18.9 (build 11.0.10+9-LTS, mixed mode, sharing) + + If it doesn't, then check the alternatives, and make sure + that java points to the JDK11 installation: + + [root@localhost ~]# alternatives --config java + + + + Set the JAVA_HOME environment variable + and update paths: + + export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac))))) +export PATH=$PATH:$JAVA_HOME/bin +export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar + + As an alternative, the variables can be written into the + .bashrc file, so that they load every time a + console is opened. To enable these settings for all users, add the + variables to /etc/environment. + + + + The JAVA_HOME variable should point + to: + + [root@localhost ~]# echo $JAVA_HOME +/usr/lib/jvm/java-11-openjdk-11.0.10.0.9-0.el7_9.x86_64 + + +
+ +
+ Installing the Enea uCPE Manager in High Availabilty + mode + + These steps must be taken on each of the CentOS 7 machines that + will host the Enea uCPE Manager. + + As the root user, go to the distribution folder of the Enea uCPE + Manager, and run: + + [root@localhost distro]#./install.sh /opt/ \ +Enea_NFV_Access_uCPEManager_2.3.0-build23.tar.gz +This will install uCPEManager into /opt/ucpemanager folder. +Select the following options, while asked by the installation script: +Are you using the embedded PostgreSQL database? [Y/N]: N +External database selected, getting user information ... +Press 1 for PostgreSQL, 2 for MariaDB, 3 for SQL Server, 4 for Oracle and 5 \ +for MySQL: 2 +Specify database server name(s) or IP Address(es): \ +192.168.10.11,192.168.10.12,…,192.168.10.16 *(see note) +Specify database ID (or name) [ucpemanager]: +Specify database server port [3306]: +Specify database user name [root]: enea +Specify database password [root]: somepassword +Specify database startup thread pool size [1]: +Creating database configuration file \ +/opt//ucpemanager/application/config/databaseConfig.xml ... +Done . +… +Installing ucpemanager service .. +Specify service username [ucpemanager]: +Specify service password [ucpemanager]: somepassword +… +Specify the IP address of the local interface: 192.168.10.11 +Is this server part of a cluster? [Y/N]: Y +Specify the name of the cluster [ucpemanager]: +Specify the shared (virtual) cluster IP address: 192.168.10.10 +Specify the netmask for the cluster IP address [255.255.255.0]: +HA Configuration files modified successfully. +Configuration complete. + + + For each Enea uCPE Manager installation, place the local + interface IP first in the list of IPs. This will optimize database + communication, since the Enea uCPE Manager uses the list of IPs + sequentially, therefore using the internal loopback interface for + communicating with the database. + + + Once the servers are up and running, log into the Primary and go to System and select Cluster + View. The list of Enea uCPE Managers should be displayed, + with one listed as Primary and the rest as Backup. +
+
+ +
+ Upgrading a High Availability Deployment + + Upgrading a High Availabilty deployment is a highly complex, + multi-step process that requires care to ensure both consistency and + high-availability. Some steps need to be done manually. + + + + We start with the assumption that + ucpeManager-1is the "PRIMARY" server. + + + + Shut down database services on one side of the network, for + example: MariaDB-4, MariaDB-5 and MariaDB-6. + + + + Disconnect the network interfaces towards the VPN for machines + MariaDB-4, MariaDB-5 and MariaDB-6. This will prevent any attempts + at failover/synchronization. + + + + Run the upgrade process on ucpeManager-3 + and ucpeManager-4. This will upgrade the service + to the current release. Once the upgrade process completes, shutdown + the Enea uCPE Manager service on both machines. + + + + Disconnect the ucpeManager-2 machine from + the network (which will take MariaDB-2 offline as well). At this + point, only the "PRIMARY" server is running, this is the start of + the interval when we are susceptible to single-server + failure. + + + + Shutdown the MariaDB-2 process and run the Enea uCPE Manager + upgrade process on ucpeManager-2. This will + upgrade the service to the current release. Once the upgrade process + completes, shutdown the uCPE Manager service on the machine. + + + + Reconnect the network interfaces towards the VPN for MariaDB-4 + (ucpeManager-3), MariaDB-5 + (ucpeManager-4) and MariaDB-6 + (ucpeManager-2). Restart database services on + MariaDB-2, MariaDB-4, MariaDB-5 and MariaDB-6. This will allow + database services on all machines to synchronize, any data that has + been modified during the upgrade process will be made + consistent. + + + + Shutdown the "Primary" server + (ucpeManager-1). At this point, the service is no + longer available. + + + + Start the Enea uCPE Manager services on + ucpeManager-2. This machine will come up as the + new "PRIMARY" with the upgraded software. As part of the startup + process, it will upgrade the database and perform any other + upgrade-related functionality. + + + + At this point (once startup completes), service is available. + However, we are still susceptible to single-server failure. + + + + Start the Enea uCPE Manager services on + ucpeManager-3 and + ucpeManager-4. At this point, we are in + highly-available mode. + + + + Upgrade the Enea uCPE Manager on + ucpeManager-1 (the one that has been shut down). + Once that upgrade is complete and the service restarts, the entire + setup has been upgraded to the new version. + + +
+
\ No newline at end of file diff --git a/doc/book-enea-nfv-access-getting-started/doc/images/high_av_setup.png b/doc/book-enea-nfv-access-getting-started/doc/images/high_av_setup.png new file mode 100644 index 0000000..e2edd67 Binary files /dev/null and b/doc/book-enea-nfv-access-getting-started/doc/images/high_av_setup.png differ -- cgit v1.2.3-54-g00ecf