Linux cluster monitoring

Introduction

Linux cluster is a group of Linux computers or nodes, storage devices that work together and are managed as a single system. A traditional clustering configuration has two nodes that are connected to shared storage (typically a SAN). With Linux clustering, an application is run on one node, and clustering software is used to monitor its operation.

A Linux cluster provides faster processing speed, larger storage capacity, better data integrity, greater reliability and wider availability of resources.

Failover

Failover is a process. Whenever a primary system, network or a database fails or is abnormally terminated, then a Failover acts as a standby which helps resume these operations.

Failover Cluster

Failover cluster is a set of servers that work together to provide High Availability (HA) or Continuous availability (CA). As mentioned earlier, if one of the servers goes down another node in the cluster can take over its workload with minimal or no downtime. Some failover clusters use physical servers whereas others involve virtual machines (VMs).

CA clusters allow users to access and work on the services and applications without any incidence of timeouts (100% availability), in case of a server failure. HA clusters, on the other hand, may cause a short hiatus in the service, but system recovers automatically with minimum downtime and no data loss.

A cluster is a set of two or more nodes (servers) that transmit data for processing through cables or a dedicated secure network. Even load balancing, storage or concurrent/parallel processing is possible through other clustering technologies.

If you look at the above image, Node 1 and Node 2 have common shared storage. Whenever one node goes down, the other one will pick up from there. These two nodes have one virtual IP that all other clients connect to.

Let us take a look at the two failover clusters, namely High Availability Failover Clusters and Continuous Availability Failover Clusters.

High Availability Failover Clusters

In case of High Availability Failover Clusters, a set of servers share data and resources in the system. All the nodes have access to the shared storage.

High Availability Clusters also include a monitoring connection that servers use to check the “heartbeat” or health of the other servers. At any time, at least one of the nodes in a cluster is active, while at least one is passive.

Continuous Availability Failover Clusters

This system consists of multiple systems that share a single copy of a computer’s operating system. Software commands issued by one system are also executed on the other systems. In case of a failover, the user can check critical data in a transaction.

There are a few Failover Cluster types like Linux Server Failover Cluster (WSFC), VMware Failover Clusters, SQL Server Failover Clusters, and Red Hat Linux Failover Clusters.

Hierarchy of Linux Cluster

Cluster
  -Nodes

Pre-Requisites

1. The OpsRamp Gateway must be installed.
2. Pre-requisites for Pacemaker:
   • Credentials: root / non-root privileges with a member of “haclient” group.
   • Cluster management: Pacemaker
   • Accessibility: All nodes within a cluster should be accessible by a single credential set.
   • For non-root users: Update “~/.bashrc” file with “pcs” command path across all cluster nodes.
     Ex: export PATH=$PATH:/usr/sbin -> as a new line in ~/.bashrc file.
3. Pre-requisites for RGManager(non-pacemaker):

   • Credentials: should provide access to both root and non-root users.

   • Cluster management: RGManager

   • Accessibility: All the nodes within a cluster should be accessible by a single credential set.

   • For non-root users: Update the following commands in “etc/sudoers” file to provide access for non-root users to execute these commands.
     
     “/usr/sbin/cman_tool nodes,/usr/sbin/cman_tool status,/usr/sbin/clustat -l,/sbin/service cman status,/sbin/service rgmanager status,/sbin/service corosync status,/usr/sbin/dmidecode -s system-uuid,/bin/cat /sys/class/dmi/id/product_serial”
     
     Note: Usually a linux cluster will be configured with a virtual-ip normally called as cluster-virtual-ip.We use this Ip for adding configurations during the installation of integration.

   • If the cluster-virtual-ip is not configured give the ip address of the reachable node associated with the cluster.

Configure and install the integration

1. Go to Setup > Integrations and Apps.
2. Click + ADD on the INSTALLED APP page. The ADD APP page displays all the available applications along with the newly created application with the version.
   Notes:
   • If there are already installed applications, it will redirect to the INSTALLED APPS page, where all the installed applications are displayed.
   • If there are no installed applications, it will navigate to the ADD APP page.
   • You can even search for the application using the search option available. Also you can use the All Categories option to search.

3. Click ADD in the Linux Failover Cluster application.
4. In the Configurations page, click + ADD. The Add Configuration page appears.
5. Enter the below mentioned BASIC INFORMATION:

Note: Select App Failure Notifications to be notified in case of an application failure that is, Connectivity Exception, Authentication Exception.

6. In the RESOURCE TYPE section, select:
   • ALL: All the existing and future resources will be discovered.
   • SELECT: You can select one or multiple resources to be discovered.
7. In the DISCOVERY SCHEDULE section, select Recurrence Pattern to add one of the following patterns:
   • Minutes
   • Hourly
   • Daily
   • Weekly
   • Monthly
8. Click ADD.

9. Now the configuration is saved and displayed on the configurations page after you save it.
   Note: From the same page, you may Edit and Remove the created configuration.
10. Click NEXT
11. In the Installation page, select an existing registered gateway profile, and click FINISH.

The application is now installed and displayed on the Installed Integrations page. Use the search field to find the installed application.

Modify the Configuration

• See Modify an Installed Integration or Application article.

  Note: Select the Linux Failover Cluster application.

View the Linux Failover Cluster details

To view the resource information, go to Infrastructure > Resources > Cluster and click on your created cluster name. You can navigate to the Attributes tab to view the discovery details, and Metrics tab to view the metric details for Linux Failover Cluster.

View resource attributes

The discovered resource(s) are displayed under Attributes. In this page you will get the basic information about the resources such as: Resource Type, Native Resource Type, Resource Name, IP Address etc.

View resource metrics

To confirm Linux Cluster monitoring, review the following:

• Metric graphs: A graph is plotted for each metric that is enabled in the configuration.
• Alerts: Alerts are generated for metrics that are configured as defined for integration.

Supported Metrics

Resource Type: Cluster

Pacemaker

Resource Type: Server

Risks, Limitations & Assumptions

• Application can handle Critical/Recovery failure notifications for below two cases when user enables App Failure Notifications in configuration
  • Connectivity Exception
  • Authentication Exception
• Application will not send any duplicate/repeat failure alert notification until the already existing critical alert is recovered.
• Metrics can be used to monitor Linux-Failover-Cluster resources and can generate alerts based on the threshold values.
• No support of showing activity log and applied time.
• No support for the option to get Latest snapshot metric.