NVIDIA Bright Cluster Manager

Introduction

NVIDIA Bright Cluster Manager offers fast deployment, and end-to-end management for heterogeneous high-performance computing (HPC) and AI server clusters at the edge in the data center, and in multi/hybrid-cloud environments. It automates provisioning and administration for clusters ranging in size from a couple of nodes to hundreds of thousands. It also supports CPU-based and NVIDIA GPU-accelerated systems, and enables orchestration with Kubernetes.

Monitoring Use cases

• In case of any threshold breach or unexpected metric behavior, based on configurations the device monitoring helps to collect the metric values with respect to time and sends alerts to the intended customer team to act up immediately.
• It helps the customer with smooth functioning of business with minimal or zero downtime in case of any infrastructure related issues occurring.

Prerequisites

The OpsRamp Gateway must be installed.

Configure and install the integration

1. Go to Setup > Integrations and Apps.

2. Click + ADD on the INSTALLED APP page. The ADD APP page displays all the available applications along with the newly created application with the version.
   Notes:

   • If there are already installed applications, it will redirect to the INSTALLED APPS page, where all the installed applications are displayed.
   • If there are no installed applications, it will navigate to the ADD APP page.
   • You can even search for the application using the search option available. Also you can use the All Categories option to search.

     

3. Click ADD in the Nvidia Bright Cluster Manager application.

4. In the Configurations page, click + ADD. The Add Configuration page appears.

5. Enter the below mentioned BASIC INFORMATION:

Note: Select App Failure Notifications; if turned on, you will be notified in case of an application failure that is, Connectivity Exception, Authentication Exception.

6. In the RESOURCE TYPE section, select:
   • ALL: All the existing and future resources will be discovered.
   • SELECT: You can select one or multiple resources to be discovered.
7. In the DISCOVERY SCHEDULE section, select Recurrence Pattern to add one of the following patterns:
   • Minutes
   • Hourly
   • Daily
   • Weekly
   • Monthly
8. Click ADD.

9. Now the configuration is saved and displayed on the configurations page after you save it.
   Note: From the same page, you may Edit and Remove the created configuration.
10. Click NEXT
11. In the Installation page, select an existing registered gateway profile, and click FINISH.

The application is now installed and displayed on the Installed Integration page. Use the search field to find the installed application.

Modify the Configuration

• See Modify an Installed Integration or Application article.

  Note: Select the Nvidia Bright Cluster Manager application.

View the NVIDIA Bright Cluster Manager details

The NVIDIA Bright Cluster Manager integration is displayed in the Infrastructure > Resources > Cluster. You can navigate to the Attributes tab to view the discovery details, and the Metrics tab to view the metric details for NVIDIA Bright Cluster Manager.

Supported metrics

Risks, Limitations & Assumptions

• Application can handle Critical/Recovery failure notifications for below two cases when user enables App Failure Notifications in configuration:
  • Connectivity Exception (ConnectTimeoutException, HttpHostConnectException, UnknownHostException).
  • Authentication Exception (UnauthorizedException).
• Application will not send any duplicate/repeat failure alert notification until the already existed critical alert is recovered.
• Application cannot control monitoring pause/resume actions based on above alerts.
• Metrics can be used to monitor NVIDIA Bright Cluster Manager resources and can generate alerts based on the threshold values.
• Macro replacement limitation (i.e, customization for threshold breach alert subject, description).
• Component level thresholds can be configured on each resource level.
• No support of showing activity log and applied time.
• No support for the option to get the latest snapshot metric.
• Application is not returning any data if any SSH connectivity issues (based on monitoring and discovery frequency our App will try to establish SSH connections to the target device).
• Application will work only with SSH credentials, with ssh port 22 in the open state.
• Application will not work for low monitoring frequencies. We recommend providing a minimum monitoring frequency of 15 minutes.
• Virtual Nodes discovery and Monitoring will not be supported.