The Cassandra Management Pack enables Evident ClearStone Live to monitor Cassandra clusters. The features included in this management pack are:

• Real-time monitoring for multiple Cassandra clusters (at 10-60 second intervals)
• Continuous monitoring and metric collection for Cassandra clusters
• Customizable Real-Time Dashboard for analyzing Cassandra performance data
• Configurable monitoring policies for Cassandra

With these monitoring and management features, organizations can use ECSL throughout the software development lifecycle thru production support. Common use cases for the product include:

• Pre-production (dev/test/UAT)
  • Capacity planning
  • Optimization & cluster resource tuning
  • Load/stress testing
  • Continuous integration testing
• Production monitoring
  • Continuous health monitoring with fine grain performance metrics.
  • Real-time notification via SMTP, SNMP, or custom
  • Set performance thresholds to monitor health, usage, and capacity
• Cluster-wide analysis
  • Monitor aggregate usage and capacity of the cluster
  • Monitor JVM performance
• JVM/Node monitoring
  • Trend heap utilization and garbage collections
  • View JVMs with high CPU, low heap, extensive GC activity

The Cassandra Management Pack supports the following statistics and metrics:

• Thread pool statistics
• Column Family Store statistics
• Commit Log statistics
• Storage Proxy statistics
• Storage Service statistics
• Streaming Service statistics
• JVM Statistics (Platform and Memory Pool)

ECSL performs parallel collections from all the Cassandra nodes in the cluster. The data from these nodes can be consolidate and aggregated into a logical cluster as define in the collection configuration. This provides an aggregated view of the Cassandra clusters. Users can utilize the pre-build perspectives in the Real-Time Dashboard to monitor the activity and performance characteristics of the Cassandra cluster. In addition, administrators can configure monitoring policies to alert operators when there are performance issues with the Cassandra JVMs, thread pools, column family stores, etc.

The current management pack produces the following metrics for Cassandra clusters. The raw statistics are collected from each Cassandra node; more statistics are aggregated/derived to produce more meaningful information for the cluster. Please refer to the Cassandra wiki for details on the metrics.

As newer releases of Cassandra are available, the Cassandra Mbeans are subject to change. Therefore, please consult Evident Software to ensure the current Casandra Management Pack can support different versions of Cassandra. Beyond the Cassandra statistics, the Cassandra management pack includes support for the JVM Platform and Memory Pool statistics.

The remainder of this document covers the following topics for the Cassandra Management Pack.

• Deployment of the Cassandra Management Packs
• Collection Configuration
• Real-Time Dashboard Perspectives

Cassandra Management Pack Deployment

A single ECSL instance can support concurrent monitoring of multiple Cassandra clusters. If the Cassandra clusters are deployed in different data centers, then Evident recommends deploying separate instances of ECSL or at minimal deploy ECSL Collectors at the remote data centers. Given these options for deployment, Evident has provided two Cassandra Management Packs to accommodate a local deployment and a remote deployment.

The two management packs for monitoring are bundles as:

• MgmtPack_Cassandra.zip - deployed locally on top of an existing ECSL instance that has been configured with the Collector, Pipeline Server, and User Interfaces. This distribution includes all the collection, pipeline plugins, and user interface artifacts for the main ECSL server.
• MgmtPack_Cassandra_Standalone.zip - deployed on a standalone remote ECSL Collector-only instance. This package only contains artifacts for collection only.

Management Pack Installation

1. Before deploying either version of the Cassandra Management Pack, stop the current ECSL system.
2. If there is only a single server where ECSL is installed, proceed to extract the contents of the MgmtPack_Cassandra.zip on top of the ECSL installation directory. This deploys all the necessary artifacts to support Cassandra.
3. If the ECSL environment is set up with a remote ECSL Collector, then proceed to extract the contents of the MgmtPack_Cassandra_Standalone.zip on top of the ECSL installation directory of the remote ECSL Collector . This deploys all the necessary artifacts to support Cassandra.
4. After deployment is completed, restart ECSL.

Collection Configuration

Prerequisites

• These management packs support Cassandra release 0.6.4 or newer.
• The Cassandra cluster is running Java 1.6 update 21 or newer.

Setup Remote JMX Monitoring

A single ECSL JMX Collector can support concurrent monitoring of multiple Cassandra clusters. If the Cassandra clusters are geographically distributed, it is recommended to deploy ECSL JMX Collectors at each local site to perform the data collection locally at each site. The Evident Collector can be configured to collect the Cassandra Mbeans via remote JMX from all the Cassandra nodes. For each Cassandra cluster, the administrator is required to have the hostname/IP address, JMX port, JMX credentials for each Cassandra node.

1. Launch the Admin Console application URL: http://localhost:8080/evidentcsadmin
2. Click on Generic JMX Collection Configuration.
   
   
3. Click on New.
4. In Step 1 of the dialog, select the Cassandra family.
5. Enable the Collection Types that ECS needs to collect for. By default, select all of them.
6. Provide a user friendly cluster name of for the ClusterName tag (i.e. Twissandra). This will name of the logical cluster that will be applied to all the JVMs in the cluster.
7. Provide a default node name for the JVMs in the cluster.
8. In Step 2 of the dialog, click on Add to populate the JMX connections for each individual Cassandra node.
9. A dialog displays a form for specifying the JMX connection information for each node.
   
   
10. Enter the Host (hostname/IP address) and JMX Port #. Select if JMX authentication is enable and supply the credentials.
11. Leave the security tag as is. The data from this specific JVM will use the security tag from this collection group.
12. Override the NodeName tag for each Cassandra node. Specify a unique node name for each Node.
13. Save the JMX connection configuration.
14. Continue to add other nodes running in the cluster.
15. In Step 3 of the dialog, specify a name for the Collection Group (i.e. Production)
16. To secure the Mbean data collected from all these JVMs in the group, select the desired security tag. Security tags are defined in the User Administration panel.
17. Select the collection interval.
18. Update the thread pool for this particular collection group. The recommendation is to limit to the pool to 48 threads for each ECS Collector.
19. Save and Commit this configuration.
20. After this configuration is saved, restart ECSL.

Cassandra Perspectives (Real-Time Dashboard)

Users can monitor and visualize the performance data collected from the Cassandra nodes with the Evident ClearStone Live Real-Time Dashboard. This web-based user interface is an Adobe Flex application that provides a number of customizable interactive visualizations such as charts (line, bar, pie, plot ), tree map, and tables.

The Cassandra Management Pack comes with a couple pre-configured perspectives to monitor thread pools and Column Family Stores. Here are a few examples:

Tree Map visualization of the Column Family Stores

Multi-charts displaying Cassandra thread pools

Beyond these examples, users can configure additional perspectives with the following data sets:

• Thread pool statistics
• Column Family Store statistics
• Aggregated Column Family Store statistics
• Aggregated Key Space statistics
• Commit Log statistics
• Storage Proxy statistics
• Storage Service statistics
• Streaming Service statistics
• JVM Statistics