Real-Time Monitoring and Analysis for Coherence-based Data Grids

Evident ClearStone Live leverages the same innovative product architecture as the Evident ClearStone historical reporting solution for the compute/data grid environments.  Through the addition an operational cache (OpCache) and custom MBeans for monitoring cluster sessions, the platform now supports real-time monitoring and analysis for Coherence-based data grid environments. Evident ClearStone Live overcomes the inherent limitations of database dependant management solutions, which reach their limits when real-time, cross domain, or large scale / multiple cluster monitoring is required.

In order to manage high performance data grid solutions for web and other high-performance computing (HPC) applications and, management tools that support real-time visualization and intelligent analytics are essential. These monitoring tools help the grid operational team see what is currently happening in the cluster so that problems can be quickly identified and pro-active steps taken to insure that the data grid runs efficiently even under dynamic or unpredictable application loads.

While distributed cache technology provides clear benefits, it also creates new challenges that cannot be met with existing system management tools:

• Cache resources are known but how well the cache is servicing the application demand, at any point in time, is not well known. This leads to decreased application performance during peak demand due to poor cluster optimization.
• Correlating and understanding cache performance and cluster level events requires extensive manual analysis of raw management data (which is impractical, or may be unavailable, for solving issues in real-time).
• Utilization and performance is not tracked in real-time, so the impact of demand changes on cache resource usage cannot be determined.
• New dynamic provisioning capabilities require the ability to validate configuration and performance in real time.

Data Grid HeatMap	Cluster Performance	Data Grid Nodes
click on thumbnail to enlarge

Features & Benefits

Feature/Function		Benefit
Advanced Architecture Using Web Technologies Based on the same technologies as the Evident ClearStone solution including integration within a Web Service Container, J2EE Application Frameworks, Enterprise Service Bus (ESB), and Event Processing Engine.		Leverage leading edge, open-source web technologies like MULE, Hibernate, and Spring
Scalable and Reliable Data Collection Support for an Oracle Coherence adapter that leverages the existing Coherence MBean Server to collect a variety of raw data grid performance metrics from nodes throughout the entire cluster.  The system includes a set of custom MBeans that extend the monitoring capabilities beyond what is currently available in Coherence v3.3 (e.g. metrics for TCP-Extend proxy nodes).		Leverages the existing Coherence management framework (less intrusive), expands the type of data that can be collected and processed (get more information), and minimizes the impact of monitoring the target cluster (low-cost monitoring).
Evident ClearStone Pipeline Processing Componentized, reusable and XML configurable data processing objects that aggregate, correlate and analyze the raw cluster data collected by the Coherence Adapter. Processing is structured as one or more data processing “pipelines”.		Reconfigurable processing logic that allows the solution to be structured to meet varying application requirements and end-user needs.
Real-Time Operational Cache (OpCache) Implemented using a high-performance embedded data grid (Oracle Coherence) to store collected and processed data from the monitored cluster. The data collection can be configured down to a 10 sec update interval and the processed data can be stored for up to 24 hrs of data grid performance statistics. The OpCache can be scaled across multiple servers for monitoring larger Coherence clusters by adding additional nodes depending the update interval and data retention requirements.		The system uses a data grid to achieve the application performance required for real-time data collection and processing from the monitored cluster. Scales to large monitored cluster(s) while maintaining the same real-time application performance (not be possible with traditional relational database technology).
Data Services Layer Makes the enriched data in the OpCache accessible to clients using the system via a standard web browser.		Provides backend services that are essential for delivering a Rich Internet Application (RIA) experience using a modern web browser (no client/server technology required) via the Real Time Console.
Real Time Console (RIA) Support a rich set of Adobe Flex-based dashboards and visualizations that provide real-time information pertaining to all aspects of the cluster including: Data Grid Health Dashboard Event Viewer Cluster Performance Charts Cache Performance Charts Data Grid HeatMap Data Grid Session Dashboard Data Grid Node Dashboard		A highly advanced, web-based user interface that can be run from most web browsers. Access all aspects of cluster, storage node, and proxy node performance information, data visualizations and events from a single pain-of-glass across a single cluster or multiple clusters.

top