Real-Time Console for Oracle Coherence
The Evident ClearStone Live Real-Time Console is a the real-time monitoring user interface for Oracle Coherence. This is a separately licensed module for customers who have licensed ECSL for Oracle Coherence. The console is an Adobe Flex web application accessible using a web browser with the Adobe Flash plug-in. The user interface provides real-time visualizations about:
- Data grid footprint (i.e. # of nodes, # of caches, # of clients, # of objects, total memory available)
- Generated events related to changes in the data grid infrastructure (i.e. nodes join/depart, client connections)
- Data grid resource allocation (i.e. storage nodes, proxy nodes) and client activity.
- Cache performance and activity across the data grid
- Partitioned cache performance and activity per node
- Node level analytics (including TCP Extend nodes)
- Extend-TCP proxy nodes and sessions activity
- Coherence Log Monitoring
The following sections describe the features and functionality for each of the visualizations in the Real-Time Console.
Data Grid Health
The Data Grid Health Monitor visualization represents a macro view of the overall health and status of the monitored cluster. The key performance indicators (KPIs) displayed in this visualization is a summary of the vital statistics of the monitored cluster.
The visualization includes a real-time chart that displays the most recent 360 samples (1 hour with 10 second updates) of real-time metric data. To monitor a specific KPI, click on the KPI item. The chart will be updated with the selected KPI’s metric data. Mouse over any data point in the chart to view the metric data and time. To customize the health monitor with different KPIs, click on the Customize button and select up to 12 KPIs to monitor.
The Event Monitor displays the most recent data grid events. These events are generated by Evident Coherence Adapter to inform users of node, cache, and Extend-TCP connection changes in the data grid. These events do not originate from the log files. As a member of the cluster, the Adapter is aware of such infrastructure changes. Here are a few examples of these synthesized events:
- When a member leaves or joins the cluster, node level events are produced to indicate storage or client node behaviors. Such events are important to inform users when caches become endangered or when the loss of storage capacity may lead to a resource problem.
- As storage nodes are departing or joining the cluster, the grid would re-balance the data across the storage members; this brief maintenance cycle may have an impact on performance.
- Extends-TCP connections and disconnects.
The events are color coded based on the severity level (NORMAL, MINOR, MAJOR, or CRITICAL).
Some of the visualizations (i.e. Cluster Dashboard, Cache Charts, etc.) in the Real-Time Console support graphical annotations of events. These annotations will be directly linked to the events in the Event Monitor.
The system keeps a history of the most recent 2 hours in the OpCache. Besides broadcasting these events to the Real-Time Console, the Pipeline Server will:
- Record all the events in log files on the Evident ClearStone Live server. The log files can be found in the <install_dir>\tomcat\logs directory on the server. The log files have a naming convention of: clusterevents.SCE.<cluster name>.log.
- Send e-mail notifications to users for ERROR and FATAL events. This assumes e-mail notification has been configured on the Evident ClearStone Live server. Please refer to the installation/setup documentation for further details on e-mail notification.
The JVM Dashboard presents a consolidated view of all the Java Virtual Machines running in the cluster. This is an optional visualization that is only available if the Coherence cluster has been properly configured to expose JVM platform MBeans thru clustered JMX. This requires Coherence 3.4.x Enterprise/Grid Edition. Each member in the cluster will require a custom mbean configuration file that is configured to expose the standard Java platform Mbeans. The JVM containers running Coherence storage, management, proxy, or client members can be monitored at the JVM level.
The JVM Dashboard consists of four sections:
- Node Selector
- JVM Grid
- Column Selector
- JVM Properties Table
The Node Selector is the tree view on the left that displays all the Java processes running on each host. Each process is identified with process ID and its parent host/server. The following actions are available with the Node Selector panel:
- The tree can be expanded by clicking on the “+” button or collapsed by clicking on the “-” button.
- Users can click on each host to expand the list of the Java processes on the host. Each process may have an icon representing if the node is a client, proxy, or storage node.
- Users can filter the JVMs listed by process ID or node type (client, proxy, storage). For larger size clusters, this is helpful to limit the number of JVM entries in the tree.
- By selecting on a single host or process, the JVM Grid will be filtered to the selected host or process.
- The "!" button resets the tree filter criteria and refreshes the tree.
The JVM Grid is the main panel that displays the Java processes and performance metrics in a tabular format. Each java process is listed by the hostname, process name (PID), and member type. The KPIs can be selected using the KPI Selector. In this table, the columns can be sorted by clicking on the column header. The columns can also be moved by dragging the column header across to other columns. If a column contains a percentage value, a percentage gauge will display a color code bar for the value.
The Column Selector panel controls the performance metrics to display in the JVM Grid. The following attributes and KPIs are available for each JVM:
- Unicast Adddress - host IP where the JVM is running on
- Process Name - Process ID or override name
- Member ID - Coherence Member ID
- Node Type - type of member
- CPU % - current CPU utilization of the JVM on the host
- Heap Available - current available heap (%)
- Available Processors – the number of processors reported to the Java process thru the OperatingSystemMXBean
- Free Physical Memory – FreePhysicalMemorySize from OperatingSystemMXBean
- Committed Virtual Memory – the amount of committed virtual memory (that is, the amount of virtual memory guaranteed to be available to the running process). CommittedVirtualMemorySize from OperatingSystemMXBean
- Heap Memory Usage Committed – committed value in HeapMemoryUsage from MemoryMXBean
- Initial Heap Memory – init value in HeapMemoryUsage from MemoryMXBean
- Heap Memory Max – max value in HeapMemoryUsage from MemoryMXBean
- Heap Memory Used – used value in HeapMemoryUsage from MemoryMXBean
- Loaded Class Count – LoadedClassCount from ClassLoadingMXBean
- Non Heap Memory Committed – committed value in NonHeapMemoryUsage from MemoryMXMbean
- Initial Non-Heap Memory – init value in NonHeapMemoryUsage from MemoryMXMbean
- Non Heap Memory Max – max value in NonHeapMemoryUsage from MemoryMXMbean
- Non Heap Memory Used – used value in NonHeapMemoryUsage from MemoryMXMbean
- Peak Thread Count – PeakThreadCount from ThreadingMXBean
- Thread Count – ThreadCount from ThreadingMXBean
- Total Physical Memory – TotalPhysicalMemorySize from OperatingSystemMXBean
- Uptime – Age of the JVM
- GC Time – the aggregate GC pause times across all the JVM's garbage collectors for the within the current collection interval
The Properties Table displays the system properties of a JVM or the JVM's hosts. Users can select any JVM in the grid to display that JVM’s Coherence and system properties.
The Cluster Dashboard visualization is a multi-chart view designed to analyze the utilization and performance of the cluster at a macro level. It provides real-time charting of one or more KPIs for a single cluster. For example, user can monitor the allocation of storage nodes and proxy nodes while comparing storage memory utilization. Users can also examine the utilization of memory by the named caches across the cluster by looking at the Cache Distribution charts.
Users can change the layout and select the KPIs to be displayed in the Cluster Dashboard by clicking on the “Customize” button.
In the Customize panel, users can change the grid layout orientation and size. Under the layout selection, the visualizations can be selected by dragging and dropping the icon into a quadrant in the layout. The visualizations include the following:
- Storage, Proxy, and Client Nodes - this trends the number of storage-enabled members, *Extend clients, and non-storage enable nodes ("clients") over time. The chart illustrates the stability of the members over time. If any members joined or departed, there would be dips or rises in the chart.
- Storage Memory -this trends the total storage memory (MB) allocated vs. the storage memory available over time. Storage memory is defined as the total JVM heap of all the storage-enabled members in the cluster. The chart provides information about the memory utilization of the cluster throughout the day. Users can visually see if there were periods of low memory availability due to data consumption or member loss.
- Cache Distribution (Objects) - this is a pie chart displays the distribution by the number of objects for each named cache across the cluster. This informs users which named caches are storing more objects.
- Cache Distribution (Memory) - this is a pie chart displays the distribution of the storage memory utilization for each named cache across the cluster. This informs users how named caches are occupying more memory in the cluster.
- Number of Caches - this trends the total number of named caches managed by the cluster.
- Number of Hosts - this trends the total number of unique hosts where the cluster members reside on.
- Number of Extend-TCP Sessions - this trends the count of concurrent active Extend-TCP sessions connected to all the Extend-TCP members in the cluster. This informs the users the # of concurrent Extend-TCP clients in the cluster.
- Weakness Factor - this trends which nodes have been weak over time. Nodes that determined to be weak by other nodes are depicted with taller bars in the chart.
- Total Request Count - this trends the aggregated client side requests for each Coherence service for the cluster.
- Request Timeouts - this trends the aggregated client-side request timeouts for each Coherence service for the cluster. Request timeouts indicate client requests are failing or stalling. This has a direct impact on Coherence and performance.
- Task Counts - this trends the aggregated tasks submitted for each Coherence service.
- Task Timeout - this trends the aggregated task timeouts for each Coherence service. Task timeouts indicate failures in completing server-side executions. This has a direct impact on performance and throughout. The service may be overloaded due to insufficient # of processing nodes and/or threads.
- Total Task Backlog - this trends the aggregated task backlog for each Coherence service. Task backlogs indicate bottlenecks for a Cohrenece service, which impacts performance and task throughput. There may not be sufficient threads allocated or the timeout threshold may need to be adjusted.
- Total Task Hung Count - this trends the aggregated task hung count for each Coherence service. These tasks exceed a hung threshold, but may not necessarily have failed.
The visualizations in the line charts contain a horizontal scrollbar which allow users to scrollbar to earlier hours in the day to view the historical real-time trend. The range of the time scrollbar is limited to the size of the Evident ClearStone OpCache configured for real-time data (up to 24 hours).
The left side of the Cluster Dashboard contains three panels:
- Options Panel - contains the toggles for the event annotations, chart data points, and the violation line segments
- KPI Panel - this bottom panel displays all the current KPI values for the cluster. This is used as a KPI selection tool to create new KPIs.
In the Options panel, users can enable the following options:
- Show Event Annotations – displays annotations in the line charts to denote when an event occurred
- Show Data Points – display marked data point markers in the charts
Upon exceeding or falling below the thresholds, a threshold violation event occurs and an alert is published (depending on system configuration). Thresholds are configured in the ECSL Administration Console.
- When violations occur the line charts for the violated KPI would be color coded to indicate when and how long did the threshold violation occurred. Minor violations appear as yellow line segments, major violations appear as orange line segments, and major violations appear as red line segments. Users can mouse-over the violating data point(s) to view the values and threshold settings.
- The system can generate an enterprise management notification via an SNMP trap that is sent to an enterprise management system,
- The system can generate an email notification via a SMTP email request to a list of pre-configured email destinations.
Event Correlation and Viewing
The time based line charts in the Cluster Dashboard include "event markers" that time correlate an event with the Key Performance Indicator (KPI) activity being currently displayed in the chart. This allows the correlation of ECSL events to changes in a specific cluster KPIs for troubleshooting or to view "cause-and-effect" relationships across the cluster. The event markers are illustrated in the above figure by the vertical lines with a number indicator on the top of the line. The number indicates an event group containing a set of events occurring in a 10 second interval. The events are grouped to ensure the chart is not over populated with event annotations. To view a specific event:
- Select the event marker of interest within the specific cluster chart
- This action will highlight the associated event with the highest severity in the event group and its description in the Event Viewer panel in the upper right-hand corner of the Real-Time Console
Note: event markers are color coded to show the severity level of the event (i.e. NORMAL, MINOR, MAJOR, and CRITICAL).
The Cache Charts visualization is focused on analyzing the behavior of named caches. The KPIs for the named caches are summarized across all managed storage nodes in the cluster. The Cache Charts provides users with an aggregate view of each named cache’s usage and size. As thresholds (if defined) are violated, the charts will illustrate the violations with the appropriate line colors in the charts. These thresholds are configured in the ECSL Administration Console.
Note: Metric values are not aggregated across both the primary and backup members. While some metrics like the get and put metrics are updated by both client and non-client specific get/put calls (i.e. get/puts from grid balancing).
The visualization is divided into two sections: chart and table (see below). The chart displays the performance trend for a single KPI across multiple named caches selected by the user. In the example below, there are a few named cache configured with a near-cache strategy. As one would expect to see, the front-tier cache is more active than the back tier cache..
Note: On initial startup, the chart is blank. Users will have to select one or more name caches to chart and a KPI to trend.
Below the chart is the tabular view of the named caches. Each row is a named cache within the cluster. The table provides access to the latest metric values for displayed KPIs. The named caches can be grouped together by selecting the grouping field in the options panel.
To chart a KPI across a set of caches, there are two steps.
- Select the checkbox next to the named cache.
- Select the KPI to chart.
To chart a specific KPI, mouse over the desired column. A drop down menu arrow will appear on the right side of the column header. Upon clicking the arrow, a menu will appear with the options to Chart Column, Remove Column, and Insert Column (see above). Click on Chart Column to chart the metric values for the selected KPI. The console will request the historical real-time metric values for the selected named caches and KPI. Only one KPI can be charted at any time for the selected named caches.
The rows can be sorted in ascending or descending order by clicking on the column headers. Columns can be added or removed by activating the drop down menu. To add an additional KPI to the table, select Insert Column to view the additional KPIs. Select from the list and the new column will be added to the table. To remove a column from the table, select Remove Column.
The left side of the Cluster Dashboard contains three panels:
- Options Panel - contains the cache grouping option and toggles for the event annotations, chart data points, and the violation line segments.
- Filter Panel – contains an option to filter caches by name and other fields.
In the Options panel, users can enable the following options:
- Show Event Annotations – displays annotations in the line charts to denote when an event occurred
- Show Data Points – display marked data point markers in the charts
- Allows users to group the named caches by a selected attribute (i.e. Type, Tier, etc.)
In the Filter panel, users can selected filter name cache entries based on the different key attributes of a named cache (i.e. cache name). To filter, perform these steps:
- Select the attribute to filter on.
- Enter the text pattern match on (case-insensitive and partial string match)
- Click on the “Apply” button
- The filtering will be activated and the table will reflect the filtered caches.
- To remove the filter, click on the “Reset” button
Similar to the Cluster Dashboard, the Cache Chart visualization also supports the ability to set thresholds and view cache-oriented threshold violations.
- When threshold violations occur, the Cache Charts will display the violations in the chart and the table.
- The table will display any current KPI(s) in violation in the appropriate color (MINOR, MAJOR, and CRITICAL) base on the severity level.
- For the chart, the user can select one or more caches to chart. If the data series for a cache contains threshold violations, the line segments will be colorized by with the appropriate severity color. The user can hover over any part of the line to view the threshold constraints for that violation. The threshold guidelines will automatically appear in the chart specifically for the selected series.
- The presentation of threshold violation line segments can also be disabled by unselecting "Show threshold violation colors" from the Options tab.
Event Correlation and Viewing
The time based line charts in the Cache Charts include "event markers" that time correlate an event with the Key Performance Indicator (KPI) activity being currently displayed in the chart. This allows the correlation of ECSL events to changes in a specific cluster KPIs for troubleshooting or to view "cause-and-effect" relationships across the cache. The event markers show up as vertical lines with a number indicator on the top of the line. The number indicates an event group containing a set of events occurring in a 10 second interval. The events are grouped to ensure the chart is not over populated with event annotations. To view a specific event:
- Select the event marker of interest within the specific cache chart
- This action will highlight the associated event with the highest severity in the event group and its description in the Event Viewer panel in the upper right-hand corner of the Real-Time Console
Note: event markers are color coded to show the severity level of the event (i.e. NORMAL, MINOR, MAJOR, and CRITICAL).
The Cache Heat Map is an analysis tool used to visualize the cache partitioning structure of the monitored cluster. It presents how the named cache partitions are distributed across the storage nodes in the cluster. The cells in the heat map represent each individual cache partition on a storage node. The cache partitions can be grouped together using any two of the following attributes:
- (None) - No grouping specified
- Cluster – by cluster
- Name – name of the cache
- Host – by physical host where the cache resides
- Persistence Type - this is the type of backing store that is used
- Tier – type of backing store
- Node - this is the JVM instance/process
To view how each cache is partitioned across the servers, choose Host for the first group and Name for the second group. This would display the cache partitions for each named cache organized by physical host.
The cell size represents relative measurement of a specific metric value for each cache based on a selected Size KPI. To configure a different metric for the cell size, click on the Size KPI option and choose an alternate KPI. The default Size KPI is set to ‘size’ (# of objects in each cache partition).
The heat map also features a Color KPI option. This option allows users to colorize the cache partitions. Similar to the Size KPI, there are over a dozen KPIS different to choose from. The default is set to ‘Number of Objects’ (# of objects in each cache partition).
A user can set one or more ranges of colors for the heat map. In addition, the user can set the color gradient for each range. The following example displays the cache partitions grouped by the names of the caches. The cell size is based on the # of objects for each partition. The color is based on the “units” measure for each cache partition. In this example, the Coherence cache configuration is configured with binary units.
To change the color scale for the previous heat map:
- Mouse over the tick marks above the color bar.
- A color palette would be displayed (as seen below).
- Select the desired color.
To update an existing range, click on the numeric value and update it. To add an additional tier, click on the color bar and specific a value within the current range. Below is an example of the same heat map with an additional range with a new color gradient.
Using a combination of the Size and Color KPIs, a user can visualize the information in numerous ways to analyze the partitioning behavior of the named caches.
To view the metric values for any cell, the user can perform either one of the following steps:
- Move the mouse pointer over the cell. A tool tip will display the Cache Name, Member Id, and the current value of the chosen 'Size' metric (see the previous figure).
- Click on a cell to select it. A table will be appear below the heat map displaying the metric values for the selected cache partition.
- Click on a group header. A table will appear below the heat map displaying the metric values for each cache partition in the group. (see next figure)
The heat map includes a filtering feature to filter the cache partitions by various cache attributes such as name, host, cluster, etc. This helps reduce the number of cells displayed in the heat map. Once the filter is applied, the heat map will only display the cache partitions that meet the filter criteria.
The Sessions Dashboard displays real-time statistics for Extend-TCP sessions in the monitored cluster. Users can monitor active sessions in real-time and view a history of disconnected Extend-TCP sessions.
Note: For Coherence 3.3.x users, Extend-TCP statistics is not available via the standard Coherence JMX management framework. To support this, Evident Software provides a custom extension for Extend-TCP nodes to monitor the proxy nodes and sessions. Therefore the Extend-TCP nodes of the monitored cluster will need to be instrumented to support this visualization.
For each Extend-TCP session, the product captures the connection usage information between the client remote address and port and the corresponding Extend-TCP proxy node. This visualization displays the network performance statistics for each session.
For active Extend-TCP sessions, users can view the session’s network activity (byte and message utilization) since the start of the session or as long as the OpCache retention. These sessions can be monitored in real-time to view the network utilization patterns and trends.
In the Options panel, users can enable the following options:
- Show Event Annotations – displays annotations in the line charts to denote when an event occurred
- Show Data Points – display marked data point markers in the charts
- Toggle between Active Sessions and Session History
- Group Sessions by Proxy or Client Host/IP
To view the disconnected sessions, click on the “Session History” radio option. The console will display a table of all disconnected sessions. The table will only provide the final statistics for each disconnected session without any time series data. In either view, the session records can be grouped or filtered by the Client address or Proxy host. This allows users to organize and limit the scope of the session records.
Event Correlation and Viewing
The time based line charts in the Session Dashboard include "event markers" that time correlate an event with the Key Performance Indicator (KPI) activity being currently displayed in the chart. This allows the correlation of ECSL events to changes in a specific cluster KPIs for troubleshooting or to view "cause-and-effect" relationships across sessions. The event markers show up as vertical lines with a number indicator on the top of the line. The number indicates an event group containing a set of events occurring in a 10 second interval. The events are grouped to ensure the chart is not over populated with event annotations. To view a specific event:
- Select the event marker of interest within the specific session chart
- This action will highlight the associated event with the highest severity in the event group and its description in the Event Viewer panel in the upper right-hand corner of the Real-Time Console
Note: event markers are color coded to show the severity level of the event (i.e. NORMAL, MINOR, MAJOR, and CRITICAL).
The Node Dashboard provides detailed monitoring and analysis of the JVMs in the cluster. This includes all storage enabled nodes, Extends-TCP proxy nodes, and Coherence clients. Some of the JVM specific metrics require enabling instrumenting the cluster with Java Platform Mbeans. Please refer to the Coherence documentation for this configuration. This dashboard enables users to select from a suite of visualizations suitable for charting node performance and activity. The following visualizations are available:
- Cache Distribution – a pie chart illustrating the current storage consumption for each named cache of the selected node. The size of each slice is measured based on “units” calculation configured in the Coherence cache configuration file.
- Memory Utilization –a time-series chart of the node’s heap allocation and memory availability
- Packet Volume – a time-series chart of the node’s packet utilization (sent & received)
- Cache Access – a stacked-bar chart of the current put and get volumes for each named cache of the current node
- Cache Access Time – a stacked bar chart of the total time (ms) of time spent accessing data for a named cache
- Extend-TCP Traffic (Extend-TCP nodes only) – a time-series chart network utilization (bytes sent/received) for the selected Extend-TCP node
- Extend-TCP Sessions (Extend-TCP nodes only) – a time-series chart of the concurrent # Extend-TCP session count for the selected Extend-TCP node
- CPU Utilization - a time-series chart of the JVM's CPU utilization for the packet publisher service (based on the Node mbean's statistic attribute)
- Success Rate - a time-series chart of the node's publisher and receiver success rate
- Weakness Factor - a time-series chart of the Coherence node's calculated weakness. The weakness value indicates how many other nodes reported this node as weak over time.
- Node Properties – a property table consisting of the most popular node level attributes.
If the system is configured to with custom Mbeans such as Java Platform Mbeans, the following visualizations are also available:
- Heap Memory Utilization - a time-series chart of the heap memory utilization (%) of the Java process over time
- Threads - a time-series chart of the number of active threads managed by the node over time
- Classes - a time-series chart of the total number of classes loaded by the process over time
- GC Time - a time-series chart representing the total garbage collection time spend during each collection interval
- Used Memory Pool (Line) - a time-series stacked bar chart of the JVM's memory pool usage.
- Used Memory Pool (Pie) - a pie chart of the JVM's current memory pool usage.
Under the Options panel, users will select a node from a list of available active nodes. The nodes are grouped by physical hosts. When a node is selected, the Node Properties panel will be updated with the selected node’s attributes. Only a single node can be selected at any time. The list of nodes includes storage, proxy, and client nodes.
The Node Properties panel is displayed under the node selection panel. This table contains the configured runtime attributes of the selected node.
The dashboard consists of a charting area where users can drag and drop various real-time visualizations into multiple quadrants. The charting area can be customized with a different layout using the “Customize” button. To pick a different visualization, drag and drop it into the desired quadrant. Not all visualizations are applicable for all types of Coherence members/JVMs.
Some visualizations such as Memory Utilization and Packet Volume includes a time slider along the x-axis. Users can use this slider to view previous real-time trends. The Cache Storage and Cache Access visualization may have a vertical scroll bar to scroll thru the chart.
Event Correlation and Viewing
The time based line charts in the Node Dashboard include "event markers" that time correlate an event with the Key Performance Indicator (KPI) activity being currently displayed in the chart. This allows the correlation of ECSL events to changes in a specific cluster KPIs for troubleshooting or to view "cause-and-effect" relationships across nodes. The event markers show up as vertical lines with a number indicator on the top of the line. The number indicates an event group containing a set of events occurring in a 10 second interval. The events are grouped to ensure the chart is not over populated with event annotations. To view a specific event:
- Select the event marker of interest within the specific node chart
- This action will highlight the associated event with the highest severity in the event group and its description in the Event Viewer panel in the upper right-hand corner of the Real-Time Console
Note: event markers are color coded to show the severity level of the event (i.e. NORMAL, MINOR, MAJOR, and CRITICAL).
The Node Heat Map is an analysis tool used to visualize all the cluster members of the monitored cluster. This type of visualization can be helpful in identifying the hot spots in the cluster from a VM perspective. The cells in the heat map represent a cluster member (JVM). The heat map features and functionality are similar to the Cache Heat Map.
The JVMs can be grouped together using any two of the following attributes:
- (None) - No grouping specified
- Hostname
- Type (i.e. storage, client, proxy)
- Process Name
- Rack Name
- Role Name
- Site Name
The cell size represents relative measurement of a specific metric value for each node based on a selected Size KPI. To configure a different metric for the cell size, click on the Size KPI option and choose an alternate KPI. The following KPIs are available for the cluster members:
- Memory Available
- AllocatedMemory
- PacketsReceived
- PacketsRepeated
- PacketsResent
- PacketsSent
- PublisherPacketUtilization
- PublisherSuccessRate
- ReceiverPacketUtilization
- ReceiverSuccessRate
- CPUUtilization
The Node Heat Map's Color KPI options are:
- MemoryAvailable
- AllocatedMemory
- PublisherPacketUtilization
- PublisherSuccessRate
- ReceiverPacketUtilization
- ReceiverSuccessRate
- CPUUtilization
Users can use the other filter and search features to limit the cluster members to analyze. For example, to limit the displayed cells by storage nodes only:
- Access the Filter panel
- Change the filter to "Node Type"
- Type in "STORAGE"
- Click the Apply button
- This will filter all non-storage enabled nodes from the heat map.
The Service Dashboard provides detailed monitoring and analysis of the Coherence services for each cluster member. The members are grouped by all the Coherence services. This dashboard enables users to diagnosis specific services performance on a particular node. The following visualizations are available:
- Request Total Count – a time-series chart of # of synchronous requests issued for the selected service for the selected node
- Request Pending Count – a time-series chart of # of pending synchronous requests issued by the selected service for the selected node
- Request Pending Duration – a time-series chart of # of the oldest pending requests for the selected service for the selected node
- Request Timeout Count – a time-series chart of # of requests that timed out. This indicates server-side failure in processing submitted tasks from the select node for the specific Coherence service.
- Request Failure Ratio – a time-series chart of # of requests that timed out over the total requests submitted for a specific Coherence service and node.
- Task Count – a time-series chart of # of server side tasks executed for the selected service for the selected node
- Task Backlog – a time-series chart of # of tasks in the backlog queue for the selected node for a specific service. A backlog indicates bottleneck on the processing nodes.
- Task Timeout Count –a time-series chart of # of tasks that experience timeout for the selected node for a specific service. Timeouts indicate failures on server-side processing of tasks.
- Task Hung Count – a time-series chart of the current # of hung tasks for the selected node for a specific service.
- Task Failure Ratio – a time-series chart of # of tasks that timed out over the total tasks submitted for a specific Coherence service and node.
- Thread Abandoned Count – a time-series chart of the # of abandoned threads for the selected service's thread pool for the selected node
- Thread Average Active Count – a time-series chart of the # of active threads for the selected service for the selected node
- Cpu Time – a time-series chart of a service's CPU time spent. Only use for comparing against other service to determine which services are consuming processing resources.
- Partitions Endangered – a time-series chart that indicated the # of partitions that were marked as endangered over time.
- Partitions Unbalanced – a time-series chart that indicated the # of partitions that were marked as unbalanced over time.
- Partitions Vulnerable – a time-series chart that indicated the # of partitions that were marked as vulnerable over time.
The Aggregated Service Dashboard provides monitoring and analysis of the Coherence services across the entire cluster. This dashboard enables users to diagnosis each service as a whole across the cluster.
The following visualizations are available:
- Max Request Pending Duration – a time-series chart of # of the oldest pending requests for the selected service across members configured with this service
- Total Request Pending Count – a time-series chart of total # of pending synchronous requests issued by clients for the selected service
- Total Request Timeout Count – a time-series chart of total # of requests timed out for the selected service. These timeouts have direct impact on clients where requests may have failed.
- Total Request Total Count – a time-series chart of total # of synchronous requests issued by clients for the selected service.
- Total Task Backlog – a time-series chart of total task backlog queue for the selected service. A backlog indicates bottleneck on the processing nodes.
- Total Task Count – a time-series chart of total # of tasks executed for the selected service across members configured with this service
- Total Task Hung Count – a time-series chart of the total # of current hung tasks for the selected service across members configured with this service
- Total Thread Count – a time-series chart of the average # of active threads in the cluster for the selected service across members configured with this service
- Total Task Timeout Count –a time-series chart of total # of tasks that experience timeout for the selected service across members configured with this service. Timeouts indicate failures on server-side processing of tasks.
- Request Failure Ratio – a time-series chart of # of requests that timed out over the total requests submitted for the selected Coherence service.
- Task Failure Ratio – a time-series chart of # of tasks that timed out over the total tasks submitted for the selected Coherence service.
- Total Cpu Time – a time-series chart of a service's aggregated CPU time spent (across all members).
|
