Data Warehouse and Operational Cache
The Evident ClearStone Data Warehouse forms the foundation for Evident Software’s reporting solution, in addition to supporting the use of 3rd party OLAP tools/applications to report on or mine the historical data populated by the Evident ClearStone Pipeline Server. The Data Warehouse is the centralized repository that contains all the data and events produced by the Evident ClearStone system. For instance, it contains a consolidated view of domain/container, services and operational data across a heterogeneous Real-Time Infrastructure (RTI), including off-grid servers and the network data associated with grid nodes/members.
The Data Warehouse is continuously updated with enriched information from the Pipeline Server. It enables RTI operational staff or business users (application owners) to source correlated and aggregated operational performance and usage data enriched with business and/or application context.
Data Warehouse Components
Once the Evident ClearStone Pipeline Server has completed processing the raw data collected by the Evident ClearStone Adapters, the enriched data is loaded into the database using database specific bulk load tools for optimal performance. During this process, database stored procedures (ETL’s) are also executed in order to perform the summarizations and analysis required to get data properly structured for reporting.
The Evident ClearStone Data Warehouse architecture contains several different tables that are maintained as separate database files. The tables include:
- Control Tables – control information that is used by ETL processing including ETL processing history and capturing errors during ETL processing
- Dimension Tables – information related to the domain/container and/or grid entities such as engines, jobs, brokers, named-caches, and system level server resources such as CPU, memory, storage and network I/O etc.
- Fact Tables – information pertaining to events such as which jobs have run, when jobs were submitted, how long it took to run a job, average or time-correlated CPU usage for a server, etc.
- Summary Tables – summarized data required for summary reports (such as long term trends), so that these reports can be generated quickly without further data processing during on-demand report generation
- Staging Table – temporary data that is generated and used during ETL processing
- Archives – data that is infrequently accessed (historical data)
Operational Cache
The near real-time reporting and alerting required for SLA monitoring is implemented within Evident ClearStone using a separate operational data cache. This high-performance data store is typically used for exception reporting or alerting and to access near real-time KPI’s associated with SLA’s for the compute grid, off-grid servers, the data cache, or the network.
The Operational Cache is populated by the Evident ClearStone pipeline and can be configured to provide alerts, email, and alarms when an SLA violation has occurred as measured by exceeding the threshold on a specific KPI.
Benefits
The Evident ClearStone Data Warehouse and Operational Cache have many benefits including:
- Single data store to maintain enriched data – allows different views of the data to generate reports for different audiences (e.g. the business user, the operational staff, and IT architects)
- Stored Procedures functions (ETLs) to summarize and format data for reporting – allows quick report generation (minimal data processing during report run)
- Open access to the data for custom reporting or data mining with 3rd party tools – access to data schema for additional custom reporting and data mining not currently available with existing 'out-of-the-box' report templates
- Standard Relational Database implementation – Data Warehouse implementation uses standard, off-the-shelf database engines, i.e. Oracle Database 10g or Microsoft SQL Server
- High-performance Operational Cache for near real-time KPI monitoring – fast access to data for SLA monitoring
- Aggregated usage and rated data for generation of service accounting reporting (RTI chargeback)
|