Articles tagged cassandra performance

The views expressed in this blog are strictly personal, and do not necessarily represent the views of Evident Software.

Inside Evident ClearStone 5.0 – Cassandra, Neo4j and ODI

Date: March 28, 2011
Tags: ,

By Bill Nigh

ClearStone 5.0 is now in GA, with compelling new functionality and a long, clear growth path due to good architecture choices. The main architectural change, with significant implications: ClearStone 5.0 was re-architected with NoSQL DBs Cassandra and Neo4J.

There were several challenges we faced in re-architecting ClearStone, and we’re proud to have handled them in stride.

  1. We were accustomed to using an RDBMS for storage of application and server performance metrics. In an RDBMS, triggers are a bulletproof way of sensing change to data and having the option to take some action, such as sending notifications. While other NoSQL technologies offer an equivalent to triggers, Cassandra does not, so we had to come up with a way to emulate this functionality. (We and others have requested this feature from the Apache Cassandra project, so it may show up in the future). We changed ClearStone’s data model to accommodate this new reality.
  2. Neo4j presented a fascinating challenge. We store and model data as a series of connected nodes; Neo4j, with its very low impedance with the way we designed, made it look like it would be fairly easy. What is challenging is the wide open nature of Neo4j qua implementation of a graph db; you can define nodes and edges (relationships) anyway you want; they can have or not have properties, and can have different types of directionality. So the challenge was to figure out how we wanted to represent all the data without fully knowing in advance how we wanted to traverse all that and get correlations; that presented a challenge from the engineering standpoint of the model.
  3. Another design challenge: we are consuming arbitrary time series data structures. ODI (Open Data Interface), our RESTful API for instrumenting most any IT resource, opens up the product to accept any data from any IT resource; because of this open API, we can’t know in advance what data model will characterize data being instrumented, so we needed maximum possible flexibility. An RDBMS would not only require a formal schema but an attendant requirement to normalize everything, or at least start with that.  Cassandra combines that needed flexibility with high accessibility; Cassandra’s column family implementation is “very forgiving”, as Ivan Ho, Evident’s CTO has said; columns can be added on the fly. (Another feature of Cassandra was especially persuasive: in prior versions of ClearStone, due to its use of RDBMS, there was a discontinuity of the data; a different application was used for history. Cassandra solved that problem, allowing display of current and historical data.)
  4. Cassandra has no query language; all access is through its API; (Lucene is used to query Neo4j, in contrast, one of several methods of retrieval). Using a globally unique key assigned during its inception in Neo4j, we can retrieve needed Cassandra metric, event and entity information.

Future versions will benefit from the flexibility we architected into the product with these NoSQL DBs, as will our customers; expect to see additional Management Packs over the next few weeks; when we ‘sprint’, we take it literally :)

View Comments

Cassandra performance – impact of embedding Cassandra within Tomcat

Date: March 24, 2011
Tags:

By Bill Nigh

Recently, I posted a tip from one of our veterans about embedding Cassandra within Tomcat for testing, including a suggestion to not do so in production, mainly for performance reasons.

This post generated a comment from a reader named Morten:

“I have actually been considering using Cassandra embedded with Tomcat 7 to run a full stack in the same process, so I am real interested in knowing more about the reason for the strong discouragement of not doing this? Are there any tests, profiling results, or similar behind this?”

I passed the question to Ching-Cheng Chen, and here is the information he furnished:

It probably highly depends on what your application is doing. If your application is very light and uses very few resources, then you might be able to live with that. However, if your application is also performing some heavy logic especially the type of logic that causes lots of heap usage, you really want to avoid embedded Cassandra in your application.

We don’t have any profiling to specifically prove this; however, we have been seeing way different GC behaviors. Our application logic creates many “temporary” objects, with a life cycle long enough to survive a few minor GCs.

If you check Cassandra’s default GC setting, it sets MaxTenuringThreshold as 1. The idea behind that setting is probably that everything that survives one minor GC most likely is meant for cache, so promote them into old gen ASAP. (I pressed Ching-Cheng for clarification of ‘old gen’, and he responded: “It’s a Java GC term; heap memory for Java is kind of divided into multiple ‘regions’: old gen/tenured, eden space, young gen, survivor space, etc. etc. Newly created data is generally put in young gen first and that data is candidate for minor GC. Long lived data will promote to old gen/tenured eventually and no longer will be processed during minor GC but will be checked during major GC. http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html“)

Anyway, this is how you can see how the GC setting for Cassandra and the application has a conflict. He observed that if we set MaxTenuringThreshold to 1 (as Cassandra’s default setting), the old gen fills up too fast and we constantly have to stop-the-world GC for a few seconds.

If we set the MaxTenuingThreshold higher, then we see way more data being copied to survivor space and more time spent on minor GC.

We then tried to decouple Cassandra with 2 JVMs; the JVM running Cassandra used the default Cassandra GC setting and the application JVM didn’t set the MaxTenuringThreshold (I think the default is 31 or 32?). Neither JVM had any stop-the-world GC anymore.

The bottom line is that if you are satisfied with the performance after you embed Cassandra, then go for it, but most likely you will get better performance by decoupling Cassandra from your application.

View Comments

Memcached Cassandra monitoring – is there a convenient way?

Date: March 3, 2011
Tags: , , ,

By Bill Nigh

Why would an organization run both Memcached and Cassandra, and how could one monitor both Memcached stats and Cassandra performance in a convenient way?

There are a number of reasons for running both technologies. Maybe a recent merger has brought both technologies in house. Maybe Cassandra is being used for time series data, such as we do with Evident ClearStone, our NoSQL DB monitoring tool, and Memcached is the front serving cache. Cassandra is pretty fast as it is, so the combination is unlikely, but if such a combination were found, then the monitoring solutions would typically be point solutions. Adding the separate consoles for these products to monitoring applications for JVM such as JConsole, WebLogic, JBoss and others, and the common problem of the NoSQL space hits you in the face: a lack of integration within a single dashboard.

Evident ClearStone’s Real Time Dashboard can display JMX-based, Linux SAR, php, perl, ruby and other scripted feeds on the same page. With this functionality, you can at least begin to glimpse the outlines of a more integrated approach to monitoring application performance.

Learn more about our performance monitoring solution for Java, NoSQL and web servers.

View Comments

Evident ClearStone v5.0 – A new product for the NoSQL DB era

Date: February 28, 2011
Tags: , , , , ,

By Scott Barnett

We are about to release our newest version of ClearStone.

We have learned a lot over the past few years, serving customers in the Oracle Coherence and Data Synapse markets, as well as over the past 6 months as we turned our focus to NoSQL DBs – see our product section on NoSQL DB Logging and Reporting.  ClearStone 5.0 introduces the concept of Application Performance Management for NoSQL and MORE.  What do we mean by NoSQL and MORE?

First, we firmly believe that NoSQL (or whatever it is ultimately called) will be the next major tier of the application stack. As such, it will need all of the tools, from development and deployment to, yes, management. There are some basic monitoring capabilities available today for NoSQL DBs, but they are hardly enough.  They lack the detail, scope and actionable capabilities that most DevOps folks are used to.  At the same time, monitoring NoSQL by itself is not satisfactory – in order to get a holistic view of any web, cloud or enterprise application, developers and operations will need to see what’s going on at all levels of the application – from the system level to in-memory cache and everything in between.

So, ClearStone 5.0 was developed to move us even more firmly in the direction of holistic application management with a direct focus on NoSQL. It’s a release intended to address the reality that many applications using NoSQL are leveraging cloud assets and need a tool that can handle dynamic environments combining traditional Java stacks, NoSQL data caching, and virtual cloud environments.  Under the hood, we’ve replaced our internal caching mechanism in 4.x with a combination of Cassandra and a graph database that allow us to store more information, and provide better correlation/relationship mapping of assets between tiers in the application.   We kept (and continue to improve) our really slick user interface. We’re working to provide additional capabilities for people to get custom reports and specific views into their data.

The biggest change in 5.0 is the introduction of an API (we’re calling it the RESTful Data Interface) that lets you instrument metrics, KPIs, and SLAs for any IT asset using a scripting-friendly, XML-over-HTTP interface. We now support built-in collection of application-level metrics from platforms such as JBoss, WebLogic, and Tomcat and will introduce built-in collection of system-level metrics, such as UNIX SAR (system activity recorder). The new back end enables correlation of all collected metrics and events, including custom metrics/events, and real-time visibility into historical performance data without requiring a relational database.

We’ve been embracing the increasingly popular DevOps model, so that both developers and operations can take advantage of the platform. Specific new features for developers (especially during testing) include the ability to:

  • Define SLAs and conditions for alerting
  • Analyze performance, capacity, and utilization across multiple applications/systems
  • Tag resources into logical groups for easy reporting
  • View both real-time and historical performance data from a single interface

ClearStone 5.0 dramatically extends what we can manage, and provides much more flexibility and openness via the API. The architecture changes also allow us to become “cloud-friendly” and has prompted a dramatic pricing change where we will charge via a subscription based on the number of “resources” you are managing, not by server/core counts. By the way, the first 10 resources will be free! Beyond that, pricing will start at $10/resource/month, so you can start small and work your way up.  Now, not only the product is elastic, our pricing is too.  For those larger companies that only want to give us large sums of money, we still support ELA and all-you-can-eat prices for environments greater than 1,000 managed resources.

We will be beta testing 5.0 starting February 14, and plan to GA in mid-March. To sign up for the beta, go to http://www.evidentsoftware.com/clearstone-5-0-beta-program and fill out the form.  We’ll be taking the first 50 users in the beta program, and I hope you’ll be one of them!  I look forward to your feedback on our new software and our new direction!

Related Articles:

Enhanced by Zemanta

View Comments

Embedding Cassandra within Tomcat for testing

Date: February 21, 2011
Tags: ,

By Bill Nigh

Evident ClearStone 5.0 uses Cassandra for its persistent storage of application performance monitoring data, as written about elsewhere in our blog. There are some cases where you might want to embed Cassandra within your Java application or run it as a webapp within your application server for testing purposes. Ching-Cheng Chen, longtime stalwart of the company, sat with me recently to help me with this blog post, furnishing all the detail. Let me start by sharing his strong suggestion that embedding Cassandra within Tomcat is not a good idea for production.

Here are some notes about embedding Cassandra within Tomcat:

Take a look at the Cassandra (for 0.7.0 release) start up script; it will give you clues on how to start up/shut down Cassandra within your own Java class.

To start up a Cassandra node, instantiate an org.apache.cassandra.thrift.CassandraDaemon object and invoke its activate() method in a separate thread.

To shut down the Cassandra node, invoke the deactivate() method on the CassandraDaemon object you created during start up.

The default Cassandra thread pool worked fine for us in our testing mode.   Everyone should modify their Cassandra thread pool configuration according to their environment, though.

But what about the configuration? Cassandra will load the configuration file cassandra.yaml if found in the classpath by default. If there is a need to use different configuration base on environment, you can always generate a configuration file on the fly. To generate Cassandra configuration file dynamically, instantiate an org.apache.cassandra.config.Config object and populate it with your preferred configuration, then write the configuration using the snakeyaml API included in Cassandra distribution.

import org.apache.cassandra.config.Config;
import org.apache.cassandra.utils.SkipNullRepresenter;
import org.yaml.snakeyaml.Dumper;
import org.yaml.snakeyaml.DumperOptions;
import org.yaml.snakeyaml.Yaml;
import org.yaml.snakeyaml.nodes.Tag;
…
// create configuation object
Config config = new Config();
config.cluster_name = “ClusterName”;
config.rpc_port = 9160;
…
// write configuration file
FileWriter fw = new FileWriter(new File(classPathDir+”/cassandra.yaml”));

DumperOptions options =DumperOptions();
options.setDefaultFlowStyle(DumperOptions.FlowStyle.BLOCK);
SkipNullRepresenter representer = newSkipNullRepresenter();
representer.addClassTag(Config.class, Tag.MAP);
Dumper dumper = new Dumper(representer, options);
Yaml yaml = new Yaml(dumper);
yaml.dump(config, fw);
fw.close();

I close with this reminder: Do NOT use this for production; your application(s) will compete for resources with Cassandra; that is certainly not a good thing. For the simpler testing environment, though, it can be useful.

Read more about our solution for Cassandra NoSQL Performance Monitoring

View Comments

Evident ClearStone 5 implements Cassandra and Neo4j as an Elastic Data Store

Date: February 9, 2011
Tags: , , , , ,

By Ivan Ho

As an Application Performance Management tool, Evident ClearStone collects and receives real-time data from disparate applications and systems in a distributed application environment. 90% of the data that ClearStone manages is the performance data from the application processes and systems. The other 10% are related to the inventory and events of the managed environment.

ClearStone 5 required an elastic data store that can handle all the scalability, performance, persistence, and flexibility requirements demanded of this class of product. The demands placed on this elastic data store ruled out the use of a traditional RDBMS backend.   The solution must have options for load balancing and high availability – being able to distribute and replicate data was essential. It was obvious that a NoSQL DB would come into play.

For example, in a single application environment with multiple servers, ClearStone would monitor system level metrics (CPU, disk, network, I/O) , application container performance, Java platform measurements, a distributed Cassandra cluster or other NoSQL clusters, and a database like PostgresDB.  The schemas of the performance data vary significantly from one type of resource to another, therefore we also needed a solution that would provide  flexibility in dynamic schema creation and updates while keeping the system running 24×7. At time when this was being developed, there was no one data caching, database system, or NoSQL solution that met all of these requirements. Ultimately, the Evident architects decided on using two NoSQL solutions, Apache Cassandra and Neo4j.

Cassandra is implemented as a time-series data store for storing all the real-time data and historical data. Our implementation uses Apache Cassandra 0.7 with the Hector client APIs for Cassandra. With Cassandra 0.7, we can dynamically create and evolve column families for storing all the performance data. The performance data is normalized by metric. We have also partitioned our column families based on the granularity of the data sets.

Neo4j is implemented as an inventory database used for maintaining all the managed resources (i.e. processes, hosts, clusters, etc.) of the application environment. It is used to store current state of all the resources, relationships among the resources, and correlated events to these resources. Anytime there are events associated with a resource, we keep a timeline of such events married to a snapshot of the associated resource(s) in the inventory at the time of the event occurrence. We felt the use of a graph database like Neo4j was ideal for storing metadata for the resources and mapping relationships and correlated events.

Beyond the performance and scalability benefits, we found that there were also some fringe benefits to each of these products. Both of these NoSQL systems have the options of being embedded into ClearStone or run as isolated clusters managed by ECS across multiple servers. Lastly both of these two products are cloud-friendly, therefore enabling ClearStone to be deployed in both traditional enterprise environment and cloud environments.

If you would like to learn more about our experience with either one of these NoSQL solutions, please come visit our Support Section and post your questions.

If you would like to try ClearStone, please join our Beta Program here.

View Comments

Apache Cassandra Notes

Date: February 8, 2011
Tags:

By Bill Nigh

Apache Cassandra 0.7 was announced recently, and downloads of the powerful and well-vetted distributed data maintenance solutions proceed apace. (Hmm, Apache, apace…synchronicity, or neighbo[u]ring memory neurons connecting through semantic patterns?)

One of the stars in the NoSQL firmament, Apache Cassandra is busily working behind the scenes worldwide to promote scalability and uptime for big names in the business and social media world.

We at Evident Software offer Application Performance Management tools for Cassandra and other NoSQL technologies. Our interest in this Apache project is two-fold however; we are also incorporating Cassandra 0.7 into Evident ClearStone v5.0, which is going to beta on Feb 14th. Won’t you be our beta tester Valentine?

View Comments

Riptano training for Apache Cassandra

Date: November 6, 2010
Tags: ,

By Ivan Ho

Riptano Several team members from Evident Software took a road trip from NJ to attended Riptano’s training event for Apache Cassandra in Reston, VA. The classroom training covered Cassandra from ground up and focused on the latest 0.7 release. This session included lab exercises where we ran Linux VMs to practice with deploying, coding, and troubleshooting Cassandra 0.7. One of the leading Cassandra experts (Matt Dennis from Riptano) delivered this training. The session was very informative to the audience. The participants were very engaging with their use cases and challenges with using a NoSQL technology like Cassandra.

It is evident that Cassandra is rapidly becoming a viable technology for large scale, high performance, distribute data storage for high performance distributed applications. With Riptano’s commercial support and services, Cassandra will only grow faster. In addition, Evident Software provides APM tools for applications that leverage Cassandra and other NoSQL and data caching products.

It was also a good opportunity to meet the Riptano team and chat with other Cassandra users in the local area. Riptano is hosting future training events in Austin, TX and Santa Monica, CA. Anyone getting involved with Cassandra or considering NoSQL, should attend upcoming training events.

Read more about our solution for Cassandra NoSQL Performance Monitoring

View Comments