Articles tagged weblogic monitoring tool

The views expressed in this blog are strictly personal, and do not necessarily represent the views of Evident Software.

WebLogic Optimization Tip #1 – Understand the database tier

Date: March 9, 2011
Tags: ,

By Bill Nigh

WebLogic optimization is a rich area for possibilities due to the numerous technologies and monitoring tools involved. One crucial area is the database; this post will talk about RDBMS, with subsquent posts in this series addressing NoSQL and data caching.

  1. JNDI lookups are relatively expensive, so use caching for any objects that initialize database connections (or ANY connections for that matter, such as JMX connections that might be done for server performance monitoring).
  2. Use resource pooling for connections to the database from WebLogic. The WebLogic JDBC connection pooling feature allows this to be configured relatively easily.
  3. The Prepared Statement Cache should be optimized and used. This cache keeps compiled SQL statements in memory, thus avoiding a round-trip to the database when the same statement is used later.
  4. Dive deeper into that realm, and ensure that whenever possible, stored procedures, packages and similar program constructs are used rather than SQL statements.
  5. Let the RDBMS server do the heavy lifting; that’s what it’s designed to do: serve.
  6. Understand the set orientation of SQL (which abhors unnecessary use of cursors).
  7. Understand that other applications may also be accessing that database, and that the root cause of sudden poor performance may be some other business entity hitting that server. Try to ensure your database administrators keep you in the loop in such matters.

These are just some of the considerations that take us out of the application tier and into the database tier, but which may result in perceived poor application performance.

View Comments

WebLogic Server performance tuning – hot spots cooled off

Date: March 1, 2011
Tags: , ,

By Bill Nigh

In a recent post, I shared the story of how Evident ClearStone had been instrumental in helping an enterprise see a pattern of interaction between an organization’s application tier (WebLogic) and data caching tier (Coherence) that was bringing a website to its knees during periods of high demand caused by promotional campaigns. Here are some tidbits on WebLogic Server performance tuning from some of the troubleshooting cycles of that engagement.

It turns out that the organization was co-locating WebLogic Server and Coherence nodes on the same server. When there were performance problems it was important to understand which of the tiers was causing the problem.

We did not see on the server (a heavy-duty Solaris box) any telltale CPU usage spike. CPU utilization was around 10%. However, when we looked at the individual cores, one core repeatedly was maxing out, and this, remember, on a multi-core server. We used prstat (process stat) to get proper granularity of information on the relationship of process to CPU to core utilization. Working from this discovery, we could then isolate for the Coherence tier.

But what was taxing the Coherence tier during those periods of peak demand in response to that promotion? It turns out that everyone who got the email blasts was being directed to the same page on the website, rather than partitioning based on information that could be gleaned from website customer profiles. Because everyone was going for the same page, ‘hot keys’ were being created. In such scenarios, the ‘hot key’ refers to a popular piece of information; that information only exists in one node (unless one counts the backup node, which is not visible to read requests). Because the same node must satisfy all requests, it can get overwhelmed and fail. Staff analyzed the URLs that were directing people to the site, and were able to trace the requests back to the email blasts tied to the promotion.

So the root cause analysis started at the server level, then focus shifted to the application, and within the application to the actual objects being retrieved, but the problem was caused by the promotional email campaign. The problem, in other words, began in a functional area of the organization that was oblivious to the infrastructure implications of their actions.

With NoSQL DB, unlike SQL, you don’t start with a normalized schema; rather, the data is (should be) structured in response to what you want. Likely queries are part of use cases, and you’re concerned with optimizing for reads or writes (or maybe a balance of both). In the case we’ve cited, the integrity of the system was compromised because of lack of understanding of how the email blasts were creating ‘hot spots’. And, since an RDBMS with all its mature and tested query optimization was not used, a possibly unexpected burden now existed.

Learn more about our performance monitoring solution for Java, NoSQL and web servers

View Comments

WebLogic-Coherence slow performance – ClearStone to the rescue

Date: March 1, 2011
Tags: ,

By Bill Nigh

I heard recently of a use case of Oracle WebLogic and Coherence that benefitted from Evident’s  performance monitoring. It seems an enterprise was running an application that used WebLogic for its application server tier and Coherence for its distributed data caching tier. The application was running on 8 Web servers, and WebLogic had been configured with about 130 service threads allocated for incoming requests.

The company sent out a promotion email blast. This got a huge response, and the Web site underperformed; it could not serve the content fast enough. The WebLogic service threads were “taking their own sweet time” to finish a request, in the endearing phraseology of one of our guys, and this was creating a big backlog. Performance bottlenecks began to form, and eventually the system collapsed a number of times.

The WebLogic system failures inspired the employment of a number of safeguards. The ops staff would monitor the number of threads. When the number of available threads began to get low, the working assumption was that the system would soon collapse, so the ops staff took action based on that threshold, taking thread dumps to try to analyze what was hanging and restarting the Web server. Only when the number of users dropped off would the system stabilize. (And some of these users dropping off were probably customers who were frustrated that the link they clicked on was leading to a site with poor performance.)

There was a relationship between WebLogic Server and Coherence in the application’s behavior; when the thread count dropped, monitoring of Coherence revealed an increase of activity in the cache, such as GETs.

As part of a troubleshooting consult, we used Evident ClearStone to monitor WebLogic and Coherence. By looking at performance metrics on application and caching tiers, we were able to identify the root cause as the application source code. Part of the application code was unintentionally retrieving all the information from the caching tier, as seen by a big spike in requests going to the caching tier. The load on the caching tier was causing it  to give up. The inter-tier relationship was key to discovering the problem, which is where ClearStone came to the rescue. In the future, ClearStone’s advanced heuristics will make it even easier to discover interplays and causal relationships between tiers in complex NoSQL systems.

Learn more about our performance monitoring solution for Java, NoSQL and web servers

View Comments