0

We have a mysterious error in our Tomcat and webapps log. A Java web application is running on Tomcat 6, uses Oracle 11g database. All request are logged. We use Commons DBCP for database connection pooling. One feature of our application is that all connection are request-scoped. We implemented that with Spring's SmartDataSource. Releasing connections is wrapped in try/finally, so even if there's an error during the request, the connection gets released. This works well the whole time we tested it.

One day our server returned Error 503 "Service unavailable". In the logs we only found one exception: org.apache.commons.dbcp.AbandonedTrace$AbandonedObjectException. So it seems that Common DBCP abandoned connection collector found an abandoned connection and reported it. It is not error by itself, it is a suggestion that an error is somewhere else.

The trace tells us the exact time and code when the connection was acquired. We explored that time in logs and there was a request that began but never ended. This would explain why the connection was not released, but what bothers us is how try/finally could have been abrupted?

I think this is a Tomcat issue, because there was no other exception in our application logs and because the error code that the server returned was not the ususal Error 500 "Internal server error".

Does anyone have any suggestions why this could happen? It is possible that Tomcat interrupts a thread so that try/finally are ignored?

Infeligo
  • 11,535
  • 8
  • 36
  • 50
  • Is is the web request or the DB request that never ended? In the first case, is your thread running into a deadlock? – Gandalf Oct 04 '11 at 09:47
  • Web request never ended. How do I tell if a thread ran into deadlock? – Infeligo Oct 04 '11 at 10:29
  • Your thread can run into a deadlock if you use unlucky synchronized blocks (or it could run into a livelock waiting for some condition that is never met). If you have debugging enabled at your web application you could connect with a debugger, suspend all threads and look at their stacktraces to see if they are waiting for some monitor that is held by another thread. Or you could do a stack dump like described here: http://stackoverflow.com/questions/2124672/java-stack-dump-on-windows To look for some deadlock this tiny example may help: http://www.myloadtest.com/java-thread-dump/ – Gandalf Oct 04 '11 at 10:49
  • We don't use synchronized blocks or some other type of locking in our code. The frameworkss and libs we use may well do. The problem with debugging is that we saw this error only twice on production server and each time after exploring the logs. – Infeligo Oct 05 '11 at 06:17

0 Answers0