Hello All,
We are hitting on a weird problem with tomcat 6 production server (which is
a serious concern for us now). Hoping to get help from the you guys
This is the problem -
1. Our tomcat 6 server (version 6.0.29) does not respond after working
for a couple of days. Browsers are unable to find application URLs (which
were previously processed normally).
2. The logs does not show any error.
3. There are only a total of 50-60 threads in the thread dump, mostly in
WAIT state. Also, it does not show any BLOCKING or LOCKED threads.
4. Memory utilization was 400MB only
5. TOMCAT restart does not fix the problem. The machine needs to be
restarted to resume normal operation.
6. Netstat did not reveal any extraneous connections
We are running Tomcat 6.0.29 on the following platform -
OS: Windows server 2003 R2 Enterprise Edition SP2
Hardware: Intel Xeon L5520 2.27 GHz (16 CPU threads )
The server.xml we are using has been attached with this mail.
Have anybody here faced similar issue before ?
Please help !
Regards
Sreekumar K.J
www.comprotechnologies.com
The fact that restarting Tomcat does not help, and that there is no error in the logs,
raises a strong suspicion that the problem may be external to Tomcat.
So, just to eliminate certain side-issues :
1) what do you mean by :
What error do you get in the browser ? (Preferably, use a browser other than IE, or if you
do use IE, then in the options disable the "friendly error messages")
2) When the problem happens again :
- does the host still respond to "ping" ?
- on the server, in a command window, do : netstat -p tcp -anb
and paste the complete result here
- from another workstation :
- open a command window
- enter : telnet (hostname) (port) (the port on which Tomcat listens, presumabyl 80)
you will not get a prompt, you will get a blank screen.
- now you have to type blind, because you will not get an echo :
GET / HTTP/1.1
<CR>
<CR>
(meaning 2 times "return"
what do you see ?
- do the same from a command window on the server itself, using "localhost" as the
hostname. Same result ?
What does this mean? They're getting 404? Or any response at
all? By Tomcat? Or some inbetween server? Can they access
other services on the machine? Can you access Tomcat locally,
i.e. from the same machine Tomcat is running on?
Might be a problem with the machine or the network, after all,
not with Tomcat.
--
Michael Ludwig
We have done the tests you had mentioned.
here are our answers to your questions (i'm adding my name before my
builds up further)-
*1.* What do you mean by : Browsers are unable to find application URLs
(which were previously processed normally).What error do you get in the
browser ? (Preferably, use a browser other than IE, or if you do use IE,
then in the options disable the "friendly error messages")
*
** Sreekumar.* Using Firefox 3.6x we get the following error
The connection was reset
The connection to the server was reset while the page was loading.
Additionally, the firebug net panel displays "Status - Aborted"
*2*. When the problem happens again : does the host still respond to "ping"?
*
**Sreekumar.* Yes, the host responds to ping. Additionally we have an Apache
server running and that continues to work normal. Other applications like
RDP and FTP are also NOT impacted.
*3.* On the server, in a command window, do : netstat -p tcp -anb
and paste the complete result here
*
** Sreekumar.* NETSTAT was output as follows
Proto Local Address Foreign Address State PID
TCP 0.0.0.0:21 0.0.0.0:0 LISTENING 1652
[inetinfo.exe]
TCP 0.0.0.0:80 0.0.0.0:0 LISTENING 1336
[httpd.exe]
TCP 0.0.0.0:135 0.0.0.0:0 LISTENING 704
RpcSs
[svchost.exe]
TCP 0.0.0.0:445 0.0.0.0:0 LISTENING 4
[System]
TCP 0.0.0.0:1025 0.0.0.0:0 LISTENING 464
[lsass.exe]
TCP 0.0.0.0:1033 0.0.0.0:0 LISTENING 1420
[erl.exe]
TCP 0.0.0.0:1039 0.0.0.0:0 LISTENING 1880
[erl.exe]
TCP 0.0.0.0:1046 0.0.0.0:0 LISTENING 1568
[cvd.exe]
TCP 0.0.0.0:1047 0.0.0.0:0 LISTENING 1568
[cvd.exe]
TCP 0.0.0.0:1059 0.0.0.0:0 LISTENING 1568
[cvd.exe]
TCP 0.0.0.0:1066 0.0.0.0:0 LISTENING 1652
[inetinfo.exe]
TCP 0.0.0.0:1086 0.0.0.0:0 LISTENING 1416
[EMCliSrv.exe]
TCP 0.0.0.0:1105 0.0.0.0:0 LISTENING 2932
[ManagementAgentNT.exe]
TCP 0.0.0.0:1106 0.0.0.0:0 LISTENING 2932
[ManagementAgentNT.exe]
TCP 0.0.0.0:1311 0.0.0.0:0 LISTENING 2904
[dsm_om_connsvc32.exe]
TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING 3844
TermService
[svchost.exe]
TCP 0.0.0.0:4369 0.0.0.0:0 LISTENING 1440
[epmd.exe]
TCP 0.0.0.0:5222 0.0.0.0:0 LISTENING 1880
[erl.exe]
from another workstation :
- open a command window
- enter : telnet (hostname) (port) (the port on which Tomcat listens,
presumabyl 80)
you will not get a prompt, you will get a blank screen.
- now you have to type blind, because you will not get an echo :
GET / HTTP/1.1
<CR>
<CR>
(meaning 2 times "return"
what do you see ?
- do the same from a command window on the server itself, using "localhost"
as the hostname. Same result ?
***Sreekumar* - The telnet output was as follows (same from local machine
and a different workstation)
HTTP/1.1 400 Bad Request
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Connection: close
0
Connection to host lost.
Next we tried with /HTTP/*1.0 *instead - In this case, when on the local
machine/server, we did not get any response and the telnet exited
immediately after the 2nd [ENTER].
Also, we checked telnet /HTTP/*1.1* after restarting the machine (when
everything was running fine), it still gives a BAD REQUEST. But telnet
requests to /HTTP/*1.0* started working.
***Sreekumar* - The performance counters (after tomcat stopped responding)
from the Window Perfmon for tomcat6.exe were as follows:
Elapsed time - 10666 (sec)
% processor time - 0.000
IO Data bytes per sec - 0.000
Handle count - 730
Page file bytes - 1252765696
Page file bytes Peak - 1293123584
Pool non paged bytes - 42576
Pool paged bytes - 76356
Priority base - 8
Private bytes - 1252765696
Thread count - 77
Virtual bytes - 1813188608
Virtual bytes Peak - 1818099712
Working set - 872779776
Working set Peak – 900476928
ALL OTHER COUNTERS WERE “0”
***Sreekumar *- Following are relevant apache error logs. Note, apache is
running on port 80 and forwarding requests (using mod_proxy) to tomcat on
port 8080.
[Sat Dec 18 04:02:54 2010] [error] [client 125.18.244.200] (OS 10054)An
existing connection was forcibly closed by the remote host. : proxy: error
reading status line from remote server localhost
[Sat Dec 18 04:02:54 2010] [error] [client 125.18.244.200] proxy: Error
reading from remote server returned by /rel/
***Sreekumar *- I'm also attaching the complete thread dump output as
"threadDump.txt"
Please let me know what could be hinted by these results and if any further
tests are required to resolve this issue.
Thanks
Console CTRL+BREAK event signaled
2010-12-18 04:00:04
Full thread dump Java HotSpot(TM) Server VM (17.0-b17 mixed mode):
"File Reaper"
daemon
prio=10 tid=0x60e52800
nid=0x868
in Object.wait()
[0x6551f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x27c28b70>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x27c28b70>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at org.apache.commons.io.FileCleaner$1.run(FileCleaner.java:62)
"File Reaper"
daemon
prio=10 tid=0x64798400
nid=0x9a0
in Object.wait()
[0x6523f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x27c28b90>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x27c28b90>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at org.apache.commons.io.FileCleaner$1.run(FileCleaner.java:62)
"http-8080-6"
daemon
prio=6 tid=0x63ce8400
nid=0x854
in Object.wait()
[0x668ff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x20d226d0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x20d226d0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-5"
daemon
prio=6 tid=0x6372d800
nid=0x1200
in Object.wait()
[0x66a0f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x196f41b0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x196f41b0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-4"
daemon
prio=6 tid=0x63531400
nid=0x1400
in Object.wait()
[0x669bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x196f43f8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x196f43f8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-3"
daemon
prio=6 tid=0x622bf800
nid=0x1740
in Object.wait()
[0x6696f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x19734260>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x19734260>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-2"
daemon
prio=6 tid=0x6377b800
nid=0x143c
in Object.wait()
[0x6685f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x197347b8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x197347b8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-1"
daemon
prio=6 tid=0x60e94400
nid=0x5f8
in Object.wait()
[0x6680f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x188b1150>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x188b1150>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Acceptor-0"
daemon
prio=6 tid=0x62adc400
nid=0x1348
runnable
[0x667bf000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1169)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-7"
daemon
prio=6 tid=0x615fd400
nid=0x1360
in Object.wait()
[0x6676f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4920>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4920>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-6"
daemon
prio=6 tid=0x62411400
nid=0x174c
in Object.wait()
[0x6671f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4ab0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4ab0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-5"
daemon
prio=6 tid=0x62357400
nid=0x14bc
in Object.wait()
[0x666cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4c40>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4c40>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-4"
daemon
prio=6 tid=0x62891400
nid=0x56c
in Object.wait()
[0x6667f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4dd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4dd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-3"
daemon
prio=6 tid=0x616c8800
nid=0x1138
in Object.wait()
[0x6662f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4f60>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4f60>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-2"
daemon
prio=6 tid=0x612f5400
nid=0xad0
in Object.wait()
[0x665df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b50f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b50f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-1"
daemon
prio=6 tid=0x62b3fc00
nid=0x13bc
in Object.wait()
[0x6658f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18391080>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18391080>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-0"
daemon
prio=6 tid=0x62434400
nid=0x1414
in Object.wait()
[0x6653f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b5280>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b5280>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-7"
daemon
prio=6 tid=0x61426800
nid=0x12dc
in Object.wait()
[0x664ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a604e8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a604e8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-6"
daemon
prio=6 tid=0x636ec400
nid=0x15a4
in Object.wait()
[0x6649f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18391210>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18391210>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-5"
daemon
prio=6 tid=0x62498800
nid=0x16d4
in Object.wait()
[0x6644f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60670>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60670>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-4"
daemon
prio=6 tid=0x63108c00
nid=0x1354
in Object.wait()
[0x65fff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b0e0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b0e0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-3"
daemon
prio=6 tid=0x6141a800
nid=0x121c
in Object.wait()
[0x65faf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a607f8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a607f8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-2"
daemon
prio=6 tid=0x619b8400
nid=0x17f0
in Object.wait()
[0x65f5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b268>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b268>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-1"
daemon
prio=6 tid=0x63900400
nid=0x16dc
in Object.wait()
[0x65f0f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b3f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b3f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-0"
daemon
prio=6 tid=0x61bd1800
nid=0x1198
in Object.wait()
[0x65ebf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60980>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60980>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Acceptor-0"
daemon
prio=6 tid=0x6233d400
nid=0xf18
runnable
[0x65e6f000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1169)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Sendfile-0"
daemon
prio=6 tid=0x625a0800
nid=0x12f0
in Object.wait()
[0x65e1f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b6b8>
(a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:1805)
- locked <0x18a7b6b8>
(a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-7"
daemon
prio=6 tid=0x62762400
nid=0xb00
in Object.wait()
[0x65dcf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60b08>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60b08>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-6"
daemon
prio=6 tid=0x627ed400
nid=0x14c8
in Object.wait()
[0x65d7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b8c0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b8c0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-5"
daemon
prio=6 tid=0x62c28400
nid=0x1374
in Object.wait()
[0x65d2f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60ca0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60ca0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-4"
daemon
prio=6 tid=0x625a6400
nid=0xadc
in Object.wait()
[0x65cdf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7ba58>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7ba58>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-3"
daemon
prio=6 tid=0x6298b400
nid=0x17c8
in Object.wait()
[0x65c8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60e38>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60e38>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-2"
daemon
prio=6 tid=0x62754400
nid=0x10e0
in Object.wait()
[0x65c3f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7bbf0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7bbf0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-1"
daemon
prio=6 tid=0x60e59400
nid=0x1768
in Object.wait()
[0x65bef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7bd88>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7bd88>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-0"
daemon
prio=6 tid=0x613c3c00
nid=0x16f0
in Object.wait()
[0x65b9f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60fd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60fd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-7"
daemon
prio=6 tid=0x61463400
nid=0x290
in Object.wait()
[0x65b4f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7bf20>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7bf20>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-6"
daemon
prio=6 tid=0x61dd4400
nid=0xa94
in Object.wait()
[0x65aff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a61168>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a61168>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-5"
daemon
prio=6 tid=0x6145b400
nid=0x16c8
in Object.wait()
[0x658af000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a612f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a612f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-4"
daemon
prio=6 tid=0x61649400
nid=0x126c
in Object.wait()
[0x6585f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b41d30>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b41d30>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-3"
daemon
prio=6 tid=0x6198e400
nid=0x1480
in Object.wait()
[0x6580f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b4cd50>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b4cd50>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-2"
daemon
prio=6 tid=0x63143400
nid=0x1684
in Object.wait()
[0x657bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b41eb8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b41eb8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-1"
daemon
prio=6 tid=0x633aac00
nid=0x768
in Object.wait()
[0x6576f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b4ced8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b4ced8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-0"
daemon
prio=6 tid=0x61976400
nid=0x1588
in Object.wait()
[0x6571f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b42040>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b42040>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
daemon
prio=6 tid=0x637e5400
nid=0x14fc
waiting on condition
[0x6560f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
at java.lang.Thread.run(Thread.java:619)
"Thread-7"
prio=6 tid=0x637c5400
nid=0xf7c
waiting on condition
[0x655bf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.pearson.rel.webapp.servlets.bosh.Janitor.run(Janitor.java:57)
at java.lang.Thread.run(Thread.java:619)
"Store userCache Spool Thread"
daemon
prio=6 tid=0x61966400
nid=0x628
waiting on condition
[0x656cf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)
at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:74)
at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)
"net.sf.ehcache.CacheManager@17ee3e5"
daemon
prio=6 tid=0x62295c00
nid=0xaa8
in Object.wait()
[0x6556f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x17ff7930>
(a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x17ff7930>
(a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Timer-0"
daemon
prio=6 tid=0x635cdc00
nid=0x13d8
in Object.wait()
[0x6567f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x17fbfc28>
(a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x17fbfc28>
(a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Thread-3"
daemon
prio=6 tid=0x62b3f400
nid=0x15e0
waiting on condition
[0x653cf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
at java.lang.Thread.run(Thread.java:619)
"GC Daemon"
daemon
prio=2 tid=0x61957800
nid=0x13a8
in Object.wait()
[0x620ff000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13d54028>
(a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x13d54028>
(a sun.misc.GC$LatencyLock)
"Thread-2"
prio=6 tid=0x60e33000
nid=0xbec
runnable
[0x6128f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x18a211c0>
(a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:662)
at org.apache.catalina.startup.Catalina.start(Catalina.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
"Low Memory Detector"
daemon
prio=6 tid=0x60df4400
nid=0xb44
runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1"
daemon
prio=10 tid=0x60df1c00
nid=0x660
waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0"
daemon
prio=10 tid=0x60debc00
nid=0x1424
waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener"
daemon
prio=10 tid=0x60dea400
nid=0x10c0
runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher"
daemon
prio=10 tid=0x60de9400
nid=0x1648
waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer"
daemon
prio=8 tid=0x60a46400
nid=0xf5c
in Object.wait()
[0x6105f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13d5ffc0>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x13d5ffc0>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler"
daemon
prio=10 tid=0x60a45000
nid=0x2b8
in Object.wait()
[0x6100f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13d54018>
(a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x13d54018>
(a java.lang.ref.Reference$Lock)
"main"
prio=6 tid=0x00616800
nid=0x14c4
runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"VM Thread"
prio=10 tid=0x60a42400
nid=0x1274
runnable
"GC task thread#0 (ParallelGC)"
prio=6 tid=0x0061e000
nid=0x11c4
runnable
"GC task thread#1 (ParallelGC)"
prio=6 tid=0x0061f400
nid=0x17e4
runnable
"GC task thread#2 (ParallelGC)"
prio=6 tid=0x60950800
nid=0x14b8
runnable
"GC task thread#3 (ParallelGC)"
prio=6 tid=0x60951c00
nid=0x15c4
runnable
"GC task thread#4 (ParallelGC)"
prio=6 tid=0x60953000
nid=0x16b4
runnable
"GC task thread#5 (ParallelGC)"
prio=6 tid=0x60954400
nid=0x4cc
runnable
"GC task thread#6 (ParallelGC)"
prio=6 tid=0x60955c00
nid=0xd80
runnable
"GC task thread#7 (ParallelGC)"
prio=6 tid=0x60957000
nid=0x1748
runnable
"GC task thread#8 (ParallelGC)"
prio=6 tid=0x60958400
nid=0x1624
runnable
"GC task thread#9 (ParallelGC)"
prio=6 tid=0x60959800
nid=0x258
runnable
"GC task thread#10 (ParallelGC)"
prio=6 tid=0x6095ac00
nid=0x10dc
runnable
"GC task thread#11 (ParallelGC)"
prio=6 tid=0x6095c000
nid=0x139c
runnable
"GC task thread#12 (ParallelGC)"
prio=6 tid=0x6095d400
nid=0x1784
runnable
"VM Periodic Task Thread"
prio=10 tid=0x60dfe400
nid=0x13d4
waiting on condition
JNI global references: 1801
Heap
PSYoungGen
total 300544K, used 58496K
[0x47000000, 0x5c550000, 0x60950000)
eden
space 250816K, 22% used
[0x47000000,0x4a817c90,0x564f0000)
from
space 49728K, 2% used
[0x594c0000,0x595c8530,0x5c550000)
to
space 48960K, 0% used
[0x564f0000,0x564f0000,0x594c0000)
PSOldGen
total 699072K, used 327254K
[0x13d50000, 0x3e800000, 0x47000000)
object
space 699072K, 46% used
[0x13d50000,0x27ce5938,0x3e800000)
PSPermGen
total 130176K, used 62003K
[0x03d50000, 0x0bc70000, 0x13d50000)
object
space 130176K, 47% used
[0x03d50000,0x079dcec8,0x0bc70000)
You could configure your mail client to use standard indentation.
There's nothing running on port 8080.
What hostnames and ports did you use for these two telnet command?
You're getting a Bad Request error because you omitted the Host header.
It should include the hostname, e.g. localhost:
GET / HTTP/1.1
Host: localhost
<CR>
<CR>
HTTP 1.0 doesn't require the Host header. But I don't understand what
worked and what didn't.
--
Michael Ludwig
Hello Michael,
Thank you for the quick reply.
It seems the complete list of netstat output somehow got omitted. They have
been appended to this mail. Tomcat is displayed as running on port 8080.
Clarifying this - telnet requests to / HTTP/*1.0* displayed the html of our
home page.
From you comments, this would mean that had the host been provided, telnet
would have printed the home page html for the request "GET / HTTP/1.1" too.
We have also noted that in between some requests are successfully served by
tomcat, while most of the resources are not fetched. (as explained in the
previous mail)
wondering what could be wrong here...
Regards
Sreekumar
Netstat output:
------------------------------------------------------------------------------------------------------------------
- Show quoted text -
TCP 192.168.103.117:3389 192.168.1.113:1432 ESTABLISHED 3844
TermService
[svchost.exe]
TCP 192.168.103.117:3389 192.168.1.22:1259 ESTABLISHED 3844
TermService
[svchost.exe]
TCP 192.168.103.117:4493 192.168.101.124:1521 ESTABLISHED 5356
[tomcat6.exe]
TCP 127.0.0.1:3450 127.0.0.1:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 127.0.0.1:5152 127.0.0.1:1054 CLOSE_WAIT 2284
[jqs.exe]
TCP 192.168.103.117:1790 184.84.255.35:80 < http://184.84.255.35/> >
CLOSE_WAIT 6008
[jucheck.exe]
TCP 192.168.103.117:1973 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:2074 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:3825 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4697 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4701 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4702 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4903 72.247.219.72:80 < http://72.247.219.72/> >
CLOSE_WAIT 2300
[jucheck.exe]
It looks like the only thing listening to a HTTP port is the above, which looks like
Apache httpd, not Tomcat. And I do not see any of the usual Tomcat ports there.
Are you sure that Tomcat really runs on the same host ?
Are you using Apache httpd as a front-end to Tomcat ?
And if yes, with which connector ?
In your Apache configuration files, do you have any
- "proxy*" statements ?
- JkMount .. statements ?
If yes, post them here.
In your Tomcat main configuration file ((tomcat_dir)/conf/server.xml), what <Connector ..>
tags (uncommented) are there ? Can you also post them here ?
Hello Andre,
Thank you for your attention to this query.
The these entries in the netstat shows tomcat running. (These were missing
in the original message and were appended in a subsequent mail.)
TCP 0.0.0.0:8009 0.0.0.0:0 LISTENING 5356
[tomcat6.exe]
TCP 0.0.0.0:8080 0.0.0.0:0 LISTENING 5356
[tomcat6.exe]
Yes we are using apache httpd as front-end to tomcat using mod_proxy (not
AJP).
The configuration in httpd.conf are
ProxyPass /rel/ http://localhost:8080/rel/
ProxyPassReverse /rel/ http://localhost:8080/rel/
Please note that to isolate the issue we had replicated this behavior (of
the application) by directly running it on tomcat (port 8080). (reason why
the apache mod_proxy settings were not included in the original mail.)
The server.xml is available as an attachment with the original message.
Could configuration errors be the cause of this problem - that tomcat works
good for a couple of days, and then starts ignoring a few requests, after a
while stops ignoring most of them.
Can a library mismatch trigger such issues ?
Sreekumar
Do the mod_proxy logs reveal anything interesting?
Do you have the access log turned on for Tomcat? Requests will only be
logged after processing, but it might still be useful to find out if
requests get processed and with what outcome.
http://tomcat.apache.org/tomcat-6.0-doc/config/valve.html#Ac[..]
Maybe I haven't followed closely enough, but it's not clear to me that
this is the application's fault. How do you know?
--
Michael Ludwig
It looks like you are not using the above port. So you could comment out the following in
your server.xml :
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
It will not solve your problem, but it will simplify what we're looking at.
Apart from the above and the other ports in LISTEN state, when "tomcat freezes", do you
have any other ports in the netstat listing, shown as "CLOSE_WAIT" for example ?
If yes, how many ?
Let me also suggest something else.
Let's suppose that your Tomcat is installed in the directory C:\Tomcat6, and let's call
this directory "CATALINA_BASE" for now.
Under that directory, there is a directory "/webapps/", and undet that one there is
(apparently) a directory "/rel/" with your current application.
In the directory /webapps/, at the same level as the directory /rel/, create a new
directory "/freeze/", and in that directory place a html file "freeze.html".
It does not matter what the content of that html file is, just make it recognisable.
Next, in your Apache configuration, add the following lines :
ProxyPass /freeze/ http://localhost:8080/freeze/
ProxyPassReverse /freeze/ http://localhost:8080/freeze/
and restart Tomcat and Apache httpd.
Then, the next time Tomcat appears to freeze, try with a browser to access the "freeze"
page, as :
http://(hostname)/freeze/freeze.html
and as
http://(hostname):8080/freeze/freeze.html
and let's see what happens.
Do they show anything at all?
It would be interesting to look at the thread dump.
If it is too long for the e-mail, you may post it somewhere else, e.g.
to a pastebin site.
Your telnet test,
shows that Tomcat processes incoming connections (and correctly
identifies your HTTP/1.1 request as an invalid one). So maybe it is
only your webapp that fails.
1. Are you sure that restart was successful? I.e., that "stop" part of
it has actually stopped Tomcat and freed resources.
2. Maybe there are some external resources (database? exhausting free
space on a hard drive?) I wonder what can it be that can be exhausted
and is not freed by Tomcat shutdown. It is easy to exhaust database
connections or open file handles, if you do not close them properly,
but all they should be freed when a Windows application shuts down.
3. Maybe there is anything in the system Events log.
4. If you stop and start Tomcat after the failure, are all the usual
startup time messages present in the logs? Maybe there are some
messages, like the webapp failing to start.
5. You may deploy the manager webapp, accessible as
http://<server>:<port>/manager/html
It will show you the status of your webapps and the requests that are
currently being processed.
Best regards,
Konstantin Kolinko
Frequently.
Usually a mis-match between the number of threads configured in httpd
and the number of threads configured in Tomcat. Since mod_jk and
mod_proxy use persistent connections by default and the BIO connector
uses one thread per connection then if httpd has more threads configured
you can eventually reach a state where all Tomcat's threads have been
allocated and the next request blocks.
There are two possible approaches.
1. Tune the time-outs
2. Disable connection re-use
I'd go with 2 - it takes much less time to set up. Ignore the dire
warnings in the docs. Most users don't notice the difference.
Mark
Ah. Did you infer that from the OP's description of most threads
being in the WAITING state?
http://download.oracle.com/javase/1.5.0/docs/api/java/lang/T[..]
Here's another take on the same matter from two years ago:
Re: mod_jk -> ajp hangs indefinitely on some requests - Rainer Jung
--
Michael Ludwig
Hello All,
Thank you for the suggestions.
We noted that we have these errors in the logs -
SEVERE: The web application [/rel] registered the JBDC driver
[oracle.jdbc.OracleDriver] but failed to unregister it when the web
application was stopped. To prevent a memory leak, the JDBC Driver has been
forcibly unregistered.
Dec 21, 2010 1:41:48 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: The web application [/rel] appears to have started a thread named
[Thread-7] but has failed to stop it. This is very likely to create a memory
leak.
Dec 21, 2010 1:41:48 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: The web application [/rel] created a ThreadLocal with key of type
[org.apache.commons.lang.builder.ReflectionToStringBuilder$1] (value
[org.apache.commons.lang.builder.ReflectionToStringBuilder$1@29fb93]) and a
value of type [java.util.HashSet] (value [[]]) but failed to remove it when
the web application was stopped. This is very likely to create a memory
leak.
The thread dump has been attached with this mail.
Hope these help.
We are now trying to replicate this issue by running tomcat on port 80
(switching off apache httpd).
I'll post the observations here after we do this.
Thanks again
Sreekumar
Console CTRL+BREAK event signaled
2010-12-18 04:00:04
Full thread dump Java HotSpot(TM) Server VM (17.0-b17 mixed mode):
"File Reaper"
daemon
prio=10 tid=0x60e52800
nid=0x868
in Object.wait()
[0x6551f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x27c28b70>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x27c28b70>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at org.apache.commons.io.FileCleaner$1.run(FileCleaner.java:62)
"File Reaper"
daemon
prio=10 tid=0x64798400
nid=0x9a0
in Object.wait()
[0x6523f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x27c28b90>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x27c28b90>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at org.apache.commons.io.FileCleaner$1.run(FileCleaner.java:62)
"http-8080-6"
daemon
prio=6 tid=0x63ce8400
nid=0x854
in Object.wait()
[0x668ff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x20d226d0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x20d226d0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-5"
daemon
prio=6 tid=0x6372d800
nid=0x1200
in Object.wait()
[0x66a0f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x196f41b0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x196f41b0>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-4"
daemon
prio=6 tid=0x63531400
nid=0x1400
in Object.wait()
[0x669bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x196f43f8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x196f43f8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-3"
daemon
prio=6 tid=0x622bf800
nid=0x1740
in Object.wait()
[0x6696f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x19734260>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x19734260>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-2"
daemon
prio=6 tid=0x6377b800
nid=0x143c
in Object.wait()
[0x6685f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x197347b8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x197347b8>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"http-8080-1"
daemon
prio=6 tid=0x60e94400
nid=0x5f8
in Object.wait()
[0x6680f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x188b1150>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1540)
- locked <0x188b1150>
(a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1565)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Acceptor-0"
daemon
prio=6 tid=0x62adc400
nid=0x1348
runnable
[0x667bf000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1169)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-7"
daemon
prio=6 tid=0x615fd400
nid=0x1360
in Object.wait()
[0x6676f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4920>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4920>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-6"
daemon
prio=6 tid=0x62411400
nid=0x174c
in Object.wait()
[0x6671f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4ab0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4ab0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-5"
daemon
prio=6 tid=0x62357400
nid=0x14bc
in Object.wait()
[0x666cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4c40>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4c40>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-4"
daemon
prio=6 tid=0x62891400
nid=0x56c
in Object.wait()
[0x6667f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4dd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4dd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-3"
daemon
prio=6 tid=0x616c8800
nid=0x1138
in Object.wait()
[0x6662f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b4f60>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b4f60>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-2"
daemon
prio=6 tid=0x612f5400
nid=0xad0
in Object.wait()
[0x665df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b50f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b50f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-1"
daemon
prio=6 tid=0x62b3fc00
nid=0x13bc
in Object.wait()
[0x6658f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18391080>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18391080>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-CometPoller-0"
daemon
prio=6 tid=0x62434400
nid=0x1414
in Object.wait()
[0x6653f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x189b5280>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x189b5280>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-7"
daemon
prio=6 tid=0x61426800
nid=0x12dc
in Object.wait()
[0x664ef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a604e8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a604e8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-6"
daemon
prio=6 tid=0x636ec400
nid=0x15a4
in Object.wait()
[0x6649f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18391210>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18391210>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-5"
daemon
prio=6 tid=0x62498800
nid=0x16d4
in Object.wait()
[0x6644f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60670>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60670>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-4"
daemon
prio=6 tid=0x63108c00
nid=0x1354
in Object.wait()
[0x65fff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b0e0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b0e0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-3"
daemon
prio=6 tid=0x6141a800
nid=0x121c
in Object.wait()
[0x65faf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a607f8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a607f8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-2"
daemon
prio=6 tid=0x619b8400
nid=0x17f0
in Object.wait()
[0x65f5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b268>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b268>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-1"
daemon
prio=6 tid=0x63900400
nid=0x16dc
in Object.wait()
[0x65f0f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b3f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b3f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Poller-0"
daemon
prio=6 tid=0x61bd1800
nid=0x1198
in Object.wait()
[0x65ebf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60980>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60980>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Acceptor-0"
daemon
prio=6 tid=0x6233d400
nid=0xf18
runnable
[0x65e6f000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1169)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Sendfile-0"
daemon
prio=6 tid=0x625a0800
nid=0x12f0
in Object.wait()
[0x65e1f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b6b8>
(a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:1805)
- locked <0x18a7b6b8>
(a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-7"
daemon
prio=6 tid=0x62762400
nid=0xb00
in Object.wait()
[0x65dcf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60b08>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60b08>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-6"
daemon
prio=6 tid=0x627ed400
nid=0x14c8
in Object.wait()
[0x65d7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7b8c0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7b8c0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-5"
daemon
prio=6 tid=0x62c28400
nid=0x1374
in Object.wait()
[0x65d2f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60ca0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60ca0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-4"
daemon
prio=6 tid=0x625a6400
nid=0xadc
in Object.wait()
[0x65cdf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7ba58>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7ba58>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-3"
daemon
prio=6 tid=0x6298b400
nid=0x17c8
in Object.wait()
[0x65c8f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60e38>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60e38>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-2"
daemon
prio=6 tid=0x62754400
nid=0x10e0
in Object.wait()
[0x65c3f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7bbf0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7bbf0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-1"
daemon
prio=6 tid=0x60e59400
nid=0x1768
in Object.wait()
[0x65bef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7bd88>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7bd88>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-CometPoller-0"
daemon
prio=6 tid=0x613c3c00
nid=0x16f0
in Object.wait()
[0x65b9f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a60fd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a60fd0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-7"
daemon
prio=6 tid=0x61463400
nid=0x290
in Object.wait()
[0x65b4f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a7bf20>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a7bf20>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-6"
daemon
prio=6 tid=0x61dd4400
nid=0xa94
in Object.wait()
[0x65aff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a61168>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a61168>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-5"
daemon
prio=6 tid=0x6145b400
nid=0x16c8
in Object.wait()
[0x658af000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18a612f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18a612f0>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-4"
daemon
prio=6 tid=0x61649400
nid=0x126c
in Object.wait()
[0x6585f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b41d30>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b41d30>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-3"
daemon
prio=6 tid=0x6198e400
nid=0x1480
in Object.wait()
[0x6580f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b4cd50>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b4cd50>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-2"
daemon
prio=6 tid=0x63143400
nid=0x1684
in Object.wait()
[0x657bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b41eb8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b41eb8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-1"
daemon
prio=6 tid=0x633aac00
nid=0x768
in Object.wait()
[0x6576f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b4ced8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b4ced8>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"http-8080-Poller-0"
daemon
prio=6 tid=0x61976400
nid=0x1588
in Object.wait()
[0x6571f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x18b42040>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1336)
- locked <0x18b42040>
(a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
daemon
prio=6 tid=0x637e5400
nid=0x14fc
waiting on condition
[0x6560f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1579)
at java.lang.Thread.run(Thread.java:619)
"Thread-7"
prio=6 tid=0x637c5400
nid=0xf7c
waiting on condition
[0x655bf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.pearson.rel.webapp.servlets.bosh.Janitor.run(Janitor.java:57)
at java.lang.Thread.run(Thread.java:619)
"Store userCache Spool Thread"
daemon
prio=6 tid=0x61966400
nid=0x628
waiting on condition
[0x656cf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at net.sf.ehcache.store.DiskStore.spoolAndExpiryThreadMain(DiskStore.java:669)
at net.sf.ehcache.store.DiskStore.access$900(DiskStore.java:74)
at net.sf.ehcache.store.DiskStore$SpoolAndExpiryThread.run(DiskStore.java:1170)
"net.sf.ehcache.CacheManager@17ee3e5"
daemon
prio=6 tid=0x62295c00
nid=0xaa8
in Object.wait()
[0x6556f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x17ff7930>
(a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x17ff7930>
(a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Timer-0"
daemon
prio=6 tid=0x635cdc00
nid=0x13d8
in Object.wait()
[0x6567f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x17fbfc28>
(a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x17fbfc28>
(a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
"Thread-3"
daemon
prio=6 tid=0x62b3f400
nid=0x15e0
waiting on condition
[0x653cf000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1080)
at java.lang.Thread.run(Thread.java:619)
"GC Daemon"
daemon
prio=2 tid=0x61957800
nid=0x13a8
in Object.wait()
[0x620ff000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13d54028>
(a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x13d54028>
(a sun.misc.GC$LatencyLock)
"Thread-2"
prio=6 tid=0x60e33000
nid=0xbec
runnable
[0x6128f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x18a211c0>
(a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
at org.apache.catalina.startup.Catalina.await(Catalina.java:662)
at org.apache.catalina.startup.Catalina.start(Catalina.java:614)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
"Low Memory Detector"
daemon
prio=6 tid=0x60df4400
nid=0xb44
runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1"
daemon
prio=10 tid=0x60df1c00
nid=0x660
waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0"
daemon
prio=10 tid=0x60debc00
nid=0x1424
waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener"
daemon
prio=10 tid=0x60dea400
nid=0x10c0
runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher"
daemon
prio=10 tid=0x60de9400
nid=0x1648
waiting on condition
[0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer"
daemon
prio=8 tid=0x60a46400
nid=0xf5c
in Object.wait()
[0x6105f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13d5ffc0>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x13d5ffc0>
(a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler"
daemon
prio=10 tid=0x60a45000
nid=0x2b8
in Object.wait()
[0x6100f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13d54018>
(a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x13d54018>
(a java.lang.ref.Reference$Lock)
"main"
prio=6 tid=0x00616800
nid=0x14c4
runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
"VM Thread"
prio=10 tid=0x60a42400
nid=0x1274
runnable
"GC task thread#0 (ParallelGC)"
prio=6 tid=0x0061e000
nid=0x11c4
runnable
"GC task thread#1 (ParallelGC)"
prio=6 tid=0x0061f400
nid=0x17e4
runnable
"GC task thread#2 (ParallelGC)"
prio=6 tid=0x60950800
nid=0x14b8
runnable
"GC task thread#3 (ParallelGC)"
prio=6 tid=0x60951c00
nid=0x15c4
runnable
"GC task thread#4 (ParallelGC)"
prio=6 tid=0x60953000
nid=0x16b4
runnable
"GC task thread#5 (ParallelGC)"
prio=6 tid=0x60954400
nid=0x4cc
runnable
"GC task thread#6 (ParallelGC)"
prio=6 tid=0x60955c00
nid=0xd80
runnable
"GC task thread#7 (ParallelGC)"
prio=6 tid=0x60957000
nid=0x1748
runnable
"GC task thread#8 (ParallelGC)"
prio=6 tid=0x60958400
nid=0x1624
runnable
"GC task thread#9 (ParallelGC)"
prio=6 tid=0x60959800
nid=0x258
runnable
"GC task thread#10 (ParallelGC)"
prio=6 tid=0x6095ac00
nid=0x10dc
runnable
"GC task thread#11 (ParallelGC)"
prio=6 tid=0x6095c000
nid=0x139c
runnable
"GC task thread#12 (ParallelGC)"
prio=6 tid=0x6095d400
nid=0x1784
runnable
"VM Periodic Task Thread"
prio=10 tid=0x60dfe400
nid=0x13d4
waiting on condition
JNI global references: 1801
Heap
PSYoungGen
total 300544K, used 58496K
[0x47000000, 0x5c550000, 0x60950000)
eden
space 250816K, 22% used
[0x47000000,0x4a817c90,0x564f0000)
from
space 49728K, 2% used
[0x594c0000,0x595c8530,0x5c550000)
to
space 48960K, 0% used
[0x564f0000,0x564f0000,0x594c0000)
PSOldGen
total 699072K, used 327254K
[0x13d50000, 0x3e800000, 0x47000000)
object
space 699072K, 46% used
[0x13d50000,0x27ce5938,0x3e800000)
PSPermGen
total 130176K, used 62003K
[0x03d50000, 0x0bc70000, 0x13d50000)
object
space 130176K, 47% used
[0x03d50000,0x079dcec8,0x0bc70000)
Hello Andre
These are the ports in the close wait state -
TCP 127.0.0.1:3450 127.0.0.1:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 127.0.0.1:5152 127.0.0.1:1054 CLOSE_WAIT 2284
[jqs.exe]
TCP 192.168.103.117:1790 184.84.255.35:80 < http://184.84.255.35/> >
CLOSE_WAIT 6008
[jucheck.exe]
TCP 192.168.103.117:1973 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:2074 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:3825 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4697 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4701 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4702 192.168.103.117:8080 CLOSE_WAIT 1836
[httpd.exe]
TCP 192.168.103.117:4903 72.247.219.72:80 < http://72.247.219.72/> >
CLOSE_WAIT 2300
[jucheck.exe]
We have another test application running on tomcat, which also fails to
respond once tomcat starts ignoring requests; so is the case with tomcat
manager too.
We are trying to replicate this behavior again by directly running tomcat on
80; will be posting the observations here.
Thanks and Regards
Sreekumar
Hi.
Ok, there do not seem to be a whole bunch of them, which is good.
One some systems, with badly-behaved applications, I have seen hundreds of those, to the
point of rendering the system totally incapable of accepting new TCP connections of any
kind. But that does not appear to be the case here.
This is one of the connections between the Apache front-end proxy_http module, and the
back-end tomcat port 8080 HTTP Connector.
The fact that it is a "CLOSE_WAIT" state indicates that one of the sides has closed its
connection, but the other has not yet. It is a normal TCP state, if it remains moderate
and does not last too long.
One source of problems - as I believe Mark pointed out recently, maybe in another thread -
is when there is a mismatch between the number of connections which the front-end is
trying to make to Tomcat, and the number of threads available in Tomcat to handle them.
If there are many more client requests than Tomcat threads available to serve them, then a
lot of them will end up in the accept queue of the back-end Tomcat, waiting for a tomcat
thread to become available to serve them. At some point, this queue reaches its maximum
size, and then further requests are being rejected.
There are a whole bunch of parameters allowing you to control this at the httpd level, see
: http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass
and at the Tomcat level, the Connector attribute "maxThreads".
The defaults are usually fine however, so I would not start experimenting with then until
you know what the problem really is.
As far as i know, the above is the "java update scheduler" service.
Nothing to do with the current problem, but I generally dislike this kind of thing on a
server, and turn them off. They use up resources, and ports which you later always wonder
about. Plus, I don't want any server of mine to decide to update himself, or even pop up
annoying dialogs all the time.
A matter of preference.
Right. What I was trying to do, is to have some application as simple as possible, and
totally independent of your own webapps. A simple html page will be served by the Tomcat
embedded "default servlet", using only Tomcat code.
If that one blocks too, then you would know that it has nothing to do with application
code, extra libraries etc..
Well, not quite, as tomcat could still be blocked by your own apps.
But if the rest blocks, and this does not, then it would be a clear sign that the block is
in your applications.
It is equivalent to the telnet test done before, just a bit easier to use.
That's a good idea, to eliminate the front-end http and the proxy connector's impact.
Thank you Andre, Michael, Konstantin and Mark.
As suggested from the thread dump (all the threads where in waiting state),
we had a thread synchronization problem in the code (committed recently). We
could find this out after a few load tests on Tomcat and further code
reviews.
Corrected this, and now Tomcat is working good.
Thank you and Wish you all a very Happy New Year,
Sreekumar
Related Topics
- Problem with apache tomcat while running load test
- Tomcat 5.5.26 stops responding to all client from browser/applets for ~3minutes few times per day
- Application stops responding, jk worker in error state
- Tomcat 6.0.32 stops responding
- Tomcat HTTPS Connector Stops working after an hour
- RE: Application stops responding, jk worker in error state
Post a Comment
