Tomcat出現大量CLOSE_WAIT

最近遇到tomcat出現大量CLOSE_WAIT的問題, 始終找不出原因
以下是TCP連線的步驟

CLIENT                                 SERVER

1.    ESTABLISHED                    ESTABLISHED
2.    (Close)
FIN-WAIT-1  –> <FIN,ACK>  –> CLOSE-WAIT
3.    FIN-WAIT-2  <– <ACK>      <– CLOSE-WAIT
4.                                   (Close)
TIME-WAIT   <– <FIN,ACK>  <– LAST-ACK
5.    TIME-WAIT   –> <ACK>      –> CLOSED
(2 MSL)

Server端出現CLOSE_WAIT表示Server"被動地"收到關閉連線通知,Server確實關閉socket連線後會發LAST_ACK回去給Client端
通常CLOSE_WAIT出現的時間很短暫,會出現大量CLOSE_WAIT有以下幾種可能:
1. CPU資源吃緊,該請求正排隊關閉連線:此為正常現象,通常一段時間後就會消失,建議換好一點的硬體設備
2. 後端code有無窮迴圈,Server無法回應該請求:基本上這種情況可以排除,通常這類有明顯bug的code很快就會被發現,除非是天兵工程師。可用jstack或visualvm等工具查看
3. 後端code的multi-thread發生deadlock,Server無法回應該請求:檢查程式有synchronized的地方,可用jstack或visualvm等工具查看
4. 後端code有直接對Remote端發TCP packet的地方沒正確關閉連線:當Server的後端code用client類別連到Remote端,連線逾時遭到Remote端斷線,此時被動關閉連線時,code若沒處理好關閉,就會造成本地端CLOSE_WAIT
5. 某一版Tomcat或JVM的bug造成無法關閉socket

查socket語法:ss -tulpn (一些fd資訊要用sudo才會出現)
[Description] :
ss : It is a command representing utility used to investigate sockets
-t : It is an additional parameter for the ‘ss’ command used to add filter for the output for displaying TCP sockets.
-u : It is an additional parameter for the ‘ss’ command used to add filter for the output for displaying UDP sockets.
-l : It is an additional parameter for the ‘ss’ command used to add filter for the output for displaying only listening sockets.
-p : It is an additional parameter for the ‘ss’ command used to add filter for the output for displaying process associated with the sockets displayed.
-n : It is an additional parameter for the ‘ss’ command used to add filter for the output in a numeric format.

(拿到pid後)
查看socket local port:
root@hostname: /proc/7112/fd# ls -al | grep socket
查看socket local port的另一個方法:
root@hostname:/proc/7112/fd# lsof -i -a -p 7112
參考文章:http://www.dark-hamster.com/operating-system/linux/ubuntu/show-list-of-listening-services-in-linux-using-ss/

—-
問題發生原因:
http://jschu.blog.51cto.com/5594807/1732414

http://m.myexception.cn/open-source/921974.html

http://serverfault.com/questions/160558/how-to-not-get-so-many-apache-close-wait-connections

http://ahuaxuan.iteye.com/blog/657511

暴力刪除法:
https://github.com/rghose/kill-close-wait-connections

https://www.experts-exchange.com/questions/20568402/How-to-clear-CLOSE-WAIT-state-of-a-TCP-connection.html

http://www.shellhacks.com/en/HowTo-Kill-TCP-Connections-in-CLOSEWAIT-State

HTTP的session, cookie和connection

Session: server端記載的用戶會話清單, 存在server記憶體, 可以視為server端的cookie, 以session-id辨認用戶端。session的expired time由web app設定的timeout決定, 一旦用戶存取server就重新計時, 若用戶閒置超過timeout時間就會把該用戶從會話清單移除掉 [現實對照: 某一天的聊天對象]

Cookie: client端的屬性資訊, 例如session cookie就記載著用戶的session-id。cookie可以存成file, 也可以設定cookie expired的期限。若是session cookie就無法存成file, 關閉瀏覽器cookie就消毀;若有HttpOnly flag的cookie只能透過Http(s)存取, Javascript不能存取。[現實對照: 某個聊天對象是誰, 及他的嗜好興趣]

Connection: 指server與client溝通時建立的TCP socket連線。當瀏覽器連到某個web server時, 瀏覽器會帶Connection: keep-alive的表頭, 告訴server端保持連線, 此時server就會開著一條與該client溝通的socket連線, 直到關閉瀏覽器或client端發出"Connection: close"的表頭, 抑或是閒置太久由server端斷開便會中止連線。(注意: curl的Connection表頭永遠是close, 就算加上-H “Connection: keep-alive"也沒用;telnet的Connection表頭預設是keep-alive, 但是可以透過更改Connection: close來斷開連線) [現實對照: 某一次的通話]

[轉] How To: ICMP Ping in Java (JDK 1.5 and above)

Programatically using ICMP Ping is a great way to establish that a server is up and running. Previously you couldn’t do ICMP ping (what ping command does in Linux/Unix & Windows) in java without using JNI or exec calls. Here is a simple and reliable method to do ICMP pings in Java without using JNI or NIO.


String host = "172.16.0.2"
int timeOut = 3000; // I recommend 3 seconds at least
boolean status = InetAddress.getByName(host).isReachable(timeOut)

status is true if the machine is reachable by ping; false otherwise. Best effort is made to try to reach the host, but firewalls and server configuration may block requests resulting in a unreachable status while some specific ports may be accessible. A typical implementation will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host.

In Linux/Unix you may have to suid the java executable to get ICMP Ping working, ECHO REQUESTs will be fine even without suid. However on Windows you can get ICMP Ping without any issues whatsoever.

PS: Linux 版必須該 JVM 使用者有 root 權限發 ICMP