Search This Blog

Monday, July 15, 2013

tuning - MySQL lowering wait_timeout value to lower number of open connections - Server Fault

tuning - MySQL lowering wait_timeout value to lower number of open connections - Server Fault:

Lowering the value is pretty trivial without a mysql restart
Let's say you want to lower timeouts to 30 seconds
First, add this to my.cnf
[mysqld]
interactive_timeout=30
wait_timeout=30
Then, you can do something like this
mysql -uroot -ppassword -e"SET GLOBAL wait_timeout=30; SET GLOBAL interactive_timeout=30"
All DB Connections after this will timeout in 30 seconds

WARNING

Make sure to use explicitly use mysql_close. I do not trust Apache as most developers do. If not, sometimes, there is a race condition where Apache closes a DB Connection but does not inform mysqld and mysqld hold that connection open until it times out. Even worse, you may see TIME_WAITs more often. Choose your timeout values wisely.

UPDATE 2012-11-12 10:10 EDT

CAVEAT

After applying my posted suggestions, create a script called /root/show_mysql_netstat.sh with the following lines:
netstat | grep mysql > /root/mysql_netstat.txt
cat /root/mysql_netstat.txt | awk '{print $5}' | sed 's/:/ /g' | awk '{print $2}' | sort -u > /root/mysql_netstat_iplist.txt
for IP in `cat /root/mysql_netstat_iplist.txt`
do
        ESCOUNT=`cat /root/mysql_netstat.txt | grep ESTABLISHED | awk '{print $5}' | grep -c "${IP}"`
        TWCOUNT=`cat /root/mysql_netstat.txt | grep TIME_WAIT   | awk '{print $5}' | grep -c "${IP}"`
        IPPAD=`echo "${IP}..................................." | cut -b -35`
        (( ESCOUNT += 1000000 ))
        (( TWCOUNT += 1000000 ))
        ES=`echo ${ESCOUNT} | cut -b 3-`
        TW=`echo ${TWCOUNT} | cut -b 3-`
        echo ${IPPAD} : ESTABLISHED:${ES} TIME_WAIT:${TW}
done
echo ; echo
netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n | sed 's/d)/d/'
When you run this, you should see something like this:
[root@*** ~]# /root/ShowConnProfiles.sh
10.48.22.4......................... : ESTABLISHED:00002 TIME_WAIT:00008
10.48.22.8......................... : ESTABLISHED:00000 TIME_WAIT:00002
10.64.51.130....................... : ESTABLISHED:00001 TIME_WAIT:00000
10.64.51.133....................... : ESTABLISHED:00000 TIME_WAIT:00079
10.64.51.134....................... : ESTABLISHED:00002 TIME_WAIT:00001
10.64.51.17........................ : ESTABLISHED:00003 TIME_WAIT:01160
10.64.51.171....................... : ESTABLISHED:00002 TIME_WAIT:00000
10.64.51.174....................... : ESTABLISHED:00000 TIME_WAIT:00589
10.64.51.176....................... : ESTABLISHED:00001 TIME_WAIT:00570


      1 established
      1 Foreign
     11 LISTEN
     25 ESTABLISHED
   1301 TIME_WAIT
If you still see a lot of mysql TIME_WAITs for any given web server, here are two escalation steps to take:

ESCALATION #1

Login to the offending web server and restart apache as follows:
service httpd stop
sleep 30
service httpd start
If necessary, do this to all the web servers
service httpd stop (on all web servers)
service mysql stop
sleep 120
service mysql start
service httpd start (on all web servers)

ESCALATION #2

You can force the OS to kill TIME_WAITs for mysql or any other app with the following:
SEC_TO_TIMEWAIT=1
echo ${SEC_TO_TIMEWAIT} > /proc/sys/net/ipv4/tcp_tw_recycle
echo ${SEC_TO_TIMEWAIT} > /proc/sys/net/ipv4/tcp_tw_reuse
This will make TIME_WAITs time out in 1 second.

No comments:

Post a Comment