ADSL slowness cause

[cah]

I felt tired today and stayed and worked from home.

I experienced the slowness from ADSL again (well, starting yesterday actually).
I checked the network packet count and certainly it was high on hsiao.net.

From my past experience, if I disconnect network cable from hsiao.net, the network connection speed would be normal for a few minutes and then goes back to sluggish state. I tried to disconnect the Ethernet cable from hsiao.net and check packet count and it confirmed the past experience.

Code: Select all

%netstat -i -I rtls4 60
    input   rtls4     output       input  (Total)    output
packets errs  packets errs  colls  packets errs  packets errs  colls 
41110341 0     40251160 0     0      67613094 0     44594086 0     0     
274     0     13      0     0      1221    0     180     0     0     
141     0     133     0     0      182     0     135     0     0     
29      0     16      0     0      79      0     17      0     0     
18      0     5       0     0      69      0     17      0     0     
62      0     46      0     0      100     0     48      0     0     
277     0     16      0     0      1068    0     19      0     0     
672     0     1015    0     0      731     0     1029    0     0     
1678    0     2352    0     0      1717    0     2352    0     0     
1687    0     2397    0     0      1740    0     2412    0     0     
1677    0     2357    0     0      1718    0     2359    0     0     
1997    0     2437    0     0      2858    0     2513    0     0     
1528    0     2029    0     0      1637    0     2104    0     0     
1545    0     2153    0     0      1595    0     2162    0     0     
1663    0     2287    0     0      1721    0     2296    0     0     
1648    0     2400    0     0      1684    0     2400    0     0     
1970    0     2342    0     0      2758    0     2346    0     0     
1774    0     2406    0     0      1815    0     2407    0     0

After 6, 7 minutes, the packet count increased dramatically.

I then tried to find out why this is happening by using snoop command.

Code: Select all

%snoop -d rtls4

From the output I found a lot of SMTP connection to Yahoo's mail server.
I then checked my mail queue:

Code: Select all

%mailq
                /var/spool/mqueue (1 request)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
o5U6Zs8i019175* 9163950 Wed Jun 30 02:35 <cah@hsiao.net>
                 (reply: read error from f.mx.mail.yahoo.com.)
                                         <sdawood@yahoo.com>

There was an email that was having issues communicating with Yahoo's mail server.
I then cleaned up the mail queue and the network came back to normal state.

It looks like the failure to communicate with Yahoo's mail server was causing the intermittent network slowness.
How very interesting!

After I cleaned up the mail queue and ran netstat for a few hours, I didn't see slowness any more.

I will do another experiment next time when I see the slowness from ADSL to see if this was a coincidence or it is the real root cause!

[cah]

The slowness occurred one more time yesterday and it is now confirmed that the slowness was caused by some failed mail sending sessions. Not only Yahoo but all other mail server communication error would cause the same behavior.

Here are the steps to correct the behavior.

1. Become root user by using sudo

Code: Select all

sudo su -

2. Check how many sendmail processes are running.

Code: Select all

%ps -ef | grep sendmail
   smmsp 14569     1   0   May 06 ?           0:08 /usr/lib/sendmail -Ac -q5m
   smmsp  2308     1   0   May 07 ?           0:07 /usr/lib/sendmail -Ac -q5m
    root 14566     1   0   May 06 ?           0:59 /usr/lib/sendmail -bd -q5m
    root  2305     1   0   May 07 ?           0:56 /usr/lib/sendmail -bd -q5m
   smmsp 19374     1   0   May 07 ?           0:07 /usr/lib/sendmail -Ac -q5m
    root 19376     1   0   May 07 ?           0:58 /usr/lib/sendmail -bd -q5m
    root  4835  2305   0 16:51:13 ?           0:00 /usr/lib/sendmail -bd -q5m

Normal situation, there should only be 6 processes (2 for each zone, global or non-global). Since there are 3 zones, hsiao.net, geminitwins.net and badmintonequipments.com, there are 6. Every sendmail process should have 1 as the GID (Group ID). The example above has an additional process that does not have 1 as its GID and that is the process to kill.

3. Check what this process is.

Code: Select all

%/usr/ucb/ps -axww | grep -v grep | grep 4835
  4835 ?        S  0:00 sendmail: ./o611Y8DV026875 nomail.tpe.yahoo.com.: user open

This proves this hung process was trying to communicate with Yahoo's mail server.

4. Check the mail queue

Code: Select all

%mailq

I had cleaned the mail queue and do not currently have examples here. This command should show a coordinating information with the above process information.

5. Kill the hung process

Code: Select all

%kill -9 4835

6. Clean the mail queue

Code: Select all

%cd /var/spool/mqueue
%rm *o611Y8DV026875 (dfo611Y8DV026875 and qfo611Y8DV026875)

I will update this document when I get another hung situation.

[cah]

However, it still does not explain why only sendmail in hsiao.net is causing the network slowness while other non-global zones do not see this issue when having the same or similar problems.

Both geminitwins.net and badmintonequipments.com have similar undelivered mail too but they did not have many input and output packets. Maybe it is because they both have much less traffic. I have add similar situation from these two NGZ later.

Still can't quiet figure out what is causing it.
Can't find any information on the internet either.