网络编程第四讲-长连接的常见问题

, , Comments

这篇文章是总结自Swoole微课程《网络编程第四讲-长连接的常见问题》。这一节学习到了很多的知识，可以说干货满满。

连接失效问题

例子

其中，Redis常见的报错就是：

1
2
配置项：timeout
报错信息：Error while reading line from the server

Redis可以配置如果客户端经过多少秒还不给Redis服务器发送数据，那么就会把连接close掉。

MySQL常见的报错：

1
2
配置项：wait_timeout & interactive_timeout
报错信息：has gone away

和Redis服务器一样，MySQL也会定时的去清理掉没用的连接。

如何解决

1
2
1、用的时候进行重连
2、定时发送心跳维持连接

用的时候进行重连

优点是简单，缺点是面临短连接的问题。

定时发送心跳维持连接

推荐。

如何维持长连接

tcp协议中实现的tcp_keepalive

操作系统底层提供了一组tcp的keepalive配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
tcp_keepalive_time (integer; default: 7200; since Linux 2.2)
        The number of seconds a connection needs to be idle before TCP
        begins sending out keep-alive probes.  Keep-alives are sent only
        when the SO_KEEPALIVE socket option is enabled.  The default
        value is 7200 seconds (2 hours).  An idle connection is
        terminated after approximately an additional 11 minutes (9
        probes an interval of 75 seconds apart) when keep-alive is
        enabled.       
                        
        Note that underlying connection tracking mechanisms and
        application timeouts may be much shorter.
        
tcp_keepalive_intvl (integer; default: 75; since Linux 2.4)
        The number of seconds between TCP keep-alive probes.
                        
tcp_keepalive_probes (integer; default: 9; since Linux 2.2)
        The maximum number of TCP keep-alive probes to send before
        giving up and killing the connection if no response is obtained
        from the other end.

Swoole底层把这些配置开放出来了，例如：

1
2
3
4
5
6
7
8
9
10
11
<?php

$server = new \Swoole\Server('127.0.0.1', 6666, SWOOLE_PROCESS);

$server->set([
    'worker_num' => 1,
    'open_tcp_keepalive' => 1,
    'tcp_keepidle' => 4, // 对应tcp_keepalive_time
    'tcp_keepinterval' => 1, // 对应tcp_keepalive_intvl
    'tcp_keepcount' => 5, // 对应tcp_keepalive_probes
]);

其中：

1
2
3
4
5
'open_tcp_keepalive' => 1, // 总开关，用来开启tcp_keepalive
'tcp_keepidle' => 4, // 4s没有数据传输就进行检测
// 检测的策略如下：
'tcp_keepinterval' => 1, // 1s探测一次，即每隔1s给客户端发一个包（然后客户端可能会回一个ack的包，如果服务端收到了这个ack包，那么说明这个连接是活着的）
'tcp_keepcount' => 5, // 探测的次数，超过5次后客户端还没有回ack包，那么close此连接

我们来实战测试体验一下，服务端脚本如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<?php

$server = new \Swoole\Server('127.0.0.1', 6666, SWOOLE_PROCESS);

$server->set([
    'worker_num' => 1,
    'open_tcp_keepalive' => 1, // 开启tcp_keepalive
    'tcp_keepidle' => 4, // 4s没有数据传输就进行检测
    'tcp_keepinterval' => 1, // 1s探测一次
    'tcp_keepcount' => 5, // 探测的次数，超过5次后还没有回包close此连接
]);

$server->on('connect', function ($server, $fd) {
    var_dump("Client: Connect $fd");
});

$server->on('receive', function ($server, $fd, $reactor_id, $data) {
    var_dump($data);
});

$server->on('close', function ($server, $fd) {
    var_dump("close fd $fd");
});

$server->start();

我们启动这个服务器：

1
2
~/codeDir/phpCode/hyperf-skeleton # php server.php

然后通过tcpdump进行抓包：

1
2
3
4
~/codeDir/phpCode/hyperf-skeleton # tcpdump -i lo port 6666
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes

我们此时正在监听lo上的6666端口的数据包。

然后我们用客户端去连接它：

1
2
~/codeDir/phpCode/hyperf-skeleton # nc 127.0.0.1 6666

此时服务端会打印出消息：

1
2
3
~/codeDir/phpCode/hyperf-skeleton # php server.php 
string(17) "Client: Connect 1"

tcpdump的输出信息如下：

1
2
3
4
5
6
7
8
9
01:48:40.178439 IP localhost.33933 > localhost.6666: Flags [S], seq 43162537, win 43690, options [mss 65495,sackOK,TS val 9833698 ecr 0,nop,wscale 7], length 0
01:48:40.178484 IP localhost.6666 > localhost.33933: Flags [S.], seq 1327460565, ack 43162538, win 43690, options [mss 65495,sackOK,TS val 9833698 ecr 9833698,nop,wscale 7], length 0
01:48:40.178519 IP localhost.33933 > localhost.6666: Flags [.], ack 1, win 342, options [nop,nop,TS val 9833698 ecr 9833698], length 0
01:48:44.229926 IP localhost.6666 > localhost.33933: Flags [.], ack 1, win 342, options [nop,nop,TS val 9834104 ecr 9833698], length 0
01:48:44.229951 IP localhost.33933 > localhost.6666: Flags [.], ack 1, win 342, options [nop,nop,TS val 9834104 ecr 9833698], length 0
01:48:44.229926 IP localhost.6666 > localhost.33933: Flags [.], ack 1, win 342, options [nop,nop,TS val 9834104 ecr 9833698], length 0
01:48:44.229951 IP localhost.33933 > localhost.6666: Flags [.], ack 1, win 342, options [nop,nop,TS val 9834104 ecr 9833698], length 0
01:48:44.229926 IP localhost.6666 > localhost.33933: Flags [.], ack 1, win 342, options [nop,nop,TS val 9834104 ecr 9833698], length 0
// 省略了其他的输出

我们会发现最开始的时候，会打印三次握手的包：

1
2
3
01:48:40.178439 IP localhost.33933 > localhost.6666: Flags [S], seq 43162537, win 43690, options [mss 65495,sackOK,TS val 9833698 ecr 0,nop,wscale 7], length 0
01:48:40.178484 IP localhost.6666 > localhost.33933: Flags [S.], seq 1327460565, ack 43162538, win 43690, options [mss 65495,sackOK,TS val 9833698 ecr 9833698,nop,wscale 7], length 0
01:48:40.178519 IP localhost.33933 > localhost.6666: Flags [.], ack 1, win 342, options [nop,nop,TS val 9833698 ecr 9833698], length 0

然后，停留了4s没有任何包的输出。

之后，每隔1s左右就会打印出一组：

1
2
01:52:54.359341 IP localhost.6666 > localhost.43101: Flags [.], ack 1, win 342, options [nop,nop,TS val 9859144 ecr 9858736], length 0
01:52:54.359377 IP localhost.43101 > localhost.6666: Flags [.], ack 1, win 342, options [nop,nop,TS val 9859144 ecr 9855887], length 0

其实这就是我们配置的策略：

1
2
'tcp_keepinterval' => 1, // 1s探测一次
'tcp_keepcount' => 5, // 探测的次数，超过5次后还没有回包close此连接

因为我们操作系统底层会自动的给客户端回ack，所以这个连接不会在5次探测后被关闭。操作系统底层会持续不断的发送这样的一组包：

1
2
01:52:54.359341 IP localhost.6666 > localhost.43101: Flags [.], ack 1, win 342, options [nop,nop,TS val 9859144 ecr 9858736], length 0
01:52:54.359377 IP localhost.43101 > localhost.6666: Flags [.], ack 1, win 342, options [nop,nop,TS val 9859144 ecr 9855887], length 0

如果我们要测试5次探测后关闭这个连接，可以禁掉6666端口的包：

1
~/codeDir/phpCode/hyperf-skeleton # iptables -A INPUT -p tcp --dport 6666 -j DROP

这样会把所有从6666端口进来的包给禁掉，自然，服务器就接收不到从客户端那一边发来的ack包了。

然后服务器过5秒就会打印出close（服务端主动的调用了close方法，给客户端发送了FIN包）：

1
2
3
4
~/codeDir/phpCode/hyperf-skeleton # php server.php 
string(17) "Client: Connect 1"
string(10) "close fd 1"

我们恢复一下iptables的规则：

1
~/codeDir/phpCode # iptables -D INPUT -p tcp -m tcp --dport 6666 -j DROP

即把我们设置的规则给删除了。

通过tcp_keepalive的方式实现心跳的功能，优点是简单，不要写代码就可以完成这个功能，并且发送的心跳包小。缺点是依赖于系统的网络环境，必须保证服务器和客户端都实现了这样的功能，需要客户端配合发心跳包。还有一个更为严重的缺点是如果客户端和服务器不是直连的，而是通过代理来进行连接的，例如socks5代理，它只会转发应用层的包，不会转发更为底层的tcp探测包，那这个心跳功能就失效了。

所以，Swoole就提供了其他的解决方案，一组检测死连接的配置。

1
2
'heartbeat_check_interval' => 1, // 1s探测一次
'heartbeat_idle_time' => 5, // 5s未发送数据包就close此连接
swoole实现的heartbeat

我们来测试一下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<?php

$server = new \Swoole\Server('127.0.0.1', 6666, SWOOLE_PROCESS);

$server->set([
    'worker_num' => 1,
    'heartbeat_check_interval' => 1, // 1s探测一次
    'heartbeat_idle_time' => 5, // 5s未发送数据包就close此连接
]);

$server->on('connect', function ($server, $fd) {
    var_dump("Client: Connect $fd");
});

$server->on('receive', function ($server, $fd, $reactor_id, $data) {
    var_dump($data);
});

$server->on('close', function ($server, $fd) {
    var_dump("close fd $fd");
});

$server->start();

然后启动服务器：

1
2
~/codeDir/phpCode/hyperf-skeleton # php server.php

然后启动tcpdump：

1
2
3
4
~/codeDir/phpCode # tcpdump -i lo port 6666
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes

然后再启动客户端：

1
2
~/codeDir/phpCode/hyperf-skeleton # nc 127.0.0.1 6666

此时服务器端打印：

1
2
~/codeDir/phpCode/hyperf-skeleton # php server.php 
string(17) "Client: Connect 1"

然后tcpdump打印：

1
2
3
02:48:32.516093 IP localhost.42123 > localhost.6666: Flags [S], seq 1088388248, win 43690, options [mss 65495,sackOK,TS val 10193342 ecr 0,nop,wscale 7], length 0
02:48:32.516133 IP localhost.6666 > localhost.42123: Flags [S.], seq 80508236, ack 1088388249, win 43690, options [mss 65495,sackOK,TS val 10193342 ecr 10193342,nop,wscale 7], length 0
02:48:32.516156 IP localhost.42123 > localhost.6666: Flags [.], ack 1, win 342, options [nop,nop,TS val 10193342 ecr 10193342], length 0

这是三次握手信息。

然后过了5s后，tcpdump会打印出：

1
2
02:48:36.985027 IP localhost.6666 > localhost.42123: Flags [F.], seq 1, ack 1, win 342, options [nop,nop,TS val 10193789 ecr 10193342], length 0
02:48:36.992172 IP localhost.42123 > localhost.6666: Flags [.], ack 2, win 342, options [nop,nop,TS val 10193790 ecr 10193789], length 0

也就是服务端发送了FIN包。因为客户端没有发送数据，所以Swoole关闭了连接。

然后服务器端会打印：

1
2
3
4
~/codeDir/phpCode/hyperf-skeleton # php server.php 
string(17) "Client: Connect 1"
string(10) "close fd 1"

所以，heartbeat和tcp keepalive还是有一定的区别的，tcp keepalive有保活连接的功能，但是heartbeat存粹是检测没有数据的连接，然后关闭它，并且只可以在服务端这边配置，如果需要保活，也可以让客户端配合发送心跳。

如果我们不想让服务端close掉连接，那么就得在应用层里面不断的发送数据包来进行保活，例如我在nc客户端里面不断的发送包：

1
2
3
4
5
6
7
8
9
10
11
~/codeDir/phpCode/hyperf-skeleton # nc 127.0.0.1 6666
ping
ping
ping
ping
ping
ping
ping
ping
ping

我发送了9个ping包给服务器，tcpdump的输出如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// 省略了三次握手的包
02:57:53.697363 IP localhost.44195 > localhost.6666: Flags [P.], seq 1:6, ack 1, win 342, options [nop,nop,TS val 10249525 ecr 10249307], length 5
02:57:53.697390 IP localhost.6666 > localhost.44195: Flags [.], ack 6, win 342, options [nop,nop,TS val 10249525 ecr 10249525], length 0
02:57:55.309532 IP localhost.44195 > localhost.6666: Flags [P.], seq 6:11, ack 1, win 342, options [nop,nop,TS val 10249686 ecr 10249525], length 5
02:57:55.309576 IP localhost.6666 > localhost.44195: Flags [.], ack 11, win 342, options [nop,nop,TS val 10249686 ecr 10249686], length 0
02:57:58.395206 IP localhost.44195 > localhost.6666: Flags [P.], seq 11:16, ack 1, win 342, options [nop,nop,TS val 10249994 ecr 10249686], length 5
02:57:58.395239 IP localhost.6666 > localhost.44195: Flags [.], ack 16, win 342, options [nop,nop,TS val 10249994 ecr 10249994], length 0
02:58:01.858094 IP localhost.44195 > localhost.6666: Flags [P.], seq 16:21, ack 1, win 342, options [nop,nop,TS val 10250341 ecr 10249994], length 5
02:58:01.858126 IP localhost.6666 > localhost.44195: Flags [.], ack 21, win 342, options [nop,nop,TS val 10250341 ecr 10250341], length 0
02:58:04.132584 IP localhost.44195 > localhost.6666: Flags [P.], seq 21:26, ack 1, win 342, options [nop,nop,TS val 10250568 ecr 10250341], length 5
02:58:04.132609 IP localhost.6666 > localhost.44195: Flags [.], ack 26, win 342, options [nop,nop,TS val 10250568 ecr 10250568], length 0
02:58:05.895704 IP localhost.44195 > localhost.6666: Flags [P.], seq 26:31, ack 1, win 342, options [nop,nop,TS val 10250744 ecr 10250568], length 5
02:58:05.895728 IP localhost.6666 > localhost.44195: Flags [.], ack 31, win 342, options [nop,nop,TS val 10250744 ecr 10250744], length 0
02:58:07.150265 IP localhost.44195 > localhost.6666: Flags [P.], seq 31:36, ack 1, win 342, options [nop,nop,TS val 10250870 ecr 10250744], length 5
02:58:07.150288 IP localhost.6666 > localhost.44195: Flags [.], ack 36, win 342, options [nop,nop,TS val 10250870 ecr 10250870], length 0
02:58:08.349124 IP localhost.44195 > localhost.6666: Flags [P.], seq 36:41, ack 1, win 342, options [nop,nop,TS val 10250990 ecr 10250870], length 5
02:58:08.349156 IP localhost.6666 > localhost.44195: Flags [.], ack 41, win 342, options [nop,nop,TS val 10250990 ecr 10250990], length 0
02:58:09.906223 IP localhost.44195 > localhost.6666: Flags [P.], seq 41:46, ack 1, win 342, options [nop,nop,TS val 10251145 ecr 10250990], length 5
02:58:09.906247 IP localhost.6666 > localhost.44195: Flags [.], ack 46, win 342, options [nop,nop,TS val 10251145 ecr 10251145], length 0

有9组数据包的发送。（这里的Flags [P.]代表Push的含义）

此时服务器还没有close掉连接，实现了客户端保活连接的功能。然后我们停止发送ping，过了5秒后tcpdump就会输出一组：

1
2
02:58:14.811761 IP localhost.6666 > localhost.44195: Flags [F.], seq 1, ack 46, win 342, options [nop,nop,TS val 10251636 ecr 10251145], length 0
02:58:14.816420 IP localhost.44195 > localhost.6666: Flags [.], ack 2, win 342, options [nop,nop,TS val 10251637 ecr 10251636], length 0

服务端那边发送了FIN包，说明服务端close掉了连接。服务端的输出如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
~/codeDir/phpCode/hyperf-skeleton # php server.php 
string(17) "Client: Connect 1"
string(5) "ping
"
string(5) "ping
"
string(5) "ping
"
string(5) "ping
"
string(5) "ping
"
string(5) "ping
"
string(5) "ping
"
string(5) "ping
"
string(5) "ping
"
string(10) "close fd 1"

然后我们在客户端那边ctrl + c来关闭连接：

1
2
3
4
5
6
7
8
9
10
11
12
13
~/codeDir/phpCode/hyperf-skeleton # nc 127.0.0.1 6666
ping
ping
ping
ping
ping
ping
ping
ping
ping
^Cpunt!

~/codeDir/phpCode/hyperf-skeleton #

此时，tcpdump的输出如下：

1
2
03:03:02.257667 IP localhost.44195 > localhost.6666: Flags [F.], seq 46, ack 2, win 342, options [nop,nop,TS val 10280414 ecr 10251636], length 0
03:03:02.257734 IP localhost.6666 > localhost.44195: Flags [R], seq 2678621620, win 0, length 0
应用层心跳
1
2
3
1、制定ping/pong协议（mysql等自带ping协议）
2、客户端灵活的发送ping心跳包
3、服务端OnRecive检查可用性回复pong

例如：

1
2
3
4
5
6
7
8
9
10
$server->on('receive', function (\Swoole\Server $server, $fd, $reactor_id, $data)
{
    if ($data == 'ping')
    {
        checkDB();
        checkServiceA();
        checkRedis();
        $server->send('pong');
    }
});

结论

1
2
3
1、tcp的keepalive最简单，但是有兼容性问题，不够灵活
2、swoole提供的keepalive最实用，但是需要客户端配合，复杂度适中
3、应用层的keepalive最灵活但是最麻烦

codinghuangPHP Developer

PHP开发者~