DynamoDB SDK ver1.5 hangs on CLOSE_WAIT after moving to instance to HVM with Curl 7.61 installed

by user3305146   Last Updated July 01, 2019 01:26 AM

I have and old code working fine on this environment:

Curl 7.19, php 5.3.17 AWS EC2 PV instance DynamoDB SDK ver 1.5

I do realize the SDK is ancient, but we need this code to work for another few months.

After moving the code to new instance (AWS EC2 HVM instance) the code started to be unstable. Every day 2-3 times, randomly and unexpectedly, the code hangs. After adding some debug we noticed a pattern:

  1. dynamodb timeouts
  2. tcp stuck in CLOSE_WAIT
  3. CPU usage goes 100%

The main difference I noticed between old and new instance is Curl version. I tried to get any information about know issues according Curl 7.61 and older DynamoDb SDK for php, but couldn't find anything.

I want to ask if anyone experienced similar problems and what might be the cause?

We tried to move the code back to PV instance, which worked fine. The code works here as expected.

Exact Curl versions and errors:

old PV instance

curl --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.12.9.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

new HVM instance

curl --version
curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.0.2k zlib/1.2.8 libidn2/0.16 libpsl/0.6.2 (+libicu/50.1.2) libssh2/1.4.2 nghttp2/1.21.1
Release-Date: 2018-09-05
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy PSL

"timeout error" that we found in logs

[2019-06-26 08:05:42][30661] * Connection #2 to host dynamodb.ap-northeast-1.amazonaws.com left intact
[2019-06-26 08:05:42][30661] * Operation timed out after 120000 milliseconds with 0 out of 0 bytes received
[2019-06-26 08:05:42][30661] * Closing connection 1
[2019-06-26 08:05:42][30661] <div style="border:1px solid #990000;padding-left:20px;margin:0 0 10px 0;">
[2019-06-26 08:05:42][30661] 
[2019-06-26 08:05:42][30661] <h4>A PHP Error was encountered</h4>
[2019-06-26 08:05:42][30661] 
[2019-06-26 08:05:42][30661] <p>Severity: Warning</p>
[2019-06-26 08:05:42][30661] <p>Message:  curl_multi_add_handle(): 110 is not a valid cURL handle resource</p>
[2019-06-26 08:05:42][30661] <p>Filename: requestcore/requestcore.class.php</p>
[2019-06-26 08:05:42][30661] <p>Line Number: 879</p>


Related Questions