Common AWS SSL Errors in Python

Self signed certificate in certificate chain

While there are a number of reasons this error can happen, most of them are related to the process in which certificates are verified by Python. When a secure SSL/TLS connection is made, the certificate presented by the server is checked against a known list of certificates provided by a CA (certificate authority). Though the certificate may be cryptographically valid, if it is not found in the CA bundle it cannot be verified and will throw this error.

This error usually happens as a result of an enterprise using an SSL intercepting proxy, often the case with Deep Packet Inspection. In order to fix this, we need to add the intermediary certificates that are present and place them in our Certificate Authority file.

The following command will pull all of the intermediate certificates from your keychain on your Mac and add them into Python’s CA file.

security find-certificate -a -p ls /Library/Keychains/System.keychain >> $(python3 -c "import ssl; print(ssl.get_default_verify_paths().cafile)")

If you open the keychain access app on your Mac, you will see that the System section contains all of the certificates that have been added to your ca-file:

If you still encounter the error after these steps, you can reference the ca pem file directly like so:

import boto3
session = boto3.Session()
client = session.client(service_name='s3', verify='/path/to/cafile.pem')

Alternatively, you can disable certificate validation as outlined in the section below:

Unable to get local issuer certificate

[SSL: CERTIFICATE_VERIFIED_FAILED] certificate verify failed: unable to get local issuer certificate

You can get this error when your local CA store cannot be found either due to permission problems or because the file is outright missing. To see which CA file python is using, run the following command:

python3 -c "import ssl; print(ssl.get_default_verify_paths().cafile)"

If it is as simple as a permission issue, try setting the CA pem file to something less restrictive like so:

chmod 644

If you are outright missing a CA-file, keep reading, the next section will outline how to create one.

Creating a CA File for Python on MacOS

The easiest way to do this on a Mac is to build a CA bundle using the system’s key store as most corporate devices already contain the root and intermediary certificates needed to allow these connections.

The following command will pull all of the certificates from the system keychains and create a certificate bundle called ca-bundle.pem that Python can reference.

(security find-certificate -a -p ls /System/Library/Keychains/SystemRootCertificates.keychain && \
security find-certificate -a -p ls /Library/Keychains/System.keychain) \
> ca-bundle.pem

• You can then rename and place this file in the location Python is expecting it. The following command will give you the file name and path that Python is trying to load:

  python3 -c "import ssl; print(ssl.get_default_verify_paths().cafile)"
  

• Alternatively, you can configure boto3 to reference this newly created pem file directly when instantiating the session like so:

  import boto3
  session = boto3.Session()
  client = session.client(service_name='s3', verify='/path/to/cafile.pem')
  

Disable certificate verification in boto3

Though this is the easiest solution, it is also not recommended as you can put your application at risk for Man-in-the-Middle attacks.You can disable certificate validation via the boto3 client by first creating a session and then setting the verify parameter to False:

import boto3
session = boto3.Session()
client = session.client(service_name='s3', verify=False)

TLS/SSL connection has been closed (EOF)

botocore.exceptions.SSLError: SSL Validation failed for https://s3.amazonaws.com TLS/SSL connection has been closed (EOF)

This is the result of a proxy configuration error, usually related to the authentication credentials being passed to the proxy server. If either your username or password have special characters you will need to percent encode them: Please see the below section on how to configure your proxy for more details:

EOF occurred in violation of protocol

botocore.exceptions.SSLError: SSL validation failed for https://dynamodb.us-east-1.amazonaws.com/ EOF occured in violation of protocol

This is almost always a proxy or port issue. It means you were attempting to communicate via TLS (HTTPS) to an HTTP endpoint. This can happen when you specify the wrong port number, or more frequently there is an enterprise proxy blocking the request.

These proxies often communicate via HTTP for performance reasons so you don’t need two TLS handshakes per connection. This can happen in the following situations:

• you should be using the proxy but are bypassing it
• you should not be using the proxy but are using it

You can configure the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables as appropriate to match your environment. For example, if you want to bypass the proxy for AWS-specific traffic you can do the following:

export NO_PROXY="s3.amazonaws.com,.s3.amazonaws.com"

If you want to route specific traffic through a proxy, you can configure the proxy settings like so:

export HTTPS_PROXY="http://my-enterprise-proxy.com:8080"
export HTTP_PROXY="http://my-enterprise-proxy.com:8080"

If your proxy requires a username and password, you can provide them like so:

export HTTPS_PROXY="http://username:password@my-enterprise-proxy.com:8080"
export HTTP_PROXY="http://username:password@my-enterprise-proxy.com:8080"

If your username or password contains any symbols using the following command to url encode (percent escape) them.

For example, I can encode mypassword!@: in the following way: Note the back slashes in front of the special characters

python -c "from urllib.parse import quote; print(quote('mypassword\!\@\:'));"

I get a result of mypassword%21%5C%40%5C%3A - it can be used in the Proxy URL without any issues.

export HTTPS_PROXY="http://username:mypassword%21%5C%40%5C%3A@my-enterprise-proxy.com:8080"

If you don’t want to use an environment variable, you can also configure the proxy for AWS using a Config class in the boto3 library like so:

import boto3
from botocore.config import ConfigParser

config = Config(proxies={http:'http://my-enterprise-proxy.com:8080', https:'http://my-enterprise-proxy.com:8080'})
dynamodb = boto3.resource('dynamodb', region_name='us-east-1', config=config)
results = dynamodb.table('my-dynamodb-table-name').scan()
print(results)