https.txt tests fail

[chaudum]

Instead of an SSL exception the test fails when trying to establish a connection

Failure in test /home/christian/sandbox/crate/crate-python/src/crate/client/doctests/https.txt
Failed doctest test for https.txt
  File "/home/christian/sandbox/crate/crate-python/src/crate/client/doctests/https.txt", line 0

----------------------------------------------------------------------
File "/home/christian/sandbox/crate/crate-python/src/crate/client/doctests/https.txt", line 80, in https.txt
Failed example:
    client.server_infos(crate_host)
Expected:
    Traceback (most recent call last):
    ...
    crate.client.exceptions.ConnectionError: Server not available, exception: ...[SSL: ...
Got:
    Traceback (most recent call last):
      File "/usr/lib/python3.8/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest https.txt[11]>", line 1, in <module>
        client.server_infos(crate_host)
      File "/home/christian/sandbox/crate/crate-python/src/crate/client/http.py", line 340, in server_infos
        response = self._request('GET', '/', server=server)
      File "/home/christian/sandbox/crate/crate-python/src/crate/client/http.py", line 439, in _request
        raise ConnectionError(
    crate.client.exceptions.ConnectionError: Server not available, exception: HTTPSConnectionPool(host='localhost', port=65534): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f0779d09730>: Failed to establish a new connection: [Errno 110] Connection timed out'))

[amotl]

This error is tripping again on CI, coming from #397, see [1].

[1] https://github.com/crate/crate-python/runs/2115670144?check_suite_focus=true#step:5:156

[amotl]

On my machine, when running src/crate/client/doctests/https.txt in isolation, everything works without any hiccup.

Just to share an observation about this. When running the https.txt doctest on my machine, this warning appears at the end of the test run:

sys:1: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 65534)>

Others are observing the same thing when HTTP requests are issued within a unittest environment, see [1,2]. I tried different countermeasures but haven't been able to resolve it.

[1] https://stackoverflow.com/questions/48160728/resourcewarning-unclosed-socket-in-python-3-unit-test
[2] psf/requests#3912

[amotl]

While I tried to get hold of this nasty error on behalf of #397, I still haven't been able to resolve it. @mfussenegger told me that #157 might be able to resolve it.

Nevertheless, I am inclined to skip the relevant single test within https.txt on behalf of the cleanup and modernization patch coming from #397 until we find a solution for it in order to get a more sane response from CI regarding the major parts of the test harness. What do you think about this?

cc @seut

[amotl]

I improved the test cases within https.txt on behalf of bf11e61, and, while everything works flawlessly when running those tests in isolation on my macOS machine, invoking them on Linux within a Docker container blocks the execution completely:

Setup

docker run -it --rm python:3.9.2 bash

python3 -m venv .venv
source .venv/bin/activate

# Workaround for Python 3.9
python -m pip install --upgrade "setuptools>=31,<51"

pip install zc.buildout==2.13.4

buildout -n

Invoke

./bin/test -vvvv --ignore_dir=testing --buffer
Running tests at level 1
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
  Running:
 /src/src/crate/client/doctests/cursor.txt (0.006 s)
 /src/src/crate/client/doctests/connection.txt (0.003 s)
  Ran 2 tests with 0 failures, 0 errors, 0 skipped in 0.009 seconds.
Running crate.client.tests.httpsserver tests:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
  Set up crate.client.tests.httpsserver listening on localhost 65534
in 0.504 seconds.
  Running:
 /src/src/crate/client/doctests/https.txt

Outcome

After 150s of execution time, the response is:

----------------------------------------------------------------------
File "/src/src/crate/client/doctests/https.txt", line 93, in https.txt
Failed example:
    client.server_infos(crate_host)
Expected:
    Traceback (most recent call last):
    ...
    crate.client.exceptions.ConnectionError: Server not available, exception: ...[SSL: ...
Got:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest https.txt[13]>", line 1, in <module>
        client.server_infos(crate_host)
      File "/src/src/crate/client/http.py", line 402, in server_infos
        response = self._request('GET', '/', server=server)
      File "/src/src/crate/client/http.py", line 501, in _request
        raise ConnectionError(
    crate.client.exceptions.ConnectionError: Server not available, exception: HTTPSConnectionPool(host='localhost', port=65534): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff5ca50e1c0>, 'Connection to localhost timed out. (connect timeout=5)'))

----------------------------------------------------------------------
File "/src/src/crate/client/doctests/https.txt", line 101, in https.txt
Failed example:
    client.server_infos(crate_host)
Expected:
    Traceback (most recent call last):
    ...
    crate.client.exceptions.ConnectionError: Server not available, exception: ...[SSL: ...
Got:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/doctest.py", line 1336, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest https.txt[15]>", line 1, in <module>
        client.server_infos(crate_host)
      File "/src/src/crate/client/http.py", line 402, in server_infos
        response = self._request('GET', '/', server=server)
      File "/src/src/crate/client/http.py", line 501, in _request
        raise ConnectionError(
    crate.client.exceptions.ConnectionError: Server not available, exception: HTTPSConnectionPool(host='localhost', port=65534): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff5ca49c2b0>, 'Connection to localhost timed out. (connect timeout=5)'))

Evaluation

So, it looks like that, on Linux, dialing to an SSL-enabled host with invalid client certificate configuration, will stall the SSL handshake in one way or another. At least, we observe this behavior when connecting to Python's http.server.HTTPServer implementation as configured within tests.py as HttpsTestServerLayer. I am running Python 3.9.2 on both environments (macOS native, Linux on Docker).

[amotl]

I have been able to narrow this down to this single test where an invalid CA key/cert file is used to emulate an invalid client certificate.

When using an invalid client certificate, the connection will fail::

    >>> client = HttpClient([crate_host], ca_cert=cacert_valid, cert_file=cacert_invalid, key_file=cacert_invalid, timeout=5)
    >>> client.server_infos(crate_host)
    Traceback (most recent call last):
    ...
    crate.client.exceptions.ConnectionError: Server not available, exception: ...[SSL: ...

When removing this single test, the whole suite within https.txt completes successfully.

[amotl]

This has finally been resolved with 9d309e1 on behalf of #397 by providing correct pairs of valid vs. invalid client certificates to the respective tests which were failing before.