removing the container is asynchronous and causes exceptions

[robnagler]

Fix with #18. Here the remove happens, but doesn't get removed
immediately, and create_object gets a collision:

20:34:58 [I 2020-08-21 20:34:58.473 JupyterHub rsdockerspawner:532] slot_alloc: found slot=19 cname=/jupyter-robnagler pool=internal host=<redact>
20:34:58 rsdockerspawner/rsdockerspawner.py:201:__docker_client /srv/jupyterhub/docker_tls/<redact>/cacert.pem
20:34:58 [W 2020-08-21 20:34:58.566 JupyterHub dockerspawner:976] Removing container that should have been cleaned up: jupyter-robnagler (id: 7aafee3)
20:34:58 [I 2020-08-21 20:34:58.567 JupyterHub dockerspawner:815] Removing container 7aafee3fcbed0065b18bcada63762eae93df00a744d4cba9b145a90f09c4e699
20:34:58 [I 2020-08-21 20:34:58.582 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/robnagler (robnagler@<redact>) 1048.82ms
20:34:58 [I 2020-08-21 20:34:58.584 JupyterHub rsdockerspawner:605] free slot=19 cname=/jupyter-robnagler user=robnagler host=<redact>
20:34:58 [I 2020-08-21 20:34:58.590 JupyterHub rsdockerspawner:544] slot_alloc: allocated slot=19 cname=/jupyter-robnagler pool=internal host=<redact>
20:34:58 rsdockerspawner/rsdockerspawner.py:201:__docker_client /srv/jupyterhub/docker_tls/<redact>/cacert.pem
20:34:58 [E 2020-08-21 20:34:58.665 JupyterHub user:640] Unhandled error starting robnagler's server: 409 Client Error: Conflict ("Conflict. The container name "/jupyter-robnagler" is already in use by
 container "7aafee3fcbed0065b18bcada63762eae93df00a744d4cba9b145a90f09c4e699". You have to remove (or rename) that container to be able to reuse that name.")
20:34:58 [I 2020-08-21 20:34:58.675 JupyterHub dockerspawner:1107] Stopping container jupyter-robnagler (id: 7aafee3)
20:34:58 [I 2020-08-21 20:34:58.675 JupyterHub dockerspawner:815] Removing container 7aafee3fcbed0065b18bcada63762eae93df00a744d4cba9b145a90f09c4e699
20:34:58 [I 2020-08-21 20:34:58.701 JupyterHub pages:347] robnagler is pending stop
20:34:58 [I 2020-08-21 20:34:58.757 JupyterHub log:174] 200 GET /hub/spawn-pending/robnagler (robnagler@<redact>) 60.03ms
20:35:03 [I 2020-08-21 20:35:03.916 JupyterHub pages:347] robnagler is pending stop
20:35:08 [I 2020-08-21 20:35:08.652 JupyterHub pages:347] robnagler is pending stop
20:35:09 [I 2020-08-21 20:35:09.030 JupyterHub rsdockerspawner:605] free slot=19 cname=/jupyter-robnagler user=robnagler host=<redact>
20:35:09 [I 2020-08-21 20:35:09.111 JupyterHub dockerspawner:784] Container 'jupyter-robnagler' is gone
20:35:09 [E 2020-08-21 20:35:09.111 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /home/vagrant/.pyenv/version
s/3.7.2/envs/py3/lib/python3.7/site-packages/jupyterhub/handlers/base.py:845> exception=APIError(HTTPError('409 Client Error: Conflict for url: https://<redact>:2376/v1.40/containers/create?name=jupyter-robnagler'))> after timeout
20:35:09 Traceback (most recent call last):
20:35:09 tornado/gen.py", line 593, in error_callback
20:35:09 future.result()
20:35:09 jupyterhub/handlers/base.py", line 852, in finish_user_spawn
20:35:09 await spawn_future
20:35:09 jupyterhub/user.py", line 656, in spawn
20:35:09 raise e
20:35:09 jupyterhub/user.py", line 560, in spawn
20:35:09 url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
20:35:09 dockerspawner/dockerspawner.py", line 983, in start
20:35:09 obj = yield self.create_object()
20:35:09 rsdockerspawner/rsdockerspawner.py", line 102, in create_object
20:35:09 res = yield super().create_object(*args, **kwargs)
20:35:09 dockerspawner/dockerspawner.py", line 879, in create_object
20:35:09 obj = yield self.docker("create_container", **create_kwargs)
20:35:09 File "/home/vagrant/.pyenv/versions/3.7.2/lib/python3.7/concurrent/futures/_base.py", line 425, in result
20:35:09 return self.__get_result()
20:35:09 File "/home/vagrant/.pyenv/versions/3.7.2/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
20:35:09 raise self._exception
20:35:09 File "/home/vagrant/.pyenv/versions/3.7.2/lib/python3.7/concurrent/futures/thread.py", line 57, in run
20:35:09 result = self.fn(*self.args, **self.kwargs)
20:35:09 dockerspawner/dockerspawner.py", line 743, in _docker
20:35:09 return m(*args, **kwargs)
20:35:09 docker/api/container.py", line 430, in create_container
20:35:09 return self.create_container_from_config(config, name)
20:35:09 docker/api/container.py", line 441, in create_container_from_config
20:35:09 return self._result(res, True)
20:35:09 docker/api/client.py", line 267, in _result
20:35:09 self._raise_for_status(response)
20:35:09 docker/api/client.py", line 263, in _raise_for_status
20:35:09 raise create_api_error_from_http_exception(e)
20:35:09 docker/errors.py", line 31, in create_api_error_from_http_exception
20:35:09 raise cls(e, response=response, explanation=explanation)
20:35:09 docker.errors.APIError: 409 Client Error: Conflict ("Conflict. The container name "/jupyter-robnagler" is already in use by container "7aafee3fcbed0065b18bcada63762eae93df00a744d4cba9b145a90f09c4e699". You have to remove (or rename) that container to be able to reuse that name.")
20:35:09 [W 2020-08-21 20:35:09.120 JupyterHub dockerspawner:757] Container not found: jupyter-robnagler
20:35:09 ERROR:asyncio:Task exception was never retrieved

[e-carlin]

I tried to repeat this today with no luck. I'm going to implement polling for port close in remove_object.