Custom server certificate

[msnelling]

How do we get the agent to accept successfully connect to the server using HTTPS if it is using a certificate signed by an internal/unknown CA?

Currently I'm having to connect over HTTP which is less than ideal.

[dtretyakov]

@msnelling, solution depends on docker image platform and tool which you use to establish connection to the HTTPS host.

TeamCity 2018.1 itself provides self-signed certificates management for bundled build runners:
https://confluence.jetbrains.com/display/TCDL/Uploading+SSL+Certificates

[msnelling]

@dtretyakov this is for the agent to connect back to the TeamCity server over HTTPS on startup.

[dtretyakov]

@msnelling, in this case you need to populate <TeamCity Agent Home>/conf/trustedCertificates directory by using one of the following ways:

• place the custom certificate to some directory and then mount it to the build agent
• create a docker image based on jetbrains/teamcity-agent and place custom certificate in it

[msnelling]

Thanks @dtretyakov, I tried the first option (my preferred) with no luck. I have the CA certificate in the directory /var/lib/teamcity/conf/trustedCertificates and started the agent container with

docker run -it \
  -e AGENT_NAME=docker-linux-full \
  -e SERVER_URL="https://teamcity.mydomain.com" \
  -v /var/lib/teamcity/conf:/data/teamcity_agent/conf \
  jetbrains/teamcity-agent

... but this doesn't work. I still get the error

Error while asking server for the communication protocols via URL https://teamcity.mydomain.com/app/agents/protocols. Will try later: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (enable debug to see stacktrace)

[dtretyakov]

@atroxaper, please take a look at the issue.

[atroxaper]

Hello, @msnelling. At first glance, it seems that you did everything right. Please check that your certificate is stored in one of the following formats: PEM, DER or PKCS#7.

For ensuring all paths are correct, you can try to do the following:

1. Connect the agent to the build server over http as you did it above;
2. Add any valid certificate to the build server as described in section Adding trusted certificates to TeamCity server;
3. Start any build on the agent;
4. After build finished you should observe the certificate under /data/teamcity_agent/system/serverTrustedCertificates.

[weyert]

@msnelling Did you ever get this working?

[RonAmihai]

I'm having the exact same issue as @msnelling described.

My TeamCity server is running behind nginx proxy & exposed as HTTPS.
I've mounted the certificates into the teamcity-agent using the conf/trustedCertificates volume.

Did you manage to solve this?

[imlight]

+1 , has anyone got this working ?

[imlight]

for me , inside container below command execute successfully
curl -iv https://TCserver:8443/app/agents/protocols

It just during agent registration i get the error

2020-02-26 05:15:56,400]   WARN - buildServer.AGENT.registration - Error while asking server for the communication protocols via URL https://TCserver:8443/app/agents/protocols. Will try later: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (enable debug to see stacktrace)

[imlight]

Alright this is what we had to do to make teamcity build agent work with https

1. Copy /opt/buildagent/bin/agent.sh from inside container to host ( docker host )

2. Edit the  "TEAMCITY_AGENT_OPTS_ACTUAL"  and mention your keystore file path which gets mounted inside container, in mu case it looked like this 

TEAMCITY_AGENT_OPTS_ACTUAL="$TEAMCITY_AGENT_OPTS -ea $TEAMCITY_AGENT_MEM_OPTS_ACTUAL -Dteamcity_logs=$LOG_DIR/ -Djavax.net.ssl.keyStore=/data/teamcity_agent/conf/trustedCertificates/keystore.jks -Djavax.net.ssl.trustStore=/data/teamcity_agent/conf/trustedCertificates/keystore.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStorePassword=changeit"

3. Edit your Dockerfile and add this line 

COPY agent.sh /opt/buildagent/bin/agent.sh

4. Build the image 

5. And run the container with the above builded image

Just to verfiy , from docker host or inside container , run ps -ef | grep java and make sure java process gets started with java keystore.

[kir]

@imlight Have you tried passing environment variable TEAMCITY_AGENT_OPTS when starting the container, like -e 'TEAMCITY_AGENT_OPTS=-Djavax.net.ssl.keyStore=/data/teamcity_agent/conf/trustedCertificates/keystore.jks -Djavax.net.ssl.trustStore=/data/teamcity_agent/conf/trustedCertificates/keystore.jks -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStorePassword=changeit' ?

[imlight]

Thats a good suggestion,
something i look forward to try out on weekend. Will update.

[imlight]

@kir That works like a charm mate! Cheers!

[vssubs]

@msnelling, in this case you need to populate <TeamCity Agent Home>/conf/trustedCertificates directory by using one of the following ways:

* place the custom certificate to some directory and then mount it to the build agent

* create a docker image based on `jetbrains/teamcity-agent` and place custom certificate in it

Is this directory mean to exists ? Or do you need to create it manually ? I cannot find this on any teamcity install I have ... ?

[anthonypburns]

@msnelling, in this case you need to populate <TeamCity Agent Home>/conf/trustedCertificates directory by using one of the following ways:

* place the custom certificate to some directory and then mount it to the build agent

* create a docker image based on `jetbrains/teamcity-agent` and place custom certificate in it

Is this directory mean to exists ? Or do you need to create it manually ? I cannot find this on any teamcity install I have ... ?

As far as I've found the directory does not exist. I was able to get a docker image based on jetbrains/teamcity-agent to work with a required intermediate cert by using ADD in a dockerfile:
ADD yourcert.pem /data/teamcity_agent/conf/trustedCertificates/yourcert.pem

Unfortunately I can't figure out how to make this work on Windows as I'm trying to do now X:/BuildAgent/conf/trustedCertificates doesn't seem to do the trick there. And there's no JetBrains or teamcity_agent entry into the hidden ProgramData folder either 🤷‍♀️

EDIT: I should note that you can get this working on Windows by importing into the Java Keystore directly using the keytool, but just dropping a file in seemed cleaner and more producible via non-manual means.

[jwalzer]

So, to summarize as I understand it: It is still not possible to use the stock containers by simply dropping a file into conf/trustedCertificates ?
Is it still needed to wiggle with TEAMCITY_AGENT_OPTS to get this working?

as of now I'm stuck in the same issue as above and what I see my evaluation of TC will fail on this :(

[alec-drw]

Old but I just ran into the same problem. This is applicable on the jetbrains/teamcity-agent:2021.1.2-linux-sudo image. I was able to fix it as follows:

Firstly, you need to grab agent.sh from /opt/buildagent/bin/agent.sh from the Dockerfile, as you will need to make two edits.

Changes to agent.sh:

TEAMCITY_AGENT_OPTS_ACTUAL="$TEAMCITY_AGENT_OPTS -ea $TEAMCITY_AGENT_MEM_OPTS_ACTUAL -Dteamcity_logs=$LOG_DIR/ -Djavax.net.ssl.keyStore=/opt/java/openjdk/jre/lib/security/cacerts -Djavax.net.ssl.trustStore=/opt/java/openjdk/jre/lib/security/cacerts -Djavax.net.ssl.keyStorePassword=changeit -Djavax.net.ssl.trustStorePassword=changeit"

And within the start|run) function of agent.sh:

keytool -importcert -noprompt -alias mycert -file /data/teamcity_agent/conf/trustedCertificates/mycert.crt \
        -keystore /opt/java/openjdk/jre/lib/security/cacerts -storepass changeit

Finally, copy the cert and updated agent.sh file to the Dockerfile:

RUN mkdir -p /data/teamcity_agent/conf/trustedCertificates
COPY certs/mycert /data/teamcity_agent/conf/trustedCertificates/mycert.crt
COPY agent.sh /opt/buildagent/bin/agent.sh

[approximate]

There is an alternative solution that doesn't require patching Dockerfiles or images.

[tlunsfordCXP]

It's this sort of core missing functionality that is pushing me to look for an alternative to TeamCity. It's still not simple after 5+ years. It's more and more important to make this simple, as security tools like zscaler make this a very common scenario.

[ron-tayler]

A solution that helps with almost all images except TeamCity:
Add a certificate with the extension .crt on Docker Host to the folder /usr/local/share/ca-certificates. And run update-ca-certificates

After launching both Docker Standalone and Docker Swarm, you can simply add Volumes /etc/ssh:/etc/ssh:ro.