A few months ago, we worked with a client to migrate websites from on-premises to Azure. As part of this migration, Jenkins was set up to do the Continuous Integration/Continuous Delivery. During the initial setup, we checked out Jenkins email notifications and they were working fine with their internal mail server.
One fine day, an email comes from the client stating that emails from Jenkins were not coming at all. Well, that’s strange…
The Diagnosis
I like troubleshooting quite a lot and the rule of thumb is to first ask what changed? While the initial diagnosis was going on, the client mentioned that their network team made upgrades to the mail server and now only TLSv1.2 & TLSv1.1 were supported. Earlier, Jenkins sent email using SSL. (Yep, you guessed it right…using the -Dmail.smtp.starttls.enable=true parameter
.)
The Frantic Search
As emails were part of the build workflow, downtime meant that the developers were not able to know whether the code they pushed worked or not and had to manually log into Jenkins and then keep on checking.
I found quite a few articles about the issue but all of them were using SSL for emails. I couldn’t find something specifically for Jenkins which enabled email sending over TLSv1.2.
Jenkins uses JavaMail to send out emails. After quite some searching around using my Google-Fu, I found an article over here via the WaybackMachine.
The interesting content of the post is as below:
We prepared two JVM versions, i.e. JDK 6u45 and JDK 7u71, and two JavaMail versions, i.e. 1.4.1 and 1.5.2. (JavaMail 1.5.2 is the latest released build as of now.) With any pair of JVM and JavaMail, the test was successful, only if the server supported TLS 1.0. Most probably, not only does JavaMail disable SSL 3.0, but it also disables TLS 1.1 and TLS 1.2 by default. I suspect it might possibly be hard-coded somewhere in JavaMail’s source code to enable TLSv1 only.
So, what can you do if you want to force it to use TLS 1.1 or TLS 1.2? Needless to say, you need to have JDK 7 or JDK 8 at first. (See CVE-2014-3566 – Instructions to Mitigate the SSL v3.0 Vulnerability (aka “Poodle Attack”) in Java SE.)
The Cure
The article talks about the mail.smtp.ssl.protocols property which can be used to set the protocols. The Jenkins configuration was changed so that the JAVA_ARGS looked like below:
JAVA_ARGS="-Djava.awt.headless=true -Dmail.smtp.starttls.enable=true -Dmail.smtp.ssl.protocols=TLSv1.2"
To resolve the issue, following steps were performed:
- Edit out the Jenkins configuration file (in Ubuntu, generally it is located at /etc/default/Jenkins) and add the arguments as highlighted above. (Ensure that you add both highlighted parameters!)
- Restart the Jenkins service by typing in “service Jenkins restart”
- Now open Jenkins Dashboard on your favorite browser and then navigate to – Manage Jenkins -> Configure System -> E-mail Notification.
- Click on the Advanced button.
- Ensure that “Use SSL” option is not selected – it needs to stay unchecked.
- Hit the Save button. Lo and behold, the email will now work over TLSv1.2.
‘Til next time!