BSides Cape Town 2024

DevOps or DevOops? Securing a Pipeline Without Losing Your Mind
2024-12-07 , Track 1

This talk will follow a light-hearted take on the mistakes and solutions I had while setting up a Gitlab to Jenkins to Tomcat CICD pipeline this year. Many of the configurations were insecure by default and when approached a mentality of "Make it work" it just compounds the issue. The talk will go through each stage of the pipeline, the issues I found, the issues I caused and the solutions for both.


Introduction:

The introduction will paint a background of how this talk came about and the components in the pipeline. The idea is that Source Code from Gitlab sends a webhook on merge to Jenkins. Jenkins then pulls the code and builds it, then deploys it to the Tomcat web server. Tomcat itself is hosted on localhost port 8080 which is served to the internet using an NGINX reverse proxy. If that sounds like a whole lot of technical gibberish, don't worry we'll understand it by the end of the talk.

Gitlab:

Gitlab is a source code repository on Linux, much like GitHub it can store your code and your code changes. However, by default it has a few problems. For example anyone can register on the application. We'll look into some of these common problems as well as common mistakes that I made while configuring a repository. Even though Jenkins is only meant to be exposed internally, that doesn't mean that everyone internally should see your typos in your commit messages.

Some of the misconfigurations I made included the creation of a public repo, being able to create and accept my own merge requests and allowing developers to see information. So by the time I had a working pipeline, anyone with access could compromise each host in the pipeline.

Jenkins:

Jenkins is a automation server which in my pipeline is used to build and deploy code from Gitlab to web servers (in the case of this example). It is basically RCE as a feature with a few added security issues on top. One of my favourites was creating a webhook that had my Jenkins Admin creds in it. We'll look at what attack paths exist in Jenkins and against Jenkins and build it back to the access an might have have with Gitlab.

Apart from embedding my credentials into a webhook I also had to work out ways to move code from the build agent to the web server. This started with some very bad ideas, like python simple server, and ended up with ssh.

Deployments:

Finally with deployments, I'll give a brief overview of the pain I experienced with trying to get Tomcat working in the first place, but then we'll go through some attacks that work and are fairly under the radar if you can deploy malicious code to production. We will also tackle the problem of how you block client-side code from calling out to a specific domain.

Some interesting things here is that any Javascript I've written for key logging, information stealing, and the works are never detected by any controls. Especially with attacks like dependency confusion attacks: How do you block a domain client-side if you can't necessarily remove the malware yourself?

Conclusions:

For the conclusions we'll take a look at where the network started, and the amount of issues it had (Security and others). Then compare it to where the network ended and how non-intuitive the fixes were. To end off, the attacks will be mapped back to a DevOps pipeline to see what type of risks each stage of the pipeline could introduce.

I'm a technically inclined ocean enthusiast who looks forward to the day I can play video games on my surfboard. And then probably also try hack said surfboard.