Oxeye is an AppSec platform that detects code vulnerabilities and issues in open source and third party packages. But how do we ensure that the tool that detects these vulnerabilities is itself not vulnerable? Since this situation would represent the dictionary definition of irony, we made sure to tackle this task and fix all possible security issues we might have in The Observer  (see description below). This blog post will describe some good guidelines on how to build and maintain secure containers. Whether you’re looking for a guide on how to build a secure container or just a good read, you’re in the right place.

For those of you who aren’t familiar with the Oxeye platform, the observer is the component running in our customers’ environments. It detects all running containers and scans them for security issues both at the container and at the application levels. It also collects configurations like network definitions, scaling policies, and other data that describe the application’s environment. All of these pieces of information help Oxeye’s engine to deduct and prioritize which are the most critical vulnerabilities.

Because of all the functions it’s expected to perform,  the observer must be a Swiss Army Knife container - one that knows how to detect and address different languages and security configurations. For example, In order to scan a container with a Java application, the observer uses a Java scanning tool - which is also written in Java and therefore must have Java installed. The same goes for Python, .net, Golang, and NodeJS. This results in a container that can do a lot - but also has many third party packages installed. The screen cap below shows the output when I executed [.inline-code] Grype [.inline-code], a well-known tool used to detect outdated and vulnerable packages inside containers.

Over 1000 vulnerabilities in one container! It won’t be enough to just update some files. Like most development teams, we didn’t intentionally use older versions of third party libraries - we just installed the tools we needed. Issues of this kind lead me to the first guideline:

‍

Guideline #1 - If possible, start from scratch
When building a container, most people choose to have the runtime they need for their application. But, in many cases, this might bring out-of-date libraries. For example, we didn’t choose to install [.inline-code] libbluetooth3.so [.inline-code] in the observer -  we don’t use Bluetooth when scanning containers. But we do use Debian because Debian makes it easy for us to install Java and Python using package managers like [.inline-code] apt [.inline-code], and Debian comes with vulnerable Bluetooth libraries. If possible, choose a clean base image. In precompiled languages like GoLang, choose [.inline-code] FROM scratch [.inline-code], and in interpreted or semi-compiled languages, choose a non-vulnerable precompiled SDK base image. You can see how many vulnerabilities an image has on its tags page on Dockerhub. For example, here’s a screenshot of Python’s tags on Dockerhub:

This is also a good time to give a shout-out to Chainguard, who provide vulnerability-free containers based on Wolfi OS.
In most cases, this step alone will solve 90% of the issues, as most vulnerabilities are not even in use.

‍

Guideline #2 - Use multi-stage Dockerfiles

In case you haven’t heard of multi-stage Dockerfiles before, it is when in the process of building your final container, you use another intermediate container. The most common use case is using a “build” container, then copying the final artifact into a published container. This building pattern allows the final container to be leaner - as no building libraries are needed. It also speeds up the build process as build steps that don’t depend on one another happen concurrently.

But, did you know that sometimes even if you don’t build anything you might need build-dependencies?

The Observer uses a Python library called [.inline-code] psutil [.inline-code]. This library provides many utilities for gathering process information, in a Pythonic way. This library depends on OS features and includes some C code, which compiles into a [.inline-code] .so [.inline-code] (shared library in Linux). So, when installing this module using the [.inline-code] pip [.inline-code] module manager, it will attempt to compile the C code. To avoid that we can run:

This command will pull an already compiled version of the library and install it without compiling it locally. But what if that is not enough? The observer supports running on arm-based processors (aarch64), unlike the [.inline-code] psutil [.inline-code] module (and the maintainers don’t plan on adding this support).

So what do we do? That’s right - a multi-stage container for building our dependencies.

Then we just:

We still need the [.inline-code] musl [.inline-code] and [.inline-code] libgmp [.inline-code] libraries as the compiled psutil .so uses them, which leads me to my next point.

‍

Guideline #3: Understand that not everything can be removed

This is a rough one - we might not be able to make it perfect. There’s a high probability that we will have to release containers that have a vulnerability in them. Our job is to understand how exploitable this vulnerability is. For example, we might have a vulnerable log4j library as part of our Java SDK that we install, but that doesn’t automatically mean that our application is vulnerable, as the package may not be loaded into memory and used at runtime. In our case, we use tools like [.inline-code] FindSecBugs [.inline-code], which, even in their latest version use outdated libraries (yes [.inline-code] bcel-6.5.0.jar [.inline-code] I’m looking at you).

However, If an SDK contains a library you don’t need - consider removing it from the container. This will make it weigh less and carry less vulnerability potential.

‍

Guideline #4: Try using as few tools as possible

This is based on simple logic. The more external binaries you bring into your container, the more external utilities you need to update. So, if possible, try reusing the ones you already have. For example, we use [.inline-code] busybox [.inline-code], which is a shell utility that includes many tools within it. This saves us many other Linux binaries like [.inline-code] rm, ls, ps, [.inline-code] etc…
In order to add many other binaries like [.inline-code] file [.inline-code], we used the multi-stage containers and copied pre-compiled binaries and libraries.

‍

Guideline #5: Make updating a habit

Just like cleaning your apartment isn’t something you do only once, but something that you must continually do - cleaning your containers should also be an ongoing activity. You must remember that even if nothing changes in your container, vulnerabilities keep being released every day. This means that even without any changes, your container might suddenly become vulnerable. You can either schedule a day every day/week/month or use automated tools. Check out automation bots like dependabot, or manually schedule scans using CI/CD steps like grype and Oxeye.

Ensuring that our observer was clear of vulnerabilities was not an easy process, but a necessary one. In the end, we had a slick new observer that weighed much less (2.5GB less!) and was way more secure! A side benefit? We documented these steps, in the hope that you or your team can benefit from the work that we invested into improving the security of our app. The gist of the process? If it is possible try maintaining a clean container from day one, but it is never too late to start cleaning it from vulnerabilities, and these steps will hopefully help you do so.