The Oxeye research team has been able to gain remote code execution in Spotify’s open source, CNCF-incubated project—Backstage, by exploiting a VM sandbox escape through the vm2 third-party library. We reported this RCE vulnerability via Spotify’s bug bounty program, and the Backstage team responded rapidly by patching it in version 1.5.1, and ranking the vulnerability with a CVSS score of 9.8.
Potential vulnerability impact – An unauthenticated threat actor can execute arbitrary system commands on a Backstage application by exploiting a vm2 sandbox escape in the Scaffolder core plugin.
Having more than 19,000 stars on Github, Backstage – a CNCF incubated project by Spotify, is one of the most popular open source platforms for building developer portals. It restores order to your microservices and infrastructure, thus enabling your product teams to ship high-quality code quickly without compromising autonomy. Backstage unifies all of your infrastructure tooling, services, and documentation to create a streamlined development environment from end to end. In addition to Spotify, it’s used by a variety of organizations, including American Airlines, Netflix, Splunk, Fidelity Investments and Epic Games.
Backstage can hold integration details to many organization systems, such as Prometheus, Jira, ElasticSearch, and others. Thus, successful exploitation has critical implications for any affected organization and can compromise those services and the data they hold.
Out of the box, Backstage includes:
Every research project we spin up starts with mapping potential inputs to an application. A key Backstage feature that caught our attention is its software templates.
Backstage has three parts:
Core – Base functionality built by core developers in the open source project
App – A Backstage app instance that is deployed and tweaked. It ties together core functionality with additional plugins. It’s built and maintained by app developers, usually by a productivity team at a company.
Plugins – Additional functionality to make your Backstage app more useful. Plugins can be specific to a company or open sourced/reusable.
Software templates let an application owner quickly spin up new projects, components, and plugins in their organization according to its best practices and guidelines.
Each template is defined by a YAML file that resembles a Kubernetes resource. It contains various fields that define how the component should behave. For example:
Evaluating user-provided strings in a template engine can be dangerous since it exposes the application to template-based attacks. The severity of such an attack depends on the features the templating engine offers. Here, Nunjucks is extremely powerful but has been notorious for being unsafe. For example, this 2016 blog post describes popular techniques to bypass restrictions that Nunjucks imposes.
In an earlier research paper, Oxeye found a vm2 sandbox escape vulnerability that results in remote code execution (RCE) on the hosting machine. Once we sorted out that payload, we wondered, Could we exploit it in Backstage?
When attempting this exploit in our previous blog post, we sought to control properties outside of the sandbox context. When our basic payload executed, we tried invoking the getThis method on any CallSite objects in the array. (These are used to manipulate the call stack generated by the raised error.)
This time we initially tried a simple approach, storing the vm2 sandbox escape payload in a Backstage template YAML and then trying to trigger that. But calling the getThis method yielded an undefined result and caused our exploit to fail.
To understand why our exploit failed, we found the following in the v8 documentation:
To maintain restrictions imposed on strict mode functions, frames that have a strict mode function and all frames below (its caller, etc.) are not allowed to access their receiver and function objects. For those frames, getFunction() and getThis() returns undefined
To understand which of the functions in our call stack run in strict mode, we added the following code snippet to the functions that appeared in the call stack:
When we ran our payload an additional time with the added code snippet above we could see that the earliest function executed in strict mode within the vm2 sandbox was the “renderString2” function.
The following screenshot shows the “renderString2” function executed in strict mode:
To overcome this limitation, we decided to try and override the “renderString2” function with our own implementation in an attempt to force it to run in non-strict mode.
Something else of interest could help us with gaining code execution: given an error raised while rendering the template, Backstage called the renderString2 function a second time. This let us divide our payload into two stages. Initially we overwrote the renderString2 function with our own implementation. Then we triggered an error, causing Backstage to call the function again, this time invoking our overridden implementation.
This next screenshot shows Backstage calling the renderTemplate function (that calls renderString2) twice in the event of an error:
Our final payload was:
Here is an explanation:
From here, the exploit runs in the context of the vm2 sandbox.
When looking at the call stack as we ran our payload, none of the functions executed until our overridden renderString2 function was declared as strict. When we accessed index 2 of the trace array, we got a CallSite object created outside the sandbox. This enabled us to execute arbitrary code.
Once we had successfully executed our payload locally, we attempted to assess the potential impact of such a vulnerability if exploited in the wild.
We started by running a simple query for the Backstage favicon hash in Shodan; it resulted in more than 500 Backstage instances exposed to the internet. We then tried to assess how they could be exploited remotely without authenticating to the target Backstage instance.
The first thing we noticed was that Backstage is deployed by default without an authentication mechanism or an authorization mechanism, which allows guest access. Some of the public Backstage servers accessible to the internet did not require any authentication.
Additionally, this warning message caught our attention when we reviewed Backstage authentication documentation:
With this in mind, next we tried setting up a local Backstage instance that requires authentication, following tutorial guidelines originally maintained by Backstage. We ended up with authentication only enforced on the client side; requests flowing to the backend API were not verified for authentication or for authorization.
When trying to send requests directly to the backend API server of some of the internet-exposed instances, we found a handful did not require any form of authentication or authorization. Thus we concluded the vulnerability could be exploited without authentication on many instances.
If you’re using Backstage in your organization, we strongly recommend updating it to the latest version to defend against this vulnerability as soon as possible.
Moreover, if you’re using a template engine in your application, make sure you choose the right one in relation to security. Robust template engines are extremely useful but might pose a risk to your organization.
And if you use Backstage with authentication, enable it for both the front and backend.
Oxeye provides a cloud-native application security solution designed specifically for modern architectures. We enable you to quickly identify and resolve all application-layer risks as an integral part of the software development lifecycle (SDLC). Our seamless, comprehensive, and effective solution ensures touchless assessment, focuses on exploitable risks, and provides actionable remediation guidance. Built for DevOps and AppSec teams, Oxeye helps shift security left while accelerating development cycles, reducing friction, and eliminating risks. Visit www.oxeye.io to learn more.
Eliminate uncertainty from the application security process, and save your development and AppSec teams time.