Python Sandboxing is Still Broken by Design

The bug hunt revealed some deep issues in Python code sandboxes and highlighted the extent the Python ecosystem implicitly trusts CPython’s memory safety. The ramifications of that are large, and ownership of the problem is messy.

The bug hunt revealed some deep issues in Python code sandboxes and highlighted the extent the Python ecosystem implicitly trusts CPython’s memory safety. The ramifications of that are large, and ownership of the problem is messy.

After our advisory, Grafana responded almost immediately. They let us know they had reproduced the issue and were working on a fix. Especially given IRM’s positioning as an incident response platform, we applaud Grafana’s security team for having exceptionally strong coordination and response time. They clearly practice what they preach.

The Complicated History of Sandboxed Python

Sandboxed Python has a tumultuous history going back well over a decade. The same expressive dynamic features of the language also make it exceptionally hard to contain in pure Python. The ecosystem is packed with sandboxing attempts, some of which are going strong, and some of which saw abrupt retirement.

‍

The solutions that persist do so in the form of secure expression evaluators, template engines, and restricted language subsets, which are adopted extensively across a wide array of business and scientific domains. Large parts of security-critical features in the Python ecosystem are increasingly built on foundations often deemed impossible to secure.

However, most of the discourse around sandboxing hinges on the expressiveness of CPython itself being impossible to tame. We are demonstrating a different angle of the problem: Python Sandboxes cannot be more secure than CPython itself, which makes no memory safety guarantees for arbitrary code. Without CPython memory safety flaws being treated as security issues, there’s no way for a sandbox to know its underlying interpreter can be trusted.

‍This dangerously blurs the line between trusted and untrusted code across the entire Python ecosystem.

Breaking Out by Breaking CPython

Grafana relies on the Jinja template engine in its incident response offering. It powers features like flexible formatting of alert payloads, grouping related alerts, and routing alerts. These functions are user driven, enabling them to directly edit sandboxed templates. Their use case is a typical well-trodden path across many types of applications. 

Given the difficulty of creating a leak-proof sandbox in pure CPython, Jinja’s sandbox has an incredibly solid track record of constraining user templates. Given that, we opted to approach a breakout from the CPython layer beneath it. A simple hypothesis:

Can we escape a Python sandbox by attacking the memory safety of its underlying interpreter?

The answer was a resounding yes. And we did it by using a (patched) off-the-shelf memory bug from CPython’s own public issue tracker.

Initial Recon

The first hurdle to binary exploitation on a remote CPython interpreter is discovering versions and memory layouts. Without directly being able to query the runtime, this information must be derived heuristically. We approached this in two phases:

• Fingerprinting CPython major and minor versions
• Discovering the exact compiled layout of the binary

We built Circaetus to help automate this process. In short, we used it to heuristically discover the CPython version by creating a concise test template for each major and minor version of Python. The template tests capabilities that became available on specific versions for the first time. Each feature's presence or absence contributes to a fingerprint that reveals the precise version when interpreted. 

Once we had the version, we checked many common distributions of CPython for a binary that had the same distance between symbols until we found a matching layout. This was possible because Python allows discovery of symbol locations in memory, and the delta between two symbols is a unique facet of each compiled CPython distribution.

The Primitive: Reading Past a Rebuilt Dictionary

A reverse dictionary iterator is created when using the reversed() builtin on a dictionary object. 

The iterator carries:

• Item count, di_used
• Position, di_pos

Vulnerable CPython versions checked whether the dictionary's item count ma_used still equals the iterator’s count di_used, but after a dictionary was cleared and refilled, CPython reused the iterator’s old position without checking whether that position still exists. Clearing and rebuilding the same dictionary could therefore preserve the expected item count while replacing a deliberately crafted eight-entry physical history with a five-entry table. A reverse iterator still positioned at six or seven would then read past the replacement table.

The breakout required a useful object immediately after the undersized replacement keys table. On Grafana’s affected version, both of the following consume a 160-byte allocator block:

• Five-entry general dictionary keys table = 160 bytes 
• Python function + garbage collector header = 160 bytes 

Because the allocations were the same size, the payload could arrange for the Python function backing a new Jinja macro to occupy the neighboring 160-byte block. The stale iterator then read beyond its dictionary’s table and interpreted fields from the live function as a dictionary entry, exposing its otherwise inaccessible globals mapping.

Exploitation: Turning a Jinja Macro into a Globals Leak

The payload made two reverse iterators before clearing its dictionary:

• A calibration_iterator which was held at position seven
• An attack_iterator was advanced once and held at position six

The dictionary was then rebuilt with only five entries so both positions were out of bounds. The surrounding 160-byte controlled block made position seven behave like a fabricated dictionary entry whose me_key was the controlled dictionary itself. That allowed the template to identify exactly which neighboring block it had reached before continuing.

Because reversed(dict) is a key iterator, its fabricated entry returns the pointer aligned with func_globals. Jinja treated the resulting pointer as an ordinary dictionary key.

‍At this point, we used the leaked function globals to traverse to os and execute unsandboxed code.

This gave us a direct foothold in Grafana’s cloud infrastructure. The position exposed credentials, network topology, internal apis, and direct data connections. This is a major concern for any cloud-hosted product.

The Python Ecosystem has a Difficult Bug Ownership Problem

We have been discussing the implications of memory safety on downstream sandboxes with the CPython team, which concluded with them reiterating their stance that: “CPython does not support sandboxing untrusted Python code as a security boundary, so escapes from such a sandbox are not vulnerabilities in Python and should be reported to the sandbox’s developers instead.”

CPython’s maintainers have held this position throughout its history. They have never promised that hostile code can be contained safely inside the interpreter, so a Python-level sandbox cannot reasonably be expected to withstand memory corruption. Changing that stance would be a fundamental shift. 

However, significant portions of the Python ecosystem operate on the assumption that memory safety remains a baseline security requirement. We don’t believe it is realistic for every widely adopted Python sandbox to respond to every memory bug in CPython. However, that leaves end users responsible for moving away from Python sandboxes in security-critical code. 

Practical guidance does not currently line up with reality. The treatment of memory bugs as non-security issues means CPython’s version status tracking is not accurate enough for real deployments within the software supply chain. For users like Grafana, it actually may be misleading. That is how patched “non-security” bugs lead to a breach of a company’s cloud infrastructure.

Applications that render user templates, evaluate scientific expressions, and use expressive language subsets are vulnerable with no clear idea what their risks are. This leads us to the same conclusion as Victor Stinner over 13 years ago when he retired pysandbox:

Python sandboxing is broken by design

Recommendations

If you use expressive template engines or safe evaluators, their security guarantees should be considered best effort and scrutinized heavily. Rely on appropriate system-level controls to contain them or move to solutions that can provide stronger safety guarantees. Aggressively update your CPython interpreter, and backport memory safety fixes outside of CPython’s own release schedule. 

This is the first part in a series of publications on sandbox escapes and memory safety in the Python ecosystem. RunSybil is continuing to explore how issues like this affect common software. Check back soon for more research on the topic.