Rust fact vs. fiction: 5 Insights from Google's Rust journey in 2022

Reaching version 1.0 in just 2015, Rust is a relatively new language with a lot to offer. Developers eyeing the performance and safety guarantees that Rust provides, have to wonder if it's possible to just use Rust in place of what they've been using previously. What would happen if large companies tried to use it in their existing environment? How long would it take for developers to learn the language? Once they do, would they be productive?

In this post, we will analyze some data covering years of early adoption of Rust here at Google. At Google, we have been seeing increased Rust adoption, especially in our consumer applications and platforms. Pulling from the over 1,000 Google developers who have authored and committed Rust code as some part of their work in 2022, we’ll address some rumors head-on, both confirming some issues that could be improved and sharing some enlightening discoveries we have made along the way.

We’d like to particularly thank one of our key training vendors, Ferrous Systems, as we started our Rust adoption here at Google. We also want to highlight some new freely available self-service training materials called Comprehensive Rust 🦀 that we and the community have worked on over the last few quarters.

Rumor 1: Rust takes more than 6 months to learn – Debunked !

All survey participants are professional software developers (or a related field), employed at Google. While some of them had prior Rust experience (about 13%), most of them are coming from C/C++, Python, Java, Go, or Dart.

Based on our studies, more than 2/3 of respondents are confident in contributing to a Rust codebase within two months or less when learning Rust. Further, a third of respondents become as productive using Rust as other languages in two months or less. Within four months, that number increased to over 50%. Anecdotally, these ramp-up numbers are in line with the time we’ve seen for developers to adopt other languages, both inside and outside of Google.

Overall, we’ve seen no data to indicate that there is any productivity penalty for Rust relative to any other language these developers previously used at Google. This is supported by the students who take the Comprehensive Rust 🦀 class: the questions asked on the second and third day show that experienced software developers can become comfortable with Rust in a very short time.

Rumor 2: The Rust compiler is not as fast as people would like – Confirmed !

Slow build speeds were by far the #1 reported challenge that developers have when using Rust, with only a little more than 40% of respondents finding the speed acceptable.

There is already a fantastic community-wide effort improving and tracking rustc performance. This is supported by both volunteers and several companies (including Google), and we’re delighted to see key developers working in this space but clearly continuing and potentially growing additional support here would be beneficial.

Rumor 3: Unsafe code and interop are always the biggest challenges – Debunked !

The top three challenging areas of Rust for current Google developers were:

• Macros
• Ownership and borrowing
• Async programming

Writing unsafe code and handling C/C++ interop were cited as something Google developers had encountered but were not top challenges. These three other areas are places where the Rust Language Design Team has been investing in flattening the learning curve overall as well as continued evolution, and our internal survey results strongly agree with these as areas of investment.

Rumor 4: Rust has amazing compiler error messages – Confirmed !

Rust is commonly regarded as having some of the most helpful error messages in the compiler space, and that held up in this survey as well. Only 9% of respondents are not satisfied with the quality of diagnostic and debugging information in Rust. Feedback from Comprehensive Rust 🦀 participants shows the same: people are amazed by the compiler messages. At first this is a surprise – people are used to ignoring large compiler errors, but after getting used to it, people love it.

The following are excerpts from an exercise some internal Googlers have been doing to practice Rust – solving Advent of Code 2021 in Rust.

On Day 5 of the exercises, we need to perform a search for entries within a table. The error below not only detects that our pattern matching on the result was missing a case, but also makes a suggestion for a fix.

On Day 11, we need to check for whether an element is within the bounds of a grid. The Rust warning below detects that we have a redundant comparison due to the fact that the types are unsigned, and suggests code that could be removed.

Rumor 5: Rust code is high quality – Confirmed!

The respondents said that the quality of the Rust code is high — 77% of developers were satisfied with the quality of Rust code. In fact, when asked to compare whether they felt that Rust code was more correct than the code that they write in other languages, an overwhelming 85% of respondents are confident that their Rust code is correct.

And, it’s not just correct—it’s also easy to review. More than half of respondents say that Rust code is incredibly easy to review. As an engineering manager, that result is in many ways at least as interesting to me as the code authoring results, since code reviewing is at least as large a part of the role of a professional software engineer as authoring.

As both we at Google and others have noted, developer satisfaction and productivity are correlated with both code quality and how long it takes to get a code review. If Rust is not only better for writing quality code, but also better for getting that code landed, that’s a pretty compelling set of reasons beyond even performance and memory safety for companies to be evaluating and considering adopting it.

Looking forward

While over a thousand developers is a good sample of engineers, we look forward to further adoption and a future survey that includes many more use cases. In addition, while many of the developers surveyed joined teams without Rust experience, this population does have more excited early adopters than we would like from a broader survey. Stay tuned over the next year for another update!

By Lars Bergstrom, PhD – Android Platform Programming Languages and Kathy Brennan, PhD - Low-level Operating Systems Sr. User Experience Researcher

Optimizing gVisor filesystems with Directfs

gVisor is a sandboxing technology that provides a secure environment for running untrusted code. In our previous blog post, we discussed how gVisor performance improves with a root filesystem overlay. In this post, we'll dive into another filesystem optimization that was recently launched: directfs. It gives gVisor’s application kernel (the Sentry) secure direct access to the container filesystem, avoiding expensive round trips to the filesystem gofer.

Origins of the Gofer

gVisor is used internally at Google to run a variety of services and workloads. One of the challenges we faced while building gVisor was providing remote filesystem access securely to the sandbox. gVisor’s strict security model and defense in depth approach assumes that the sandbox may get compromised because it shares the same execution context as the untrusted application. Hence the sandbox cannot be given sensitive keys and credentials to access Google-internal remote filesystems.

To address this challenge, we added a trusted filesystem proxy called a "gofer". The gofer runs outside the sandbox, and provides a secure interface for untrusted containers to access such remote filesystems. For architectural simplicity, gofers were also used to serve local filesystems as well as remote.

Isolating the Container Filesystem in runsc

When gVisor was open sourced as runsc, the same gofer model was copied over to maintain the same security guarantees. runsc was configured to start one gofer process per container which serves the container filesystem to the sandbox over a predetermined protocol (now LISAFS). However, a gofer adds a layer of indirection with significant overhead.

This gofer model (built for remote filesystems) brings very few advantages for the runsc use-case, where all the filesystems served by the gofer (like rootfs and bind mounts) are mounted locally on the host. The gofer directly accesses them using filesystem syscalls.

Linux provides some security primitives to effectively isolate local filesystems. These include, mount namespaces, pivot_root and detached bind mounts¹. Directfs is a new filesystem access mode that uses these primitives to expose the container filesystem to the sandbox in a secure manner. The sandbox’s view of the filesystem tree is limited to just the container filesystem. The sandbox process is not given access to anything mounted on the broader host filesystem. Even if the sandbox gets compromised, these mechanisms provide additional barriers to prevent broader system compromise.

Directfs

In directfs mode, the gofer still exists as a cooperative process outside the sandbox. As usual, the gofer enters a new mount namespace, sets up appropriate bind mounts to create the container filesystem in a new directory and then pivot_root(2)s into that directory. Similarly, the sandbox process enters new user and mount namespaces and then pivot_root(2)s into an empty directory to ensure it cannot access anything via path traversal. But instead of making RPCs to the gofer to access the container filesystem, the sandbox requests the gofer to provide file descriptors to all the mount points via SCM_RIGHTS messages. The sandbox then directly makes file-descriptor-relative syscalls (e.g. fstatat(2), openat(2), mkdirat(2), etc) to perform filesystem operations.

Earlier when the gofer performed all filesystem operations, we could deny all these syscalls in the sandbox process using seccomp. But with directfs enabled, the sandbox process's seccomp filters need to allow the usage of these syscalls. Most notably, the sandbox can now make openat(2) syscalls (which allow path traversal), but with certain restrictions: O_NOFOLLOW is required, no access to procfs and no directory FDs from the host. We also had to give the sandbox the same privileges as the gofer (for example CAP_DAC_OVERRIDE and CAP_DAC_READ_SEARCH), so it can perform the same filesystem operations.

It is noteworthy that only the trusted gofer provides FDs (of the container filesystem) to the sandbox. The sandbox cannot walk backwards (using ‘..’) or follow a malicious symlink to escape out of the container filesystem. In effect, we've decreased our dependence on the syscall filters to catch bad behavior, but correspondingly increased our dependence on Linux's filesystem isolation protections.

Performance

Making RPCs to the gofer for every filesystem operation adds a lot of overhead to runsc. Hence, avoiding gofer round trips significantly improves performance. Let's find out what this means for some of our benchmarks. We will run the benchmarks using our newly released systrap platform on bind mounts (as opposed to rootfs). This would simulate more realistic use cases because bind mounts are extensively used while configuring filesystems in containers. Bind mounts also do not have an overlay (like the rootfs mount), so all operations go through goferfs / directfs mount.

Let's first look at our stat micro-benchmark, which repeatedly calls stat(2) on a file.

The stat(2) syscall is more than 2x faster! However, since this is not representative of real-world applications, we should not extrapolate these results. So let's look at some real-world benchmarks.

We see a 12% reduction in the absolute time to run these workloads and 17% reduction in Ruby load time!

Conclusion

The gofer model in runsc was overly restrictive for accessing host files. We were able to leverage existing filesystem isolation mechanisms in Linux to bypass the gofer without compromising security. Directfs significantly improves performance for certain workloads. This is part of our ongoing efforts to improve gVisor performance. You can learn more about gVisor at gvisor.dev. You can also use gVisor in GKE with GKE Sandbox. Happy sandboxing!

---

¹Detached bind mounts can be created by first creating a bind mount using mount(MS_BIND) and then detaching it from the filesystem tree using umount(MNT_DETACH).

By Ayush Ranjan, Software Engineer – Google

OpenTitan RTL freeze

We are excited to announce that the OpenTitan® coalition has successfully reached a key milestone—RTL freeze of its first engineering sample release candidate! A snapshot of our high quality, open source silicon root of trust hardware implementation has been released for synthesis, layout and fabrication. We expect engineering sample chips to be available for lab testing and evaluation by the end of 2023.

This is a major achievement that represents the culmination of a multi-year investment and long-term, coordinated effort by the project’s active community of commercial and academic partners—including Google, G+D Mobile Security, ETH Zurich, Nuvoton, Winbond, Seagate, Western Digital, Rivos, and zeroRISC, plus a number of independent contributors. The OpenTitan project and its community are actively supported and maintained by lowRISC C.I.C., an independent non-profit.

Hitting this milestone demonstrates that large-scale engineering efforts can be successful when many organizations with aligned interests collaborate on an open source project. It also matters because traditionally, computing ecosystems have had to depend heavily on proprietary hardware (silicon) and software solutions to provide foundational, or “root,” trust assurances to their users. OpenTitan fundamentally changes that paradigm for the better, delivering secure root of trust silicon technology which is open source, high quality, and publicly verifiable.

Our belief is that core security features like the authenticity of the root of trust and the firmware it executes should be safely commoditized rights guaranteed to the end user—not areas for differentiation. To that end, we have made available a high-quality, industrial strength, reusable ecosystem of OpenTitan blocks, top-levels, infrastructure, and software IP adaptable for many use cases, delivered under a permissive, no-cost license and with known-good provenance. OpenTitan's now-complete, standalone “Earl Grey” chip implementation, design verification, full-chip testing, and continuous integration (CI) infrastructure are all available on GitHub today.

The silicon process and OpenTitan

This release means the OpenTitan chip digital design is complete and has been verified to be of sufficiently high quality that a tapeout is expected to succeed. In other words, the logical design is judged to be of sufficient maturity to translate into a physical layout and create a physical chip. The initial manufacturing will be performed in a smaller batch, delivering engineering samples which allow post-silicon verification of the physical silicon, prior to creating production devices in large volume.

Earl Grey: Discrete implementation of OpenTitan

Source: https://opentitan.org/documentation/index.html

Design Verification

Industrial quality implementation has been a core tenet of the OpenTitan project from the outset, both to ensure the design meets its goals—including security—and to ensure the first physical chips are successful. OpenTitan’s hardware development stages ensure all hardware blocks go through several gating design and verification reviews before final integration signoff. This verification has required development of comprehensive testbenches and test infrastructure, all part of the open source project. Both individual blocks and the top-level Earl Grey design have functional and code coverage above 90%—at or above the standards of typical closed-source designs—with 40k+ tests running nightly and reported publicly via the OpenTitan Design Verification Dashboard. Regressions are caught and resolved quickly, ensuring design quality is maintained over the long term.

Software tooling

OpenTitan has led the way in making open source silicon a reality, and doing so requires much more than just open source silicon RTL and Design Verification collateral. Successful chips require real software support to have broad industry impact and adoption. OpenTitan has created generalizable infrastructure for silicon projects (test frameworks, continuous integration infrastructure, per-block DIFs), host tools like opentitantool to support interactions with all OpenTitan instances, and formal releases (e.g. the ROM to guarantee important security functionality such as firmware verification and ownership transfer).

Documentation

A good design isn’t worth much if it’s hard to use. With this in mind, thorough and accurate documentation is a major component of the OpenTitan project too. This includes a Getting Started Guide, which is a ‘from scratch’ walkthrough on a Linux workstation, covering software and tooling installation, and hardware setup. It includes a playbook to run local simulations or even emulate the entire OpenTitan chip on an FPGA.

Furthermore, OpenTitan actively maintains live dashboards of quality metrics for its entire IP ecosystem (e.g. regression testing and coverage reports). If you’re new to open source silicon development, there are comprehensive resources describing project standards for technical contribution that have been honed to effectively facilitate inter-organizational collaboration.

Thriving open source community

OpenTitan’s broad community has been critical to its success. As the following metrics show (baselined from the project’s public launch in 2019), the OpenTitan community is rapidly growing:

• More than eight times the number of commits at launch: from 2,500 to over 20,000.
• 140 contributors to the code base
• 13k+ merged pull requests
• 1.5M+ LoC, including 500k LoC of HDL
• 1.8k Github stars

Participating in OpenTitan

Reaching this key RTL freeze milestone is a major step towards transparency at the very foundation of the security stack: the silicon root of trust. The coordinated contributions of OpenTitan’s project's partners—enabled by lowRISC’s Silicon Commons™ approach to open source silicon development—are what has enabled us to get here today.

This is a watershed moment for the trustworthiness of systems we all rely on. The future of free and open, high quality silicon implementations is bright, and we expect to see many more devices including OpenTitan top-levels and ecosystem IP in the future!

If you are interested in contributing to OpenTitan, visit the open source GitHub repository or reach out to the OpenTitan team.

By Cyrus Stoller, Miguel Osorio, and Will Drewry, OpenTitan – Google