How Chrome OS works upstream

Google has a long and interesting history contributing to the upstream Linux kernel. With Chrome OS, Google has tried to learn from some of the mistakes of its past and is now working with the upstream Linux kernel as much as it can. In a session at the 2019 Open Source Summit North America, Google software engineer Doug Anderson detailed how and why Chrome OS developers work upstream. It is an effort intended to help the Linux community as well as Google.

The Chrome OS kernel is at the core of Google's Chromebook devices, and is based on a Linux long-term support (LTS) kernel. Anderson explained that Google picks an LTS kernel every year and all devices produced in that year will use the selected kernel. At least once during a device's lifetime, Google expects to be able to "uprev" (switch to a newer kernel version). Anderson emphasized that if Google didn't upstream its own patches from the Chrome OS kernel, it would make the uprev process substantially more difficult.

Simply saying that you'll work upstream and actually working upstream can be two different things. The process by which Chrome OS developers get their patches upstream is similar to how any other patches land in the mainline Linux kernel. What is a bit interesting is the organizational structure and process of how Google has tasked Chrome OS developers to work with upstream. Anderson explained that developers need to submit patches to the kernel mailing list and then be a little patient, giving some time for upstream to respond. A key challenge, however, is when there is no response from upstream. "When developing an upstream-first culture, the biggest problem anyone can face is silence," Anderson said.

Anderson emphasized that when submitting a patch to the mailing list, what a developer is looking for is some kind of feedback; whether it's good or bad doesn't matter, but it does matter that someone cares enough to review it. What the Chrome OS team does in the event that there is no community review is it will have other Chrome OS engineers publicly review the patch. The risk and worry of having Chrome OS engineers comment on Chrome OS patches is that the whole process might look a little scripted and there could be the perception of some bias as well. Anderson noted that it is important that only honest feedback and review is given for a patch.

Yak shaving

While feedback from the mailing list is what the developers want, there can be situations where that feedback gets into what Anderson referred to as "yak shaving". It means that a small patch is submitted and the feedback that comes back covers more than just the patch itself, asking the developer to, for example, clean up a related subsystem or redo an API. "Basically, your little patch ends up being someone requesting you to do a month's worth of work," Anderson said.

Sometimes if a yak shaving request is not a lot of work; he suggested that the best thing is to just do the work in that case. Other times, the work is far outside the scope of a Chrome OS developer's efforts and it's not something that the developer is able to do. Anderson said that there are times that upstream will allow a developer to land a patch without completing all of the yak shaving, while there are other times when that doesn't happen.

The Chrome OS kernel tree

Even with a stated policy of trying to be upstream first, the Chrome OS kernel tree isn't entirely pulled from the mainline kernel. Anderson explained that developers generally try to tag all the commits in the Chrome OS tree with some information about where the commits came from. He emphasized that whenever Google has something in its Chrome OS kernel tree, developers are pushing hard for it to go upstream into Linus Torvalds's tree or into the right maintainer's tree.

The UPSTREAM tag is used to identify a commit that was pulled directly from Torvalds's tree, while the FROMGIT tag refers to a commit that was picked from a maintainer's tree that will eventually feed the mainline. The FROMLIST tag is used by Chrome OS developers for commits that are picked from a mailing list. The CHROMIUM tag is a bit different than the others, in that it refers to a commit that is not going to go upstream and, ideally, is just for Chrome-OS-specific configuration items.

Fundamentally, Anderson argued that it is a win-win for both the overall kernel development community and Chrome OS to work upstream. He noted that working upstream provides extra testing and review bandwidth, as well as pushing hardware vendors that work with Chrome OS devices to work upstream as well. For example, Anderson said that the Rockchip rk3288 and rk3399 Arm systems-on-chip (SoCs) are fairly well supported upstream because of Chrome OS.

Learning from the mistakes of Android

Google hasn't always had an upstream-first approach for its operating systems, which is something that Chrome OS developers are aiming not to repeat. "We're trying not to do what Android did 10 years ago," Anderson said. Android phones tended to have large stacks of patches that had not been upstreamed, he said. In addition, Android devices had come up with all kinds of different ways of doing things that were not aligned with the upstream kernel. Rather than follow the path of Android, Chrome OS developers want to get as much as possible into the mainline, giving kernel developers the chance to comment and review along the way.

During the question-and-answer segment that followed the session, a member of the audience pointed out that even today Android development has lots of code that isn't upstream. They asked if in fact the Chrome OS kernel is basically the same as the upstream kernel. Anderson explained that Google has a full fork of the mainline kernel. So for example, there is a Chrome OS 4.19 tree, where patches are picked and tagged so it's clear where the patches come from. "So it's totally a whole fork of the upstream kernel, but the idea is that we try to keep posting things upstream and landing them there," he said. "And in a perfect world, our Chrome OS 4.19 would be just pure 4.19 with a whole bunch of upstream bits from newer kernels, but it doesn't always work out that way."

Keeping everything upstream, as Anderson explained more than once, during his session, isn't easy. But it seems clear that Google's Chrome OS development team is committed to that path, and has a well-defined process that will hopefully keep them from straying too far.