Large-scale cluster management at Google with Borg

Google’s Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines. This paper presents an architectural overview.

Download

   Google-Borg.pdf

Discussion notes

We reviewed Borg’s architecture at the cell-level – in particular, the role of Paxos and the link-shards. Somebody observed that since the sophisticated scheduler functions were split off (into the Scheduler) from the relatively mundane BorgMaster functions like adding jobs to the persistent store, it might be possible to substitute in a different distributed log for the Paxos store. It would certainly be neat to hear about specific alternatives that might be suitable.

Touching upon the Scheduler/ job admittance and scheduling, we learned about Google’s quota allocation system and debated about the merits/ demerits of an auction system for resource quotas. We discussed the need for fine-grained resource request and resource reclamation policies in such a setting where users provide the resource requirements (as both soft and hard requirements).

Reading through the Evaluation section of the paper was interesting too — specifically, the utilization results reported from running prod and non-prod jobs on the same cluster (!); I for one would like to hear of a start-up that does this!

We also talked about the security and performance isolation offered by the use of chroot and cgroups, and how this compares to Docker and co. Talking about the introspection tools built into Borg and hearing about those used at our respective companies was neat too.

At the end, there was substantial discussion about similar systems — Kubernetes in particular, and how it organizes multi-job services (which is complicated to do with Borg) and manages IP addressing between services. The article Wiktor/ Calvin provided above compares Borg, Omega and Kubernetes in further detail and is worth checking out!