Kevin's Blog

The only way to discover the limits of the possible is to go beyond them into the impossible. - Arthur C. Clarke

Mar 16, 2018 - 7 minute read - Comments - workshop

Got Namespaces? (Part 1)

In this workshop we’ll create our own small Go program which interfaces with the Linux kernel extensions that provide applications like Docker and LXC the ability to create isolated containers. Along the way we’ll discuss some of the history behind these features and set off a fork bomb at the end of Part 2!


Getting Started

You should have a linux workstation with Golang installed (or a VM). I highly recommend using a VM if you plan on setting off the fork bomb at the end of this exercise. If you’re unsure on how to install or configure Go you should read their documentation.

You’ll want to modify your sudoers file since we’ll be launching go via sudo.

  • run sudo visudo, modify the secure_path parameter by adding the go binary path (typically, /usr/local/go/bin). your sudoers files should look something like this:
#
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

...

Open up your favorite editor and begin by create a new file named main.go

main.go
 1 2 3 4 5 6 7 8 910111213141516171819202122232425262728293031323334
package main

import (
	"fmt"
	"os"
	"os/exec"
)

func main() {
	switch os.Args[1] {
	case "run":
		run()
	default:
		panic("nope")
	}
}

func run() {
	fmt.Printf("Running %v \n", os.Args[2:])

	cmd := exec.Command(os.Args[2], os.Args[3:]...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	must(cmd.Run())
}

func must(err error) {
	if err != nil {
		fmt.Printf("has error: %v \n", err)
	}
}

In this program we’re simply performing a fork and exec. When you execute this code by issuing the go run main.go run echo hello you’ll see the results of the echo hello command having been executed.

19:33:37 $ go run main.go run echo hello
Running [echo hello] 
hello

Our First Container

So let’s begin by adding a SysProcAttr to our forked process. This will allow us to attach any number of flags which enable certain features available to Namespaces. We’re going to give our newly forked process the ability to set it’s own hostname.

main.go
 1 2 3 4 5 6 7 8 910111213141516171819202122232425262728293031323334353637
package main

import (
	"fmt"
	"os"
	"os/exec"
	"syscall"
)

func main() {
	switch os.Args[1] {
	case "run":
		run()
	default:
		panic("nope")
	}
}

func run() {
	fmt.Printf("Running %v \n", os.Args[2:])

	cmd := exec.Command(os.Args[2], os.Args[3:]...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	cmd.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWUTS,
	}

	must(cmd.Run())
}

func must(err error) {
	if err != nil {
		fmt.Printf("has error: %v \n", err)
	}
}

So, let’s learn about syscall.CLONE_NEWUTS. I cannot find any concrete documentation about this particular flag, other than the patch for the code change which implemented the functionality. You can learn more here or, just take my word that this particular flag allows the forked process to assign its own host / domain name.

Let’s execute this program, call forth a new shell, and set our hostname within our “container”. Note: you will need to elevate your permissions here (ie, sudo) as CLONE_NEWUTS requires CAP_SYS_ADMIN.

15:04:04 $ sudo go run main.go run /bin/bash
Running [/bin/bash] 
linux-host go # hostname -b containerland
linux-host go # hostname
containerland

Now, let’s run this same command in another shell outside of the newly minted container:

15:03:36 $ hostname
linux-host

It’d be helpful to launch our forked process and instantiate the hostname prior to executing the shell. This is possible within the construct of our run() function by executing syscall.Sethostname([]byte("containerland")), however, we must fork and exec prior to doing so otherwise we just set the hostname on the root namespace.

Just keep forking!

In our next exercise we’ll setup an additional case argument for our program, allowing the invocation of a fork within the program execution. This will allow us to setup our namespaces and execute important configuration changes to the process before handing control over to the user.

main.go
 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930313233343536373839404142434445464748495051
package main

import (
	"fmt"
	"os"
	"os/exec"
	"syscall"
)

func main() {
	switch os.Args[1] {
	case "run":
		run()
	case "fork":
		fork()
	default:
		panic("nope")
	}
}

func run() {
	fmt.Printf("run() executing %v \n", os.Args[2:])

	cmd := exec.Command("/proc/self/exe", append([]string{"fork"}, os.Args[2:]...)...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	cmd.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID,
	}

	must(cmd.Run())
}

func fork() {
	fmt.Printf("fork() executing %v \n")

	cmd := exec.Command(os.Args[2], os.Args[3:]...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	must(syscall.Sethostname([]byte("containerland")))
	must(cmd.Run())
}

func must(err error) {
	if err != nil {
		fmt.Printf("has error: %v \n", err)
	}
}

Let’s take a look at line 24, we’re doing a bit of recursion here and calling the same program again but prepending the argument fork to the array of arguments passed during the initial execution of the program. In addition, on line 29, we’ve introduced a new namespace called CLONE_NEWPID. This clones a new process tree which is completely detached from the parent. If you execute the above program with go run main.go run /bin/bash, the new PID for /bin/bash is 1, but if you execute ps, you’ll see that the /bin/bash process is certainly not 1. The primary reason is that we’ve not setup a chroot jail. We’ll cover more about chroot in the next exercise, however, if you’re interested in the CLONE_NEWPID you can read about it here.

Groot! cough Chroot!

You will need to download this archive to continue with the following exercise. Once you’ve download it, extract it into a safe location, for example $HOME/go/ubuntufs.

main.go
 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354
package main

import (
	"fmt"
	"os"
	"os/exec"
	"syscall"
)

func main() {
	switch os.Args[1] {
	case "run":
		run()
	case "fork":
		fork()
	default:
		panic("nope")
	}
}

func run() {
	fmt.Printf("run() executing %v \n", os.Args[2:])

	cmd := exec.Command("/proc/self/exe", append([]string{"fork"}, os.Args[2:]...)...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	cmd.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID,
	}

	must(cmd.Run())
}

func fork() {
	fmt.Printf("fork() executing %v \n", os.Args)

	cmd := exec.Command(os.Args[2], os.Args[3:]...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	must(syscall.Sethostname([]byte("containerland")))
	must(syscall.Chroot(fmt.Sprintf("%s/go/ubuntufs", os.Getenv("HOME"))))
	must(os.Chdir("/"))

	must(cmd.Run())
}

func must(err error) {
	if err != nil {
		fmt.Printf("has error: %v \n", err)
	}
}

Those who are familiar with linux chroot will be aware of what’s happening in L44-45, for those who’d like a refresher I recommend reading the wikipedia article about this feature. We’re effectively instructing the child process to set the rootfs to our newly downloaded ubuntufs. Let’s play around on our container and point out the isolation.

First, let’s explore our jailed filesystem, you’ll benefit from having two terminal sessions open via tmux, iterm, etc.

In terminal #1 run main.go with the chroot changes above:

10:40:07 $ sudo go run main.go run /bin/bash
Alias tip: _ go run main.go run /bin/bash
run() executing [/bin/bash] 
fork() executing [/proc/self/exe fork /bin/bash] 
root@containerland:/# ls
bin   dev  home  lib64	mnt  proc  run	 srv  tmp  var
boot  etc  lib	 media	opt  root  sbin  sys  usr
root@containerland:/# touch whereami

In terminal #2, our host system, lets explore the rootfs and try to find the file we just created above whereami:

10:42:46 $ ls /    
bin    dev   initrd.img  lost+found  opt   run   srv       tmp  vmlinuz
boot   etc   lib         media       proc  sbin  swapfile  usr
cdrom  home  lib64       mnt         root  snap  sys       var
10:43:53 $ ls ~/go/ubuntufs
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr  whereami

As you can see, the file we touched inside the container is not in the root filesystem of the host, but in the chroot folder that the container launched within.

Back to our container let’s try and see our process tree and how that is isolated:

root@containerland:/# ps
Error, do this: mount -t proc proc /proc

Hmm, well, at least the system told us what was wrong and we can take that as a queue on how to solve it. The /proc folder is a pseudo-filesystem that must be created with some special parameters to function as the kernel intended. This is actually very straightforward, as we’ll demostrate below:

main.go
 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657
package main

import (
	"fmt"
	"os"
	"os/exec"
	"syscall"
)

func main() {
	switch os.Args[1] {
	case "run":
		run()
	case "fork":
		fork()
	default:
		panic("nope")
	}
}

func run() {
	fmt.Printf("run() executing %v \n", os.Args[2:])

	cmd := exec.Command("/proc/self/exe", append([]string{"fork"}, os.Args[2:]...)...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	cmd.SysProcAttr = &syscall.SysProcAttr{
		Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID,
	}

	must(cmd.Run())
}

func fork() {
	fmt.Printf("fork() executing %v \n", os.Args)

	cmd := exec.Command(os.Args[2], os.Args[3:]...)
	cmd.Stdin = os.Stdin
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	must(syscall.Sethostname([]byte("containerland")))
	must(syscall.Chroot(fmt.Sprintf("%s/go/ubuntufs", os.Getenv("HOME"))))
	must(os.Chdir("/"))
	
	must(syscall.Mount("proc", "proc", "proc", 0, ""))
	must(cmd.Run())
	must(syscall.Unmount("proc", 0))

}

func must(err error) {
	if err != nil {
		fmt.Printf("has error: %v \n", err)
	}
}

Let’s run our newly modified code and see what happens when we execute a ps:

10:47:45 $ sudo go run main.go run /bin/bash
run() executing [/bin/bash] 
fork() executing [/proc/self/exe fork /bin/bash] 
root@containerland:/# ps
  PID TTY          TIME CMD
    1 ?        00:00:00 exe
    6 ?        00:00:00 bash
    9 ?        00:00:00 ps

We’ve successfully isolated our forked process into its own process namespace and its own host namespace. In Part 2 we’ll explore the security implications of containers and the /proc directory on the host, namespace our jailed filesystem, and learn how we can isolate a fork bomb using our new container!

Tags: docker Namespaces cgroups golang

Self-Hosting for Dummies

comments powered by Disqus