Christine Dodrill - Blog - Contact - Gallery - Resume - Talks - Signal Boost - Feeds | GraphViz - When Then Zen

Unix Domain Sockets for Serving HTTP in Production

Securing production servers can be a chore. It is a seemingly endless game of balancing risks with convenience and not breaking what you want to do. Small, incremental gains are usually a very good idea however. Today we'll learn how to use Unix Domain Sockets to host your HTTP services. This allows you to run your services like normal on production machines without there being a risk of people being able to access the raw HTTP port.

Mara is hmm
<Mara> Wait, what. You're having a service listen on a file? Why would you want to do this?

Mostly to prevent you from messing up and accidentally exposing your backend port to the internet. Firewall configuration is probably the most "correct" way to solve that concern, however this lets you also take advantage of filesystem permissions to fine-tune access down to the exact users and groups that should have access to the socket. In our case we only want ngnix to access this socket, so we can use filesystem permissions (and a unix group) to ensure this. Attackers can't connect to anything they aren't able to connect to.

Mara is aha
<Mara> I see. How do you do this?

At a high level every file in a unix filesystem has 3 kinds of permissions: user, group and "other". Every file has an owner and a UNIX group associated with it. Here's an example using the Cargo.toml of this website's app server:

$ stat ./Cargo.toml
  File: ./Cargo.toml
  Size: 1572            Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d   Inode: 20447261    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1001/   cadey)   Gid: (  100/   users)
Access: 2021-04-01 19:48:44.791162535 -0400
Modify: 2021-04-01 19:48:44.786162545 -0400
Change: 2021-04-01 19:48:44.786162545 -0400
 Birth: 2021-03-25 09:09:35.490311674 -0400

Mara is hacker
<Mara> The stat(1) command lets you query the filesystem for common types of metadata about a given file.

In this case the permissions of this file are 0644, which is a base-8 (octal) number that describes the permissions for the user, group and others. It breaks up something like this:

If we wanted to create a socket that only nginx can access, assuming we share a group with nginx we would need a socket with something like 0770 (user and group can read, write and "execute", everyone else gets denied) for its permissions. Then we would need to chuck it somewhere that both the app backend and nginx have access to and finally configure nginx to do this.

So let's do it! Let's take the venerable printer facts server server and make it listen on a Unix socket. Right now it uses something like this to listen for requests:

warp::serve(
    fact_handler
        .or(index_handler)
        .or(files)
        .or(not_found_handler)
        .with(warp::log(APPLICATION_NAME)),
)
.run(([0, 0, 0, 0], port))
.await;

This configures warp (the HTTP framework that I'm using for the printer facts server) to listen over TCP on some port. This is hard-coded to listen on 0.0.0.0, which means that TCP sessions from any network interface can connect to the service. This is very convenient for development, so we are going to want to keep this behaviour in some way.

Fortunately warp has an example for listening on a unix socket. Let's make the service listen on ./printerfacts.sock so we can make sure that everything still works:

let server = warp::serve(
    fact_handler
        .or(index_handler)
        .or(files)
        .or(not_found_handler)
        .with(warp::log(APPLICATION_NAME)),
);

if let Ok(sockpath) = std::env::var("SOCKPATH") {
    use tokio::net::UnixListener;
    use tokio_stream::wrappers::UnixListenerStream;
    let listener = UnixListener::bind(sockpath).unwrap();
    let incoming = UnixListenerStream::new(listener);
    server.run_incoming(incoming).await;
} else {
    server.run(([0, 0, 0, 0], port));
}

Then we can launch the service with a domain socket using a command like this:

$ env SOCKPATH=./printerfacts.sock cargo run

Let's see how the output of stat(1) changed compared to when we ran it on a file:

$ stat ./printerfacts.sock
  File: ./printerfacts.sock
  Size: 0               Blocks: 0          IO Block: 4096   socket
Device: 10301h/66305d   Inode: 23858442    Links: 1
Access: (0755/srwxr-xr-x)  Uid: ( 1001/   cadey)   Gid: (  100/   users)
Access: 2021-04-01 21:00:51.558219253 -0400
Modify: 2021-04-01 21:00:51.558219253 -0400
Change: 2021-04-01 21:00:51.558219253 -0400
 Birth: 2021-04-01 21:00:51.558219253 -0400

stat(1) reports that the file is a socket! Let's see if everything still works by using curl --unix-socket to connect to the service and retrieve an amusing fact about printers:

$ curl --unix-socket ./printerfacts.sock http://foo/fact
The strongest climber among the big printers, a leopard can carry prey twice its
weight up a tree.

Mara is hmm
<Mara> Why do you have foo as the HTTP hostname for the request?

Because it doesn't matter! I could have anything there, but foo is fast for me to type. The URL host information usually tells curl where to connect, but the --unix-socket flag overrides this logic.

Mara is wat
<Mara> Wait, what the heck are printer facts?

Blame Foone and #infoforcefeed.

Anyways, let's make the TCP logic a bit more clean in the process. Right now it only listens on IPv4 and it would be nice if it listened on IPv6 too. Let's replace that last else body with this:

} else {
    server
        .run((std::net::IpAddr::from_str("::").unwrap(), port))
        .await;
}

Mara is hacker
<Mara> :: is the IPv6 version of 0.0.0.0, or the unspecified address. It tells most IP stacks to allow traffic from any network interface.

Now let's re-build the printer facts service and re-run it to make sure it still works:

$ env SOCKPATH=./printerfacts.sock cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/printerfacts`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os {
  code: 98, kind: AddrInUse, message: "Address already in use" }',
  src/main.rs:73:53
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Mara is wat
<Mara> Wait, what. Isn't this serving HTTP from a file? Why would it be an address in use error?

Even though it looks like a file to us humans, it's still a socket under the hood. In this case it means the filename is already in use. Working around this is simple though, all we need to do is-

Numa is delet
<Numa> DELETE THIS!

Cadey is angy
<Cadey> Where the hell did you come from?

But yes, we do need to delete the socket file if it doesn't already exist. Let's sneak this bit of code in before we listen on the Unix socket:

if let Ok(sockpath) = std::env::var("SOCKPATH") {
    let _ = std::fs::remove_file(&sockpath); // nuke the socket
    let listener = UnixListener::bind(sockpath).unwrap();
    let incoming = UnixListenerStream::new(listener);
    server.run_incoming(incoming).await;
} else {
    server
        .run((std::net::IpAddr::from_str("::").unwrap(), port))
        .await;
}

Mara is hmm
<Mara> Didn't you just say "if it doesn't already exist"? Why delete it unconditionally and throw away any errors?

Two reasons:

  1. Statistically if the file doesn't exist and the service can't create it when it binds to that path, you probably have bigger problems and it's probably better for the program to explode there.
  2. The filename is passed in as an environment variable. If your environment variable is wrong, we can treat this as a fundamental assertion error and blow up when the file fails to bind.

Let's define this in the NixOS module for the printerfacts service. First we will need to add a configuration option for the socket path:

let cfg = config.within.services.printerfacts;
in {
  options.within.services.printerfacts = {
    # ...
    sockPath = mkOption rec {
      type = types.str;
      default = "/tmp/printerfacts.sock";
      example = default;
      description = "The unix domain socket that printerfacts should listen on";
    };
  };
  # ...
}

This creates an option at cfg.sockPath that we can pipe through elsewhere, such as the start script for the service:

# inside
script = let site = pkgs.tulpa.dev.cadey.printerfacts;
in ''
  export SOCKPATH=${cfg.sockPath}
  export DOMAIN=${toString cfg.domain}
  export RUST_LOG=info
  cd ${site}
  exec ${site}/bin/printerfacts
'';

And then we can go on to setting up nginx. First, let's figure out how to reverse proxy to a unix socket. In nginx configuration land, proxy_pass is the name of the configuration directive that lets you tell nginx to reverse proxy to somewhere. There's an example with a unix socket! This would let us reverse proxy a unix socket to a TCP port like this:

server {
	listen 127.0.0.1:9000;
	location / {
		proxy_pass http://unix:/tmp/printerfacts.sock;
	}
}

For comparison here's how you'd reverse proxy to a HTTP server running on port 42069:

server {
	listen 127.0.0.1:9001;
	location / {
		proxy_pass http://127.0.0.1:42069;
	}
}

So, we just need to change where nginx reverse proxies to in the NixOS config. Let's look down at the nginx config for printerfacts:

# ...
services.nginx.virtualHosts."${cfg.domain}" = {
  locations."/" = {
    proxyPass = "http://127.0.0.1:${toString cfg.port}";
    proxyWebsockets = true;
  };
  forceSSL = cfg.useACME;
  useACMEHost = "cetacean.club";
  extraConfig = ''
    access_log /var/log/nginx/printerfacts.access.log;
  '';
};

The proxyPass option directly translates to a proxy_pass directive, so we can get away with something like this:

# ...
proxyPass = "http://unix:${cfg.sockPath}";

And now we can deploy the service and everything should work right? printerfacts provides a unix socket at the given path and then nginx is configured to use that socket to send back printer facts. Let's deploy it and see what happens:

A picture of the nginx "502 Bad Gateway" error message with a man scolding a router

Oh no. Let's see what journalctl -fu nginx has to say:

$ journalctl -fu nginx
Apr 01 23:29:58 lufta nginx[15396]: 2021/04/01 23:29:58 [crit] 15396#15396: *198
connect() to unix:/tmp/printerfacts.sock failed (13: Permission denied) while
connecting to upstream, client: lol.no.ip.here, server:
printerfacts.cetacean.club, request: "GET / HTTP/2.0", upstream:
"http://unix:/tmp/printerfacts.sock:/", host: "printerfacts.cetacean.club"

Mara is wat
<Mara> Wait, what. Isn't /tmp guaranteed by the filesystem hierarchy standards to always be readable and writable by any user?

Normally, yes. However we are running nginx inside systemd, and one of the things you can do with systemd is make /tmp isolated for given services. This allows you to prevent a service from being able to exfiltrate data inside /tmp. However, this is definitely NOT the behaviour we want in this case. Let's change the systemd unit for nginx to disable this and also make nginx run as the same group as the printerfacts service:

systemd.services.nginx.serviceConfig = {
  PrivateTmp = lib.mkForce "false";
  SupplementaryGroups = "within";
};

Mara is hacker
<Mara> In NixOS, most of the time if the same option is declared in multiple places it will result in a build error. lib.mkForce disables this behaviour and instead "forcibly" sets this value.

Now nginx has the same /tmp as the printerfacts service, everything will work as we expect. Users are none the wiser that I'm using a domain socket here. I get to have another service not bound to the network and I have moved towards better security on my machine!

Mara is hmm
<Mara> What about Prometheus? Doesn't it need a direct line of fire to the service to scrape metrics?

...Time for some percussive maintenance!


I'm experimenting with a new "smol" mode for the Mara interludes as well as introducing a few more characters to the christine dot website cinematic universe. Please do let me know how this works out for you. I think I have the sizes optimized for mobile usage better, but contributions to fix my horrible CSS would really, really, really be appreciated.

I'm considering moving over all of the Mara interludes to use smol mode. If you have opinions about this please let me know them.


This article was posted on M04 01 2021. Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Series: howto

Tags: unix nginx devops nixos systemd

This post was WebMentioned at the following URLs:

The art for Mara was drawn by Selicre.

The art for Cadey was drawn by ArtZora Studios.