Using Consul’s transparent proxy on virtual machines

Transparent proxy helps secure services in a service mesh by forcing traffic through a proxy like Envoy to prevent unauthorized direct access to the application. It allows a service to resolve and connect to another service using DNS instead of through proxy upstreams listening on localhost. In HashiCorp Consul, a service can connect to an upstream service using transparent proxy and Consul’s DNS as long as an intention allows traffic between the services.

This post shows you how to configure Consul’s transparent proxy on virtual machines (VMs). Your operating system must support iptables in order for transparent proxy to restrict traffic on the VM. Furthermore, you must run Consul 1.11.1 or later to use transparent proxy. For a full example, check out my Terraform configuration that uses a user data script to set up transparent proxy. While the example uses Terraform to set up Consul with transparent proxy through user data, you can adapt its scripts for your own VM configuration.

Configure Consul client

Before your service can use transparent proxy, set up a Consul client on your virtual machine with service mesh and DNS forwarding enabled. Make sure Consul runs under the consul Linux user. You will need its user ID to make it easier to exclude Consul-related traffic from transparent proxy redirection.

If you have access control lists (ACLs) enabled, create a Consul ACL token for the node identity. This token includes a policy to write nodes and read services in Consul.

consul acl token create -node-identity=${node_name}

The Consul client configuration sets both agent and default tokens to register the virtual machine and resolve services with Consul for service discovery.

{
  "server": false,
  "acl": {
    "tokens": {
      "agent": "${CONSUL_HTTP_TOKEN}",
      "default": "${CONSUL_HTTP_TOKEN}"
    }
  },
  "connect": {
    "enabled": true
  },
  "ports": {
    "grpc": 8502
  }
}

In order to use Consul service discovery on the virtual machine, forward the virtual machine’s DNS to Consul. When you use Consul service discovery, you can configure your service to look up another service in the mesh using its virtual service name. Set up DNS forwarding for your DNS server. For example, add Consul to the systemd-resolved configuration file and restart the process to forward any DNS requests to the service mesh to Consul.

[Resolve]
DNS=127.0.0.1:8600
Domains=~consul

Start the Consul client. The virtual machine should register as a node in Consul. Before registering the service and starting the proxy, create a user for the proxy named envoy. Consul needs this user ID in order to exclude traffic to and from the proxy.

Register the service

Define a service to add it to the mesh and enable transparent proxy. The service definition sets the service’s name, port, and proxy mode.

{
 "service": {
   "connect": {
     "sidecar_service": {
       "proxy": {
         "mode": "transparent"
       }
     }
   },
   "name": "${service_name}",
   "port": ${service_port}
 }
}

Consul assigns a virtual IP in the 240.0.0.0/4 range to a service with transparent proxy enabled. This allows downstream services to connect to upstream services using a unique IP assigned by the service mesh. You can register the service with the Consul API or add the service definition to the Consul configuration directory on your VM. Adding the service definition to the configuration directory registers the service each time the Consul client restarts.

Redirect traffic to proxies

After installing and configuring Consul and Envoy, redirect traffic to and from the virtual machine using transparent proxy. If you have a mesh configuration entry defined, check if TransparentProxy.MeshDestinationsOnly is true. This setting will deny all inbound and outbound requests to the VM once you redirect traffic. As a result, make sure you install the application and its dependencies before redirecting traffic since transparent proxy will deny requests to download packages by default.

Issue the consul connect redirect-traffic command to enable transparent proxy on the VM for the named service. The command should include the user ID for consul and envoy in order for transparent proxy to exclude traffic from Consul and Envoy from redirection.

consul connect redirect-traffic \
   -proxy-id="${service_name}-sidecar-proxy" \
   -proxy-uid="$(id --user envoy)" \
   -exclude-uid="$(id --user consul)"

Once you issue the command, any connection you have to the VM (including SSH) will disconnect. The command adds iptables rules that will deny existing inbound traffic unless you have specified exclusions. If you need to exclude inbound or outbound traffic from specific ports or IP addresses, append additional arguments specifying exclusions. For example, you may need to exclude inbound traffic to the SSH port for login:

consul connect redirect-traffic \
   -proxy-id="${service_name}-sidecar-proxy" \
   -proxy-uid="$(id --user envoy)" \
   -exclude-uid="$(id --user consul)" \
   -exclude-inbound-port=22

Next, install and run Envoy on the VM. Issue the consul connect envoy command for Consul to run Envoy. If you have ACLs enabled, make sure that you create a service token and pass it to the command using the -token argument.

consul connect envoy -sidecar-for=${service_name} -token=${CONSUL_HTTP_TOKEN}

To automate transparent proxy configuration, you can add a script enabling traffic redirection to the proxy’s system file before you start the proxy. After stopping the proxy, you may want a script to clean up iptables. Use a systemd unit file to use the Consul ACL token from an environment file and redirect traffic before starting the proxy:

[Unit]
Description=Consul Envoy
After=syslog.target network.target consul.service
Wants=consul.service

ConditionFileIsExecutable=

from HashiCorp Blog https://bit.ly/41hUj3G
via IFTTT