Using Proxyman to debug TLS Docker traffic
Recently I found myself wanting to dig into how a GoLang application was chatting with the outside world. Running the application locally used the Serverless Framework Offline Plug-in, which is incredibly useful for running manual and automated tests on your development machine.
My tool of choice has traditionally been Charles Proxy. It’s been around for years and makes inspecting HTTP traffic incredibly easy. It made me sad to see it still running on an Intel compiled codebase on the Apple silicon processor. Out of curiosity, I went looking for an alternative and I found a great new option, Proxyman.
Running the code on the host works a treat with Proxyman’s automated proxy override feature. Unfortunately, it’s not as straightforward when intercepting requests originating from a Docker container.
This post will start with proxying traffic on the host machine. We’ll then work up to proxying from Docker and then specifically a container used by the Serverless Framework Offline plug-in.
Using Proxyman locally
Let’s start by stepping through setting up SSL proxying locally, before introducing Docker into the mix. We’ll first reduce the level of noise and switch off Proxyman’s Override macOS proxy setting found in the Tools > Proxy Settings menu. This means Proxyman will only capture requests that are explicitly routed to the proxy server.
We’ll start by making a request without Proxyman involved, using a noddy Go file to make a secure request to https://www.httpbin.org/get. If you want to follow along, make sure you have GoLang installed and save the Go code as perform-request.go.
Running the code should result in a request being made and httpbin responding. Straight forward enough.
| |
With Proxy Override off, you shouldn’t see the above request captured in Proxyman. Let’s start proxying the HTTP request through Proxyman by explicitly telling the Go runtime to use the Proxyman server. Setting the HTTPS_PROXY specifies the proxy settings that the GoLang HTTP client will use when performing the outbound request.
| |
After running the command above, take a look in Proxyman and you’ll see https://www.httpbin.org in the list of intercepted requests. Selecting it you’ll see information about the request, but you won’t be unable to see the body contents of the response. You’ll need to enable SSL Proxying first by clicking the ‘Enable only this domain’ button. This will configure Proxyman to capture and decrypt the traffic for that domain when handled by the proxy application.
Run the same command again and return to Proxyman.
| |
You should see the full response payload in the Proxyman app 🎉. If you’re seeing a x509: certificate signed by unknown authority error, you’ll need to install Proxyman’s custom CA onto your machine. From the menu select Certification > Install Certificate on this Mac... and follow the instructions and then try running the command again.
Proxying Docker traffic
Let’s start intercepting traffic when it originates from a Docker container. We’ll run the same code, but this time from a container using the GoLang v1.17 image.
| |
We’ve passed the HTTPS_PROXY env var into the container, but rut roh, a x509 error!
| |
This is because our Docker container doesn’t have the custom CA certificate installed. It does not trust the certificate that is being presented by Proxyman when requesting httpbin.org. Your host machine may trust it, but your Docker container is in its own little world. It needs to be explicitly told to trust the Proxyman certificate.
We’ll first need to grab the certificate in PEM format and pass it into the container. Download a copy of the custom CA cert by visiting Certificate > Export > Root Certificate as PEM... from the Proxyman app. Save the certificate as proxyman.crt without a password. For our containerised GoLang example, we will place the certificate in a location we know the runtime will pick up. A list of locations can be found at https://go.dev/src/crypto/x509/root_linux.go. We’ll choose /etc/pki/tls/certs/ for this example.
The command we used before will need an additional config flag. We’ll mount the Proxyman certificate into the known certificate location for the Go runtime to pick up.
| |
🎉 When the request is performed successfully we can see the request and response in the Proxyman app once again.
Serverless Framework
The Serverless Framework allows us to provision resources and deploy code using serverless cloud services. Combined with the Serverless Offline plugin, you can test your setup with a locally emulated API Gateway and Lambda setup. Using the command sls offline start --stage local --useDocker will start the emulated API Gateway server for you to make requests to. On request, the gateway will run a Docker container with the lambda of choice and respond accordingly.
The problem with this setup is we can’t control which certificates the container trusts. The equivalent of docker run ... is happening deep in the depths of Serverless Offline. One option would be to pass the certificate into the container as a base64 encoded env var and have the application code do something with it, but that’s a faff. We’re just trying to debug what we have, quick and easily.
To update a Serverless Offline setup we’ll need to perform two steps.
- Update the
serverless.ymlfile to include the environment variables that point to the Proxyman proxy, just like we did above. - Update the
lambci/lambda:go1.xcontainer to include the Proxyman certificate.
First, the easy bit, updating your serverless.yml. Under provider.environment you’ll want to add the following key/value pairs.
| |
Next, the more involved bit. We need to copy the Proxyman CA cert into the lambci/lambda container image so it’s available to the GoLang runtime. It’s important to note that you’ll need to do this after you’ve started serverless offline, and every time you restart it.
Create a new Dockerfile with the contents below. Make sure that your proxyman.crt file is in the same directory as your Dockerfile.
| |
Run docker build --no-cache -t lambci/lambda:go1.x . to update the container image with your version containing the Proxyman certificate.
Then go ahead and hit your Serverless Offline endpoint and you should see the proxied requests passing through Proxyman 🎉! Remember, you’ll need to build the updated image after you’ve started serverless offline and every time you restart it.
Charles Proxy
The above steps will also work with Charles Proxy. You’ll just need to grab the custom CA certificate from Help > SSL Proxying > Export Charles Root Certificate and Private Key instead. The quickest way to convert that file into something we can use above is to import it into macOS Keychain Access. We can then export the newly imported certificate using File > Export items... and save it in .pem format. When running locally, you’ll need to explicitly trust the Charles proxy root certificate by visiting Keychain access, opening the certificate and from under the Trust dropdown select ‘Always Trust’ for Secure Sockets Layers (SSL).
Key takeaways
To make Proxyman and Docker work well together two things needed to happen. First, the operating system, be it your host machine or a Docker container, needs to trust the Proxyman custom CA. In the example above we placed the certificate somewhere we knew the GoLang runtime would pick it up. Depending on your OS, language runtime, or configuration, that location will likely differ.
Second, the proxy setup works better when configuring the running container with the HTTP[S]_PROXY vars. When configuring the global Docker transparent proxy with the Proxyman address, I found that key DNS information gets lost along the way. Instead of the hostnames you’d expect, you’re presented a list of resolved IPs. It’s not the end of the world, but having the original hostname makes reviewing and filtering a lot more straightforward.