BSidesSF 2025

Thesis

Security bug discovery and research is open to everyone. If I can discover a bug like this then you can too.

Detailed Description

This talk introduces the concept of Cross Site Request Forgery and the commonly employed defenses against it in the context of beginner web application security. After this introduction I will walk through the discovery of a CSRF vulnerability in the popular gorilla/csrf Go library in an accessible manner, explaining the tools and process.

The goal of the talk is to demystify the process of vulnerability research and to encourage the audience to perform their own by verifying their working knowledge of their defensive tools through creating small applications that demonstrate the exploits they protect against.

Intended Audience

This talk is intended for an audience of web developers or those interested in web application security. Though the talk focuses on a Go library the lessons are not intended to be language-specific. It is intended to be an introductory-level talk with minimal prior web application security knowledge required.

Session Outline

CSRF & related concept introductions (10 minutes)

Build up the relevant context to explain the specifics of the vulnerability.

1. Personal Introduction & background (1 min)
   - This work was undertaken as part of my role on the Tailscale security team where we are responsible for a large amount of Go software.

2. Introduction: Illustrated example CSRF scenario & default browser behaviors (4 min)
   - Introduction to context of web application security & CSRF definition.
   - Set the stage with browser & server actors and the goal: to prevent malicious form submission.

• The setup: Eve can control the contents of the page popular.com that she knows that Alice will visit.
• She uses this to inject a form pointed to bank.com that creates a bank transfer from Alice to Eve and some Javascript to exercise it when the page loads in Alice's browser.
• Alice, who is logged into bank.com, navigates to popular.com. Her browser loads the malicious form that creates a bank transfer, and her browser submits it due to the malicious Javascript.
• HTML form submission was the basis for some early inter-website internet interactions, and thus continues to be permitted behavior even between sites of different origins. By default browsers will send all cookies that are associated with the target domain alongside HTTP POST requests from form submissions originating from other websites as if you were navigating directly.
• This allows for malicious interactions e.g. if an attacker can substitute the form destination & convince you to submit.
• Later rules known as the "Same Origin Policy" limit interactions between origins that originate from Javascript scripts executing.

1. Walk through how "double submit" CSRF protections work (3 min)
   - The "double-submit" CSRF protection pattern is one whereby forms are templated with a random secret value that is also stored as an authenticated cookie in the browser.
   - The server then requires that every protected form submission include this new value, compares it against that stored in the authenticated cookie and rejects form submissions where they do not match.

2. But wait there's more: script injection, domain allow-listing and/or the need to permit inter-origin request submission. (2 min)
   - But what happens when attackers like Eve get complete content control and can set cookies?
   - Additionally today it's common for companies to have web applications scattered across many different domains that need to interact with each other. For example what if your marketing site needs to host a form that submits to another application in your company?
   - Consider also a scenario where instead of using attacker.com Eve performs a Machine-in-the-Middle attack between Alice and bank.com, and uses this vantage point to serve a malicious HTTP-only version of bank.com containing forms pointing to the real https://bank.com.
   - In these scenarios servers rely on the fact that spec-compliant browsers will send Origin and Referer headers that they can scrutinize to weakly implement their own version of the same origin policy.
   - Many CSRF libraries do exactly this, and enforce "Origin" checks on all requests based on these headers to prohibit requests from unauthorized domains.

Bug introduction and discovery (~10-12 minutes)

Walking through the steps of discovery.

1. It starts with an email (1 min)
   - An email to security@ asks specifically how does the gorilla/csrf library that we use protect against the cross-origin request forgery if Eve can set Alice's cookies? For example what happens if an attacker were to gain XSS access to the Tailscale marketing site, could they use this to attack login.tailscale.com?

2. My first answer was absolutely incorrect. (1-2 min)
   - My first response explaining how it all worked was wrong! I misremembered exactly how cookie rules worked and thought that Browsers would protect us.
   - The original reporter politely pointed out to me that I was incorrect. I'm grateful for their patience with me.
   - I set about to thoroughly research a correct answer so that we could both leave the conversation feeling comfortable in our working knowledge of CSRF and its defenses.
   - Encouraged by my illustrious colleagues I built a small demo web application to verify the workings of each of the Cross-Origin request scenarios in my written answer.

3. The demo application (3-4 min)
   - The demo application is a Go binary that serves a HTTPS server that listens on multiple hostnames, namely
   - target.example.com - this is the target application hosting the form that Eve wants Alice to submit
   - attack.example.com - this page hosts a malicious form that posts cross-origin to target.example.com
   - To allow for testing the behaviors of browsers in a secure context where pages are served over HTTPS we use the mkcert tool by Filippo Valsorda that simplifies provisioning TLS certificates and ensuring they are installed in our OS trust store so that they can be used by browsers.
   - This ensures that browsers treat them as valid TLS certificates i.e. exactly as if they had been issued by a trusted Certificate Authority such as Let's Encrypt but only for browsers on your single machine.
   - Importantly this allowed me to test the behaviors of the application in a manner as similar to our production environment as I could feasibly make it.
   - I plugged it all together and hit the "submit" button expecting to see a "403 unauthorized", but lo and behold it worked!

4. The bug hiding in plain sight (2 min)
   - The code in question has no obvious flaws, and enforces the Referer check when it determines the current request is being served over HTTPS.
   - There were even unit tests that explicitly cover the scenarios that we were testing in our application.
   - Why then would this branch not run and prevent my malicious form submission?

5. The (sometimes meaningful) discrepancies between development & production environments. (3 min)
   - The net/http#Request documentation gives a hint as to what the critical difference is between the unit test environment and that of our test application.
   - The URL object is not always fully populated on http.Request objects as in Go they can be used to make outbound requests as a client but also to serve inbound requests as a server.
   - In the client and unit test context this URL object has Scheme properties specified, and so the unit test passes without issue,
   - But in production URL.Scheme is empty and so the branch that contains all of the security defenses is never executed in practice!
   - This bug had existed since the very first introduction of the Gorilla library. In effect it was only performing Request Forgery protection, but not any of the Cross-Site request forgery protections for the majority of consumers.

6. We need to Go deeper: Unfortunate coincidental accidents in the Go standard library (3 min)
   - This bug and misunderstanding of the http.Request object behavior in gorilla/csrf was enabled by an underlying bug in the Go standard library's httptest.NewRequest function that was used to test it.
   - Specifically: the req.URL.Scheme property is set on the request objects that http.NewRequest returns and that many programs consume in unit tests. This behavior is subtly, but importantly different to how requests are parsed by servers in production
   - Can we fix this transparently for all users by patching the net/http library and correctly setting req.URL in production in servers?
   - Unfortunately this is not possible as to do so we would need to conclusively ascertain whether the request is being served over TLS or not.
   - While we can determine whether our specific Go program is terminating TLS by inspecting the req.TLS property, the potential existence of HTTP proxies that terminate TLS upstream of our program means that we may get false-negative results and so this is not a suitable determining factor.

7. Lessons learned (4 min)
   - Unit tests are the start of testing, they don't have to be the end. Nothing beats kicking the tires. This bug lay undiscovered for nearly a decade with passing unit tests all this time. In contrast, testing in a production-like environment surfaced the vulnerability immediately.
   - The details can matter and are worth testing when it comes to differences between your dev, test, and production environments.
   - This bug relied on a complicated environment setup with TLS working for domains to evaluate, a setup friction that likely deterred attempts. Tools like mkcert can make a difference here lowering the cost of throwaway experiments vs having to use a real production setup.
   - Ambiguous types can lead to dangerous behavior in applications where people mis-handle them. While it is fair to say that the net/http.Request documentation warned in relatively clear terms that the properly would not be set, there was nothing explicitly prohibiting users from "holding it wrong". When writing APIs and related documentation we should give consideration to how they might be misinterpreted and the potential consequences. An ideal type system would have prohibited this from ever being considered a valid program to execute.

8. Thanks / remainder Q/A