As part of our continuous effort to improve our Clean Code technology and the security of the open-source ecosystem, our R&D team is always on the lookout for new 0-day security vulnerabilities in prominent software.
We recently uncovered two critical code vulnerabilities in a personal cloud solution named CasaOS. CasaOS can be installed on any machine thanks to Docker and comes with end-user NAS devices like the ZimaBoard or the X86Pi. Users deploy CasaOS to store their personal data on devices they trust and access it from anywhere.
CasaOS is developed by IceWhale in Go and has close to 17,000 stars on GitHub as we're writing this article.
A CasaOS dashboard.
These security vulnerabilities, tracked as CVE-2023-37265 and CVE-2023-37266, allow attackers to get around authentication requirements and gain full access to the CasaOS dashboard. From here, attackers can access the data stored on the device, but that's not all.
Because of CasaOS' extensibility and support for third-party applications, they can also execute arbitrary commands on the system to gain persistent access to the device or pivot into internal networks. There are reports that an exploit for Plex Media Server, another personal cloud system, was used in the LastPass breach, and this initial foothold to get access to the employee's internal network.
While we now are releasing the technical details of our findings several months after the vendor addressed them, we were made aware of public exploits based on the study of the patch only 10 days after the security release. That means that all unpatched instances are already at risk. We urge all CasaOS users to upgrade their instances to the latest available release (v0.4.4-1 at the time of writing this article).
Let's dive into the technical details of these security vulnerabilities and see what we can learn from them!
To follow this vulnerability, we must first understand that CasaOS is not a standalone software but a set of services you install on top of a distribution like Ubuntu or Debian. That means that by default, CasaOS has control over all components processing incoming HTTP(S) requests.
The first component to receive users' requests is
casaos-gateway, the only service to be directly exposed to the network. In a common fashion in the Go ecosystem, it forwards requests to other local microservices depending on the request path.
Many services are listening only on
localhost, waiting for
casaos-gateway to send them traffic:
This can be confirmed by reading the configuration of
casaos-gateway. The entry
runtimepath defines where it will store the route that services later declare by notifying
casaos-gateway on its management port.
These routes are persisted in
Internally, the gateway uses this list and Go's
net/http/httputil/ReverseProxy to forward incoming requests to the right service.
Image taken from https://wiki.casaos.io/en/contribute/development.
A common problem caused by such reverse proxies is that the final service will see all requests coming from the reverse proxy; in this case, the source IP address at the network layer level would always be
localhost! To solve this problem, it's commonly agreed that the reverse proxy uses a header named
X-Forwarded-For to give this information to the application.
Two scenarios can arise:
- The client's request doesn't have an
X-Forwarded-Forheader: the reverse proxy will create one, and put the client's IP address in it, i.e.,
- The client's request already has an
X-Forwarded-Forheader because of another reverse proxy placed in front of
casaos-gateway. In this case, it will forward the header after appending the previous proxy's IP address, i.e.
X-Forwarded-For: 18.104.22.168, 192.168.42.42. The current proxy won't add its own IP address because the next hop can get it from the network layer.
The HTTP RFC does not specify how one should deal with invalid
X-Forwarded-For headers, so most implementations will simply copy the value found in the header when relaying requests.
RFC 7239 introduced a new header named
Forwarded, intending to replace many headers of the
X-Forwarded-* family. Still, its deployment stays very sparse and has the same implications security-wise.
There's a very common foot gun here: the application should only trust this family of headers if it knows there's a reverse proxy in front. Otherwise, the header could come directly from the client and can contain anything!
So far, so good–now it's up to the application to handle this header to find the client's IP address.
But why would we need to know the client's real IP address? Logging is the first thing that comes to most developers' minds. If you put your security hat on, you may also want to use it for rate limiting, access control, or authentication.
Looking at the code of the microservices listed above, there are several cases of security decisions based on the client's IP address. For instance, in the repository CasaOS-Common, used by
casaos-local-storage, there's this snippet:
The authentication middleware is skipped if a request comes from
::1! That means that
gin.Context.ClientIP() probably has logic to handle application-level IP address information (e.g., through
X-Forwarded-For). Otherwise, all requests wouldn't require authentication since they come from
casaos-gateway through the loopback interface.
Digging further into Gin's implementation, we see the following documentation around the
The important bit is "If the headers are not syntactically valid [...] the remote IP (coming from
Request.RemoteAddr) is returned". Here is the validation function applied to all comma-separated values of
X-Forwarded-For starting from the rightmost one:
Interestingly, at , the code bails out of the loop if it finds an invalid IP address, even if it previously found valid IP addresses.
That means that if we're sending an invalid
X-Forwarded-For header with our request, for instance with the value
foobar, it gets relayed by
casaos-gateway after appending its IP address, and the validation of this header in Gin fails. It then falls back to the source IP address of the client–
casaso-gateway, talking from 127.0.0.1!
Requests from this IP address do not require authentication, resulting in an authentication bypass on most API endpoints. It is trivial to demonstrate it on our test instance. Our first request without the
X-Forwarded-For header gets an error 401, while a second request with an invalid
X-Forwarded-For gets through:
We'll now look into our second finding, CVE-2023-37266, before discussing post-exploitation risks.
While investigating CVE-2023-37265, we noticed a strange behavior of the session JWT. Modifying the token's claims and signature did not result in errors, and something was likely wrong with it.
How can we validate this theory? There are now tools aimed at the Bug Bounty community to detect weak secrets, like Ian Carroll's cookiemonster, but even the venerable John the Ripper is good for this job. It immediately confirms the use of an empty HMAC-SHA256 secret:
If the secret isn't really secret, anybody can craft an arbitrary JWT as they please. By crafting valid-looking but unsigned tokens, attackers could bypass the authentication and gain administration privileges on vulnerable CasaOS instances.
As always and out of caution, we won't be sharing exploitation scripts but we believe that it is important to share the full extent of these findings' impact to help users protect themselves. It is also a good opportunity to discuss security best practices when designing software features.
One of the most appealing features of CasaOS is its support of third-party applications through the internal application store or manually through the web interface and their Custom Install wizard.
The Custom Install wizard.
These applications are simply Docker containers deployed on the same host as CasaOS. Docker containers only provide thin isolation between the application and other processes, and CasaOS users can mount devices and folders from the host in the context of the application. Both Docker and the application run as
root, the former on the default user namespace and the latter on its own, so malicious applications can compromise the host through these mounts.
Such abusable features are widespread, especially in software aimed at tech-savvy users who always like to have control over what they use. Attackers can also leverage these features to compromise the system. It grants them persistent access to the device, even across software updates, and helps them to pivot into the victim's internal network.
A good practice is to allow users to opt-out, or even better opt-in, of these features in a configuration file. It lets advanced users turn off dangerous features if they don't need them, resulting in a reduction of the overall attack surface.
Shortly after discovering, confirming, and documenting these security vulnerabilities, our Vulnerability Researchers responsibly disclosed them to the CasaOS maintainers through GitHub's Security Advisories feature. We've had great discussions with them to identify and validate the robustness of their patches before releasing CasaOS v0.4.4.
While we initially recommended stripping all incoming
X-Forwarded-For and similar headers, maintainers wanted to keep the support of potential reverse proxies in front of CasaOS instances.
Another solution was found in 391dd7f, where they decided to rewrite outgoing
X-Forwarded-For headers in a way they would never contain
From a security design standpoint, this solution is not entirely satisfactory because internal services still don't require authentication for requests from
localhost. Attackers could still gain arbitrary code execution from a simple Server-Side Request Forgery on the same host. In the same way, if any of these services are exposed to untrusted users by mistake, attackers would not have to provide credentials to compromise the server.
After validating this finding on our local instance, we wanted to confirm it on the public demonstration instance in a non-intrusive way. We collected a valid JWT but noticed that the signature seemed to use another secret, and unleashing John didn't yield any result.
In fact, CasaOS maintainers already addressed this security vulnerability in the development branch with the commit
705bf1f, a few weeks before our research. The patch updates both route groups to use the configuration's private key and ensure the token's signature is valid.
We still reported our finding because we didn't find an official security advisory for it, and CasaOS decided to assign a CVE since it has been vulnerable for quite some time now.
|2023-07-03||We report all issues to CasaOS maintainers.|
|2023-07-03||The vendor confirms the issues and creates GitHub private advisories to coordinate disclosure and collaborate on the patches.|
|2023-07-14||CasaOS v0.4.4 is released.|
|2023-07-17||CVE-2023-37265 and CVE-2023-37266 are assigned to our findings.|
In this article, we came back to the details behind CVE-2023-37265 and CVE-2023-37266, two critical code vulnerabilities in CasaOS. These are fairly simple security vulnerabilities to identify and exploit, and we are aware of public exploits for them; we encourage you again to update all your CasaOS instances.
Interestingly, even with modern languages such as Go, whole classes of design bugs are still likely to arise in new software. Education is essential to security, and we hope this article helps you spot and prevent similar pitfalls in your code.
In general, identifying IP addresses at the application layer is risk-prone and shouldn't be relied on for security decisions. Many different headers may transport this information (
Forwarded, etc.), and the language APIs sometimes need to interpret nuances of the HTTP protocol the same way. Similarly, all frameworks have their own quirks and can be tricky to navigate without expert knowledge of these common security footguns. Again, the first finding was caused by a documented feature: the framework does what it claims to do.
If you would like to learn more about this topic and how it became very hard for developers to make sensible decisions based on these headers, we strongly recommend reading Adam Pritchard's The perils of the “real” client IP–we only scratched the surface of what can go wrong in our article.
We also recommend users of any personal NAS solutions to consider restricting their network exposure, for instance, with a VPN tunnel. These devices often contain personal data and are connected to our internal networks–they are goldmines to attackers and should be secured as such.