Every day, our team has woken up from a nightmare while screaming “The audit is coming soon, I swear!” This went on for quite some time, and we’re happy to say that the nightmares are over.
We’ve been working on a very large revamp of our VPN stack code-named “FreshScribe”. This new architecture provides several advantages to the reliability of our platform, while also serving as the foundation for quick and flexible updates to the stack with new components as we build out more valuable features.
We are happy to finally share the results of an extensive third-party privacy & security audit of our new VPN stack. Although this comprehensive audit (two actually) took longer than anticipated, we believe it was crucial to ensure the highest standards of privacy and security for our users. We sincerely appreciate your patience and trust in our commitment to your online safety.
Audit Scope
Many components of the new infrastructure were running in production for over a year now, but not all-together as a complete system. These components include:
- Provisioning infrastructure
- RAM-disk VPN servers
- Various micro-services
The audit was conducted on a brand new derivative codebase and VPN host infrastructure that is not yet accessible to all customers. It has been running in production in entirety for several months now, available to a select group of beta testers.
Code and Infrastructure Review
A thorough review of our code and infrastructure was conducted, covering 20 code repositories and micro-services (around 80,000 lines of code). This included custom privacy-focused forks of open-source software such as OpenVPN and Wireguard. The review focused on key areas such as:
- All inputs and outputs
- Linux system capabilities and permissions
- Filesystem and Syscall usage and access
- Distribution and rotation of secrets
- Sanitization of all output to protect user PII
In addition to the source code of the micro-services themselves, the infrastructure-as-code repositories that deploy and maintain the machines were also reviewed.
External and Internal Penetration Testing
There are two classes of machines that either directly or indirectly participate in establishing and maintaining VPN & Proxy connections. Both were tested from outside (external) and with full access (internal) to the machines.
- Distributed Core machines responsible for authentication and connection setup orchestration
- VPN & Proxy Entry Servers
Privacy, Security and Authenticity of Communications
All inter-process and machine-to-machine communication leverages a common library and security scheme for ease of development and audibility.
This stack will be made available to all users in select locations starting the week of July 15th 2024, with the entire fleet expected to be upgraded by the end of Q3 2024.
"Overall, the penetration test against the Windscribe stack yielded minimal security concerns."
"During the retest, all findings that posed any potential risk were promptly remediated."
"Based on the changes deployed during the retest and the source-code reviewed during the original penetration test, it's evident that notable development work has been implemented as a part of the Windscribe
microservice stack to reduce, and/or prevent the disclosure of end user information."
(Concluding Statements - PacketLabs Audit Report - Page 17)
Pre-Production Alpha Audit from December 2022
We also completed a third-party audit back in Dec 2022 on a pre-production alpha version of this system. For those of you that are curious, that report can be reviewed below. Be mindful that the newer report supersedes this one.
Key Changes in FreshScribe
Distributed Authentication and Configuration
The current production VPN node infrastructure must communicate with our centralized core infrastructure to authenticate incoming connections. This means that depending on which VPN node you connect to, there will be inconsistentcies in latency, performance and availability during the connection setup process.
While our centralized core data center has many layers of redundancy (power redundancy, server cluster redundancy, network redundancy), it still leaves a single point of failure datacenter-wise for both authenticating the users’ VPN connection and configuring it with customer-specific settings.
FreshScribe utilizes a distributed-core architecture that both brings the data required for connection setup closer to the VPN nodes and distributes this data geographically to reduce network related availability issues, reduce the first-time-to-packet connection setup and eliminate the single point of failure (availability) issue.
Improved Maintainability and Reliability
There are hundreds of small but meaningful improvements to many of the micro-services involved in the setup, configuration and tear down of VPN connections. We won’t go into every detail but suffice it to say we’ve removed a lot of legacy cruft that gets in the way of continuously shipping improvements. We’ve consolidated codebases to improve maintainability, revamped how many components communicate with each other and added improved health observability to find issues sooner and automate resolution of common problems.
With our new distributed authentication and configuration system, you'll experience faster and more reliable VPN connections. By decentralizing our infrastructure, we've reduced latency and eliminated single points of failure, ensuring a smoother and more consistent connection experience for you.
As a customer, aside from a more consistent experience, you shouldn’t notice any major differences yet, but here is a taste of what we will be adding to the new stack in Q4 of 2024.
Future Plans for the FreshScribe Platform
Zero-Knowledge Connection Configurations
We already go to great lengths to avoid collecting user data by modifying open source tools and libraries to not output source IP information. However when it comes to your DNS configuration with R.O.B.E.R.T or port forwards/static IPs, the legacy stack requires the storage of your settings in our database to reference upon a new connection.
FreshScribe enables our client applications to push this configuration data during the connection setup process. This means that we no longer store this data centrally (and instead it lives on your own device), which is a win-win for both your privacy and our infrastructure overhead.
IP Pinning and Rotation
Static IPs are a great way to improve user experience when using a VPN. The user will experience fewer security related logouts, prompts from services and CAPTCHA requests, as their IP is not changing as often. While this is useful, it can reduce anonymity for two reasons: 1) Static IPs are shared between a smaller pool of users than IPs offered on our regular servers. 2) We need to store which user has access to which Static IP address.
With the zero-knowledge configuration push mechanism, our clients will be able to “pin” to an IP address when connecting to the same location, getting the best of both worlds.
FastPath will also facilitate IP rotation, allowing users to change exit nodes while remaining connected to the intermediary node.
FastPath Routing
FastPath improves your VPN performance and anonymity by introducing the concept of entry, intermediary, and exit nodes that are latency-aware. By offloading your VPN traffic to an entry node close to you, it allows us to better tailor the network experience by ensuring your connection takes the fastest path available on our network. As mentioned earlier, another added benefit is being able to switch your exit location without needing to reconnect!
Anti-censorship Mechanisms
Anti-censorship mechanisms have always been an important part of our product goals and mission. While we have made great progress over the years in combatting the various types of censorship globally, it is a constant cat-and-mouse game of testing and pushing out mitigations for the latest censorship tactic.
With FreshScribe being a more modular and flexible system, we will be able to test, iterate and deploy mitigation experiments in a much more rapid and consistent fashion.
Conclusion
We’re really excited about everything that's in the works, and we're committed to delivering the best experience possible. Thank you for your patience and trust as we continue to enhance our service. We're looking forward to introducing more practical privacy features and innovations and a steady audit cadence going forward. If you have any questions or feedback, please reach out to [email protected]
