VPN testing methodology: How we test VPNs

Our VPN testing methodology is how we ensure that each provider is assessed to the same criteria and that we're only recommending VPNs of the highest quality.

One of the biggest problems with VPN reviews is that many sites will say anything to get you to sign up for whichever service they’re promoting that week. Aside from the ethical issues, this leads to half-baked coverage, with the writer simply repeating outdated or unverified information found during a cursory Google search.

That simply is not good enough. While it’s true that we are reader-supported and may receive a commission from VPN reviews, if you choose to purchase a VPN through BleepingComputer, our primary goal is to inform and educate. We cannot do this without your trust, which is why we’ve decided to reveal exactly how we test VPNs, which characteristics we value the most, and what our process is for keeping our reviews accurate and up-to-date.

This post is going to be quite long and detail-oriented. Before we begin, here are a few short, simple promises:

• We will never recommend a VPN without having thoroughly tested it ourselves
• We call out flaws or shortcomings when we see them
• Where possible, we’ll do our own research to confirm a provider’s claims rather than taking them at face value
• Our team never accepts paid reviews or provides positive coverage for money
• None of our writers have commercial links with the services we review

We’ll now walk you through our testing process and explain what we look for when considering a VPN for recommendation. For easier reading, we’ve split this up into several key areas:

Privacy

The first objective of a VPN is to ensure its users’ activities remain private. Any service that cannot do this, or which chooses to undermine its own protections by keeping activity logs, is simply not worth using.

We strongly prefer VPNs with a no-logging policy (sometimes called “zero-logs providers”). This means that they do not record any data that could personally identify you, including:

• What time you connected
• How long your session lasted
• Your real IP address
• Which websites you visited and how long you spent on each
• Search activity, download history, and the contents of your messages

Unfortunately, we have seen instances where VPN providers claim to be log-free when they actually aren’t. That’s why we look into past audit history too. An unscrupulous company might lie about their product, but an internationally-known auditor is far less likely to. Trusting privacy policies will only take you so far, so look to third-party audits. 

It’s worth noting that not all logs are inherently bad. Some providers gather limited amounts of diagnostic data to help them identify and resolve problems. This is why we won’t mark a provider down for collecting data such as the date of your last connection, the total bandwidth used, and your chosen server location, provided it’s aggregated in such a way as to protect your privacy. However, we will note in our reviews when that type of diagnostic data logging is conducted to allow users to make a fully informed choice.

Finally, we consider where the VPN is based. Some countries have stricter data-retention laws than others, and in places like Russia, the government has the right to examine a VPN provider’s servers, which is a major privacy issue. Ideally, the country of origin would not be a member of a global intelligence-sharing alliance such as the 14 Eyes. The risk here can be mitigated to some degree by a no-logging policy and privacy-first infrastructure, but it’s still worth considering.

Security

Your VPN’s main weapon against surveillance is its encryption. Strong encryption prevents your traffic from being read and would take many centuries to crack, whereas weak encryption effectively offers no protection whatsoever. Here are our current minimum encryption requirements:

• Supports the AES-256 or ChaCha20 algorithms
• Works with respected, secure VPN protocols like OpenVPN, WireGuard, or IKEv2
• Uses SHA-256 authentication or better
• RSA and Diffie-Hellman keys must be at least 2,048 bits long
• Service must provide perfect forward secrecy; that is, compromised keys must not allow an attacker to decrypt session data

This is just the beginning, however. The best encryption in the world doesn’t mean anything if user data can be compromised in other ways. For this reason, we run a series of tests to see if it provides DNS leak protection, along with preventing IP leaks and WebRTC leaks. We also intentionally cause internet drop-outs to verify that the kill switch and automatic reconnection features work as expected.

VPN companies are beginning to realize that third-party organizations represent a critical flaw in their infrastructure. To minimize the risk of outside interference or unpatched vulnerabilities, we expect providers to use their own DNS servers rather than renting them or sending requests through a public provider. If a company has gone even further, for instance, by becoming its own hosting provider, that’s even better.

Speed

VPN speeds are highly variable and can fluctuate based on a wide range of factors including the time of day and your distance from the chosen server. Using a VPN will almost always result in a slight speed decrease but the biggest limiting factor on your speeds is how fast your base internet connection is. For clarity, we perform speed tests using a virtual machine and a 5 Gbps connection. This ensures no other network activity or bottlenecks can interfere with our results.

To provide the fairest comparison, we test every VPN’s speed in the exact same way. We choose the service’s fastest secure protocol (usually WireGuard or OpenVPN) and run Ookla’s Speedtest at three different times of day, using three different servers (usually one in the US, one in Germany, and one in Japan). This gives us nine results, which we average to get a single, representative download speed.

Theoretical data is one thing but we also have to make sure that the VPN services work well in practice. To do this, we’ll perform a series of stress tests such as streaming 4K video, playing competitive online games, and making video calls to see if there are any noticeable problems. It’s important to note that we only use general-purpose servers for these experiments – streaming-optimized or P2P-friendly servers are great but won’t give us a good idea of the average user’s experience. Slow speeds should definitely be avoided. 

Multi-platform support

We prioritize providers that make it as easy as possible to protect a wide range of devices. The most obvious route here is to create dedicated apps for specific platforms, but while the vast majority of VPN companies offer apps for Windows, Android, macOS, and iOS, many still lack Linux GUI apps or support for streaming devices such as Amazon Fire TV.

There must also be a robust set of guides and tutorials for systems that don’t support full VPN apps, such as routers, games consoles, and older smart TVs. This not only streamlines the configuration process but also reduces the chance of damage to the user’s device by limiting the opportunity for anything to go wrong.

If a VPN offers browser extensions, we will test these in addition to the regular apps. However, given that these generally only include a fraction of the functionality that the main apps do, and as they only secure browser traffic, they aren’t intended as a representative example of the service and as such, won’t be the main focus of our review.

Crucially, if there is a limit on the number of devices you can connect, it must be reasonable – most people have more than one, and it’s important they’re able to browse securely across all of them. Five simultaneous connections is a good starting point but we’d rather see these limits removed entirely.

Regional restrictions and censorship

One of the biggest draws of a VPN is that it can help you regain access to your usual services, even in places where they may not be available. This works because websites can only see the VPN server’s IP address, making it seem like you’re browsing from wherever your chosen server is. Region-locking technology has only gotten stricter in recent years, and it’s becoming more difficult to find services that work with all of the most popular platforms.

If your VPN software is detected by a geo-blocked service like Netflix, you’ll be asked to turn it off. This isn’t just an inconvenience; it’s also a risk because sooner or later, you’ll forget to reconnect. To minimize the risk of this happening, we check to see if it’s possible to access popular streaming services such as Disney+, Hulu, and BBC iPlayer while connected. If the service has an app as well as a website, we’ll test that too, as we’ve found that VPNs sometimes work with one but not the other.

A handful of VPNs have historically been able to beat governmental censorship in places like China. We’ve verified these claims ourselves in the past, but the situation can change very rapidly, meaning there may be intermittent periods of disruption. These VPNs often have multiple copies of their website (called mirror sites), off-site support channels, and anonymous payment options.

Ease of use

We’ve seen several competitive VPNs with confusing, cluttered interfaces. This doesn’t remove them from the running entirely, but it does make them harder to recommend to newcomers, especially if they insist on hiding important settings in submenus.

VPN users should not need a degree in computer networking. Services that understand this simple fact can show it in several small ways. They might provide a small caption explaining what each setting does, let users connect in one click, or have key security features enabled by default. Essentially, we’d like you to be able to download the service and start browsing securely right away, with no need to configure anything for yourself if you don’t want to.

Customer service

A first-rate customer support team is one of the things that really sets top-tier VPNs apart. We’ll always reach out at some point during our review, making sure to record how long it took to get a response and whether our query was answered to a satisfactory degree.

We prefer VPNs with multiple avenues of support. Live chat is usually a good first resort as these allow you to resolve problems quickly and easily but sometimes this isn’t an option, either because it’s offline or because your question is too technical. In these cases, submitting a ticket system or sending support an email is likely your best bet.

If customer support isn’t available 24/7, we’ll be sure to mention its hours of operation. We’ll also try to find out whether the desk is manned by in-house staff or outsourced to a company like Zendesk, simply because it can be hard to get detailed answers from third-party support.

One thing every VPN should have is a dedicated support section on its website. This must include a variety of troubleshooting guides, frequently asked questions, and tutorials for common tasks. Without this, a service will not qualify for a spot on our list. We’d also like to see video guides, though these are rarer and generally hosted on the company’s YouTube channel.

Network

The scale of a VPN’s network isn’t actually all that important. If every provider’s servers were identical, the larger network would be more reliable overall but that’s just not the case in reality. Some may have more system resources or be better-optimized than their rivals, leading to more consistent performance despite potentially having a smaller network.

Instead of focusing on server numbers, we look at the number of locations on offer. Generally, we’d expect to see a paid service offer servers in around 60 countries, though this varies, with some VPNs covering up to 100. We pay close attention to which countries are supported too –  if a provider has servers somewhere unusual, like Greenland, we’ll be sure to bring that up.

It’s becoming increasingly common to see virtual servers. These aren’t physically located where they claim to be and are instead run on a machine somewhere else in the world. The advantage of this is that users can get an IP address pointing to somewhere like Egypt, Turkey, or Pakistan, without any risk of governmental interference. On the other hand, many VPNs fail to properly disclose where their virtual servers are actually located, making it near-impossible to tell which country’s data-protection laws apply to you.

Price

A high monthly price is not an effective indicator of quality. We’ve seen many extremely barebones services attempt to charge top-market rates based on nothing more than brand recognition, and that simply won’t fly. In short, the higher the price, the higher our expectations.

First, we look at how much a provider charges for a single month of coverage and ask a few key questions:

• How does this service compare to similarly-priced competitors?
• Can users reduce the price by signing up for a longer term? Is the discount worth it?
• Does this VPN ever run promotions where the price is lower?
• Are there any additional fees to unlock extra functionality in the app?
• Is there a clearly-defined refund policy? If not, why not?
• Could I easily get in touch to cancel if I had to?
• Can I pay anonymously (for example, with cryptocurrency or prepaid cards)?

Some VPNs offer a free-to-use plan with reduced functionality. You’ll often find they limit your speed, access to servers, and the amount of bandwidth you can consume per day (or month). While these freemium offerings can be useful for short periods, they’re not a reliable long-term option. We may experiment with the free plan if one is available but in most cases, our tests will be performed using the premium version of the software.

VPN Testing Methodology: FAQs

Can I expect to get the same VPN speeds?

At Bleeping Computer, we test VPN performance using a dedicated test server in Ohio. That 5 Gigabit server allows us to check the VPN's peak performance at server locations in three different countries (the US, the UK, and Hong Kong). We use the data from these international locations (which we test three times each at three-hour intervals) in order to arrive at a global download average for each VPN. 

The important thing to remember is that this average is the top speed provided by the VPN and you should not expect your internet to be this fast at home because a VPN cannot speed up your internet. If this is confusing, we will do our best to explain more about how it works below.

Most internet users do not have a 5 Gigabit connection like the one on our test server. Instead, they pay for normal home broadband or mobile internet with speeds of between 30 and 100 Mbps. The speed of the internet you have at home or on your mobile device depends on the subscription you pay for. This is controlled by your ISP and your VPN cannot make this any faster. Thus, if you want faster internet speeds, the solution is to contact your ISP and change your internet plan.

In fact, the reality is that even the fastest VPNs will slow down your internet a little The benefit of a fast VPN is that it has a lot of headroom and can move your traffic through its service and onto its final destination (and then back to you) without impacting your overall speeds too much. However, it still has to encrypt your data and tunnel your traffic to its server, which inevitably incurs some lag. 

It is also worth noting that the hardware you are using can also impact your speeds. If your smartphone or computer is very old and lacks memory or processing power, it may struggle to run the VPN app and maintain the VPN connection. This can also negatively impact the speed of your internet connection while using a VPN.

Is there any way a VPN can speed up my internet?

It is worth noting that although a VPN will usually slow down your internet, there is one specific circumstance when a VPN can help. Some ISPs engage in bandwidth throttling when you carry out data-intensive tasks such as gaming and streaming. If you believe your ISP is throttling you when you carry out certain tasks or visit specific web services, then using a VPN may help. The encryption provided by a VPN stops ISPs from detecting your web visits, which can stop them from throttling your bandwidth. This means it is worth testing a VPN to see if it can help you combat throttling.

Did you know

The following information is available to any site you visit:

Your IP Address:

 

Your Location:

 

Your Internet Provider:

 

BLEEPINGCOMPUTER RECOMMENDS:

Using a VPN will hide these details and protect your privacy. We recommend using NordVPN - #1 VPN in our tests. It offers outstanding privacy features and is currently available with three months extra free.

Get NordVPN