Tens of Thousands of Chromebooks Fail Due to Bug in Security Product

A large chunk of the 120,000 Chromebooks deployed at Maryland's Montgomery County schools went down last week after computers using Symantec BlueCoat security software weren't able to handle TLS 1.3 connections that Google started supporting with the release of Chrome and Chrome OS 56.

According to a report filed on Google's bug tracker, the problem affected both Chrome OS users, who weren't able to log into their devices, and Chrome browser users, who weren't able to connect to websites (such as many Google services).

Tens of thousands of school computers affected

A representative of the Maryland Montgomery County said this issue surfaced after around 50,000 of the county's Chromebooks updated to Chrome OS 56. As soon as this happened, problems started arising.

"Anywhere upwards of 30% of those 50,000 Chromebooks are stuck in a state of flickering between a login screen and a 'Network not available' screen," the county representative said. "Occasionally, you can see a SSL_HANDSHAKE_ERROR briefly at the login screen before switching back to the 'Network not available' screen."

Similarly, "45,000 - 46,000 [non-Chromebook] PCs have updated themselves to Chrome 56. Not all PCs are broken, but some are," he added.

The Montgomery County school representative solely blames BlueCoat 6.5, "which doesn't appear to have native support for TLS 1.3."

Buggy TLS 1.3 support caused the problem

TLS 1.3 is the next iteration of the TLS protocol used in negotiating and establishing HTTPS connections. The TLS 1.3 standard isn't finished yet, but browser makers have already started supporting it in their products.

For example, Mozilla plans to roll out TLS 1.3 in Firefox 52, scheduled for release next week. Similarly, Microsoft is also experimenting with TLS 1.3 in Edge and has plans to roll it for Internet Explorer as well.

The Google engineers who investigated the recent issue said it was entirely the security product's fault.

"When Chrome attempts to connect via TLS 1.3, BlueCoat hangs up connection," the engineer said.

This happened because BlueCoat, acquired last year by Symantec, performs a man-in-the-middle operation to intercept HTTPS communications, inspects traffic for malicious threats, and reestablishes a TLS connection to relay traffic to its destination.

BlueCoat blamed for failing to support TLS 1.3

Google says that BlueCoat has failed to support TLS 1.3 even if the browser maker informed the company months in advance of its plans to add TLS 1.3 support to Chrome and Chrome OS 56.

"They were made aware of TLS 1.3 several months ago, but evidently did not test their software per our instructions," another Google engineer said.

New research published last month has shown that many antivirus vendors trash HTTPS connections in order to inspect traffic, but fail to set them up at the same level of security and privacy as they were before the MitM operation.

Google and Firefox security experts have also been one of the loudest critics of antivirus vendors who engage in this type of practice.

Google says that a proper way to have handled the TLS 1.3 error was if the incompatible security product would have dropped TLS to 1.2 if it couldn't handle 1.3.

Problems fixed by downgrading to Chrome OS 55

In the meantime, the Montgomery County schools have reverted to Chrome and Chrome OS 55 and are now up and running.

Another way to mitigate the issue was if users would set up chrome://flags/#ssl-version-max to 1.2, which is a setting that tells the browser to never go above TLS 1.2.

Furthermore, even if it wasn't at fault, Google has stopped the rollout of TLS 1.3 in Chrome and ChromeOS 56.

"To be clear, ultimately this is an issue with proxies/firewalls that are not compatible with TLS 1.3," said a Google engineer. "A future version of Chrome will re-enable TLS 1.3."