Apple search bot leaked internal IPs via proxy configuration

A security researcher discovered that Apple's search bots that had been crawling his podcast series had been leaking internal IPs, due to a misconfigured proxy server.

And, it took Apple just a little over 9 months to fix this leak, for no obvious reason.

What are proxy servers?

Proxy servers act as a middle agent between a device attempting to connect to a destination on the internet, and the destination itself.

For example, if you are accessing bleepingcomputer.com from a corporate setting, your workstation is likely making the request via your company's proxy agent sitting in the middle, which further communicates with our website to serve you the requested pages.

There are many reasons a proxy server might be used.

In workplaces, proxies allow the network administrators to both intercept and filter the traffic. This is useful in blocking access to malicious websites.

Similarly, search engine bots responsible for crawling and indexing web resources may be behind a proxy for security reasons.

Unless anonymity is expected (as is the case with some VPNs), most proxy servers, when connecting to a server on behalf of another device, include the originating device's IP information within the HTTP request.

For example, a proxy request may contain the X-Forwarded-For or Via HTTP headers revealing the source device's IP address, and inform the destination that the request is coming from a proxy.  

Applebot exposes internal IP addresses

Applebot refers to Apple's web crawler that sweeps the web to find content for its users.

"Applebot is the web crawler for Apple. Products like Siri and Spotlight Suggestions use Applebot," according to Apple's knowledgebase.

Last month, Security researcher and podcast creator David Coomber found out that Applebot had been using a proxy that leaked Apple's internal IP addresses.

"On any given day, I see a fair amount of noise directed at my webserver, from bots scraping content or scanning for 'research' to attacks via Tor and thought it would be interesting to see how many connections were identifying themselves as being routed through a proxy," wrote the researcher.

Coomber is indeed referring to the Via and X-Forwarded-For headers being sent by the Applebot crawler.

A sample request made to Coomber's website contained both of these headers that revealed the internal IP address of the device behind the proxy.

The fields listed respectively are the proxy's external IP address, requested path,  HTTP response code, user agent/web browser information, and the Via and X-Forwarded-For header values.

"Although I’ve seen a couple of bots that were misconfigured, I was surprised to see Apple’s Podcast bot look for updates to my podcast (Deep House Mixes) using a proxy which leaked internal IPs and hostnames from the 'Via' & 'X-Forwarded-For' headers," Coomber continued in his blog post. 

Took Apple nine months to fix it

According to Coomber, Apple had resolved the leak on September 29, 2020, approximately nine months after he had reported it to them and it is not clear why.

Coomber told BleepingComputer, "I provided the details to the Apple Product Security team on December 21, 2019. Once they confirmed the issue, I worked with them to remove the 'Via' and 'X-Forwarded-For' headers from their internal proxy infrastructure, which is configured to scan for updates to content available on Apple Podcasts."

How to prevent IP leaks through proxies? 

The recommended method to prevent originating IPs from being exposed in the HTTP requests made by proxy is to inspect your proxy server's configuration.  

It should be ensured, the proxy product is not sending the originating IP information using the Via, X-Forwarded-For, X-ProxyUser-Ip, or similar headers.

"If you're running a forward proxy in your environment, you may want to consider removing the 'Via' & 'X-Forwarded-For' headers," advised Coomber.

He shared sample configuration rules that network admins using Squid proxy servers could implement.

In July 2020, Coomber had reported a separate Applebot issue where the crawler had not been fully honoring the rules specified in robots.txt files.

When asked for comment concerning these issues, Apple did not provide one to BleepingComputer.