Open Radar

 
Summary:
Our company network administrators have flagged machines as unexpectedly utilizing internet network bandwidth overnight, when the machines should be idle as users have gone for the day (see Solarwinds NTA Report.pdf in ForBugReport.zip for an example). Looking into the issue, we discovered a commonality between the flagged machines: all were left connected to the network overnight, all running 10.10.5, all with Safari 9.1 and with Safari being left running with open tabs or pages. The same behavior is not present if another browsers, such as Google Chrome, is left running overnight. The behavior occurs on both machines with anti-virus software installed (either Sophos Anti-virus or Symantec Endpoint Protection) or without anti-virus software. 

Digging into the issue further, using tools such as nettop and tcpdump on the client machines and Solarwinds Netflow Traffic Analyzer at the network level, we can see that this overnight traffic is going to a constantly changing set of Akamai servers, e.g. a104-106-248-67.deploy.static.akamaitechnologies.com or a23-208-26-224.deploy.static.akamaitechnologies.com. The traffic between the client machines and the servers is encrypted, so we are unsure of what data is being transmitted between the clients and the remote hosts. As Akamai is a content delivery service for multiple companies, including Apple, Adobe and Microsoft, it's difficult to even offer a conjecture to what the traffic might be. Even more puzzling is how the traffic is generated when a machine is left with Safari open to the default favorites page (see Safari Default Favorites.png in ForBugReport.zip), which theoretically should not be producing any or at least limited network traffic.

Steps to Reproduce:
1. Prepare a machine with OS X 10.10.5 and Safari 9.1 with the stock configuration and preferences
2. Connect the machine to a wired network that can reach the public internet
3. Configure the machine such that the display can sleep but the machine will remain active
4. Launch Safari
5. Safari should open to the default favorite page. Leave Safari open to this page without proceeding to access other sites on the internet. 
6. Leave the machine for a period of time.

You can then use nettop -m tcp to examine TCP traffic for the Safari process. See nettop.png in ForBugReport.zip for an example. You can also use tcpdump -i [interface] -s 0 -B 52488 -w [/path/to/output/file] (that’s -s zero, not oh) to collect network traffic that can be examined in a tool such as Wireshark. 

Expected Results:
Safari, as it is not connect to a particular internet site with content available for download, should have a minimal network load. It is possible that Safari would prefetch sites listed in the favorites page, but that should be a one-time action with limited data exchange.

Actual Results:


Version:
OS: 10.10.5
Safari Version: 9.1 (10601.5.17.4)

Notes:


Configuration:
Left overnight, a machine can download in excess of 35GB of network traffic. See 20160414-0830_chedit4.pcap for an example of five minutes worth of network capture on a machine with Safari left with an open favorites page during which time 345MB was exchanged with a23-208-26-224.deploy.static.akamaitechnologies.com.

18-Apr-2016 Update:
These are Akamai servers we are repeatedly seeing being communicated with by client machines:
a104-106-248-67.deploy.static.akamaitechnologies.com 
a23-192-60-175.deploy.static.akamaitechnologies.com 
a23-208-26-224.deploy.static.akamaitechnologies.com 
a23-65-248-185.deploy.static.akamaitechnologies.com 

This is not a complete list but servers we've seen at least three times during our monitoring period.

20-Apr-2016 Update:
n original submission, in the ForBugReport.zip bundle was a file, Solarwinds NTA Report.pdf, that showed Top 10 Endpoints for network activity overnight for the night of 4/12-4/13. For comparison is CHI_CORE_4192016 800 PM to 4202016 800 AM.pdf, which is a report from last night. The difference between the two is that we explicitly prevented Safari from being launched and running overnight on machines that had previously been flagged on similar overnight reports. On the report from the 12th, the top client machine had 57.9 Gbytes of Ingress data. This was a machine with Safari left running all night with a single web page open to an internal website (the web portal for a Xerox production printer). On the CHI_CORE_4192016 800 PM to 4202016 800 AM report, the top machine has a total of 3.6GB. Not present on the report, but previously featured on all reports, are deploy.static.akamaitechnoligies.com servers.

22-Apr-2016 09:25 AM Update: 
This issue is now present not only in the [COMPANY NAME] Chicago office but has been replicated via testing in the [COMPANY NAME] Minneapolis office and witnessed on a production machine our New York office. 

On the test machine in Minneapolis, using nettop -m tcp to examine open connections, a23-7-189-161.deploy.static.akamaitechnologies.com and a23-208-26-224.deploy.static.akamaitechnologies.com were seen as the focal point of heavy traffic. At the time the test machine was connected to these machines (4/21 and 4/22), I was able to determine that configuration.apple.com was resolving to these Akamai servers.

_on 4/21_
host 23.7.189.161
161.189.7.23.in-addr.arpa domain name pointer a23-7-189-161.deploy.static.akamaitechnologies.com
host configuration.apple.com
configuration.apple.com is an alias for configuration.apple.com.edgekey.net.
configuration.apple.com.edgekey.net is an alias for e5153.a.akamaiedge.net.
e5153.a.akamaiedge.net has address 23.7.189.161

and 

_on 4/22_
host 23.208.26.224
224.26.208.23.in-addr.arpa domain name pointer a23-208-26-224.deploy.static.akamaitechnologies.com.
host configuration.apple.com
configuration.apple.com is an alias for configuration.apple.com.edgekey.net.
configuration.apple.com.edgekey.net is an alias for e5153.a.akamaiedge.net.
e5153.a.akamaiedge.net has address 23.208.26.224

06-May-2016 Update:
I have attached the following files, per request from our Apple SE, Christos Drosos:
• 20160506_SafariDefaultsDumpForApple.txt - Requested Safari defaults dump
• sysdiagnose_2016.05.06_09-05-06-0500.tar.gz - Requested sysdiagnose run
• 20160506_mn-arcade.pcap.zip - Packet capture made right before running sysdiagnose
• 20160506_BehaviorExampleForApple.mov - Screencast of misbehavior in action

Notes on the machine (mn-arcade-imac) in question:
- Is running Safari 9.0.1 Please note that this behavior has been seen with Safari 8.0.6, 9.0.1, 9.0.3 and 9.1. The requested captures just happened to come from a machine running a not up-to-date version of Safari.
- Is running Sophos Anti-virus 9, which proxies all internet traffic through the Sophos Web Intelligence process. Please note this behavior has been seen on machines running Sophos Anti-virus, Symantec Endpoint Protection and no AV software at all. 

As of this morning when I ran the test, the Akamai server,  a96-6-182-199.deploy.akamaitechnologies.com,  in question resolved to configuration.apple.com:
As of this morning when I ran the test, the Akamai server,  a96-6-182-199.deploy.akamaitechnologies.com,  in question resolved to configuration.apple.com:
admin$ host configuration.apple.com
configuration.apple.com is an alias for configuration.apple.com.edgekey.net.
configuration.apple.com.edgekey.net is an alias for e5153.a.akamaiedge.net.
e5153.a.akamaiedge.net has address 96.6.182.199
admin$ host 96.6.182.199
199.182.6.96.in-addr.arpa domain name pointer a96-6-182-199.deploy.akamaitechnologies.com.