assetcache fails to serve client requests with increasing incidents of "Failed with status code 503, reason code 36" in the Debug.log

Number:rdar://34956416 Date Originated:October 12 2017, 12:16 PM
Status:Closed Resolved:January 24 2018, 2:16 PM
Product:Server Product Version:5.3.1 (16S4128)
Classification:Serious Bug Reproducible:Sometimes
After an undefined period, the Caching Service will stop servicing client requests (see attached screenshot). A typical log entry:

2017-10-12 11:05:46.787 #QIkMQ8l6yl4/ Request from [cloudd/651.15 CFNetwork/811.5.4 Darwin/16.6.0] to store®ion=amereast&Expires=1507824106& denied because the server is too busy; see MaxConcurrentClients setting
2017-10-12 11:05:46.787 #QIkMQ8l6yl4/ Failed with status code 503, reason code 36

Increasing the MaxConcurrentClients setting appears to have no effect.

These entries seems to increase over time. For example, there were 9,927 instances of "Failed with status code 503, reason code 36" in Debug-2017.10.10-21/43.log when the service still appeared to be servicing clients and 17,602 instances in Debug-2017.10.12-10/11.log when it had stopped completely.

Steps to Reproduce:
Run the caching service with a large number of clients.

Expected Results:
If the server is too busy, it should deny service to some clients, but should eventually recover when demand goes down.

Actual Results:
If the server is too busy, it'll soon stop servicing all clients and never recovers without restarting the service.

macOS Sierra 10.12.6 (16G29) 5.3.1 (16S4128)

Mac Mini (Late 2014)
3 GHz Intel Core i7
caching:MaxConcurrentClients = 50000
Sonnet xMac mini Server
Sonnet Presto 10GbE 2-Port 10 Gigabit Ethernet PCI Express Card

Note that there are no alerts issued when the service stops and I am only aware of it because I see no traffic on the Stats page.


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!