Open Radar

 
Summary:
On recent versions of iOS, devices are inconsistent about sending client certificates during the TLS handshake.

The Beta by Crashlytics (“Beta”) web service provides a Configuration Profile for installation on tester devices. This profile includes a unique client certificate that authenticates Beta testers and authorizes access to the apps to that they have been invited to test. 

This profile also includes a web clip, which loads the tester user interface via Safari. The Beta by Crashlytics web services are configured to require a matching client certificate when accessing the authenticated pages in the Beta web clip.

This has been working well for many years. But starting in early/mid April 2017 we began to see an increase in customer reports about intermittent/inconsistent authentication problems on devices with installed certificates. If users were being consistently authenticated or rejected, it would be an easier problem, but what we actually see is inconsistent behavior that leads to seemingly random authentication problems

Steps to Reproduce:
We started by trying to analyze traffic using the Charles web debugging proxy. We quickly discovered that the problem was occurring before application data started being exchanged, making Charles ineffective for diagnosis.

After realizing that the problem was with the TLS handshake, and that the TLS handshake would not be encrypted, we turned to using tcpdump on our server. By reviewing server logs while using a test device, we were able to determine when we’d reproduced the problem, and stop the packet capture. Packet capture files were then reviewed in Wireshark to make visualizing the TLS handshake easier.

Of particular interest was whether we could determine any server side behavior that could be inducing clients to behave inconsistently. We examined the whole TLS handshake looking for clues, but paid special attention to the Certificate Request message. We found that the Certificate Request message was identical for both successful (the client sent its certificate) and failed (the client did not send its certificate) requests. That message is the following byte sequence:

0000   0d 00 00 a0 03 01 02 40 00 1e 06 01 06 02 06 03
0010   05 01 05 02 05 03 04 01 04 02 04 03 03 01 03 02
0020   03 03 02 01 02 02 02 03 00 7a 00 78 30 76 31 1a
0030   30 18 06 03 55 04 0a 0c 11 43 72 61 73 68 6c 79
0040   74 69 63 73 2c 20 49 6e 63 2e 31 16 30 14 06 03
0050   55 04 08 0c 0d 4d 61 73 73 61 63 68 75 73 65 74
0060   74 73 31 0f 30 0d 06 03 55 04 07 0c 06 42 6f 73
0070   74 6f 6e 31 0b 30 09 06 03 55 04 06 13 02 55 53
0080   31 22 30 20 06 03 55 04 03 0c 19 43 72 61 73 68
0090   6c 79 74 69 63 73 20 53 53 4c 20 43 6c 69 65 6e
00a0   74 20 43 41

Our next step was to establish that the device’s view of the handshake matched that which were able to see at the server. To determine this, we used rvictl to create a remote virtual interface for the device on a connected Mac. We were then able to use the same tcpdump + Wireshark + device testing workflow to reproduce the problem and examine the TLS handshake data.

This analysis confirmed that the device’s view of the TLS handshake matched that which we recorded from the server, removing the possibility of problems being introduced in transit.

The same inconsistent Certificate Response behavior can be seen in the device captures. Some of the TLS handshake process can be seen in the device logs, as “TIC TLS Event” entries, but we weren’t able to determine anything new from these logs.

Expected Results:
Beta by Crashlytics provides a web app at https://apps-ios.crashlytics.com with certain routes (e.g. https://apps-ios.crashlytics.com/projects) that request and require a client certificate during the TLS handshake for authentication.

We expect that devices with the Crashlytics configuration profile installed send the matching client certificate on every request when prompted for it by our servers.

Observed Results:
Mobile Safari clients occasionally don’t send their installed client certificate when requested to do so by the server. Sometimes the client provides the certificate on the next request. Other times, testers have experienced longer periods of time where the device gets “stuck” and refuses to send the certificate.
 
We have not been able to determine any pattern or set of steps that reliably reproduce the problem. Seemingly random patterns of navigation within the web app, web clip page reloading, force quitting Safari, and returning to the web clip can trigger the problem.
 
Devices running iOS >= 10.3 seem to have a much higher likelihood of experiencing this behavior. We have run tests with devices running iOS 10.2.1 and iOS 9.3.1. The 10.2.1 device was made to experience the problem once, but otherwise seems mostly immune. Our testing was never able to make the 9.3.1 device reproduce the problem. We don’t have perfect evidence that this is behaving worse on iOS >= 10.3, but anecdotally this seems to be the case.
 
We define the problem as having occurred when the server issues a Certificate Request message as part of the TLS handshake, but the client responds with a Client Certificate message containing no certificates while a matching certificate is installed. The same client may have already responded to previous requests from the same server with the expected Client Certificate, or may do so on subsequent such requests. In both the success and failure cases, the Certificate Request message issued by the server is byte-for-byte identical.

Version:
AppleWebKit/603.2.4 Version/10.0 Mobile/14F89 Safari/602.1 and others

Notes:


Configuration:
iOS devices with version >= 10.3 are the most susceptible. The problem was reproduced only once on a device running iOS 10.2.1. We were not able to reproduce the problem on iOS 9.3.1