Open Radar

 
31-Aug-2012 04:52 PM Mike Boylan:
Summary:

This is an incredibly important bug affecting several hundred machines in a production environment with the potential to affect hundreds of students.

When a 10.8.1 Mac is bound to a 2003 operational level active directory domain and configured to create a mobile account on login, if that user ever enters their password incorrectly at the login window, that bad password (or a bad password of some type) seems to get cached and is used during the regular mobile account check-ins/refreshes with the domain controllers to see if the mobile account's credentials are up-to-date. This results in numerous and frequent failed authentication requests logged on the domain controllers, ultimately leading to locked Active Directory accounts.

This problem is exasperated by a separate, but ultimately related bug, where 10.8.1, on a single failed login attempt at the login window, logs at a minimum of 2, and often times as many as *seven* failed authentication attempts against the domain controllers.

Steps to Reproduce:

- Bind a 10.8.1 machine to Active Directory with the following ad plugin configuration options:
Advanced Options - User Experience
  Create mobile account at login = Enabled
     Require confirmation        = Disabled
  Force home to startup disk     = Enabled
     Mount home as sharepoint    = Enabled
  Use Windows UNC path for home  = Enabled
     Network protocol to be used = smb
  Default user Shell             = /bin/bash

Advanced Options - Mappings
  Mapping UID to attribute       = not set
  Mapping user GID to attribute  = not set
  Mapping group GID to attribute = not set
  Generate Kerberos authority    = Enabled

Advanced Options - Administrative
  Preferred Domain controller    = not set
  Allowed admin groups           = domain admins,enterprise admins,workstation admins
  Authentication from any domain = Enabled
  Packet signing                 = allow
  Packet encryption              = allow
  Password change interval       = 14
  Restrict Dynamic DNS updates   = not set
  Namespace mode                 = domain

- Login successfully as a user to trigger the creation of a mobile account
- Logout

To observe the higher than normal failed authentication attempts:
- Purposefully type a bad password once, twice, or three times at the login window for that user
- Inspect the failed password count for a user in Active Directory using the LockoutStatus tool
- Observe that the failed password count increments by at least 2, but often times more per every failed request. We saw up to 7 for a single failed attempt.

To observe the continued attempts by opendirectoryd to authenticate using kerberos with the incorrect information:

- Login to said machine and enable opendirectoryd debug logging
- Logout
- Leave said machine at the login window
- Continue to monitor the user's failed password count using the LockoutStatus tool
    -- Notice it increases over time while the machine continues to sit at the login window
    -- Observe log entry time correlation between the DC security logs logging event 675 audit failure with kerberos error code 0x18, otherwise known as bad password with opendirectoryd logs on the clients. The 675 AUDIT FAILURE messages will be triggered from the machine's IP address.

Log entries on DCs will appear as such:
675,AUDIT FAILURE,Security,Fri Aug 31 07:00:44 2012,NT AUTHORITY\SYSTEM, cos01 %{S-1-5-21-3125130454-1535616552-1580254349-1385} krbtgt/AD.SCHOOL.EDU 0x2 0x18 10.17.42.23

Log entries on the client for correlating time stamp(s) will appear as:
2012-08-31 07:00:44.720183 EDT - 124.830.837, Node: /Active Directory/AD/ad.school.edu, Module: Kerberosv5 - Audit - Invalid credentials (5000) - Verify password for record type Users 'cos01' node '/Active Directory/AD/ad.school.edu'
2012-08-31 07:00:44.720238 EDT - 124.830, Node: /Local/Default - Audit - Invalid credentials (5000) - Verify password for record type Users 'cos01' node '/Local/Default'
2012-08-31 07:00:44.720248 EDT - 124.830.837, Node: /Active Directory/AD/ad.school.edu, Module: Kerberosv5 - ODRecordVerifyPassword failed with error 'Invalid credentials' (5000)
2012-08-31 07:00:44.720281 EDT - 124.830, Node: /Local/Default - ODRecordVerifyPassword failed with error 'Invalid credentials' (5000)

Again, this is happening continuously in the background while a machine sits at the login window.

- Eventually enough failed attempts will occur leading to an account lockout in Active Directory. This will depend on the site configuration. Ours is set to lockout at 20 failed attempts for 20 minutes.

Expected Results:

I'm unsure of the exact inner-workings of the Mobile Account, but I would expect that it would use the password from the last successfully authenticated session, not the last password entered at the login window (or some other random bad password), to see if its credentials are still valid. If so, no more checks should be performed until the timer expires and it's time to check in again. If found to be invalid, the account would be flagged as such and an update performed through a successful login at the login window would occur.

Actual Results:

The refresh mechanism for AD mobile accounts seems to be using a password that is NOT the last successful password from a user login when performing its checks against the domain controllers. If a user mistypes his or her password at the login window, that password, or a incorrect password (not sure exactly from where?) is being used to perform the account checks and refreshes against the directory. When found to be invalid, the checks still continue on a regular interval, increasing the failed authentication request count on the domain controllers each and every time. Because of a separate but seemingly related bug of OS X sending multiple failed authentication requests at one time, the lockout grace count is diminished more quickly than anticipated causing fast account lockouts.

Regression:
Unsure/Untested

Notes:
We successfully duplicated this several times. Attached are all the logs for a test machine, including opendirectoryd logging in debug mode, syslog, and other various output including dsconfigad, system_profiler, mcxquery, and last. We included last to show the successful login. Also included is the klist output of a successful login showing functional kerberbos. The testing last night was performed with test account cos01. We left the machine at the login window after a failed authentication attempt and the failed count on the domain controllers was 3. By morning, the account had increased to 18 failed authentication attempts logged, with hundreds of entries of event 675 AUDIT FAILURE on the domain controllers. Time stamps with those messages correlate with refresh events in the opendirectoryd debug logs. Logs from the domain controllers showing the audit failures are also included.