netstat (pcblist_n sysctl) does not display TIME_WAIT connections

Originator:jhawk
Number:rdar://22975100 Date Originated:5 Oct 2015
Status:Open Resolved:
Product:OS X Product Version:10.10.5 (14F27)
Classification: Reproducible:Always
 
Summary:
netstat does not display tcp connections that are in the TIME_WAIT state.
This appears to because it uses the net.inet.tcp.pcblist_n sysctl to obtain the table of connections. Using the net.inet.tcp.pcblist sysctl *does* display connections in TIME_WAIT state.

So the problem is presumably in netinet/in_pcblist.c's get_pcblist_n() function.

I've written an alternate tool (pcblist2.c) that uses the net.inet.tcp.pcblist sysctl and correctly shows TIME_WAIT connections. Source attached.

Steps to Reproduce:
1. Compile pcblist2
2. Open and close a TCP connection, e.g.
  echo foo | nc scripts.mit.edu 80
3. With some haste (TIME_WAIT connections time out in about 30 seconds), run
4. pcblist2
5. netstat -np tcp

Expected Results:
pcblist2 displays the connection in TIME_WAIT:

RecvQ SendQ Local Address        Foreign Address       srtt (ms)     (state)
    0     0 18.111.17.239.51972  18.181.0.43.80            2 ± 0     TIME_WAIT

netstat should do so as well

Actual Results:
netstat does not display the TIME_WAIT connection.

Version:
10.10.5 (14F27)

Notes:
Other BSD kernels have no problem displaying TIME_WAIT connections

Configuration:
The problem appears to be universal.

Attachments:
'pcblist2.c' was successfully uploaded.

Code follows for pcblist2.c:
#include <sys/types.h>

#include <sys/attr.h>
#include <sys/socket.h>
#include <sys/socketvar.h>
#include <sys/sysctl.h>

#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/tcp.h>

#include <netinet/in_pcb.h>
#include <netinet/tcp_timer.h>
#include <netinet/tcp_var.h>

#define TCPSTATES
#include <netinet/tcp_fsm.h>

#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>

/* From the private xnu-2782.40.9/bsd/netinet/tcp_var.h lines 622-636: */
/*
 * The rtt measured is in milliseconds as the timestamp granularity is 
 * a millisecond. The smoothed round-trip time and estimated variance
 * are stored as fixed point numbers scaled by the values below.
 * For convenience, these scales are also used in smoothing the average
 * (smoothed = (1/scale)sample + ((scale-1)/scale)smoothed).
 * With these scales, srtt has 5 bits to the right of the binary point,
 * and thus an "ALPHA" of 0.875.  rttvar has 4 bits to the right of the
 * binary point, and is smoothed with an ALPHA of 0.75.
 */
#define TCP_RTT_SCALE           32      /* multiplier for srtt; 3 bits frac. */
#define TCP_RTT_SHIFT           5       /* shift for srtt; 5 bits frac. */
#define TCP_RTTVAR_SCALE        16      /* multiplier for rttvar; 4 bits */
#define TCP_RTTVAR_SHIFT        4       /* shift for rttvar; 4 bits */
#define TCP_DELTA_SHIFT         2       /* see tcp_input.c */


int main () {
  
  int rv;
  size_t len;
  struct xtcpcb *pcblist, *p;
  rv = sysctlbyname("net.inet.tcp.pcblist",
		    NULL, &len, NULL, 0);
  if (rv < 0) {
    perror("sysctl");
  }
  p = pcblist = malloc(len);
  rv = sysctlbyname("net.inet.tcp.pcblist",
		    pcblist, &len, NULL, 0);
  if (rv < 0) {
    perror("sysctl");
  }

  printf("%-5s %-5s %-21s%-21s %-13s %s\n",
	 "RecvQ", "SendQ",
	 "Local Address", "Foreign Address",
	 "srtt (ms)", "(state)");

  char laddr[25], faddr[25];
  
  for (;
       (char*)p < (char*)pcblist+len;
       p = (struct xtcpcb*)((char*)p + p->xt_len)) {
    
    if (p->xt_socket.xso_family == AF_INET &&
	p->xt_socket.xso_protocol == IPPROTO_TCP) {

      struct in_addr *lia = &p->xt_inp.inp_laddr;
      struct in_addr *fia = &p->xt_inp.inp_faddr;
	
#define INADDR_LOCALHOST 0x7f000001

      int lip = ntohl(lia->s_addr);

      if (lip == INADDR_LOCALHOST ||
	  lip == INADDR_ANY
	  ) { continue; }

      sprintf(laddr, "%s.%d", inet_ntoa(*lia), ntohs(p->xt_inp.inp_lport));
      sprintf(faddr,"%s.%d", inet_ntoa(*fia),  ntohs(p->xt_inp.inp_fport));

      printf("%5d%6d %-21s%-22s %4d ± %-5d %s\n",
	     p->xt_socket.so_rcv.sb_cc,
	     p->xt_socket.so_snd.sb_cc,
	     laddr, faddr,
	     (int)((double)p->xt_tp.t_srtt / TCP_RTT_SCALE),
	     (int)((double)p->xt_tp.t_rttvar / TCP_RTTVAR_SCALE),
	     tcpstates[p->xt_tp.t_state]);
    }

  }

  return 0;
}

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!