Open Radar

 
On Monterey, the pf `route-to` directive fails to work correctly.  Re-routing  a TCP stream results in retransmission errors which prevents a connection being established. This is a major regression from behaviour under Big Sur and Catalina - where `route-to` works as expected. 

This regression breaks a major feature of our app .

Reproduction steps:

We use `route-to` to redirect traffic from a virtual utun interface (which has the default route) to a physical WIFI interface (`en0`). 

A minimal reproduction requires us to setup a `utun` interface as the default route before defining a `route-to` rule using `pfctl`

Set a utun interface as the default route. We do this by creating the following two routes:
    - (Set an ip on the utun device first: `sudo ifconfig utun0 10.1.1.3 10.1.1.3`)
    - `sudo route add -inet 0.0.0.0/1 -interface utun0`
    - `sudo route add -inet 128.0.0.0/1 -interface utun0`

Turn on packet forwarding (necessary for `route-to` to work)
    - `sudo sysctl -w net.inet.ip.forwarding=1`

Setup the pf `route-to` rule to reroute all packets heading to port 8888 to the physical interface (`en0`) instead of the default `utun0` interface:
    - `echo "nat on en0 inet all -> (en0)\npass out inet proto {udp,tcp} from any to any port 8888 no state tag BYPASS4\npass out route-to (en0 192.168.1.1) inet all no state tagged BYPASS4" | sudo pfctl -f -`
        - Note that `192.168.1.1` should be the address of the router.
        - Also note the `nat` rule is required to ensure the correct source ip.
- Create the TCP stream on port 8888:
    - `echo testing | nc -4 <remote TCP server> 8888`
    - Best to use a raw ip for the remote server, as setting the `utun` interface as the default route may break DNS (as well as other non-rerouted packets). This is only the case for this minimal reproduction where we use a non-functional `utun` device.

Expected behaviour:

We expect that the TCP stream to port `8888` will be redirected through the `en0` interface and  a TCP connection to the remote server will be established. 

The expected behaviour is observed on Big Sur and Catalina.

Observed behaviour:

On Monterey, the TCP stream fails to establish. WireShark indicates there are TCP retransmission errors. This breaks any connection we attempt to re-route. 

Further notes:

The correct/expected behaviour has been seen on some hardware even with Monterey installed. A 2017 Macbook pro behaved as expected, but a 2018 Macbook pro did not. It is not clear the link between certain hardware and the presence of this issue - but in about 90% of cases the incorrect behaviour on Monterey was observed. It's suspected this issue may primarily be on newer hardware.

Serial number of a Macbook pro where issue occurs: C02X42NMJG5K
Serial number of a Macbook pro where the issue does NOT occur: C02VD002HTD9

We always observed CORRECT behaviour on Catalina and Big Sur, irrespective of the hardware.