ReplayKit, Broadcast Upload Extension: Determining Orientation of Video Buffers

Originator:o.schnurr
Number:rdar://33828771 Date Originated:8/10/2017
Status:open Resolved:no
Product:iOS Product Version:iOS 11 beta 5 (15A5341f)
Classification:ReplayKit Reproducible:always
 
Area:
ReplayKit

Summary:
If you do end up shipping iOS 11 where the com.apple.broadcast-services-setupui extension is _not_ used for global screen recordings with 3rd-party upload extensions, then this bug report merely suggests that there doesn't seem to be a way for a broadcast upload extension to determine video orientation on its own. That orientation information (whether it's video orientation or device orientation) should be made available somewhere.

Details:
I am able to receive sample buffers in my broadcast upload extension (NSExtensionPointIdentifier: com.apple.broadcast-services-upload). The video sample buffers appear to be aligned to the camera sensor. For three of the four device orientations, I need to rotate the display of the video frame to show it correctly.

I know how to apply this kind of rotation to a video file or a video broadcast. My problem is determining what the rotation should be. The sample buffer itself doesn't appear to provide details about orientation or a preferred transform. The complicating factor is that queries to both UIDevice and UIScreen _from within a broadcast upload extension_ appear to lack correct device orientation as well. Both of them either report device orientation to be unknown or they incorrectly report the device to be unrotated.

I am guessing that the the broadcast setup UI (NSExtensionPointIdentifier: com.apple.broadcast-services-setupui) will solve this problem. It's a view controller that can resolve orientation. You already provide a way for a setupui extension to send this kind of data as setupInfo to an upload extension.

However, as of beta 5, the setupui extension is not used when initiating a global screen recording from the Screen Recording button. (I'm hoping that will be fixed before iOS 11 is released.)

Steps to Reproduce:
- Run the attached VideoOrientationTest project to install the VideoOrientationTest app and associated "Broadcast" extension on an iOS 11 device.
- Leave your device attached to Xcode on your Mac.
- Install the Screen Recording button into your Control Center if it's not already installed.
- Bring up the Control Center.
- 3D touch the Screen Recording button.
- Choose the "Broadcast" extension.
- Tap "Start Broadcast". Wait for the count down and let the recording continue running.

- In Xcode, set a breakpoint the processSampleBuffer() call in SampleHandler.swift (line 35).
- In Xcode select Debug -> Attach to Process.
- Wait for the process list to resolve.
- Select Broadcast at the top of the list.
- Wait for the break point to trap.

Expected Results:
At the breakpoint, in that call stack, in that process, there should be a way to query or determine which way the video sampleBuffer is oriented.

Lot's of possible solutions. (You guys would know better than me.) Some possibilities:

- Run the setupui extension before running a 3rd-party broadcast upload extension for a global Screen Recording. Give the setupui extension the opportunity to send device orientation to the upload extension.
- Or, add orientation to the setupInfo of the broadcastStarted() call to the upload extension.
- Or, embed the orientation somewhere into each video sample buffer.
- Or, get UIDevice or UIScreen capable of communicating correct device orientation into a broadcast upload extension.

Observed Results:
At the breakpoint, in that call stack, in that process, there doesn't appear to be a way to query or determine which way the video sampleBuffer is oriented.

Version:
- iOS 11 beta 5 (15A5341f)
- Xcode 9 beta 5 (9M202q)

Notes:
The SampleHandler can determine the size and aspect ratio of a video sample buffer but that doesn't tell us which way the content inside the video frame is oriented.

Without a rotation, three of the four device orientations will render content incorrectly. It's not clear how code inside a SampleHandler could query or determine what that rotation should be.

Configuration:
This has been the case for all the iOS 11 betas so far (up to and including beta 5).

Comments


Please note: Reports posted here will not necessarily be seen by Apple. All problems should be submitted at bugreport.apple.com before they are posted here. Please only post information for Radars that you have filed yourself, and please do not include Apple confidential information in your posts. Thank you!