Thoughts after Apple iPad event with implications for #VisionPro: Today, Apple positioned iPad and VisionPro for professional use, including movie production and sound editing (e.g., FinalCut & Logic Pro on the iPad), and training (VisionPro). They also updated the Apple Pencil. Here's an exciting idea:
An issue to some with Vision Pro has been the lack of strong integration of hand controllers, especially compared to more gaming-centric headsets. For serious use of VisionPro's initial major pro app, Excel, I think it helps to use a physical keyboard and trackpad, which it does support. But that's not rich enough for many more advanced uses.
I think in the not-too-distant future we’ll see the iPad integrated with VisionPro like the Mac started, if not more so. You’ll use an iPad, perhaps with a Magic Keyboard, and the new Apple Pencil Pro for professional-level control. Having both a pencil, with squeeze, twirl, haptic-feedback, hover, etc., along with the current full-motion hand and arm movement in 3D-space, gives you the start of a very rich and precise way of interacting with spatial computing. Moving on the hard iPad surface could be quite superior to waving something in the air or using a joystick. The Mac is not for using a pen, but the iPad is. I’m thinking long-term, not just the current headset. The videos they showed of their pro-apps on iPad, and the VisionPro update which included touting a film director using it to oversee the editing and visual effects for an upcoming film, hinted towards this convergence to me. I wonder if it's true.
Note: For those who don't know my full background, I co-founded a company called Slate in the early 1990's that made apps for early "pen computers". I even hold some patents for our work on a spreadsheet with special gestures and handwriting recognition. On the iPad, I personally wrote one of the very early notetaking apps (NoteTakerHD).
@danb The AvP can essentially project you any "device" that seems appropriate for a task in your hands. For haptic feedback all you might need is a plain plate.
@helge I wasn't thinking of the iPad just for an image. Yes, the AvP can do that, and probably would to some extent. Their pencil, though, is tuned to writing on their screen and may need some electronics and processing there. (AI for good tracking?)
@danb I think what I'm saying is that it doesn't make much sense to use devices designed for the real world, as-is, within visionOS. Yes, you might need a smarter plate, but it wouldn't need that fancy OLED display, as it goes through cameras anyways. Rather have the device directly render into your AvP (which is what the macOS screen does).
@helge Yes, I don't think a great OLED display is needed. Yes, like the Mac. I was thinking more of having the thing you are manipulating being not below the pen but elsewhere. Like a mouse on the desk and the screen vertical in front of you. I guess I wasn't clear enough.
@danb remove the iPad and you might be onto something. visionOS already supports many iPad apps. Just give it Apple Pencil support and allow for apps to more easily be pinned to a flat surface. Any surface could be a drawing surface or take it off a surface and work directly in 3D space with Zbrush or other sculpting apps.
If all the other spatial devices can track well enough with controllers not on a surface so could Apple Pencil with Vision Pro.
@kaplag Just on a plain surface could be interesting. I think the iPad helps with precise positioning (more than 150 dpi at 100 or more samples per second helps for a pen as I recall from my pen-computing days), charging, more computation, and an image under your hand. Not sure what support the pencil requires of the surface under it. (Remember the Wacom, etc., from the 1990's and later - it's been a long time since I was last in the precise pen world.) This is not gross motions but rather light feathering or more. Of course, Apple doesn't always seem that concerned with cost of a pro accessory.
@danb I love this notion so much. I agree there’s a new approach slowly coming together these days. I’m not sure what it is. But between tiny cheap wireless sensors, remarkable inference engines, and incredible new display capabilities we’re on the cusp of something big. I’m less sure any *one* of these is the Next Thing. More like the whole will be greater than the sum of the parts. Like bitmapped displays, pointing devices, and networking. How it comes together is going to be exciting.
Reading some comments, I think I wasn't clear enough about the role of the iPad. I was assuming the pen would normally manipulate things you see elsewhere, not under it. Like a mouse. The iPad could add context, but especially to add electronics and computing power to interact with the pen, and an appropriate surface on which to move the pen.