[SOLVED] First attempt to integrate ARkit and first questions...
Edit: I found a solution, see here https://forum.omz-software.com/topic/4362/first-attempt-to-integrate-arkit-and-first-questions/29
Following some objective-c/swift ARkit examples, I made some python tests using the objc_util module on my iPhone 6S+ with iOS 11 GM.
I move forward step by step... as I'm a complete beginner in "calling objective-c from python"...
I can open the camera through ARkit but I'm thinking my arscnview is not correctly initialized and my 'delegate' is not correctly set. For example, despite my call to setDebugOptions_, I can't display the feature points for example, or if I insert a beep in the renderer_didAdd_for_ method, I can't ear any sound even if a plane is supposed to be detected by my camera.
So if you have any advice, I'm more than interrested !o)
Here my code:
# coding: utf-8 from objc_util import * import ui import os import sys #from myDebugToolKit import * load_framework('SceneKit') load_framework('ARKit') # Some 'constants' used by ARkit # But i can't transfer them to the ARKit framework, why ????? ARWorldAlignmentGravity = 0 ARWorldAlignmentGravityAndHeading = 1 ARWorldAlignmentCamera = 2 ARPlaneDetectionNone = 0 ARPlaneDetectionHorizontal = 1 << 0 ARPlaneDetectionVertical = 1 << 1 ARSCNDebugOptionNone = 0 ARSCNDebugOptionShowWorldOrigin = 1 << 0 ARSCNDebugOptionShowFeaturePoints = 1 << 1 SCNScene = ObjCClass('SCNScene') ARSCNView = ObjCClass('ARSCNView') ARWorldTrackingConfiguration = ObjCClass('ARWorldTrackingConfiguration') ARSession = ObjCClass('ARSession') UIViewController = ObjCClass('UIViewController') ARPlaneAnchor = ObjCClass('ARPlaneAnchor') # I should refactor te following line in a class but I need to learn more the create_objcc_class function sceneview = None # Here two little set up functions used by the main class def createSampleScene(): # an empty scene scene = SCNScene.scene() return scene def createARSceneView(x, y, w, h, debug=True): v = ARSCNView.alloc().initWithFrame_((CGRect(CGPoint(x, y), CGSize(w, h)))) v.setShowsStatistics_(debug) # Problem here... feature points are not shown.... despite the method call v.setDebugOptions_(ARSCNDebugOptionShowWorldOrigin | ARSCNDebugOptionShowFeaturePoints) return v # Some callback definitions used by create_objc_class def CustomViewController_touchesBegan_withEvent_(_self, _cmd, _touches, event): touches = ObjCInstance(_touches) for t in touches: loc = t.locationInView_(sceneview) sz = ui.get_screen_size() print(loc) def CustomViewController_viewWillAppear_(_self, _cmd, animated): configuration = ARWorldTrackingConfiguration.alloc().init() # Another problem here...constants aren't well communicated... (my assumption...) configuration.setPlaneDetection_ (ARPlaneDetectionHorizontal) configuration.setWorldAlignment_(ARWorldAlignmentGravityAndHeading) session = sceneview.session() session.runWithConfiguration_(configuration) def CustomViewController_viewWillDisappear_(_self, _cmd, animated): session = sceneview.session() session.pause() def MyARSCNViewDelegate_renderer_didAdd_for_(_self, _cmd, scenerenderer, node, anchor): if not isinstance(anchor, (ARPlaneAnchor)): return # to be implemented... # The main class... class MyARView(ui.View): def __init__(self): global sceneview self.flex = 'WH' screen = ui.get_screen_size() # set up the scene scene = createSampleScene() # set up the ar scene view delegate methods = [MyARSCNViewDelegate_renderer_didAdd_for_] protocols = ['ARSCNViewDelegate'] MyARSCNViewDelegate = create_objc_class('MyARSCNViewDelegate', NSObject, methods=methods, protocols=protocols) delegate = MyARSCNViewDelegate.alloc().init() # set up the ar scene view sceneview = createARSceneView(0, 0, screen.width, screen.height) sceneview.scene = scene sceneview.delegate = delegate # set up the custom view controller methods = [CustomViewController_touchesBegan_withEvent_, CustomViewController_viewWillAppear_, CustomViewController_viewWillDisappear_] protocols =  CustomViewController = create_objc_class('CustomViewController', UIViewController, methods=methods, protocols=protocols) cvc = CustomViewController.new().init().autorelease() cvc.view = sceneview # last set up self_objc = ObjCInstance(self) self_objc.addSubview_(sceneview) # workaround : I need to call manually viewWillAppear as otherwise my callback is not called... cvc.viewWillAppear_(False) def will_close(self): session = sceneview.session() session.pause() @on_main_thread def main(): v = MyARView() v.present('full_screen', hide_title_bar=True) if __name__ == '__main__': main()
Adding the scene below gives me a floating cube, which hoverever is not fully ”connected” to the surroundings but keeps slipping away as I try to walk around it.
def createSampleScene(): # an empty scene scene = SCNScene.scene() root_node = scene.rootNode() cube_geometry = SCNBox.boxWithWidth_height_length_chamferRadius_(1, 1, 1, 0) cube_node = SCNNode.nodeWithGeometry_(cube_geometry) cube_node.setPosition((0, 0, -5)) light = SCNLight.light() light.setType_('omni') light_node = SCNNode.node() light_node.setLight_(light) light_node.setPosition((1.5, 1.5, 1.5)) root_node.addChildNode_(light_node) root_node.addChildNode_(cube_node) return scene
@mikael : thanks for your support. I continue to seek a solution to this 'arkit' challenge ;o)
As a roadmap, my first step is to be able to see the feature points (I need it because I don't know if the arkit framework is correctly set), then, I will try to use the plane detection and then, I will try to add a 3D object to this plane...my dream...
I'm still thinking the major problem lies in the fact that my custom view controller is badly added to the main hierarchy. It's not normal that I need to call manually 'cvc.viewWillAppear_(False)'. So I think some callbacks are not called because my hierarchy is badly "mounted". If someone can check this, I would be grateful...
Another part of the jigsaw : I've noticed if I use 'setDebugOptions_' with a value, this value is clamped to (1<<11)-1 (1<<10 is the last bit flag you can use without any ARkit stuff). It's like my ARWorldTrackingConfiguration was not taken into account...
Moreover, if I inspect the "session" variable in the "CustomViewController_viewWillAppear_" function, "configuration" is actually null :o(
Caution : the following sentence contains Beatles lyrics inside :
"help!, i need somebody... help..." ;o)
Have you tried on_main_thread in
__init__? I think things like adding subviews are supposed to be on the main thread.
@JonB, is it not already on main thread, since the call to main is decorated?
good point, yes
Sorry, a side note guys. Has anyone written anything either in a gist or in the forum that really explains @on_main_thread as it applies specifically to Pythonista and some simple logic to know wether you need to be using it. Or is it safe to say if you are not using objC or your own threads that you can safley say you dont need to be concerned with it?
According to Ole's reply in one issue. @on_main_thread is there for objc_util only. Every other module should handle this internally. If not and main thread is required, it's a bug.
See https://github.com/omz/Pythonista-Issues/issues/461 for more info.
@Brun0oO, both @JonB and @zrzka have replied on this thread, and neither pointed out anything obvious wrong with your code, so we are out of luck. :-) Unless @omz dips in, but I think he is too busy with the next release of Pythonista to spend time on the fun stuff.
@zrzka , ok thanks thats very clear! Just good to know
Back to the original problem:
per apple docs, you have to add the child view controller first:
The problem i think is that ui.Views don't have a viewcontroller until they are presented
@JonB, allright! So, by changing the
initializeand only calling it after
v = MyARView() v.present('full_screen', hide_title_bar=True) v.initialize()
... we can follow the right order, using the "forbidden"
self_objc.nextResponder().addChildViewController_(cvc) self_objc.addSubview_(sceneview) cvc.didMoveToParentViewController_(self_objc)
... after which
CustomViewController_viewWillAppear_no longer needs to be called manually, i.e. things work as they should.
(Here's a gist with this version of the code.)
But still, debug info does not appear, nor does the cube stay anchored to the surroundings.
@omz, just checking: for ARKit to really work in Pythonista, would Pythonista need to include the
Nope, the key is not required. See https://developer.apple.com/documentation/arkit (Important section at the beginning). You only need device with >= A9 chip. Use isSupported to check if your device is supported.
Again, just to be sure, I checked the property and got
Trueon my 6S+.
Would be nice if someone could run the gist from the previous post, maybe on a newer device, just to check that the results are similar.
I'm investigating the ARSession problem and i'm trying to understand why its configuration stays null after a "runWithConfiguration_" call (I'm going to use some delegate methods in order to spy the initialization process...).
I will update my github ASAP.
I'm rewriting the Apple demo (candles, ...) into Python(ista). Will share it when finished.
@zrzka, just curious, did you progress with this? And if not, was it due to some problems or just other more important things taking up your bandwidth?
@mikael simple answer, pretty busy these days :/
If anyone needs a bridge for SceneKit I have https://github.com/scj643/objc_tools/blob/master/objc_tools/scenekit/sk_scene.py
@scj643, looks good, thanks. As soon as some real programmer gets ARKit working, I am sure to use your code.