Access to pyObject() of ObjCInstance is crashy?

JonB

In your simple example, on_main_thread wasn't used... so I was just guessinh.

Does this cause crashes?

v=ui.View()
v.present('sheet')
@on_main_thread
def test():
    v._objc_instance.pyObject(argtypes=[], restype=c_void_p)

for i in range(10):
    test()

Also... Does the original code ever work, just unreliability? Does it work from console, manually?

shinyformica

@JonB now we're getting somewhere!

Running your test gave me an interesting result, so I set up a series of three tests which nicely illustrate the issue:

import ui
import objc_util
import ctypes
v=ui.View()
#v.present('sheet')

@objc_util.on_main_thread
def test_which_works(inview):
    p=inview.objc_instance.pyObject(argtypes=[], restype=ctypes.py_object)
    print p

    v2=ui.View()
    p=v2.objc_instance.pyObject(argtypes=[], restype=ctypes.py_object)
    print p

for i in range(10):
    test_which_works(v)

@objc_util.on_main_thread
def test_which_is_interesting():
    p=v.objc_instance.pyObject(argtypes=[], restype=ctypes.py_object)
    print p

for i in range(10):
    test_which_is_interesting()

@objc_util.on_main_thread
def test_which_crashes():
    for i in range(10):
        v3=ui.View()
        p=v3.objc_instance.pyObject(argtypes=[], restype=ctypes.py_object)
        print p
#test_which_crashes()

Run the above, uncommenting the crashing one to see it crash Pythonista.
So, the first test gets the pyObject of a local reference to a globally held view (it doesn't appear to matter if the view is presented or not). It also does the same thing to a locally created and held view.

The crashing test is the same scenario as in my simplest example: in a loop, create and access the pyObject of a locally created view.

The second test produces an interesting result: multiple calls to a function which attempts to access the pyObject of a global reference to a view. This one doesn't crash, but it does produce a traceback:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 315, in 'calling callback function'
  File "/var/containers/Bundle/Application/7BBC6AD0-AD54-4A7F-A649-D1A5084BD15B/Pythonista3.app/Frameworks/Py2Kit.framework/pylib/site-packages/objc_util.py", line 1066, in OMMainThreadDispatcher_invoke_imp
    retval = func(*args, **kwargs)
  File "/private/var/mobile/Containers/Shared/AppGroup/EE2056B2-9D54-47EC-AF60-07E05B54F00C/Pythonista3/Documents/test.py", line 29, in test_which_is_interesting
    p=v.objc_instance.pyObject(argtypes=[], restype=ctypes.py_object)
AttributeError: 'weakref' object has no attribute 'objc_instance'

And sure enough, if I print out what p and v are in that second test, they suddenly become weakrefs a couple iterations in. And then a few more iterations in, they suddenly lose the ability to get the objc_instance. The question is why? How does it suddenly change from a _ui.View to a weakref (with the same memory location)? Why does it continue to work for a few more iterations?

The first test is accessing a global view which is definitely held to the end. But it also accesses a local view created within the function, but that view is also technically held until the end of the function.
The crashing test creates views inside a loop inside a function, which ought to work the same as the first test.
The interesting test starts to make it clear: the only difference between it and the first test is that it accesses the global view, not a local reference to it. And that global is becoming a weakref at some point.

This must be related to what happens with @on_main_thread and the execution of the code in the interpreter thread. It's getting into some thick weeds with the inner workings of things...and I think this gets me enough info to make it work for my purposes.

JonB

I vageuly recall some issue a long time ago, where the view objcinstance might not be guaranteed to be completely initialized until it is presented.

my guess is that the interaction between python to objc and back in a tight loop (python creates a view, which somewhere the needs to go create a the actual objc view, then calling pyObject i think probably has c code calling back into python code, which screws with the GIL) and somewhere things are just not quite thread safe.

using c_void_p, however seems to mostly resolve the issue. the check for None is important, and you can retry. the code below doesnt complain at all for me, though i have an old, slow device. if i remove on_main_thread, it complains that pyobj returned None, but the second retry always works.

import ui
from objc_util import *
from ctypes import *
import gc,time

@on_main_thread
def test(restype):
	for i in range(100):
		v=ui.View()
		id_v=id(v)		
		pyobj= v.objc_instance.pyObject(argtypes=[], restype=restype)
		if restype is c_void_p:
			print(i)
			while not pyobj:
				print('returned none... retrying')
				pyobj= on_main_thread(v.objc_instance.pyObject)(argtypes=[], restype=restype)
			assert(id_v == pyobj.value)
			pyobj=cast(pyobj, py_object)
		elif restype is py_object:				
			print(i)
			if not(id_v ==id(pyobj)):
				print('fail: pyobj=', pyobj)

			
test(c_void_p)
#test(py_object) #crashes!

mikael

@JonB, this runs peacefully on iPhone X.

shinyformica

@JonB well, that cracks it then, definitely something in the deep woods of the interaction between the objective-c runtime and the python interpreter. I'm going to try grafting the retry technique above to the NavigationView wrapper to see if it will make it stable.

mikael

@shinyformica, try just using c_void_p and on_main_thread first, without any retries?

JonB

I'll say when I removed on_main_thread, and only wrapped the pyObject call, it originally failed every 10th time. Later, I couldn't get it to reproduce! But c_void_p is at least safer.

shinyformica

@mikael, @JonB I wish I had better news, but it seems that no matter whether I attempt ctypes.py_object, ctypes.c_void_p with a cast, multiple retries, on_main_thread, etc. I always eventually crash. So I am going to give up for now...the solution I have now is less efficient, but stable, and works well for my purposes.

It's quite a bit harder to debug than the simple tests demonstrated throughout this thread. I can create tests which crash, and ones which don't by switching around the order of things and where objects are created. In the actual code, there is a ui.NavigationView, which hosts views with ui.TableView children, and those tables have data sources which respond via their action callbacks by eventually calling the code which tries to get the pyObject of the current navigation view and then push or pop a view in response. So where the issue really arises is somewhere in the complex interaction of ui thread and main thread.

Thanks for the ideas and help!

JonB

Are you getting crashes just getting c_void_p? Or during cast? C_void_p shouldn't give you a seg fault. Then your can check it is valid before using it.

If you have access to the views you are going to push, there may be two other options.

First, set the tag parameter on both the objc_instance and ui.View, to unique integers.

Then you can query the tag attribute, and look up the tag in python (not sure if the tag lookup methods in python views work with navigation views)

Or, you can actually store your own pyObject, via objc_addAssociatedObject. When the views are first created.

objc_addAssociatedObject(view.objc_instance, sel('pyObject'), ns(id(view)), 3)

Then the view can be retrieved as

py_object.from_address(ObjCInstance(objc_getAssociatedObject(objc_view, 
sel('pyObject'))).integerValue()).value

I am using this approach to develop a more pythonic way to use create_objc_class -- using a python class to encapsulate the objc class, with method decorators that replace the |_self, _sel ` arguments with a regular self (python object) argument, which is automatically looked up from the objc object.

You can store whatever python data you want using associated objects, just change the key (which should be a selector, of your choosing).

(Some of this is from memory, I'll post a working example later)

shinyformica

@JonB I don't really have time to dig further into it. It doesn't matter whether I attempt to get the python ui.View from the objc instance via:

nc = self.navigationview.objc_instance.navigationController()
view = nc.topViewController().view()
pobj = view.pyObject(argtypes=[], restype=ctypes.c_void_p)

or

nc = self.navigationview.objc_instance.navigationController()
view = nc.topViewController().view()
pobj = view.pyObject(argtypes=[], restype=ctypes.py_object)

in either case, things become unstable and eventually crashes occur, especially after trying it again. The py_object seems to crash immediately, while the c_void_p way just makes things highly unstable. Oddly, getting the value via c_void_p gives me back the actual ui.View instance, not a c_void_p object which I need to cast(), which is what I was expecting...which to me is a sign of something fishy going on. It never appears to be None, and since it's coming in as an actual ui.View instance and not ctypes.c_void_p, there's no "value" property to check.

I'll look at the objc_addAssociatedObject() idea, though at this point that seems about the same as just keeping a mapping entirely in python of the id of the objc view to the python ui.View object, since I'm already going to the trouble of tracking things manually.

mikael

@JonB:

I am using this approach to develop a more pythonic way to use create_objc_class -- using a python class to encapsulate the objc class, with method decorators that replace the |_self, _sel ` arguments with a regular self (python object) argument, which is automatically looked up from the objc object.

Yes, please!

JonB

@mikael
Still a little work in progress, I need to find something useful to do with it.

https://gist.github.com/e3ea36bfd652e7ace38987714c01622c

@shinyformica This includes a get_associatedObject, that you could call when initializing a view.

set_associated_object(v.objc_instance, v)
...
pyobj=get_associated_object(n.objc_instance.navigationController().viewControllers()[0].view())

shinyformica

@JonB regardless of the troubles I was having...that objc decorator idea is brilliant, and ought to be rolled into objc_util.

re: my problems with pyObject being crashy. After going over the way my code was executing, I rearranged things to be absolutely certain that all attempts to access internals via pyObject() and cast() were within a @on_main_thread...and that seemed to finally make it stable. So, though I'm not entirely clear where it was going wrong, somewhere at least one of those calls was being made outside the main thread, and was therefore making the whole thing unstable at any future call.