Pythonista 1.6 Beta

[omz]

So, with Apple's new TestFlight, I can finally have a lot more beta testers, and won't have to worry about device limits so much.

If you'd like to play with the next version of Pythonista before it hits the App Store, please send me an email with your Apple ID, and I'll see what I can do. I don't plan to use up all the slots yet (I might need some later), but if you've been posting here, there's a very good chance that I can give you access to the beta.

As for what to expect, here's a list of the new features in the current beta:

• Improved support for the iPhone 6 and 6 Plus screen sizes
• Removing files from the library moves them to a trash folder instead of deleting them immediately.
• URLs in console output are 'linkified' automatically
• New reminders module for accessing the iOS Reminders database (read/write)
• New cb module for connecting to Bluetooth LE peripherals (experimental)
• New dialogs module as an easy-to-use alternative to the ui module when you just need simple data entry. This also contains functions to import files from the iCloud Drive document picker (iOS 8 only). – note: this is broken in the current beta, but will be fixed in the next few days

Anyone?

[dgelessus]

@blmacbeth, this is due to the unusual way that Pythonista uses to print output to the console. What is stored in sys.__stdout__ is the default Python output stream (a file object pointing to file descriptor 1, which is Unix stdout) but (presumably) because of iOS limitations that goes into nowhere.

Instead Pythonista has a "secret" module named _outputcapture with a few built-in functions to handle text I/O through the Pythonista console. sys.stdout is also replaced with a StdoutCatcher instance, whose write method encodes the string and shows it using _outputcapture. Because the StdoutCatcher class isn't stored anywhere by default, it is lost when you change sys.stdout and it is hard to restore afterwards.

The reason why the interactive code completion stops working as well is simple - whenever Pythonista needs a list of possible completions, it creates and runs a function named _pythonista_complete_line, which uses some standard Python module to do the completion and prints all possible results to sys.stdout, where they are caught by Pythonista and displayed in the completion list. (Yes, it is possible to monkey-patch sys.stdout and hack the code completion mechanism.)

Here's an emergency script that you can run in case the in/out/err streams get lost. It will replace them with objects practically identical to Pythonista's.

import importcompletion as _importcompletion
import _outputcapture

if (
    sys.stdin.__class__.__name__ != "StdinCatcher"
    or sys.stdout.__class__.__name__ != "StdoutCatcher"
    or sys.stderr.__class__.__name__ != "StderrCatcher"
):
    _outputcapture.CaptureStdout(b"I'm alive.\n")
    
    class StdinCatcher(object):
        def __init__(self):
            self.encoding = "utf8"
        
        def read(self, limit=-1):
            return _outputcapture.ReadStdin(limit)
        
        def readline(self):
            return _outputcapture.ReadStdin()
    
    sys.stdin = StdinCatcher()
    
    _outputcapture.CaptureStdout(b"Rebuilt StdinCatcher and sys.stdin...\n")
    
    class StdoutCatcher(object):
        def __init__(self):
            self.encoding = "utf8"
        
        def flush(self):
            pass
        
        def write(self, s):
            if isinstance(s, str):
                _outputcapture.CaptureStdout(s)
            elif isinstance(s, unicode):
                _outputcapture.CaptureStdout(s.encode("utf8"))
        
        def writelines(self, lines):
            for line in lines:
                self.write(line + "\n")
    
    sys.stdout = StdoutCatcher()
    
    _outputcapture.CaptureStdout(b"Rebuilt StdoutCatcher and sys.stdout...\n")
    
    class StderrCatcher(object):
        def __init__(self):
            self.encoding = "utf8"
        
        def flush(self):
            pass
        
        def write(self, s):
            if isinstance(s, str):
                _outputcapture.CaptureStderr(s)
            elif isinstance(s, unicode):
                _outputcapture.CaptureStderr(s.encode("utf8"))
        
        def writelines(self, lines):
            for line in lines:
                self.write(line + "\n")
    
    sys.stderr = StderrCatcher()
    
    _outputcapture.CaptureStdout(b"Rebuilt StderrCatcher and sys.stderr...\n")

[blmacbeth]

@dgelessus Thank you for that excellent explanation. Where did you find all of that out? Is it hidden deep within the documentation, or have you looked through some source code? Just wondering.

[dgelessus]

As far as I know none of this is documented anywhere, but Python has great introspection capabilites and an interactive prompt. With a little brute-forcing I managed to figure out much of the iOS/Python interaction internals.

The _outputcapture module is in the list of sys.builtin_module_names and the syntax of its functions is (relatively) easy to guess. It is always imported (because it is necessary for console IO), but it is hidden from the autocompletion list. If you want to experiment with it, use import _outputcapture as outputcapture to import it under a name that isn't hidden from autocompletion.

Pythonista's custom in/out/err stream replacements are normal Python classes/objects, and although there is no source code for them, they can be decompiled using the dis module. Python bytecode is not too hard to read in its decompiled form, so it was relatively easy to translate it back into Python source code.

Figuring out the interactive autocompletion was a little more challenging. At first I only noticed that when typing a single _ into the console without hitting Return, one of the completions is _pythonista_complete_line. However when trying to use that function it isn't there - it is only created when Pythonista needs a line completion, and once the results are returned it is immediately deleted. It still lists itself in the completion list though, because it exists in the main namespace until it is fully executed once.

Later I found out how I could get a reference to the actual function object - I (accidentally) shadowed the built-in set type with another function, which caused the line completion function to raise exceptions every time it ran. (This made the interactive prompt completely unusable, so from then on I had to use short Python scripts as a console replacement.) Because the exceptions occurred inside of _pythonista_complete_line, their tracebacks had a reference to the function, which I could dis-assemble to reconstruct the code:

def pythonista_complete_line():
    import string
    import sys
    from rlcompleter import Completer
    completions = _importcompletion.complete("current input text", "text")
    if completions is not None and len(completions) > 0 and completions[0] != "importcompletion":
        sys.stdout.write(string.join(completions, "|"))
    else:
        completer = Completer()
        completions = []
        exclusions = set(["___omz_complete___(", "_importcompletion", "_outputcapture"])
        for i in range(1000):
            c = completer.complete("current input text")
            if not c: break
            if c not in exclusions: completions.append(c)
        if len(completions) > 0: sys.stdout.write(string.join(completions, "|"))
        else: sys.stdout.write("|")

[omz]

Nice detective work there, @dgelessus! :)

[JonB]

Omg, thank you! I had discovered the function, but was never able to figure out the interface, or hoe it got access to the current input text line. Will this work for allowing completions within Cmd loops? Or ... Ok, your current input text Is really a placeholder for some inaccessible string?

I would LOVE to implement a history system usable within pdb.pm() for example.

[dgelessus]

Right, I forgot to explain the placeholder strings I used. For whatever reason _pythonista_complete_line is constructed with the current input string in it as literals. (Who needs function arguments anyway?) What is "current input text" in my example is the full input line, and "text" is the last word of the input (i. e. "current input text".rsplit(" ", 1)) because importcompletion requires it.

@JonB, the line completion is only called when in interactive Python mode. Pythonista doesn't attempt any completion during raw_input and such, so there's nowhere to inject your own code into. I don't think there would be any way to get the input before Pythonista executes it, so there isn't much use in hacking the line completion at the moment. In any case you'd need some way to differentiate normal output and completion lists - _pythonista_complete_line uses sys.stdout and there's no way to change that.

As far as I can tell the input history is separate from line completion, the two last matching history items are always added under the result list no matter what the code completer returns.

PS: Found something. This code should work as it is with no preparations necessary.

def write(self, s):
    ##if s == "|":
    inside_pcl = False
    try:
        raise Exception()
    except Exception:
        if sys.exc_info()[2].tb_frame.f_back.f_code.co_name == "_pythonista_complete_line":
            inside_pcl = True
    
    if inside_pcl:
        self.real_write("A suggestion" if s == "|" else (s + "|A suggestion"))
    else:
        self.real_write(s)

write_im = type(sys.stdout.write)(write, sys.stdout, type(sys.stdout))
if not hasattr(sys.stdout, "real_write"):
    sys.stdout.real_write = sys.stdout.write
sys.stdout.write = write_im

[mteep]

I have recently installed the latest 1.6 beta over an existing 1.5 on both an iPad and an iPhone. On both devices I lost all keychain data (dropbox credentials) during the update, which has never happened before. As described by others, I also once got a file blanked out. It was a file that I just had left open when switching to another app and at some later time Pythonista got unloaded.

I've mainly been testing the cb module, successfully connecting to a WowWee MiP self-balancing (segway like) toy robot. The robot uses a single characteristic to send commands, and another characteristic with notifications for responses. Though I am new to BLE, everything seems to work. At least when I'm just sending a few commands and printing responses to the console.

However, when I tried to make a simple driving app using the ui module, I started to get a lot of crashes. I suspect these are due to threading issues, but I'm not proficient enough in Python yet to understand how to avoid them. The robot can drive continously at a given speed, but the command needs to be repeated every 50 ms or the robot will stop as a safety measure. I tried to accomplish this using ui.delay(), which crashed immediately when used by itself. But in combination with ui.in_background it sometimes worked for a while before crashing.

But I assume the crashes due to threading problems aren't specific to the beta or the cb module, and should be discussed elsewhere. Unless Ole wants to share any special threading considerations that only apply to the cb module.

[ccc]

The MiP sounds like a fun thing to hack with. Have you tried using the yaw, pitch, and roll of your iOS devise to control the MiP? motion.get_attitude() gives you access to these values.

[wradcliffe]

I believe that there are issues with threads but have not been able to pin anything down enough to give omz anything to work with. I spent a great deal of time working on a way of logging that did not interfere with the cb modules natural functionality. I kept having the issue that a crashing problem (crash or weird behavior) would go away when I got the logging going. I can say for sure that each callback is run in a new thread each time and it looks a lot like IOS is using some kind of thread pool. The thread names seem to count up and then start over. I have been able to get dozens of different threads going using several peripherals. What all this means is that any code you write in a callback needs to be thread safe and some elements of the python libraries may not be thread safe in these callbacks. What omz needs is a sample app that clearly shows if one of the callbacks can be running simultaneously in multiple threads. Perhaps your app is just doing something that takes a long time in a callback and somehow gets python to context switch and allow the callback to get re-enterred? I am drawing at straws here. I would be willing to get a WowWee MIP in order to help repo this. This would be a good one to use as the sample app for cb in the future IMHO. Anything with a fairly high data transfer rate would be the best. I have been using a BLE MIDI interface and these tend to generate lots of data.

[JonB]

@dgelessus nice!
So, I gather the history is handled through some builtin that we cannot access? (Builtins can't be disassembled, right?)
I was hoping for the ability to add to history programatically, oh well. That leads to a feature request: @omz Add console.history command to get / set history. And/or some way to enable completion on raw_input ( which we could finesse into our own history system :)

[ccc]

Reported elsewhere but repeated here for completeness:

import photos                     # When you tap 'Cancel' in the image picker...
x = photos.pick_image()                # x = None which is the correct behavior.
y = photos.pick_image(include_metadata=True)  # Exception thrown which is a bug.

[mteep]

@ccc
No, I haven't tried using the device orientation to control the MiP. I first need to be able to update its speed continuously. Also, in my experience, virtual joysticks works better to control physical things, mainly because you easily can let go of them. So, like the original app by default, I use two of them.

@wradcliffe
I don't get any crashes if I just send the speed to the robot whenever the joystick changes (in the action function). But, unless I constantly move my finger, the robot stutters. And even if I do this, the robot turns slightly counter-clockwise every now and then. (I only look at the joystick y position and always sends straight forward or backward speeds.) This never happens with the original app. So my current theory is that I send data too fast in this case. That is, that the value of the characteristic changes before it has been fully parsed within the robot. This turning also didn't seem to occur during the short periods I was able to drive while limiting the rate using ui.delay().

So, I don't think the crashing issue is related specifically to the cb module. (But possibly to ui.delay(), since it tends to crash Pythonista also if I try to update the UI in a posted function.) I will try different approaches, but it might take a few days.

[wradcliffe]

@mteep - one big question I have about CoreBluetooth is whether it buffers the data being transferred to and from a characteristic. I believe it does and you can get it to crash be overrunning its buffering. This is just a theory. I have read quite of posts on stackoverflow having to do with people not getting enough speed and what to expect for speed of transfer. Some of this is confusion about Bluetooth LE (Smart) being both for low power and high speed. It is NOT for high speed and it seems difficult to actually make it low power.

[JonB]

Mteep, do you call ui.cancel_delays at the start of the function that was calling ui.delay? That might not do exactly what you want, but it would keep you from calling the cb function simultaneously.

One pattern for dealing with this issue is that you've got one thread only that talks to the resource, and other threads send messages. You might consider using a threading.Thread, which gets commands from a queue.Queue. Your ui would put values into the queue on touch moved, or whatever your interface is.

The Thread would manage how frequently it sends cb commands, for instance maybe you send with a 10msec delay if the queue.get returned a value, but if you get Queue.Empty, send the last command then wait for 50 msec. You'd probably want to handle the Queue getting too full, in which case you'd be falling behind.. Like check qsize, and if more than some value, you could read multiple values from the queue, but only send the latest.

[mteep]

I got my MiP driving app to work like the original app once I set up a dedicated thread to send the speed every 50 ms.

@JonB
The ui.delay() seems to be more or less the same as creating a threading.Timer, which is a new thread. I had hoped/guessed that it would instead schedule the function to be run in the UI thread, as this is something most UI frameworks I am familiar with supports and often requires. (As a side note, that erroneous approach worked for a longer time when I tried it on a newer (A8X class) device.)

The turning seems to have been caused by a misreading of the MiP protocol description. You send it a command byte followed by a number of argument bytes that depend on the command. The problem is that the robot often accept commands with fewer arguments, if there is sufficient delay afterwards. So I sent two too short driving commands after each other, and the second one was interpreted the turning argument of the first one.

So, as long as you observe thread safety and send commands of the correct length, controlling the MiP using the cb module seems entirely deterministic.

@wradcliffe
I will see if I can come up with a test of your buffering theory, but in the MiP case, there is really no point in trying to push data faster than the robot physically can act. So throttling on the host side seems reasonable. (A possible exception is if you open up the robot and use the hacker port UART to communicate with some other micro-controller.)

[wradcliffe]

@mteep - can you post your script as an example. I would like to see how you setup the dedicated thread and did the timing and communication between the callback and the thread.

[wradcliffe]

@omz - I found a real bug (probably in 1.5 as well). The animation function curve_bounce_in_out calls two non-existent functions.

def curve_bounce_in_out(x):
    t = x
    d = 1.0
    if t < (d / 2.0):
        return curve_ease_in_bounce(x * 2) * 0.5
    return curve_ease_out_bounce(x * 2 - d) * 0.5 + 0.5

Should be curve_bounce_in and curve_bounce_out - right?

[0942v8653]

@wradcliffe: confirmed in 1.5.

[mteep]

@wradcliffe
Sorry about the late reply. The forum was down last time I tried answering. My script is a little too long for this thread, so I have removed almost everything except the sending thread. (I will post the whole thing to GitHub once it's a little more complete.) The MipManager starts the SpeedUpdater thread once it gets the write characteristic in a callback. My UI calls action methods in MipManager (not shown), which either sets the current speed or turn in the SpeedUpdater, or occasionally queues a special command. Currently, I only send other commands when the robot has fallen over in order to attempt to get up (something the original app doesn't do at all), so the simple timing works. Finally, when my UI is closed, I call the shutdown method of SpeedUpdater via the MipManager.

Note that I'm not sure of the thread safety of this code or of Queue.Queue. I just noticed that the threading support was closely modeled after Java and hoped the memory model would be too.

import cb
import threading
import Queue

class SpeedUpdater(threading.Thread):
    def __init__(self, peripheral, characteristic):
        threading.Thread.__init__(self)
        self.peripheral = peripheral
        self.characteristic = characteristic
        self.state_lock = threading.Condition()
        self.keep_alive = True
        self.speed_code = 0
        self.turn_code = 0
        self.queue = Queue.Queue()
    
    def run(self):
        keep_alive = True
        while keep_alive:
            try:
                msg = self.queue.get_nowait()
                self.peripheral.write_characteristic_value(self.characteristic, msg, False)
            except Queue.Empty:
                with self.state_lock:
                    self.state_lock.wait(0.05)
                    speed_code = self.speed_code
                    turn_code = self.turn_code
                if (speed_code != 0) or (turn_code != 0):
                    msg = chr(0x78)+chr(speed_code) + chr(turn_code)
                    self.peripheral.write_characteristic_value(self.characteristic, msg, False)
            with self.state_lock:
                keep_alive = self.keep_alive
                
    def set_speed(self, speed_code):
        with self.state_lock:
            self.speed_code = speed_code

    def set_turn(self, turn_code):
        with self.state_lock:
            self.turn_code = turn_code
            
    def queue_cmd(self, cmd):
        self.queue.put(cmd)

    def shutdown(self):
        with self.state_lock:
            self.keep_alive = False
            self.state_lock.notifyAll()
        self.join()

class MiPManager (object):
    def __init__(self):
        self.state_lock = threading.RLock()
        self.peripheral = None
        self.speed_updater = None

    def did_discover_characteristics(self, s, error):
        log('Did discover characteristics...')
        for c in s.characteristics:
            if c.uuid == 'FFE9':
                with self.state_lock:
                    self.speed_updater = SpeedUpdater(self.peripheral, c)
                self.speed_updater.start()

[wradcliffe]

@mteep - did you determine that the RLock used in the MiPManager object was necessary to prevent crashes?