omz:forum

    • Register
    • Login
    • Search
    • Recent
    • Popular

    Welcome!

    This is the community forum for my apps Pythonista and Editorial.

    For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.


    Pythonista 1.6 Beta

    Pythonista
    55
    301
    474102
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Saturn031000
      Saturn031000 last edited by

      Just wondering, what happens when the beta 'expires'? Does it revert back to 1.5?

      1 Reply Last reply Reply Quote 0
      • dgelessus
        dgelessus last edited by

        @blmacbeth, this is due to the unusual way that Pythonista uses to print output to the console. What is stored in sys.__stdout__ is the default Python output stream (a file object pointing to file descriptor 1, which is Unix stdout) but (presumably) because of iOS limitations that goes into nowhere.

        Instead Pythonista has a "secret" module named _outputcapture with a few built-in functions to handle text I/O through the Pythonista console. sys.stdout is also replaced with a StdoutCatcher instance, whose write method encodes the string and shows it using _outputcapture. Because the StdoutCatcher class isn't stored anywhere by default, it is lost when you change sys.stdout and it is hard to restore afterwards.

        The reason why the interactive code completion stops working as well is simple - whenever Pythonista needs a list of possible completions, it creates and runs a function named _pythonista_complete_line, which uses some standard Python module to do the completion and prints all possible results to sys.stdout, where they are caught by Pythonista and displayed in the completion list. (Yes, it is possible to monkey-patch sys.stdout and hack the code completion mechanism.)

        Here's an emergency script that you can run in case the in/out/err streams get lost. It will replace them with objects practically identical to Pythonista's.

        import importcompletion as _importcompletion
        import _outputcapture
        
        if (
            sys.stdin.__class__.__name__ != "StdinCatcher"
            or sys.stdout.__class__.__name__ != "StdoutCatcher"
            or sys.stderr.__class__.__name__ != "StderrCatcher"
        ):
            _outputcapture.CaptureStdout(b"I'm alive.\n")
            
            class StdinCatcher(object):
                def __init__(self):
                    self.encoding = "utf8"
                
                def read(self, limit=-1):
                    return _outputcapture.ReadStdin(limit)
                
                def readline(self):
                    return _outputcapture.ReadStdin()
            
            sys.stdin = StdinCatcher()
            
            _outputcapture.CaptureStdout(b"Rebuilt StdinCatcher and sys.stdin...\n")
            
            class StdoutCatcher(object):
                def __init__(self):
                    self.encoding = "utf8"
                
                def flush(self):
                    pass
                
                def write(self, s):
                    if isinstance(s, str):
                        _outputcapture.CaptureStdout(s)
                    elif isinstance(s, unicode):
                        _outputcapture.CaptureStdout(s.encode("utf8"))
                
                def writelines(self, lines):
                    for line in lines:
                        self.write(line + "\n")
            
            sys.stdout = StdoutCatcher()
            
            _outputcapture.CaptureStdout(b"Rebuilt StdoutCatcher and sys.stdout...\n")
            
            class StderrCatcher(object):
                def __init__(self):
                    self.encoding = "utf8"
                
                def flush(self):
                    pass
                
                def write(self, s):
                    if isinstance(s, str):
                        _outputcapture.CaptureStderr(s)
                    elif isinstance(s, unicode):
                        _outputcapture.CaptureStderr(s.encode("utf8"))
                
                def writelines(self, lines):
                    for line in lines:
                        self.write(line + "\n")
            
            sys.stderr = StderrCatcher()
            
            _outputcapture.CaptureStdout(b"Rebuilt StderrCatcher and sys.stderr...\n")
        
        1 Reply Last reply Reply Quote 0
        • blmacbeth
          blmacbeth last edited by

          @dgelessus Thank you for that excellent explanation. Where did you find all of that out? Is it hidden deep within the documentation, or have you looked through some source code? Just wondering.

          1 Reply Last reply Reply Quote 0
          • dgelessus
            dgelessus last edited by

            As far as I know none of this is documented anywhere, but Python has great introspection capabilites and an interactive prompt. With a little brute-forcing I managed to figure out much of the iOS/Python interaction internals.

            The _outputcapture module is in the list of sys.builtin_module_names and the syntax of its functions is (relatively) easy to guess. It is always imported (because it is necessary for console IO), but it is hidden from the autocompletion list. If you want to experiment with it, use import _outputcapture as outputcapture to import it under a name that isn't hidden from autocompletion.

            Pythonista's custom in/out/err stream replacements are normal Python classes/objects, and although there is no source code for them, they can be decompiled using the dis module. Python bytecode is not too hard to read in its decompiled form, so it was relatively easy to translate it back into Python source code.

            Figuring out the interactive autocompletion was a little more challenging. At first I only noticed that when typing a single _ into the console without hitting Return, one of the completions is _pythonista_complete_line. However when trying to use that function it isn't there - it is only created when Pythonista needs a line completion, and once the results are returned it is immediately deleted. It still lists itself in the completion list though, because it exists in the main namespace until it is fully executed once.

            Later I found out how I could get a reference to the actual function object - I (accidentally) shadowed the built-in set type with another function, which caused the line completion function to raise exceptions every time it ran. (This made the interactive prompt completely unusable, so from then on I had to use short Python scripts as a console replacement.) Because the exceptions occurred inside of _pythonista_complete_line, their tracebacks had a reference to the function, which I could dis-assemble to reconstruct the code:

            def pythonista_complete_line():
                import string
                import sys
                from rlcompleter import Completer
                completions = _importcompletion.complete("current input text", "text")
                if completions is not None and len(completions) > 0 and completions[0] != "importcompletion":
                    sys.stdout.write(string.join(completions, "|"))
                else:
                    completer = Completer()
                    completions = []
                    exclusions = set(["___omz_complete___(", "_importcompletion", "_outputcapture"])
                    for i in range(1000):
                        c = completer.complete("current input text")
                        if not c: break
                        if c not in exclusions: completions.append(c)
                    if len(completions) > 0: sys.stdout.write(string.join(completions, "|"))
                    else: sys.stdout.write("|")
            
            1 Reply Last reply Reply Quote 0
            • omz
              omz last edited by

              Nice detective work there, @dgelessus! :)

              1 Reply Last reply Reply Quote 0
              • JonB
                JonB last edited by

                Omg, thank you! I had discovered the function, but was never able to figure out the interface, or hoe it got access to the current input text line. Will this work for allowing completions within Cmd loops? Or ... Ok, your current input text Is really a placeholder for some inaccessible string?

                I would LOVE to implement a history system usable within pdb.pm() for example.

                1 Reply Last reply Reply Quote 0
                • dgelessus
                  dgelessus last edited by

                  Right, I forgot to explain the placeholder strings I used. For whatever reason _pythonista_complete_line is constructed with the current input string in it as literals. (Who needs function arguments anyway?) What is "current input text" in my example is the full input line, and "text" is the last word of the input (i. e. "current input text".rsplit(" ", 1)) because importcompletion requires it.

                  @JonB, the line completion is only called when in interactive Python mode. Pythonista doesn't attempt any completion during raw_input and such, so there's nowhere to inject your own code into. I don't think there would be any way to get the input before Pythonista executes it, so there isn't much use in hacking the line completion at the moment. In any case you'd need some way to differentiate normal output and completion lists - _pythonista_complete_line uses sys.stdout and there's no way to change that.

                  As far as I can tell the input history is separate from line completion, the two last matching history items are always added under the result list no matter what the code completer returns.

                  PS: Found something. This code should work as it is with no preparations necessary.

                  def write(self, s):
                      ##if s == "|":
                      inside_pcl = False
                      try:
                          raise Exception()
                      except Exception:
                          if sys.exc_info()[2].tb_frame.f_back.f_code.co_name == "_pythonista_complete_line":
                              inside_pcl = True
                      
                      if inside_pcl:
                          self.real_write("A suggestion" if s == "|" else (s + "|A suggestion"))
                      else:
                          self.real_write(s)
                  
                  write_im = type(sys.stdout.write)(write, sys.stdout, type(sys.stdout))
                  if not hasattr(sys.stdout, "real_write"):
                      sys.stdout.real_write = sys.stdout.write
                  sys.stdout.write = write_im
                  
                  1 Reply Last reply Reply Quote 0
                  • mteep
                    mteep last edited by

                    I have recently installed the latest 1.6 beta over an existing 1.5 on both an iPad and an iPhone. On both devices I lost all keychain data (dropbox credentials) during the update, which has never happened before. As described by others, I also once got a file blanked out. It was a file that I just had left open when switching to another app and at some later time Pythonista got unloaded.

                    I've mainly been testing the cb module, successfully connecting to a WowWee MiP self-balancing (segway like) toy robot. The robot uses a single characteristic to send commands, and another characteristic with notifications for responses. Though I am new to BLE, everything seems to work. At least when I'm just sending a few commands and printing responses to the console.

                    However, when I tried to make a simple driving app using the ui module, I started to get a lot of crashes. I suspect these are due to threading issues, but I'm not proficient enough in Python yet to understand how to avoid them. The robot can drive continously at a given speed, but the command needs to be repeated every 50 ms or the robot will stop as a safety measure. I tried to accomplish this using ui.delay(), which crashed immediately when used by itself. But in combination with ui.in_background it sometimes worked for a while before crashing.

                    But I assume the crashes due to threading problems aren't specific to the beta or the cb module, and should be discussed elsewhere. Unless Ole wants to share any special threading considerations that only apply to the cb module.

                    1 Reply Last reply Reply Quote 0
                    • ccc
                      ccc last edited by

                      The MiP sounds like a fun thing to hack with. Have you tried using the yaw, pitch, and roll of your iOS devise to control the MiP? motion.get_attitude() gives you access to these values.

                      1 Reply Last reply Reply Quote 0
                      • wradcliffe
                        wradcliffe last edited by

                        I believe that there are issues with threads but have not been able to pin anything down enough to give omz anything to work with. I spent a great deal of time working on a way of logging that did not interfere with the cb modules natural functionality. I kept having the issue that a crashing problem (crash or weird behavior) would go away when I got the logging going. I can say for sure that each callback is run in a new thread each time and it looks a lot like IOS is using some kind of thread pool. The thread names seem to count up and then start over. I have been able to get dozens of different threads going using several peripherals. What all this means is that any code you write in a callback needs to be thread safe and some elements of the python libraries may not be thread safe in these callbacks. What omz needs is a sample app that clearly shows if one of the callbacks can be running simultaneously in multiple threads. Perhaps your app is just doing something that takes a long time in a callback and somehow gets python to context switch and allow the callback to get re-enterred? I am drawing at straws here. I would be willing to get a WowWee MIP in order to help repo this. This would be a good one to use as the sample app for cb in the future IMHO. Anything with a fairly high data transfer rate would be the best. I have been using a BLE MIDI interface and these tend to generate lots of data.

                        1 Reply Last reply Reply Quote 0
                        • JonB
                          JonB last edited by

                          @dgelessus nice!
                          So, I gather the history is handled through some builtin that we cannot access? (Builtins can't be disassembled, right?)
                          I was hoping for the ability to add to history programatically, oh well. That leads to a feature request: @omz Add console.history command to get / set history. And/or some way to enable completion on raw_input ( which we could finesse into our own history system :)

                          1 Reply Last reply Reply Quote 0
                          • ccc
                            ccc last edited by

                            Reported elsewhere but repeated here for completeness:

                            import photos                     # When you tap 'Cancel' in the image picker...
                            x = photos.pick_image()                # x = None which is the correct behavior.
                            y = photos.pick_image(include_metadata=True)  # Exception thrown which is a bug.
                            
                            1 Reply Last reply Reply Quote 0
                            • mteep
                              mteep last edited by

                              @ccc
                              No, I haven't tried using the device orientation to control the MiP. I first need to be able to update its speed continuously. Also, in my experience, virtual joysticks works better to control physical things, mainly because you easily can let go of them. So, like the original app by default, I use two of them.

                              @wradcliffe
                              I don't get any crashes if I just send the speed to the robot whenever the joystick changes (in the action function). But, unless I constantly move my finger, the robot stutters. And even if I do this, the robot turns slightly counter-clockwise every now and then. (I only look at the joystick y position and always sends straight forward or backward speeds.) This never happens with the original app. So my current theory is that I send data too fast in this case. That is, that the value of the characteristic changes before it has been fully parsed within the robot. This turning also didn't seem to occur during the short periods I was able to drive while limiting the rate using ui.delay().

                              So, I don't think the crashing issue is related specifically to the cb module. (But possibly to ui.delay(), since it tends to crash Pythonista also if I try to update the UI in a posted function.) I will try different approaches, but it might take a few days.

                              1 Reply Last reply Reply Quote 0
                              • wradcliffe
                                wradcliffe last edited by

                                @mteep - one big question I have about CoreBluetooth is whether it buffers the data being transferred to and from a characteristic. I believe it does and you can get it to crash be overrunning its buffering. This is just a theory. I have read quite of posts on stackoverflow having to do with people not getting enough speed and what to expect for speed of transfer. Some of this is confusion about Bluetooth LE (Smart) being both for low power and high speed. It is NOT for high speed and it seems difficult to actually make it low power.

                                1 Reply Last reply Reply Quote 0
                                • JonB
                                  JonB last edited by

                                  Mteep, do you call ui.cancel_delays at the start of the function that was calling ui.delay? That might not do exactly what you want, but it would keep you from calling the cb function simultaneously.

                                  One pattern for dealing with this issue is that you've got one thread only that talks to the resource, and other threads send messages. You might consider using a threading.Thread, which gets commands from a queue.Queue. Your ui would put values into the queue on touch moved, or whatever your interface is.

                                  The Thread would manage how frequently it sends cb commands, for instance maybe you send with a 10msec delay if the queue.get returned a value, but if you get Queue.Empty, send the last command then wait for 50 msec. You'd probably want to handle the Queue getting too full, in which case you'd be falling behind.. Like check qsize, and if more than some value, you could read multiple values from the queue, but only send the latest.

                                  1 Reply Last reply Reply Quote 0
                                  • mteep
                                    mteep last edited by

                                    I got my MiP driving app to work like the original app once I set up a dedicated thread to send the speed every 50 ms.

                                    @JonB
                                    The ui.delay() seems to be more or less the same as creating a threading.Timer, which is a new thread. I had hoped/guessed that it would instead schedule the function to be run in the UI thread, as this is something most UI frameworks I am familiar with supports and often requires. (As a side note, that erroneous approach worked for a longer time when I tried it on a newer (A8X class) device.)

                                    The turning seems to have been caused by a misreading of the MiP protocol description. You send it a command byte followed by a number of argument bytes that depend on the command. The problem is that the robot often accept commands with fewer arguments, if there is sufficient delay afterwards. So I sent two too short driving commands after each other, and the second one was interpreted the turning argument of the first one.

                                    So, as long as you observe thread safety and send commands of the correct length, controlling the MiP using the cb module seems entirely deterministic.

                                    @wradcliffe
                                    I will see if I can come up with a test of your buffering theory, but in the MiP case, there is really no point in trying to push data faster than the robot physically can act. So throttling on the host side seems reasonable. (A possible exception is if you open up the robot and use the hacker port UART to communicate with some other micro-controller.)

                                    1 Reply Last reply Reply Quote 0
                                    • wradcliffe
                                      wradcliffe last edited by

                                      @mteep - can you post your script as an example. I would like to see how you setup the dedicated thread and did the timing and communication between the callback and the thread.

                                      1 Reply Last reply Reply Quote 0
                                      • wradcliffe
                                        wradcliffe last edited by

                                        @omz - I found a real bug (probably in 1.5 as well). The animation function curve_bounce_in_out calls two non-existent functions.

                                        def curve_bounce_in_out(x):
                                        	t = x
                                        	d = 1.0
                                        	if t < (d / 2.0):
                                        		return curve_ease_in_bounce(x * 2) * 0.5
                                        	return curve_ease_out_bounce(x * 2 - d) * 0.5 + 0.5
                                        

                                        Should be curve_bounce_in and curve_bounce_out - right?

                                        1 Reply Last reply Reply Quote 0
                                        • 0942v8653
                                          0942v8653 last edited by

                                          @wradcliffe: confirmed in 1.5.

                                          1 Reply Last reply Reply Quote 0
                                          • mteep
                                            mteep last edited by

                                            @wradcliffe
                                            Sorry about the late reply. The forum was down last time I tried answering. My script is a little too long for this thread, so I have removed almost everything except the sending thread. (I will post the whole thing to GitHub once it's a little more complete.) The MipManager starts the SpeedUpdater thread once it gets the write characteristic in a callback. My UI calls action methods in MipManager (not shown), which either sets the current speed or turn in the SpeedUpdater, or occasionally queues a special command. Currently, I only send other commands when the robot has fallen over in order to attempt to get up (something the original app doesn't do at all), so the simple timing works. Finally, when my UI is closed, I call the shutdown method of SpeedUpdater via the MipManager.

                                            Note that I'm not sure of the thread safety of this code or of Queue.Queue. I just noticed that the threading support was closely modeled after Java and hoped the memory model would be too.

                                            
                                            import cb
                                            import threading
                                            import Queue
                                            
                                            class SpeedUpdater(threading.Thread):
                                            	def __init__(self, peripheral, characteristic):
                                            		threading.Thread.__init__(self)
                                            		self.peripheral = peripheral
                                            		self.characteristic = characteristic
                                            		self.state_lock = threading.Condition()
                                            		self.keep_alive = True
                                            		self.speed_code = 0
                                            		self.turn_code = 0
                                            		self.queue = Queue.Queue()
                                            	
                                            	def run(self):
                                            		keep_alive = True
                                            		while keep_alive:
                                            			try:
                                            				msg = self.queue.get_nowait()
                                            				self.peripheral.write_characteristic_value(self.characteristic, msg, False)
                                            			except Queue.Empty:
                                            				with self.state_lock:
                                            					self.state_lock.wait(0.05)
                                            					speed_code = self.speed_code
                                            					turn_code = self.turn_code
                                            				if (speed_code != 0) or (turn_code != 0):
                                            					msg = chr(0x78)+chr(speed_code) + chr(turn_code)
                                            					self.peripheral.write_characteristic_value(self.characteristic, msg, False)
                                            			with self.state_lock:
                                            				keep_alive = self.keep_alive
                                            				
                                            	def set_speed(self, speed_code):
                                            		with self.state_lock:
                                            			self.speed_code = speed_code
                                            
                                            	def set_turn(self, turn_code):
                                            		with self.state_lock:
                                            			self.turn_code = turn_code
                                            			
                                            	def queue_cmd(self, cmd):
                                            		self.queue.put(cmd)
                                            
                                            	def shutdown(self):
                                            		with self.state_lock:
                                            			self.keep_alive = False
                                            			self.state_lock.notifyAll()
                                            		self.join()
                                            
                                            class MiPManager (object):
                                            	def __init__(self):
                                            		self.state_lock = threading.RLock()
                                            		self.peripheral = None
                                            		self.speed_updater = None
                                            
                                            	def did_discover_characteristics(self, s, error):
                                            		log('Did discover characteristics...')
                                            		for c in s.characteristics:
                                            			if c.uuid == 'FFE9':
                                            				with self.state_lock:
                                            					self.speed_updater = SpeedUpdater(self.peripheral, c)
                                            				self.speed_updater.start()
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Powered by NodeBB Forums | Contributors