StaSh - Shell Like an Expert in Pythonista

dgelessus

Well this is awkward, I've been working on basically the same thing, except with no UI and less functionality... Nicely done though, I have yet to play around with it some more, but having working pipes and some of the more advanced shell features sounds useful. And it looks like you figured out how to reliably scroll down the console output, the method I used in ShellistaUI always started to have problems after a hundred lines or so.

Regarding executable files, have you thought of using shebangs?

#!/bin/sh
# ^ that is a shebang, it tells the shell which executable to use to run this file.
# Normally the file must have the executable bit set for the shell to look at the shebang.

(I'll see about making a pull request so I can merge some of my code into your repo - your core shell is definitely better than mine, and since mine also loads commands from Python scripts the systems should be compatible.)

zencuke

Woah! Thanks.

wradcliffe

This is very nice. Thanks so much for contributing this.

You mention in your README that scripts can interact with the shell. How does that work? Sounds like it is intended to include scripts running a UI. I notice that when I open the history UI and then type a new command, the UI does not update. Could it? Is clicking on an item in the history list and example of script communication back to the shell?

ywangd

@dgelessus

My first attempt was trying to work from shellista. Then I realized it is impossible to achieve what I want without re-writing the whole thing. ShellistaUI points the right direction by employs the UI system. But it still depends on shellista and has many the same limitations. So I decided to start over as a new project.
Please be my guest to add your code to it. Sorry if it means double effort for you.

The code was compiled in past 3 weeks. It is not clean or elegant. There sure is much room for improvements. In fact I wasn't entirely happy with the pyparsing bit and may have some major overhaul when I have the time.

I thought about shebang as well. I didn't implement it mainly because I am too lazy to do it. The parser in its current state does not parse comments, i.e. #. Also I am not sure how the performance would like if the Shell needs to read the first line of every file to tell the type. In Pythonista's controlled environment, I decided to take the lazy way and rely on file extensions. But, in fact, a file without extension will still run as StaSh script, e.g script will run if a file called script is found. It will just not show up in the auto-completion possibilities when your type scri and press Tab.

On a side note, I feel my development workflow is somewhat awkward. I effectively use GitHub as a staging area. Once I made changes on my computer, I push to GitHub and retrieve them using selfupdate on my phone. Please let me know if you have a better way to streamline the whole workflow.

ywangd

@wradcliffe

I have to admit I did most of the development on a Phone and haven't thought about updating the history UI in real-time while typing commands. It is definitely a nice feature and I'll work on it when time permits.

However, it is worth to note that the history UI is NOT an external script. It is an inherent part of the Shell. Yes it runs in its own thread. So it does communicate back to the Shell. But it is not exactly the interaction between Shell and an external script that mentioned in the README.

The interaction between shell and running external script are mainly for two type of things:

Allow a running script access shell's status and act accordingly.

I also thought about (not implemented) a callback system that the running script can register to. For an example, a running script can register for being called when the shell's input textfield returns or a button tapped. One use of this feature is to allow running thread being terminated from the shell gracefully. But it does require the script to be written specifically to utilize this feature. For regular script, StaSh relies on Thread._Thread__stop() to terminate the running thread. It often works but not guaranteed.

Allow a running script to ask the shell to run another shell command

StaSh does not have a full set of shell language. It does not have keywords and program control constructs, such as if, for. Hence a StaSh script can only do limited things (basically just a list of commands to be executed from the top to bottom. But with this feature, it is possible to write a Python script that acts like a real Shell script. Here is a contrived example:


_stash = globals()['_stash']  # get the shell object

# Commands to get a sequence of files from an url
cmd_base = r'''wget http://www.test.com/file_%d.txt -o saved_%d.txt'''
for i in range(10):
    cmd = cmd_base % (i, i)
    worker = _stash.runtime.run(cmd, add_to_history=False, reset_inp=False)
    if worker.isAlive():  
        # wait for the thread to finish as StaSh does not allow parallel threads
        pass

I confess it is rather a convoluted way to interact with the Shell. But it is available if you really wants it.

jmv38

@ywangd this looks fantastic! My question may sound totally stupid to you, but i have no idea of what a 'shell' is, so could you put explainations (or a link to some) about the global context of your addon? You are giving a lot of explainations on the details, but not the global picture. Thanks!

JonB

Few comments

awesome work!
the slight flashing due to the multiple scrolls can be eliminated by using an ui.animation(). This probably deserves some more testing. I think the animate must force the drawing queue to finish before calling the animate function. Thus content_size gets properly refreshed the first time, without having to keep trying to scroll.

    def _scroll_to_end(self):
        # small animation to force proper loading of content_size
        def ani():
            self.out.content_offset = (0, self.out.content_size[1] - self.out.height)
        ui.animate(ani,0.01)

EDIT. Never mind, while this works for some cases, an empty line seems to not scroll to the end using this method. Oh well.

maybe consider adding some additional useful keys to virtual keyboard? On ios I find it particularly annoying to get to _*|~>, basically anything on the symbol page.
the gist install script didn't install all of the scripts. I had to run the selfupdate script to get things like ls, etc. also, the gist installed to Documents/stash/bin, but the BIN_PATH pointed to /Documents/bin.

ywangd

@JonB

I just had some test using ui.animate. As you found out, it does not in all cases. I really like the effect that it scrolls really smoothly. Pity it still is not perfect.
Adding more buttons for symbols is a good idea. I am still clumsy with the ui system. But this addition should be really handy for the shell.
The gist script was intended to just install the minimal set of files so that selfupdate can run. This is because the gist script use a very primitive approach to issue an URL request for every single file (so it does not rely on unzip). So I decided to have a two stage installation. This is mentioned in my original post. Maybe I didn't emphasize it enough and it could cause confusions.
The BIN_PATH is set to ~/Documents/stash/bin:~/Documents/bin if you type echo $BIN_PATH. However, due to a bug, the path ~/Documents/bin actually does Not work. So BIN_PATH is just ~/Documents/stash/bin in its fully expanded form. I have fixed the bug along with some other fixes. You can selfupdate to get the latest changes.

Thanks for the comments.

ywangd

@jmv38

A shell is a command line interpreter that sits between the end-user and operating system. It is a traditional text based user interface.

Here is the wiki page about the Unix Shell. Also a wiki page about Bash, often called the Shell as it is so influential (for people like C Shell, please, I am not trying to start a religious war here).

Graphic interfaces are so popular and dominant in current world. But a shell is still a very valuable tool, especially at performing administrative works. Basically if a program often needs to be called with very different arguments and it can benefit when used with other programs, a shell comes in handy.

Please note that I am talking about real Shells, e.g. Bash. I am not claiming that StaSh is a real shell. It just models after Bash and provide some opportunities for like-minded people. Personally I wanted a shell because I often need to run some CLI (command line interface) python tools, which could be cumbersome to run using action menus (because of constantly changing arguments).

briarfox

@ywangd Very very nice. Thank you. I really like the console look. Would you mind If I borrowed some of the code? I am working on a console that connects to a Raspberry Pi via bluetooth.

Are you planning to add ssh access?

briarfox

I've had a chance to play with it some. I'm really liking the layout. I do like how easy it is to add Python scripts. IT would be great to see a git , ssh client/server, Python interactive interpreter. I've added a small script that allows running of Python modules and scripts(main use is to call SimpleHTTPServer, Pylint, nose.

I haven't looked through the code to deeply but what would be the best way to take over the UI input for a Python script such as an ssh client.

With a few additions I will be using this instead of shellista. Great Job.

jmv38

@ywangd thanks a lot for your explanations.

ywangd

@briarfox

Thanks for your comments. Please feel free to use any code from StaSh. The UI layout has its root from ShellistaUI. So credits also goes to ShellistaUI (I believe @dgelessus is the original author of ShellistaUI). I also borrowed code from your ShellistaExt for various commands such as ls, cp etc. It would be even more appreciated if you could share your changes or even send pull requests.

StaSh is more of a framework that manages calls to python scripts. So I am hoping that the community would provide the scripts. In principle, if a script can be executed on a PC, it would also be usable from within StaSh. I/O should be taken care of automatically in most cases.
For an example, your ssh code (https://github.com/briarfox/ShellistaExt/blob/master/ShellistaExt/plugins/extensions/ssh/ssh.py) is pretty immediately usable in StaSh. Just place it into the BIN_PATH, e.g. ~/Documents/stash/bin and just type ssh in Stash. You'll see the command gets executed. If you set proper host/user/pass in __main__, it would connect to the host and all I/O is handled by StaSh. Apparently, for the script to be more useful, you'll need some arguments handling in __main__, a primitive approach could be as follows (add to the end of ssh.py):

if __name__ == '__main__':
    if len(sys.argv) == 1:
        print 'Usage: connect hostname user password'
        sys.exit(0)
    cmd = sys.argv[1]
    params = ' '.join(sys.argv[2:])
    if cmd == 'connect':
        ssh = SSH()
        ssh.connect(params)
    else:
        print 'NYI'

Pythonista builtin interactive prompt is available even when StaSh is running. Just tap on the first button of the segmentation control on the top-left side. Sure you can add your own interactive interpreter, but the builtin one just has more features such as code completion etc.
I haven't thought about running Python modules. I would like to see SimpleHTTPServer gets run in StaSh. Please let me know your changes if it is OK with you.
I haven't had my hands on a Raspberry Pi yet. But I am a Lego Mindstorms fan. I'd love to know how your code works and see if it is adaptable to Mindstorms.

briarfox

@ywangd I do like the drag and drop with stash. I'm still working on the running of modules. Code works great until I drop it in StaSh. I keep getting a package set to non string Error. I need to track down the cause. I'm hoping to run it on its own thread so the command:
<pre>
python -m SimpleHTTPServer &
</pre>
Will run it as a thread and allow you to load a client to work with it. It's a simple script that uses the module runpy and passes the args. I'll submit a pull request when I get the bugs out.

edit Fixed

briarfox

@ywangd Would you mind checking out this script? It works when ran my itself. But ShaSh when trying:
<pre>
python -m SimpleHTTPServer &
</pre>
It throws a "package set to non string"
python.py

As for the question on the Rpi and BTLE, I have a bluetooth adapter on the Rx and Tx pins of the RPi. I'm just reading and sending data with the pythonista cb beta module. I'll post it up for you.

ywangd

@briarfox

This error is tricky. It is due to the module_name variable being unicode type instead of an plain ASCII str. So once you add

module_name = str(module_name)

into the code, everything works.

This error is probably due to the fact that Python 2.x and its libraries are not fully unicode compliant. In fact, if you type runpy.run_module(u'SimpleHTTPServer', run_name='__main__') into a PC version python interpreter, it reports exactly the same error.

When the script runs by itself, sys.argv is set to plain strings. But it is set to the same strings of unicode version when executed by StaSh. I am not entirely clear about the reason. Anyway, the fix is easy. I'll also see if it is worthwhile to always convert all arguments to str before calling an external script.

dgelessus

@ywangd, the ui module uses unicode strings exclusively. Since the command input field is just a regular text field, it also returns its contents as unicode, and since there's no reason for Python to coerce down to str it doesn't. Also instead of a plain str() conversion it would be better to use the encode() method of unicode objects, that way the string is guaranteed to be properly encoded as UTF-8.

ywangd

@dgelessus Thanks for the explaination. That makes sense.

I am still thinking about whether it is really necessary to always encode unicode arguments to str before executing a command.

dgelessus

@ywangd, it isn't really necessary, because you'll almost never need to pass non-ASCII characters to a command. But when you do try, you'll very likely run into problems. I'm not sure what exactly happens when you do str(u"üñîçø∂ε"). encode is also guaranteed to work in future Python versions. Python 3 (if/when Pythonista supports it) is much more strict when it comes to byte data and unicode text.

I checked, and Python 2.7 on my Raspberry Pi encodes runtime args as UTF-8 str objects. It is possible that the encoding is OS-dependent, but UTF-8 is usually a safe choice. Python 3 of course uses proper Unicode str objects instead.

PS:

You're lucky, str encodes unicode strings as UTF-8 anyway. encode still is safer ;)

briarfox

@ywand Thank you very much, can't believe I missed that. Spend hours looking for that problem.

It seems that '&' is not passed by the input. I was using that to flag its own thread. I added it to _word_chars and it now passes &. Is it held for a reason?

@dgelessus Thanks for the explanation