StaSh - Shell Like an Expert in Pythonista

[ywangd]

[EDIT]: 2016-05-26

Inspired by shellista and its variants, StaSh is a serious attempt to implement a Bash-like shell for Pythonista.

Since its initial release, valuable contributions and advices have been received constantly from the Pythonista community. The two most popular utilities are pip (authored by @briarfox) and git (authored by @jsbain). Remarkable contributions are also made by
@dgelessus, @pudquick, @oefe, @cclauss, @georg.viehoever, @BBOOXX and @bennr01.

StaSh stands for Pythonista Shell. While Sta may not be the best abbreviation for Pythonista, it forms a concise and meaningful word with the following Sh part. So the name StaSh was chose to indicate it is a confined environment and great treasures may be found within.

Installation

StaSh can be easily installed via one line of python command (courtesy of @whitone).

import requests as r; exec(r.get('http://bit.ly/get-stash').text)

Simply copy the above line, paste into Pythonista interactive prompt and execute. It installs StaSh as a Python module under the site-packages folder (~/Documents/site-packages/stash) and copies a launching script, ~/Documents/launch_stash.py for easy access.

If you have previous versions of StaSh installed (e.g. v0.4.x), You may need to restart Pythonista BEFORE the installation.

If you have a GitHub tool available in Pythonista, such as gitview or gitrepo, you can choose to directly clone or download the repository.

StaSh requires Pythonista v2.0 as the new ObjC feature is heavily used. For older Pythonista 1.5 compatible version, please refer to the v0.4 branch.

Upgrade

Once StaSh is installed, it can be easily updated by running the selfupdate command from within the shell.

• selfupdate defaults to the master branch. To update from a different branch, e.g. dev, use selfupdate dev.
• By default, selfupdate compares local and remote versions and only performs update if newer version is found. You can however force the update without version checking via selfupdate -f.
• To check for newer version without actually install it, use selfupdate -n.
• selfupdate manages StaSh installation folder and may delete files in the process. It is therefore recommended to not place your own scripts under $STASH_ROOT/bin. Instead, save your own scripts in~/Documents/bin or customise the locations with the BIN_PATH environment variable.
• You may need to restart Pythonista after the update for changes to take full effects.

selfupdate cannot be used for version 0.4.x and under. A fresh installation is needed.

Notable Features

StaSh has a pile of features that are expected from a real shell. These
features are what really set the difference from shellista.

• Panel UI program that is completely event driven

  • No blocking thread, builtin interactive prompt is accessible at all time
  • Consistent look and feel as a proper PC terminal
  • Almost all scripts can be called from within StaSh, including programs using UI and Scene packages.
  • Attributed text (color and style) support
  • Multiple sessions are possible by opening additional Panel tabs
  • Being a pure UI program, it is possible to launch and forget. The program
    stays active indefinitely. Non-UI scripts can only run for 10 minutes in background. But StaSh can stay up forever (till memory runs out due to other Apps). You can just launch StaSh to run a few commands and leave it.
    It will still be there for you when you return later.

• Comprehensive command line parsing and handling using pyparsing

  • Environmental variables, e.g echo $HOME, NAME=value
  • Aliases, e.g. alias l1='ls -1'
  • Single and double quotes behave like Bash, e.g. "*" means literal *, "$HOME" expands while '$HOME' does not.
  • Backslash escaping, e.g. ls My\ Script.py
  • Glob, e.g. ls ~/*.py
  • Backtick quotes for subprocess, e.g. touch `ls *.py`
  • Pipes to chain commands, e.g. find . -name "*.txt" | grep interesting
  • IO redirect (actually just Output redirect), e.g. ls *.py > py_files.txt. Input redirect can be achieved by using pipes.
    • It is possible to redirect to the Pythonista builtin console,
      e.g. ls > &3
  • Bang(!) to search command history, e.g. ls -1, !l. Bang commands like !! and !-1 also works.

• Smart auto-completion just as expected

  • One UI button, Tab, is provided to enable command line auto-completion.
  • It is smart to complete either commands or files based on the cursor position
  • It also completes environment variables and aliases.
  • It also features a sub-command auto-completion system. For an example, type git sta and press Tab. It will auto-completes to git status . You can easily add your own sub-commands completion via JSON files.

• Thread management allows multiple commands running in parallel

  • One foreground jobs and unlimited number of background jobs can run simultaneously.
  • A foreground job can be stopped by pressing the CC button or Ctrl-C on an external keyboard.
  • A background job is issued by appending an ampersand character (&) at the end of a normal command, e.g. httpserver &. It can be terminated by the kill command using its job ID.
  • A few utilities are provided for thread management.
    • jobs to list current running background jobs.
    • kill to kill a running job.
    • fg to bring background jobs to foreground
    • CZ button (Ctrl-Z) to send a foreground job to background

• Command line history management. Three UI buttons are provided to navigate through the history.

• On-screen virtual keys - an extra row of keys on top of the on-screen keyboard to provide control functions and easier access to symbols

  • Virtual keys for control functions including:
    • Tab - command line auto-completion
    • CC (Ctrl-C) - terminate the running job
    • CD (Ctrl-D) - end of Input
    • CU (Ctrl-U) - kill line
    • CZ (Ctrl-Z) - Send current running foreground job to background
    • KB - show/hide keyboard
    • H - display a popup window to show command history
    • Up - recall the previous command in history
    • Dn - recall the next command in history
  • Customisable virtual keys for commonly used symbols, e.g. ~/.-*|>.
    • The Symbols can be customized via the VK_SYMBOLS option in stash config file (default is .stash_config).

• Swipe on the virtual key row to position cursor (similar to what Pythonista builtin editor offers)

• External keyboard support

  • Tab key for auto-completion
  • Up (↑) / Down (↓) for navigating through command history
  • Ctrl-A and Ctrl-E to jump to the beginning and end of the input line, respectively
  • Ctrl-U to erase the input line
  • Ctrl-W to erase one word before cursor
  • Ctrl-L to clear the screen

• You can run (almost) any regular python scripts from within StaSh

  • There is no need to customize them for the shell. If it can be executed by a python interpreter via python your_script.py, you can just call it from within StaSh by just typing your_script
  • The shell object is made available to scripts being called. This enables a range of complex interactions between the shell and called scripts. For an example, the running script can use the shell object to execute more commands, e.g. _stash('pwd').

• You can give it a resource file, similar to .bashrc, to customize its behaviour. Like the Bash resource file, aliases, environment variables can be set here. The default resource file is .stashrc under StaSh installation root (i.e. ~/Documents/site-packages/stash).

  • The prompt is customizable with the PROMPT environment variable.
    • \w - current working directory with HOME folder abbreviated as ~
    • \W - last path component of current working directory
    • All other strings are displayed literally
    • The default setting is PROMPT='[\W]$ '

• Easy self update to keep update with the development by running a single selfupdate command from within the shell.

• The UI can be configured via configuration file to customize its font size and color. The default config file is .stash_config or stash.cfg under StaSh installation root.

Usage

The usage of StaSh is in principle similar to Bash. A few things to note are:

• The search paths for executable scripts is set via an environment variable called BIN_PATH as PATH is used by the system. The default BIN_PATH is ~/Documents/bin:$STASH_ROOT/bin.

• The executable files are either Python scripts or StaSh scripts. The type of script is determined by looking at the file extensions ".py" and ".sh". A file without extension is considered as a shell script.

  • When invoking a script, you can omit the extension, StaSh will try find the file with one of the extensions. For an example, StaSh interprets the command selfupdate and find the file selfupdate.py to execute.
  • Files without extension won't show up as an auto-completion possibility.

• Commands can only be written in a single line. No line continuation is possible. However, multiple commands can be written in a single line by separating them with semicolons, e.g. ls -1 > file_list; cat file_list.

• There are many Python scripts provided along with StaSh. These scripts range from performing regular shell tasks to advanced utilities like ssh and git. Note the scripts are by no means complete when compared to a real Linux shell. The collection will be gradually expanded should the need arise. It is also expected and appreciated that the community would come up with more scripts.

  • alias.py - Define or print aliases
  • cat.py - Print contents of file
  • cd.py - Change current directory
  • clear.py - Clear console
  • cp.py - Copy file
  • crypt.py - File encryption using AES in CBC mode
  • curl.py - Transfer from an URL
  • cut.py - Cut out selection portions of each line of a file
  • du.py - Summarize disk usage of the set of FILEs, recursively for directories
  • echo.py - Output text to console
  • edit.py - Open any text type files in Pythonista editor
  • find.py - Powerful file searching tool
  • fg.py - Bring a background job to foreground
  • `gci.py - Interface to Python's built-in garbage collector
  • git.py - Git client ported from shellista
  • grep.py - search contents of file(s)
  • httpserver.py - A simple HTTP server with upload function (ripped from https://gist.github.com/UniIsland/3346170)
  • jobs.py - List all jobs that are currently running
  • kill.py - Terminate a running job
  • ls.py - List files
  • mail.py - Send emails with optional file attachment
  • man.py - Show help message (docstring) of a given command
  • mc.py - Easily work with multiple filesystems (e.g. local and FTP) synchronously.
  • md5sum.py - Print or check MD5 checksums
  • mkdir.py - Create directory
  • mv.py - Move file
  • openin.py - Show the open in dialog to open a file in external apps.
  • pbcopy.py - Copy to iOS clipboard
  • pbpaste.py - Paste from iOS clipboard
  • pip.py - Search, download, install, update and uninstall pure Python packages from PyPI.
  • printenv.py - List environment variables
  • printhex.py - Print hexadecimal dump of the given file
  • pwd.py - Print current directory
  • python.py - Run python scripts or modules
  • quicklook.py - iOS quick look for files of known types
  • rm.py - delete (remove) file
  • scp.py - Copy files from/to remote servers.
  • selfupdate.py - Update StaSh from its GitHub repo
  • sha1sum.py - Print of check SHA1 checksums
  • sha256sum.py - Print of check SHA256 checksums
  • sort.py - Sort a list, also see unique
  • source.py - Evaluate a script in the current environment
  • ssh.py - SSH client to either execute a command or spawn an interactive session on remote servers. pyte is used for terminal emulation and gives the command the feel of a full-fledged SSH client.
  • ssh-keygen.py - Generate RSA/DSA SSH Keys.
  • stashconf.py - Change StaSh configuration on the fly
  • tar.py - Manipulate archive files
  • touch.py - Update timestamp of the given file or create it if not exist
  • uniq.py - Remove duplicates from list, also see sort
  • unzip.py - Unzip file, also see zip
  • version.py - Show StaSh installation and version information
  • wc.py - Line, word, character counting
  • wget.py - get data from the net
  • which.py - Find the exact path to a command script
  • wol.py- Wake on LAN using MAC address for launching a sleeping system
  • xargs.py - Command constructing and executing utility
  • zip.py - Zip file, also see unzip

Acknowledgements

• Pythonista is a wonderful piece of software.
• StaSh is inspired by
  shellista and its variants, including ShellistaExt and ShellistaUI.
• The UI part of StaSh has its root from ShellistaUI.
• Many of the command scripts, e.g. ls.py, cp.py, mv.py, are taken from ShellistaExt with some modifications.

Known Issues

• Pickled objects are not restored correctly and generate AttributeError as if the class definition cannot be found.

Contributing

• Check any open issues or open a new issue to start discussions about your ideas of features and/or bugs
• Fork the repository, make changes, and send pull requests
  • Please send pull requests to the dev branch instead of master

[dgelessus]

Well this is awkward, I've been working on basically the same thing, except with no UI and less functionality... Nicely done though, I have yet to play around with it some more, but having working pipes and some of the more advanced shell features sounds useful. And it looks like you figured out how to reliably scroll down the console output, the method I used in ShellistaUI always started to have problems after a hundred lines or so.

Regarding executable files, have you thought of using shebangs?

#!/bin/sh
# ^ that is a shebang, it tells the shell which executable to use to run this file.
# Normally the file must have the executable bit set for the shell to look at the shebang.

(I'll see about making a pull request so I can merge some of my code into your repo - your core shell is definitely better than mine, and since mine also loads commands from Python scripts the systems should be compatible.)

[zencuke]

Woah! Thanks.

[wradcliffe]

This is very nice. Thanks so much for contributing this.

You mention in your README that scripts can interact with the shell. How does that work? Sounds like it is intended to include scripts running a UI. I notice that when I open the history UI and then type a new command, the UI does not update. Could it? Is clicking on an item in the history list and example of script communication back to the shell?

[ywangd]

@dgelessus

My first attempt was trying to work from shellista. Then I realized it is impossible to achieve what I want without re-writing the whole thing. ShellistaUI points the right direction by employs the UI system. But it still depends on shellista and has many the same limitations. So I decided to start over as a new project.
Please be my guest to add your code to it. Sorry if it means double effort for you.

The code was compiled in past 3 weeks. It is not clean or elegant. There sure is much room for improvements. In fact I wasn't entirely happy with the pyparsing bit and may have some major overhaul when I have the time.

I thought about shebang as well. I didn't implement it mainly because I am too lazy to do it. The parser in its current state does not parse comments, i.e. #. Also I am not sure how the performance would like if the Shell needs to read the first line of every file to tell the type. In Pythonista's controlled environment, I decided to take the lazy way and rely on file extensions. But, in fact, a file without extension will still run as StaSh script, e.g script will run if a file called script is found. It will just not show up in the auto-completion possibilities when your type scri and press Tab.

On a side note, I feel my development workflow is somewhat awkward. I effectively use GitHub as a staging area. Once I made changes on my computer, I push to GitHub and retrieve them using selfupdate on my phone. Please let me know if you have a better way to streamline the whole workflow.

[ywangd]

@wradcliffe

I have to admit I did most of the development on a Phone and haven't thought about updating the history UI in real-time while typing commands. It is definitely a nice feature and I'll work on it when time permits.

However, it is worth to note that the history UI is NOT an external script. It is an inherent part of the Shell. Yes it runs in its own thread. So it does communicate back to the Shell. But it is not exactly the interaction between Shell and an external script that mentioned in the README.

The interaction between shell and running external script are mainly for two type of things:

1. Allow a running script access shell's status and act accordingly.

I also thought about (not implemented) a callback system that the running script can register to. For an example, a running script can register for being called when the shell's input textfield returns or a button tapped. One use of this feature is to allow running thread being terminated from the shell gracefully. But it does require the script to be written specifically to utilize this feature. For regular script, StaSh relies on Thread._Thread__stop() to terminate the running thread. It often works but not guaranteed.

2. Allow a running script to ask the shell to run another shell command

StaSh does not have a full set of shell language. It does not have keywords and program control constructs, such as if, for. Hence a StaSh script can only do limited things (basically just a list of commands to be executed from the top to bottom. But with this feature, it is possible to write a Python script that acts like a real Shell script. Here is a contrived example:

_stash = globals()['_stash']  # get the shell object

# Commands to get a sequence of files from an url
cmd_base = r'''wget http://www.test.com/file_%d.txt -o saved_%d.txt'''
for i in range(10):
    cmd = cmd_base % (i, i)
    worker = _stash.runtime.run(cmd, add_to_history=False, reset_inp=False)
    if worker.isAlive():  
        # wait for the thread to finish as StaSh does not allow parallel threads
        pass

I confess it is rather a convoluted way to interact with the Shell. But it is available if you really wants it.

[jmv38]

@ywangd this looks fantastic! My question may sound totally stupid to you, but i have no idea of what a 'shell' is, so could you put explainations (or a link to some) about the global context of your addon? You are giving a lot of explainations on the details, but not the global picture. Thanks!

[JonB]

Few comments

1. awesome work!

2. the slight flashing due to the multiple scrolls can be eliminated by using an ui.animation(). This probably deserves some more testing. I think the animate must force the drawing queue to finish before calling the animate function. Thus content_size gets properly refreshed the first time, without having to keep trying to scroll.

    def _scroll_to_end(self):
        # small animation to force proper loading of content_size
        def ani():
            self.out.content_offset = (0, self.out.content_size[1] - self.out.height)
        ui.animate(ani,0.01)

EDIT. Never mind, while this works for some cases, an empty line seems to not scroll to the end using this method. Oh well.

3. maybe consider adding some additional useful keys to virtual keyboard? On ios I find it particularly annoying to get to _*|~>, basically anything on the symbol page.

4. the gist install script didn't install all of the scripts. I had to run the selfupdate script to get things like ls, etc. also, the gist installed to Documents/stash/bin, but the BIN_PATH pointed to /Documents/bin.

[ywangd]

@JonB

• I just had some test using ui.animate. As you found out, it does not in all cases. I really like the effect that it scrolls really smoothly. Pity it still is not perfect.

• Adding more buttons for symbols is a good idea. I am still clumsy with the ui system. But this addition should be really handy for the shell.

• The gist script was intended to just install the minimal set of files so that selfupdate can run. This is because the gist script use a very primitive approach to issue an URL request for every single file (so it does not rely on unzip). So I decided to have a two stage installation. This is mentioned in my original post. Maybe I didn't emphasize it enough and it could cause confusions.

• The BIN_PATH is set to ~/Documents/stash/bin:~/Documents/bin if you type echo $BIN_PATH. However, due to a bug, the path ~/Documents/bin actually does Not work. So BIN_PATH is just ~/Documents/stash/bin in its fully expanded form. I have fixed the bug along with some other fixes. You can selfupdate to get the latest changes.

Thanks for the comments.

[ywangd]

@jmv38

A shell is a command line interpreter that sits between the end-user and operating system. It is a traditional text based user interface.

Here is the wiki page about the Unix Shell. Also a wiki page about Bash, often called the Shell as it is so influential (for people like C Shell, please, I am not trying to start a religious war here).

Graphic interfaces are so popular and dominant in current world. But a shell is still a very valuable tool, especially at performing administrative works. Basically if a program often needs to be called with very different arguments and it can benefit when used with other programs, a shell comes in handy.

Please note that I am talking about real Shells, e.g. Bash. I am not claiming that StaSh is a real shell. It just models after Bash and provide some opportunities for like-minded people. Personally I wanted a shell because I often need to run some CLI (command line interface) python tools, which could be cumbersome to run using action menus (because of constantly changing arguments).

[briarfox]

@ywangd Very very nice. Thank you. I really like the console look. Would you mind If I borrowed some of the code? I am working on a console that connects to a Raspberry Pi via bluetooth.

Are you planning to add ssh access?

[briarfox]

I've had a chance to play with it some. I'm really liking the layout. I do like how easy it is to add Python scripts. IT would be great to see a git , ssh client/server, Python interactive interpreter. I've added a small script that allows running of Python modules and scripts(main use is to call SimpleHTTPServer, Pylint, nose.

I haven't looked through the code to deeply but what would be the best way to take over the UI input for a Python script such as an ssh client.

With a few additions I will be using this instead of shellista. Great Job.

[jmv38]

@ywangd thanks a lot for your explanations.

[ywangd]

@briarfox

Thanks for your comments. Please feel free to use any code from StaSh. The UI layout has its root from ShellistaUI. So credits also goes to ShellistaUI (I believe @dgelessus is the original author of ShellistaUI). I also borrowed code from your ShellistaExt for various commands such as ls, cp etc. It would be even more appreciated if you could share your changes or even send pull requests.

• StaSh is more of a framework that manages calls to python scripts. So I am hoping that the community would provide the scripts. In principle, if a script can be executed on a PC, it would also be usable from within StaSh. I/O should be taken care of automatically in most cases.

• For an example, your ssh code (https://github.com/briarfox/ShellistaExt/blob/master/ShellistaExt/plugins/extensions/ssh/ssh.py) is pretty immediately usable in StaSh. Just place it into the BIN_PATH, e.g. ~/Documents/stash/bin and just type ssh in Stash. You'll see the command gets executed. If you set proper host/user/pass in __main__, it would connect to the host and all I/O is handled by StaSh. Apparently, for the script to be more useful, you'll need some arguments handling in __main__, a primitive approach could be as follows (add to the end of ssh.py):

if __name__ == '__main__':
    if len(sys.argv) == 1:
        print 'Usage: connect hostname user password'
        sys.exit(0)
    cmd = sys.argv[1]
    params = ' '.join(sys.argv[2:])
    if cmd == 'connect':
        ssh = SSH()
        ssh.connect(params)
    else:
        print 'NYI'

• Pythonista builtin interactive prompt is available even when StaSh is running. Just tap on the first button of the segmentation control on the top-left side. Sure you can add your own interactive interpreter, but the builtin one just has more features such as code completion etc.

• I haven't thought about running Python modules. I would like to see SimpleHTTPServer gets run in StaSh. Please let me know your changes if it is OK with you.

• I haven't had my hands on a Raspberry Pi yet. But I am a Lego Mindstorms fan. I'd love to know how your code works and see if it is adaptable to Mindstorms.

[briarfox]

@ywangd I do like the drag and drop with stash. I'm still working on the running of modules. Code works great until I drop it in StaSh. I keep getting a package set to non string Error. I need to track down the cause. I'm hoping to run it on its own thread so the command:
<pre>
python -m SimpleHTTPServer &
</pre>
Will run it as a thread and allow you to load a client to work with it. It's a simple script that uses the module runpy and passes the args. I'll submit a pull request when I get the bugs out.

edit Fixed

[briarfox]

@ywangd Would you mind checking out this script? It works when ran my itself. But ShaSh when trying:
<pre>
python -m SimpleHTTPServer &
</pre>
It throws a "package set to non string"
python.py

As for the question on the Rpi and BTLE, I have a bluetooth adapter on the Rx and Tx pins of the RPi. I'm just reading and sending data with the pythonista cb beta module. I'll post it up for you.

[ywangd]

@briarfox

This error is tricky. It is due to the module_name variable being unicode type instead of an plain ASCII str. So once you add

module_name = str(module_name)

into the code, everything works.

This error is probably due to the fact that Python 2.x and its libraries are not fully unicode compliant. In fact, if you type runpy.run_module(u'SimpleHTTPServer', run_name='__main__') into a PC version python interpreter, it reports exactly the same error.

When the script runs by itself, sys.argv is set to plain strings. But it is set to the same strings of unicode version when executed by StaSh. I am not entirely clear about the reason. Anyway, the fix is easy. I'll also see if it is worthwhile to always convert all arguments to str before calling an external script.

[dgelessus]

@ywangd, the ui module uses unicode strings exclusively. Since the command input field is just a regular text field, it also returns its contents as unicode, and since there's no reason for Python to coerce down to str it doesn't. Also instead of a plain str() conversion it would be better to use the encode() method of unicode objects, that way the string is guaranteed to be properly encoded as UTF-8.

[ywangd]

@dgelessus Thanks for the explaination. That makes sense.

I am still thinking about whether it is really necessary to always encode unicode arguments to str before executing a command.

[dgelessus]

@ywangd, it isn't really necessary, because you'll almost never need to pass non-ASCII characters to a command. But when you do try, you'll very likely run into problems. I'm not sure what exactly happens when you do str(u"üñîçø∂ε"). encode is also guaranteed to work in future Python versions. Python 3 (if/when Pythonista supports it) is much more strict when it comes to byte data and unicode text.

I checked, and Python 2.7 on my Raspberry Pi encodes runtime args as UTF-8 str objects. It is possible that the encoding is OS-dependent, but UTF-8 is usually a safe choice. Python 3 of course uses proper Unicode str objects instead.

PS:

You're lucky, str encodes unicode strings as UTF-8 anyway. encode still is safer ;)