Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Pipeline Of Stages In Pythonista
-
I would like to allow the user of a progam I'm developing to create a pipeline of stages, probably separated by the | character (or a replacement). They would pass this text string to my program.
For example:
fred.txt | takeLines 5 | takeColumns 3 | makeHTML | fred.html
These are not the real stages and I probably haven't got the Unix file handling syntax quite right but bear with me, please.
Has anyone done this? And do you know of any modules that make it easy?
The pipeline is probably going to be invoked by:
-
Command line parameters when running (from the console?)
-
As part of an iOS extension.
I'm not sure if there are any gotchas or if I can get any help with parsing the stages.
I also can tweak the syntax. If I can't use | within the input string I could choose something else; I don't mind much.
One thing I'm sure of is I don't want to have the pipeline invoke multiple Pythonista programs. The data is too cumbersome to keep parsing and externalising at each stage. Or, put another way, it would introduce unnecessary inefficiency.
Thoughts?
-
-
There is nothing magic about putting all your stages on a single line with a pipe in between and a return being the end of the pipe
vs. putting each stage on a line by itself with a blank line or end of file being the signal of the end of a pipe.The first starts with
commands = user_text.split('|')
and the second starts withcommands = user_text.splitlines()
.It looks like a nice pattern would be for the first word of each stage/command to be the action verb and the later words are the modifiers.
I might rewrite it adding the
read
andwrite
verbs as:read fred.txt | takeLines 5 | takeColumns 3 | makeHTML | write fred.html [ or ] read fred.txt takeLines 5 takeColumns 3 makeHTML write fred.html
There is an implication of a single, unnamed variable (I will call it
state
) that carries the state of the workflow from stage to stage. Thusread
puts data into that variable and andwrite
gets data out of that variable. The twotake
verbs reduce the size of the data in the variable via selection.makeHTML
transforms the data in the variable from one format to another.I would start by writing 5 functions: read, takeLines, takeColumns, makeHTML, and write. Once you have those five done, the rest will become much simpler.
def execute_command(command, state): # [ ... put in logic to call the appropriate command and put the execution result into `state` ... ] return state def main(commands): state = None for command in commands: state = execute_command(command, state) if state: print(state) return state
-
I would use a dictionary to call your functions. Do what @ccc said and under
def execute_command(command, state): d = {'someCommand':someCommand, 'command2':command2} for key in d.keys(): if command in key: func = d[key] tokens = key.split() tokens.remove(tokens[0]) #get rid of function name func(state, *tokens)
-
stash supports piping of commands. in that case the stdout from one command is fed into the stdin of the next.
-
May be you can look at the following python recipe.
http://code.activestate.com/recipes/276960-shell-like-data-processing/It needs to be changed slightly to run it on python 3.
-
Thanks to everybody for your responses so far.
I like the dictionary idea and @ccc had a particularly good thought in terms of stages being in a file, perhaps new line separated.
@ccc I'll be in touch with you during the week as you emailed me. For those who don't know, we both work for IBM - which has a positive stance on Open Source but some governance around it.
-
@MartinPacker I have been working on a script that does something similar with a UI where you have functions and data is passed from one to the other, I had to keep track of input and output types to make sure the function the data is passed to can handle it. Anyway if you would like to take a look it is available here this might be totally overkill to what you want but you can have a look how I pass data etc