Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Critique my async
-
While this is not Pythonista specific, I use it with Pythonista, and would be happy to get some feedback on it. Please find the file on Github if you want to play with it.
Note that while this is working code that I use in my (simple) projects, it is work in progress, and is mainly intended to act as a concrete expression of my need for simpler async coding.
Usage
yield
as a synchronization pointFunctions and generators with
genr
decorator are executed in a new thread, with a special handling of theyield
s. For example, in the following:@genr def example1(): task1() task2() yield task3()
Tasks 1 and 2, also decorated with
genr
, are executed in parallel.yield
waits until they are both complete, after which task 3 gets to run.End of the function also acts as a collection point, i.e. we wait for all called functions to complete before returning. Thus, in the example above, there is little practical difference whether
task3
is executed in a separate thread or not.yield
to get resultsDecorated functions return a
concurrent.futures
Future
, which can be used in the usual way, but for more readable code, we use a Tornado-inspired way ofyield
ing the result. The classical example below demonstrates retrieving several web pages in parallel:@genr def example2(): return yield [ fetch(url) for url in ( 'https://python.org', 'https://pypi.org' )] @genr def fetch(url): return requests.get(url).text
Ad hoc threads
Being a decorator,
genr
can be used to start parallel execution of individual functions inline, with the following syntax:genr(func_name)(param1, param2, ...)
While this avoids having to create named one-line functions just for the sake of parallel execution, the syntax is not especially readable, and quickly becomes untenable if you need to add timeouts, exception management etc.
Interrupting execution
Aligned with
concurrent.futures
,genr
plays nice with beingKeyboardInterrupt
ed. To wait for agenr
thread to complete in the main thread, you can simply wait for the result:main_function().result()
Discussion
Pros
- Minimum cognitive load, no need to even understand the concept of a future.
- Move quickly from need to implementation without needing to understand asyncio.
- For relatively easy cases, functional need for parallel execution translates into a clear story in a single main function.
- One decorator for the different use cases:
- Launching a single parallel function
- Multiple parallel threads
- Collecting results
Cons
- Likely to suffer from the same challenge that drove the creation of asyncio: it is not very visible whether a function is being called in a separate thread or not, which can lead to hard-to-see bugs.
- Overloads generator semantics in a way that is not Pythonic.
-
One thing that I am currently trying to understand is how exactly is this different from the deprecated
asyncio.coroutine
, and why was it deprecated in the first place. -
Hmm, so, no interest in this topic. Would be very interesting to hear whether it is because:
a) you find existing thread or asyncio support straightforward to use in a way that fills your needs,
b) you do not really need async in your projects,
c) you deliberately avoid async because it always leads to problems, or
d) other, please specify.
-
@mikael I've never used asyncio, I really don't know what it is, thus I'm not able to criticize anything. I often use threading, is that far from asyncio?
-
@mikael Thanks for this, you awoke my interest and I started reading up on the topic. But I cannot (yet) contribute to the discussion.
-
To not die silly, I read this, interesting
-
I like the intuitive and simplistic API approach as always when mikael comes out with a new thing ;-) Next time I have a need for parallel execution in my code I will definitly come back to genr ...