Using Process Pools, Pythonista beta - should this code work?
I just come across this interesting Article today that shows how to use Process Pools for making thumb nails from JPEG images using a seperate process for each core you have. So at least in this example getting a big reduction in execution speed.
Code seems ok to run, but gets an exception no such file or directory. I have some .jpeg files in the local dir, but this is not the problem.
Oops, as I was writing this I thought I should look it up in the manual. There is a simpler example in the manual. It also fails with the same Execption as below.
It appears this should work in Pythonista, but I am guessing about that :) so not sure if this a bug/file omission or pc only etc. I can't see anything that says pic only. Look it's not important for me, just seems like a nice thing to have if you want to do a lot of processing of something that fits into this model.
Traceback (most recent call last):
File "/private/var/mobile/Containers/Shared/AppGroup/A83A4237-28DA-41B4-AA91-84BC6770F30C/Pythonista3/Documents/MyProjects/process/process2.py", line 28, in <module>
File "/private/var/mobile/Containers/Shared/AppGroup/A83A4237-28DA-41B4-AA91-84BC6770F30C/Pythonista3/Documents/MyProjects/process/process2.py", line 23, in main
with concurrent.futures.ProcessPoolExecutor() as executor:
File "/var/containers/Bundle/Application/EA1EEECB-E4B2-4C5F-ACE0-B7287FEACF04/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/concurrent/futures/process.py", line 376, in init
File "/var/containers/Bundle/Application/EA1EEECB-E4B2-4C5F-ACE0-B7287FEACF04/Pythonista3.app/Frameworks/Py3Kit.framework/pylib/concurrent/futures/process.py", line 344, in _check_system_limits
nsems_max = os.sysconf("SC_SEM_NSEMS_MAX")
FileNotFoundError: [Errno 2] No such file or directory
Code from Python Manual 3.6
import concurrent.futures import math PRIMES = [ 112272535095293, 112582705942171, 112272535095293, 115280095190773, 115797848077099, 1099726899285419] def is_prime(n): if n % 2 == 0: return False sqrt_n = int(math.floor(math.sqrt(n))) for i in range(3, sqrt_n + 1, 2): if n % i == 0: return False return True def main(): with concurrent.futures.ProcessPoolExecutor() as executor: for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): print('%d is prime: %s' % (number, prime)) if __name__ == '__main__': main()
Pythonista (like all other third-party iOS apps) cannot create subprocesses, so
ProcessPoolExecutorcan't work. You could probably use
concurrent.futures.ThreadPoolExecutorwith pretty much the same API, but due to the GIL (global interpreter lock), the performance improvements would probably be neglibible, unless your task is IO-bound (e.g. downloading multiple files in parallel should work fairly well with a
ThreadPoolExecutor, though I haven't tried this yet in Pythonista).
@ccc , ok. Thanks for the link. It seems like in this case if one core was reported back then the code possibly could at least work, although not with the desired result. It would stop a hard fail/ execption. Maybe the most Pythonetic thing to do is throw an exception rather than to work , but then maybe the exception should be a not implemented exeception or more descriptive exception
Did you try ThreadPoolExecutor?
@ccc , no I didn't. Was not the point of the post really. As I said I don't have a need for it. I just wanted to understand why it wasn't working. But with your and omz posts, it's clear why it does not work. I just went on to make the further point that it could have failed a better way with a different expection error. Or it could have possibly run but issued a warning like 'sub processes not supported, running a single process'. Or something like that