AntPool - Python Minimalist multi-computer distributed computing framework
wolf71 last edited by wolf71
- AntPool provides a server and resource client that makes it easy and fast to set up a multi-computer distributed environment.
- AntPool provides a development library consistent with Python's native concurrent.futures.Executor asynchronous library, allowing you to quickly deploy your code to run in the multi-computer distributed environment.
- In some application scenarios, we want programs to run on multiple machines in parallel to gain performance, network access improvements;
- Computational Performance Boost: For complex computation and analysis in Python, multi-threaded Python can't boost on multi-core CPUs(limit by GIL), so Python introduced the concurrent.futures.ProcessPoolExecutor, Multi-process development is easy to implement; but when higher performance is needed, a multi-computer distributed environment quickly allows you to change just a few lines of code and take performance to new heights.
- Network Capability Improvement: Crawling data on the Internet, analyzing and extracting it, is often limited by the traffic of the crawled server. Therefore, it is necessary to deploy the program to multiple machines with different network addresses for crawling, which is tedious and error-prone to do manually, but a multi-machine distributed environment can quickly solve such problems.
- Everyone Can Using Instead of requiring complex configurations, specialized equipment, and relearning how to develop, AntPool is available to everyone.
- Any single device can be used as a server or resource client or both. These include PC/Mac/Linux computers, and even devices like your iPhone/iPad/Android that can run Python.
- You can use all your devices at home to help you improve your computing performance, or you can ask your colleagues at work to contribute some of their CPU cores to help you achieve complex computations.
- All computing resources are connected to a server-driven cluster network by running a resource client, which can be joined and quit flexibly.
- No need to learn new concepts, as you only need to understand Python's native concurrent.futures.Executor, as the AntPool classes are same.
- Mobile Device Support iPad is a great tool, and there are already many great apps that run Python and iPython Notebook, but due to iOS security restrictions, many libraries can't be installed, Python also don't support multi-processing. With AntPool, you can call a multi-machine cluster directly on the iPad and run features on the cluster that are not available on the iPad or get more performance.
- iPad Python Apps
- Pythonista(requires stash , for pip support)
- iPad iPython Notebook/Jupyter Apps
- iPad Python Apps
- Runtime requirements:
- Python 3.5 or above
- Quick Start:
- pip3 install antpool
- for Pythonista user, please pip install tornado sorry for a bug, need Manual install tornado.
- Standalone deployment server and resource client
- Run in a terminal environment: antpoolsrv /v
- Open a new terminal and run: antpoolclient ws://127.0.0.1:8086
- If you need to access more resource clients to improve cluster performance, you can do so on other machines:
- pip install antpool
- antpoolclient ws://server IP:8086
- Each antpoolclient client running on the same machine will use one CPU core, so if you want to exploit the performance of the whole machine, you can run multiple clients (e.g. for a 4-core CPU machine, you can run 4 clients)
- Run the demo program
- Structure diagram
!!! Security Tip !!!
- The resource client is execution of the submitted modules, so the device running the resource client will be vulnerable to attacks by users with bad intentions, the system does not perform security checks on the executed modules, so it is recommended that the user running the resource client needs to verify that the user is trustworthy;
- For the sake of simplicity, the entire system currently does not use user/password authentication mechanisms (interfaces are reserved in the code); therefore, it is recommended that the server be deployed on the public network with security in mind;
check it on GitHub
simplexity last edited by
Thank you for this very useful post, coming from some one who is trying to downside enterprise scale heavy weight distributed architectures such as CORBA to iPad these days ;-)
Have you heard of the new and exciting Ray Framework - https://www.ray.io/ ?
I wonder if you or anyone else you know have tried to install it on an ipad as well. That would be so awesome!
ccc last edited by ccc
Pythonista is great a pure-Python content but the other languages are a problem.
simplexity last edited by
Oh, do you mean to say it will be near impossible to port Ray over to Pythonista?