Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Will pandas and scipy be available on Pythonista in the near future ??
-
@dgelessus and @JonB
Hi, thanks guys for reply, I would like to clarify that I don’t want to import in Pythonista any precompiled not-pure python libraries (it is impossible for now), but to import them in a computer (named for example “X”) with a working python environment, and I’m interested in doing that with the remote computer provided by SageCellServer, that is a virtual machine with Linux OS; if user can create his/her own server with a home computer linked to Internet, it is possible to use it instead of the SageMathCell one, solving the limits related to impossibility to install own libraries.
Sorry but I try to explain what I have in mind with other words as below.
I’d like to use Pythonista with my idevice:
-
to send some scripts (like the library “httpimport” by John Torakis, but it works only for pure-python libraries without any dll, lib, etc… files), to the remote server (computer “X”) (these scripts are the main reason of my question: Do they exist? Has someone created them, what do you know?);
-
after the remote server has received these scripts, it should run them in order to import only in the RAM (not on hard disk) of computer “X” a full not-pure python library properly compiled with other computer with suitable compilers, same OS, etc… of the remote one (“X”); let’s suppose this library I want to import only in RAM of computer “X” is stored in a folder of my dropbox and I have its shared link.
-
after the importing, I’d like to use the remote python interpreter of SageMathCell (or other own available remote computer) using Pythonista like a tool that sends python code to server and receives output from that server.
I know that a OS (windows, linux, mac, other…) needs to have an hard disk in order to work (is it true for all? I’m not sure). I think that nowadays no tools exist that allow user to decide if to install any external software, python library, fully portable executable program, full IDE with any kind of compiler
-
in the hard disk, or
-
directly in RAM, as if a portion of RAM, decided by OS, behaved the same way as a hard disk (so all files are stored in ram with the same folder hierarchy of hard disk).
As example, let’s suppose I have in my pc (named “Z”) a python environment installed in the hard disk in the path “C:\Python27\” of windows OS (as example, but I ask if it is possible also for linux, mac, other…).
Now suppose that
- I want to import a python library in my pc (without store it in hard disk with “pip install”). With httpimport library by Torakis it is possible to import temporarily only in ram a pure-python library, but, unfortunately, suppose I want to import in RAM a not-pure python library, like scipy, so I can’t use httpimport lib.
- Suppose also that the library I want to import only in RAM needs some existing libraries in my OS, that are stored in some folders of hard disk.
Well, can I:
a) compile the not-pure library with a suitable compiler following the common way, that is by installing the compiler in hard disk of a pc and using it to compile the python library,
b) copy the compiled python library, named “Y” and stored in a cloud service accessible via Internet connection, only in a portion of my pc’s ram (as if it was in “C:\Python27\Lib\site-packages\my-compiled-python-lib”) with a python command/function like the one I use to download something , that is the "urllib" python lib,
c) tell to python interpreter of pc “Z” to import the library “Y” in order to use it?The library “Y”, that needs some external libraries in my OS (see point 2. above), in my idea, would search for the external libraries stored in pc "Z" hard disk even if it is only in RAM.
So, is it possible?Thank you guys
Regards -
-
@JonB I don't know how
ctypes
creates callback functions in Pythonista.ctypes
internally useslibffi
, for which you can find the source at https://github.com/libffi/libffi. However I don't know if @omz had to modifylibffi
to make it work on iOS, so the official source code may be inaccurate.What I did find however are the internals of the Objective-C runtime's
imp_implementationWithBlock
, which effectively lets you create anIMP
that calls the given block. Seeobjc-block-trampolines.mm
anda1a2-blocktramps-arm64.s
in https://opensource.apple.com/source/objc4/objc4-723/runtime/. The way it works is that the assembly code defines a page full of identical trampolines. The trampoline code invokes a block located exactly one page before the entry point of the trampoline. That way all trampolines use the same code, but end up invoking different blocks.Now to make use of these trampolines,
imp_implementationWithBlock
allocates two pages of read-write memory. Using the Machvm_remap
function, the page of trampolines from the assembly code is mapped into the second newly allocated page, which also changes the permissions of that page to read-execute. The first page is still read-write and is used to store the blocks, which can then be called by the corresponding trampoline in the next page.None of this requires an executable stack or any other sort of write-execute memory, since the executable code is loaded unmodified from the library, and then mapped into memory multiple times without modification. I don't think iOS allows write-execute memory, that would make it very easy to circumvent the code signing restrictions.
-
@Matteo Importing a native Python module from RAM is simply not possible if the OS doesn't allow loading a native library from RAM, and I think Linux doesn't. As far as I know, on Unix systems Python uses the standard
dlopen
function to load native modules, and that requires the library to be in a file.I don't know what the SageCellServer Linux environment does and doesn't allow, but if possible you could make a temp directory, download the libraries you want into it, and add it to
sys.path
. Then any native modules in there will be found by the normalimport
mechanism. As long as the libraries are compiled correctly for the system used by SageCellServer, they should import without problems. -
This post is deleted! -
This post is deleted! -
@dgelessus Hi, thank you for suggestion, unfortunately it seems to me that it is not possible to create a folder with remote server by SageMathCell.
I had a simpler idea thanks to your reply: can python (any version) add to
sys.path
also remote folders (for example any folder stored in a dropbox or github account)?Example: I want to add to
sys.path
of my local python environment (for example Pythonista) a folder like this one:https://github.com/jsbain/objc_hacks
without downloading it neither storing it, with all scripts, in local storage memory of Pythonista.
Can I do it?
Thanks
Regards -
Is the issue that you don't have control over your server? I thought you were trying a run a sagecell server on your personal pc. if so, why not just install the packages you need. Or is the issue that sagecell is "locked down" even running on your own pc?
You cannot change sys.path to point to a non-path. on a pc, you could install a drobox client, which automatically syncs a folder, but i think that doesnt satisfy your goals. You could download files using drobox client, import them, then delete them. but if your issue is with sagecell limitations or security, that pribably won't work either.
-
@Matteo Linux can only load native libraries from the regular filesystem. That doesn't necessarily mean "on a real physical hard disk", it could also be on a network share or RAM disk mounted somewhere in the file system. However if the system doesn't even let you create temp files, it probably won't let you mount remote filesystems either.
On a regular Linux system that you have full access to, that wouldn't be a problem though. You can mount a network share under
/mnt/myshare
or whatever, add/mnt/myshare/pythonlibs
tosys.path
, and then you can import any module in that folder. I don't know Linux very well, so I don't know what commands exactly are used to mount remote filesystems. I also don't know if there are ways to mount a Dropbox or a Git repo as a filesystem. -
Hi @JonB , I have not my own sagecell provided by my online pc, because of lack of ram and cpu resources of my pc.
I use only the main SageMathCell mainteined by Andrey. But it (seems to me) doesn't allow user to install any external own python library: user can use only built-in libraries like these ones.
So I ask here if we can by-pass this limit by using a set of python scripts or a python library to import in ram (no hard disk) a remote python library compiled in linux stored in any cloud service, in order to use it not with Pythonista but with the remote python interpreter provided by SageCellMath (Pythonista would be a "remote controler" for the SageMathCell).
Hi @dgelessus, ok, I don't know Linux very much, I've tried some times ago to create a virtual machine with linux on my windows pc by using Virtual Box, but it was too slow.
After this long discussion (thanks both dgelessus and JonB!) I'd like if, in a near future, python could allow user to import any working python library (pure and not-pure) stored only in web (cloud or https), without downloading and saving it in the local filesystem. The web would become in this way a read-only big memory resource for own local python env and if user wants to change/modify the imported library, then he/she could do it by downloading the library, changing it and finally reloading the modified library in his/her own cloud account in order to use it with own local python env.
About sys.path python function, now I know that it can be used only for paths in filesystem (hard disk) and not for paths in the cloud (in a read-only mode).
Thanks both again for explanation,
Regards -
If you want a full shell, you likely need to pay, such as with Amazon aws, or any number of of full shell accounts. Anders is hosting and paying for the service he provides free, hence the limits on size,etc.
From what I can tell,though, Sagecell is running in it's own VM, complete with access to ’subprocess’ and urllib, you may have be able to do what you want,within the limits of that 64k which might be the limiter.
You might checkout numpy.f2py, which supports interfacing with Fortran libraries.
-
Actually, check out cocalc, which is just what you need... You get private projects with persistent storage, ability to compile code,etc.
-
@Matteo
Actually, check out cocalc, which is just what you need... You get private projects with persistent storage, ability to compile code,etc. -
@JonB Hi yes, if I will be able to create my own cocalc account and a way to interact with it using Pythonista or any online python environment it would be very useful.
With a cocalc account user could install any library supported by anaconda (I suppose).For now I will try to create a temporary folder with SageMath, starting from the following piece of code
import os import tempfile directory_name = tempfile.mkdtemp() print(directory_name) sys.path.append(directory_name)
in order to save inside it, during any math session, some not-built-in libraries I have in my dropbox.
Inside 'directory_name' I will download some external libraries from dropbox and I will test if I will be able to import these libraries with the common 'import ....'. If sagemathcell can edit sys.path and user can add any temp folder in it, maybe the remote python interpreter would import the libs.Thank you
Regards -
This thread seems like it has become the definitive thread for all things pandas/scipy so I’ll just post this here.
Some of you may be interested in the following app:
I’ve played with it a little and you can in fact use scipy and pandas on it. Those are preinstalled. If you upgrade for $5 you can plot as well. Which is nice.
The problem, and for me it’s a pretty big problem, is that your code is sent to a server and compiled remotely (with a max running time of 30s). There are of course many reasons not to do that. But for just learning to work with scipy or pandas or for just playing with generic ideas, I thought perhaps someone could find it interesting. It is buggy but I’ve been successful getting a few basic things to work.
I corresponded a bit with the developer and he is quite responsive. He also said he is thinking of having the code run locally in the future, but there is certainly no reason to expect that to be successful - I would think he will likely hit the same problems as omz.
Anyway for real tasks, I’m still hoping for at least pandas on pythonista one day if there’s no hope for scipy (I would gladly pay $20 more than I have already paid for those - just saying). But I thought someone may find this interesting since it is a relatively new app.
-
Hi, since cocalc was mentioned, I thought it would be worthwhile to suggest Juno (https://juno.sh), which runs on iPad. It’s basically a Jupyter notebook client running in an iOS app, and you have a choice of different back-ends: Azure Notebooks, Cocalc, or an AWS backend. I’m using the AWS backend for maximum flexibility.
-
I sure wish I knew pandas won't be supported, before I paid $10.
Sincerely,
Scott -
@scotttmiller I would be happy to pay you the $10 back if you create a working version of Pandas in an iOS app and publish it to the AppStore for $10. Please let me know.
-
In a recent post I tried to see if @omz could share something about what he was working on (no reply). I realize that telling us is a mixed bag for him but perhaps he could say if Scipy and Pandas, which are often requested, are never happening, imminent, or still a possibility.
-
I can assure those of you interested that Juno is easily worth the $10. Honestly, I’d be happy with my purchase even if I spent $20 or $30. It would be great to be able to use pandas and scipy directly on iOS, but Juno serves most of my purposes just fine. You can use cocalc or azure, or what I usually do is just run both a selenium server and a jupyter server on my laptop on the same network and just run it via Juno. And then I can do anything I want in either Python or R (or many other languages) from my iPad or phone. It’s true you can do that just via safari or other browser too, but I find it unworkable, especially on the phone. The app does make the experience pretty good.
-
Me too, pandas plz.