Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Will pandas and scipy be available on Pythonista in the near future ??
-
Analyser looks quite interesting. Personally, I would not want to give up the (many) superb features of Pythonista. I hope that @omz will (someday) find a way to add these much requested modules. (no drink for this one :-)
-
@Analyser, ok cool. I think my post may have looked a bit mean or cold. Was not meant to be. I think i was rushing. I was just trying to point out as a developer (which i have not been for so many years it does not matter), but on the surface some decisions look like they should be no brainers. But I am sure you are aware when you add any third party dependencies to your app/development you have to ask yourself many questions about the now and the future. I dont pretend to understand what would be involved here. I just received know sometimes the smallest things can have a huge domino effect. And to be fair that could be in a positive way also.
Anyway, for me I have only a single motivation. That is that Pythonista is the best it can be for all its users. -
push
add pandas and scipy and i will instantly buy the app :D -
With pandas not being integrated, I was always interested in using a remote environment of python. Running and editing code with Pythonista is so much more fun than any other method, I just wish I could use it more.
-
It is possible to use the sagemath servers for doing remote work. @Matteo has some scripts for interacting with sagecell
https://forum.omz-software.com/topic/4200/data-analysis-workflow/17 -
@ericbaranowski Hi, I have no more ideas about how to improve sage_interface, but if you have some hints to improve it post here your ideas!
sage_interface has the following features:
- You must be online.
- Max 64 kb of input scripts.
- Max 2 minutes for calculation for any single script (must be tested, I’m not sure, Andrey the author said me that user could have also 2 hours for a full math session, but these limits could change if Andrey will decide to change them).
- About max 4-5 Gb of Ram available (info by Andrey).
- No way to install/compile your personal libraries (not-pure, like last version of scipy for example or any other library you want).
- sage_interface: a)sends to server a script, b)waits for server output, c)processes the output string simply saving each string in a Pythonista variable you decide, with image and numpy array capabilities. a) and b) are available with the original script by sagemathcell authors (see here), while c) is written by me with the help of @JonB and @ccc. I use a lot also the wrench version by JonB because when I want to test a script that in future I will use often, I find very useful to test it by touching a single key in the same way I touch the built-in key run of Pythonista (Pythonista is very customizable if you know how to customize it ;-)). Wrench version has a powerful feature about error checking in the script send to server, so you can easily find any error returned by server. My function version has a very basic error checking feature, but I use it because Pythonista allows you to open several files and to surf easily among them.
For the second limit above, I say that if you must solve a very big linear equations system, for example, you could create a txt file with the big system as a sequence of numbers/coefficents (A and b for the classical Ax=b), save it in your dropbox (or any online storage service), use the server to open remotely your file with data input text (A and b) in order to save content in variables of the server python environment, and finally run your script for linear solver sending the solver script to server and using the variables for input data created previously.
My only target now about sage_interface is to write a series of standard scripts (standard for me, but they can be adapted by user) that send to sagemathcell server any script for calculation in the following numeric math fields (that I use often, for work or for hobby):
- PDE solver
- ODE solver
- Big linear equations system solver (sparse, dense)
- Non linear equations system solver
- Optimization (linear or non linear – w/o contraints fitting data with plots).
Another thing is that with sage_interface you can import to python environment of the server a remote pure python library with httpimport by John Torakis, but it works obviously with pure python libraries, no way for now to use it with c-fortran based python libraries. See here.
I asked to Jonh Torakis if he knows about some python libraries that allow user to import any other library (pure or not pure python) as a full folder with compiled c-fortran code (for example the last scipy library compiled in the same os environment the server uses, for sagemathcell is linux), but he doesn’t know. Httpimport is great for pure-python libraries and you can use it if you want to maintain clean your site-packages folder in Pythonista (but if you use often a library then you would prefer to install it instead of remotely importing it every time you need).
Does someone know if a native (Linux/Windows/Mac) linker-loader using HTTP/S (or networking in general) exist (citing John in his answer to my question)?
Regards
Bye -
Sorry if I insist, but I'm yet interested to the question I posted a few days ago:
does someone know if a native (Linux/Windows/Mac) linker-loader using HTTP/S (or networking in general) exist?
In other words:
- Let's say I have two computers (A and B) with the same OS;
- In computer A I have a full working python distribution with a properly installed scipy version;
- in computer B I have an other full working python distribution (same python version of computer A) but I haven't the scipy library and I can/want not install it;
- I copy/paste the full scipy folder of computer A in a remote service (github, dropbox, icloud, etc...); the full scipy folder contains not only pure-python scripts but also compiled files like .dll or .lib. or any kind of file required by scipy. Let's suppose that the full scipy folder is fully portable (it should, because some times ago I was able to compile a scipy version in my pc, then I copied the scipy folder created in site-packages and I pasted it in a different python distribution in a different pc with same OS: without compilation in the second pc, I was able to use scipy compiled in the first pc);
- I know the HTTPS link of the full scipy folder stored in the remote service (as shared link);
Question: does exist nowadays a way to use the existing python environment of computer B (with Internet connection) to import only in RAM memory of B the full scipy folder with HTTPS link (of point 5) in order to use it with python, as if the full scipy folder was stored in the hard drive of B?
I only ask if any of you are aware of the existence of a tool like this and where I can find it.
Thank you for any hint
Regards -
This post is deleted! -
@Matteo For pure Python modules that wouldn't be too difficult, you could write an import hook that tries to download the requested module from a server and then loads that. With native modules that's not as easy - I don't know of any OS that lets you load a native library from RAM. For example, the Unix
dlopen
function only takes a file path. You could download the library to a local file and load that, but if you can do that, you could install the library permanently as well.In any case, this would be of no use on iOS anyway. Because of iOS app sandboxing, an app is only allowed to load native libraries if they are code-signed by Apple or the app developer. This is why you cannot for example compile SciPy for iOS on a Mac and copy over the compiled libraries. Even if you code-signed them with your own iOS development certificate, iOS would refuse to let Pythonista load them, because the library was signed by a different developer than the app.
-
@dgelessus I have been thinking about this recently... Does iOS actually prevent remapping RW memory to RX?
If so, How does CFUNCTYPE callbacks work under the hood?
I have not tried this yet, but it looks interesting..
https://csl.name/post/python-jit/Of course, even if that works, the tricky bit would be writing one's own loader in python.
-
In thinking about this more, though not looking at the source, CFUNCTYPE almost certainly works by having an executable stack, or else by having a WX page full of trampoline functions. So it seems like it should be possible to use the same method to call object code in memory.... I can see it maybe working for statically linked code, but a loader would be ugly for something like pandas.
-
@dgelessus and @JonB
Hi, thanks guys for reply, I would like to clarify that I don’t want to import in Pythonista any precompiled not-pure python libraries (it is impossible for now), but to import them in a computer (named for example “X”) with a working python environment, and I’m interested in doing that with the remote computer provided by SageCellServer, that is a virtual machine with Linux OS; if user can create his/her own server with a home computer linked to Internet, it is possible to use it instead of the SageMathCell one, solving the limits related to impossibility to install own libraries.
Sorry but I try to explain what I have in mind with other words as below.
I’d like to use Pythonista with my idevice:
-
to send some scripts (like the library “httpimport” by John Torakis, but it works only for pure-python libraries without any dll, lib, etc… files), to the remote server (computer “X”) (these scripts are the main reason of my question: Do they exist? Has someone created them, what do you know?);
-
after the remote server has received these scripts, it should run them in order to import only in the RAM (not on hard disk) of computer “X” a full not-pure python library properly compiled with other computer with suitable compilers, same OS, etc… of the remote one (“X”); let’s suppose this library I want to import only in RAM of computer “X” is stored in a folder of my dropbox and I have its shared link.
-
after the importing, I’d like to use the remote python interpreter of SageMathCell (or other own available remote computer) using Pythonista like a tool that sends python code to server and receives output from that server.
I know that a OS (windows, linux, mac, other…) needs to have an hard disk in order to work (is it true for all? I’m not sure). I think that nowadays no tools exist that allow user to decide if to install any external software, python library, fully portable executable program, full IDE with any kind of compiler
-
in the hard disk, or
-
directly in RAM, as if a portion of RAM, decided by OS, behaved the same way as a hard disk (so all files are stored in ram with the same folder hierarchy of hard disk).
As example, let’s suppose I have in my pc (named “Z”) a python environment installed in the hard disk in the path “C:\Python27\” of windows OS (as example, but I ask if it is possible also for linux, mac, other…).
Now suppose that
- I want to import a python library in my pc (without store it in hard disk with “pip install”). With httpimport library by Torakis it is possible to import temporarily only in ram a pure-python library, but, unfortunately, suppose I want to import in RAM a not-pure python library, like scipy, so I can’t use httpimport lib.
- Suppose also that the library I want to import only in RAM needs some existing libraries in my OS, that are stored in some folders of hard disk.
Well, can I:
a) compile the not-pure library with a suitable compiler following the common way, that is by installing the compiler in hard disk of a pc and using it to compile the python library,
b) copy the compiled python library, named “Y” and stored in a cloud service accessible via Internet connection, only in a portion of my pc’s ram (as if it was in “C:\Python27\Lib\site-packages\my-compiled-python-lib”) with a python command/function like the one I use to download something , that is the "urllib" python lib,
c) tell to python interpreter of pc “Z” to import the library “Y” in order to use it?The library “Y”, that needs some external libraries in my OS (see point 2. above), in my idea, would search for the external libraries stored in pc "Z" hard disk even if it is only in RAM.
So, is it possible?Thank you guys
Regards -
-
@JonB I don't know how
ctypes
creates callback functions in Pythonista.ctypes
internally useslibffi
, for which you can find the source at https://github.com/libffi/libffi. However I don't know if @omz had to modifylibffi
to make it work on iOS, so the official source code may be inaccurate.What I did find however are the internals of the Objective-C runtime's
imp_implementationWithBlock
, which effectively lets you create anIMP
that calls the given block. Seeobjc-block-trampolines.mm
anda1a2-blocktramps-arm64.s
in https://opensource.apple.com/source/objc4/objc4-723/runtime/. The way it works is that the assembly code defines a page full of identical trampolines. The trampoline code invokes a block located exactly one page before the entry point of the trampoline. That way all trampolines use the same code, but end up invoking different blocks.Now to make use of these trampolines,
imp_implementationWithBlock
allocates two pages of read-write memory. Using the Machvm_remap
function, the page of trampolines from the assembly code is mapped into the second newly allocated page, which also changes the permissions of that page to read-execute. The first page is still read-write and is used to store the blocks, which can then be called by the corresponding trampoline in the next page.None of this requires an executable stack or any other sort of write-execute memory, since the executable code is loaded unmodified from the library, and then mapped into memory multiple times without modification. I don't think iOS allows write-execute memory, that would make it very easy to circumvent the code signing restrictions.
-
@Matteo Importing a native Python module from RAM is simply not possible if the OS doesn't allow loading a native library from RAM, and I think Linux doesn't. As far as I know, on Unix systems Python uses the standard
dlopen
function to load native modules, and that requires the library to be in a file.I don't know what the SageCellServer Linux environment does and doesn't allow, but if possible you could make a temp directory, download the libraries you want into it, and add it to
sys.path
. Then any native modules in there will be found by the normalimport
mechanism. As long as the libraries are compiled correctly for the system used by SageCellServer, they should import without problems. -
This post is deleted! -
This post is deleted! -
@dgelessus Hi, thank you for suggestion, unfortunately it seems to me that it is not possible to create a folder with remote server by SageMathCell.
I had a simpler idea thanks to your reply: can python (any version) add to
sys.path
also remote folders (for example any folder stored in a dropbox or github account)?Example: I want to add to
sys.path
of my local python environment (for example Pythonista) a folder like this one:https://github.com/jsbain/objc_hacks
without downloading it neither storing it, with all scripts, in local storage memory of Pythonista.
Can I do it?
Thanks
Regards -
Is the issue that you don't have control over your server? I thought you were trying a run a sagecell server on your personal pc. if so, why not just install the packages you need. Or is the issue that sagecell is "locked down" even running on your own pc?
You cannot change sys.path to point to a non-path. on a pc, you could install a drobox client, which automatically syncs a folder, but i think that doesnt satisfy your goals. You could download files using drobox client, import them, then delete them. but if your issue is with sagecell limitations or security, that pribably won't work either.
-
@Matteo Linux can only load native libraries from the regular filesystem. That doesn't necessarily mean "on a real physical hard disk", it could also be on a network share or RAM disk mounted somewhere in the file system. However if the system doesn't even let you create temp files, it probably won't let you mount remote filesystems either.
On a regular Linux system that you have full access to, that wouldn't be a problem though. You can mount a network share under
/mnt/myshare
or whatever, add/mnt/myshare/pythonlibs
tosys.path
, and then you can import any module in that folder. I don't know Linux very well, so I don't know what commands exactly are used to mount remote filesystems. I also don't know if there are ways to mount a Dropbox or a Git repo as a filesystem. -
Hi @JonB , I have not my own sagecell provided by my online pc, because of lack of ram and cpu resources of my pc.
I use only the main SageMathCell mainteined by Andrey. But it (seems to me) doesn't allow user to install any external own python library: user can use only built-in libraries like these ones.
So I ask here if we can by-pass this limit by using a set of python scripts or a python library to import in ram (no hard disk) a remote python library compiled in linux stored in any cloud service, in order to use it not with Pythonista but with the remote python interpreter provided by SageCellMath (Pythonista would be a "remote controler" for the SageMathCell).
Hi @dgelessus, ok, I don't know Linux very much, I've tried some times ago to create a virtual machine with linux on my windows pc by using Virtual Box, but it was too slow.
After this long discussion (thanks both dgelessus and JonB!) I'd like if, in a near future, python could allow user to import any working python library (pure and not-pure) stored only in web (cloud or https), without downloading and saving it in the local filesystem. The web would become in this way a read-only big memory resource for own local python env and if user wants to change/modify the imported library, then he/she could do it by downloading the library, changing it and finally reloading the modified library in his/her own cloud account in order to use it with own local python env.
About sys.path python function, now I know that it can be used only for paths in filesystem (hard disk) and not for paths in the cloud (in a read-only mode).
Thanks both again for explanation,
Regards