Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Data analysis workflow
-
Its clear Pandas, Scipy and other data modules are not available in Pythonista. But, Numpy, Matplotlib and other python only modules are available. So are feedparser and beautiful soup for web scaping. I've parts of nltk to work as well.
I use the Anaconda stack with pandas, numpy, scipy, keras, tensorflow, nltk etc etc. on desktop with Jupyter as my playground. But, havent tried much in Pythonista since am not sure if i will miss the modules not available in Pythonista too much, or perhaps thats just me.
Curious how many people do data analysis in pythonista, which modules they use, their workflow and examples of what kind of analysis they do. Thanks in advance!
-
Hi,
I’m new about python, I used (and use) fortran 90, so I’ve no depth programming knowledge. I’m still learning python that is my choice among all programming languages that I know (the main reason is Pythonista that I discovered some times ago in Apple Store since I have an iPhone).
In spare time I like to code mainly:
-) data analysis scripts (eg rolling average of different types of data or optimization of a function or some functions with or without constraints),
-) simple physics processes scripts, math (symbolic and numeric) processing (eg plotting graphs of parametric supposed-not-infinite summations that make me curious),
-) simple interactive plots (ie with the feature to run indefinitely until user’s stop and the feature to allow user to change some parameters of the plotting function or constrained functions without aborting the calculation).I’m trying to improve my python programming knowledge with the following python libraries (not pure python):
-) Scipy
-) Sfepy
-) Pydy
-) PyDSToolI’m learning to use these libraries with a Miniconda distribution and the PyScripter IDE (I use Windows PC).
I’d like if these libraries worked with Pythonista, but unfortunately they are not full pure python due to, I think, calculation speed reasons (that is the developers of these libraries decided to use existing Fortran and/or C libraries, for some tasks or calculations, optimized for some platforms and they added them to their python libs).
I’d like if python users could use different versions of all existing not pure python libraries (like Scipy or Sfepy) in order to use them without tedious compilation processes for a specific platform.
For example, It would be nice if I could download an official modified version of Scipy (created by Scipy developers) that uses only pure python libraries.
That is: the not-pure python internal libraries are replaced by pure python scripts that perform the same tasks. For example a nonlinear numerical optimization algorithm written in Fortran and implemented in Scipy could be replaced by an identical pure python algorithm: I think that for standard numerical simulations (eg for hobby) the speed decrease would be minimal, thinking of the hardware we find today…). If user must do calculations with a great number of data, so he would be invited to use a compiled version and a specific platform, but in the meantime he could do tests and experiments on the fly with a small device (phone).
In conclusion: I'm still studying to find a simple, easy-to-use solution (set of pure python scripts) that can be used on any platform with python (computers, any smartphone) to use it as a computing language for any tasks.
Regards
Matteo -
In the absence of
pandas
, I previously usedagate
to do some basic analysis. The gist below shows an example of creating a test dataset of employee data, constructing a data table and assigning each record to a random group according to a specific frequency distribution. A summary table is then produced to provide some descriptive salary statistics.This was a start for a Monte Carlo simulation model. I completed the rest of the project using
pandas
on my Mac -
Hi, in order to use Pythonista like a powerful tool for any numerical computation using a little device like a smartphone, I discovered SageMathCell. The author is Andrey Novoseltsev. The online service allows user to code python scripts with SageMath (you can use also Octave, Maxima, etc...: to change the available computational tool you must choose it at the 'Language' form).
SageMathCell is based on SageMath.Kindly Andrey showed me a python script that uses SageMathCell to return the output of any calculation that SageMath can perform. The script is sagecell-client.py that requires the pure python library websocket-client (I installed it with StaSh 'pip install websocket-client').
A more complex example is the script below (it is the script 'sagecell-client.py' slightly modified to calculate the numerical solution of a ODE system):
""" A small client illustrating how to interact with the Sage Cell Server, version 2 Requires the websocket-client package: http://pypi.python.org/pypi/websocket-client """ import websocket import json import requests class SageCell(object): def __init__(self, url, timeout=10): if not url.endswith('/'): url += '/' # POST or GET <url>/kernel # if there is a terms of service agreement, you need to # indicate acceptance in the data parameter below (see the API docs) response = requests.post( url + 'kernel', data={'accepted_tos': 'true'}, headers={'Accept': 'application/json'}).json() # RESPONSE: {"id": "ce20fada-f757-45e5-92fa-05e952dd9c87", "ws_url": "ws://localhost:8888/"} # construct the websocket channel url from that self.kernel_url = '{ws_url}kernel/{id}/'.format(**response) print self.kernel_url websocket.setdefaulttimeout(timeout) self._ws = websocket.create_connection( self.kernel_url + 'channels', header={'Jupyter-Kernel-ID': response['id']}) # initialize our list of messages self.shell_messages = [] self.iopub_messages = [] def execute_request(self, code): # zero out our list of messages, in case this is not the first request self.shell_messages = [] self.iopub_messages = [] # Send the JSON execute_request message string down the shell channel msg = self._make_execute_request(code) self._ws.send(msg) # Wait until we get both a kernel status idle message and an execute_reply message got_execute_reply = False got_idle_status = False while not (got_execute_reply and got_idle_status): msg = json.loads(self._ws.recv()) if msg['channel'] == 'shell': self.shell_messages.append(msg) # an execute_reply message signifies the computation is done if msg['header']['msg_type'] == 'execute_reply': got_execute_reply = True elif msg['channel'] == 'iopub': self.iopub_messages.append(msg) # the kernel status idle message signifies the kernel is done if (msg['header']['msg_type'] == 'status' and msg['content']['execution_state'] == 'idle'): got_idle_status = True return {'shell': self.shell_messages, 'iopub': self.iopub_messages} def _make_execute_request(self, code): from uuid import uuid4 session = str(uuid4()) # Here is the general form for an execute_request message execute_request = { 'channel': 'shell', 'header': { 'msg_type': 'execute_request', 'msg_id': str(uuid4()), 'username': '', 'session': session, }, 'parent_header':{}, 'metadata': {}, 'content': { 'code': code, 'silent': False, 'user_expressions': { '_sagecell_files': 'sys._sage_.new_files()', }, 'allow_stdin': False, } } return json.dumps(execute_request) def close(self): # If we define this, we can use the closing() context manager to automatically close the channels self._ws.close() if __name__ == "__main__": import sys if len(sys.argv) >= 2: # argv[1] is the web address url = sys.argv[1] else: url = 'https://sagecell.sagemath.org' a = SageCell(url) import pprint ####################### Modified part: ######################## string_for_SageMathCell = """ ### SageMath - ODE System Solver: import numpy as np # Parameters: t_begin = 0 t_end = 5 step = 0.5 h , g = var(' h , g ') t = var(' t ') # ODEs and ICs: # 'dhdt=g-t-2' et # 'dgdt=h+g+t' with # 'h(t=0)=1' et # 'g(t=0)=3' functions = [ h , g ] indep_var = t system = [ g-t-2 , h+g+t ] init_conds = [ 1 , 3 ] # Solver: time_interval = srange(t_begin, t_end+step, step) solution = desolve_odeint(system, init_conds, time_interval, functions, indep_var) # Output matrix: number_of_steps = Integer((t_end-t_begin)/step)+1 time_interval_Matrix = np.reshape(time_interval, (1, number_of_steps)) solution_t_functions = np.concatenate((time_interval_Matrix.T, solution), axis=1) print(solution_t_functions) """ pprint.pprint(a.execute_request(string_for_SageMathCell)) ########################################################
If you run this script (you must be online) with Pythonista, it works very well.
My question is: can someone kindly show me how to parse the full output string of the script above to save the solution (a matrix) in a python variable?
In other words, in the full output I have:
... {u'channel': u'iopub', u'content': {u'name': u'stdout', u'text': u'[[ 0.00000000e+00 1.00000000e+00 3.00000000e+00]\n [ 5.00000000e-01 2.02793180e+00 5.98376746e+00]\n [ 1.00000000e+00 5.04296841e+00 1.23947375e+01]\n [ 1.50000000e+00 1.26637518e+01 2.64136058e+01]\n [ 2.00000000e+00 3.07328697e+01 5.74525026e+01]\n [ 2.50000000e+00 7.23430485e+01 1.26665529e+02]\n [ 3.00000000e+00 1.66876083e+02 2.81571125e+02]\n [ 3.50000000e+00 3.80299576e+02 6.28890743e+02]\n [ 4.00000000e+00 8.60745868e+02 1.40829552e+03]\n [ 4.50000000e+00 1.94086754e+03 3.15801982e+03]\n [ 5.00000000e+00 4.36769632e+03 7.08677988e+03]]'}, u'header': {u'date': u'2017-09-05T10:36:17.671623', u'msg_id': u'63a3229a-f030-4c42-93cd-188740e569af', u'msg_type': u'stream', u'session': u'942b9e24-096f-43df-81d3-75b94d7fb42d', u'username': u'sc_work', u'version': u'5.0'}, ...
and I'd like to extract only the text '[[ 0.00000000e+00 1.00000000e+00 ...' to save it (like a matrix) in a variable to use with Pythonista for any other analysis.
Thanks
Regards
Matteo -
This post is deleted! -
data = a.execute_request(string_for_SageMathCell)['channel']['content']['text'] print(data)
-
@ccc Hi, thanks but it doesn't work, I have the error (python 2.7):
wss://sagecell.sagemath.org/kernel/33a8268c-e16a-4a5b-84f2-f8ec530ea63a/ Traceback (most recent call last): File "SageMathCell - example with 'desolve_odeint' from sagemath.py", line 148, in <module> data = a.execute_request(string_for_SageMathCell)['channel']['content']['text'] KeyError: 'channel'
Can you please give me some info about your code? I have added it in place of
pprint.pprint(a.execute_request(string_for_SageMathCell))
I've seen that the output is related to the word "stdout", so I will try to search this text and to print into the python shell (maybe I will need regex).
Thanks again for your time.
Matteo -
I think he probably meant iopub instead of channel (i think your output example was misformatted).
Once you get the text key, This appears to be json, you likely want to use json.loads on the textified arrays to get lists of lists -- any then pass to numpy.array() to turn into something useable as a numpy array (for plotting in matplotlib, etc)
-
@Matteo First start with
data = a.execute_request(string_for_SageMathCell)
# now you have the data and can play around with it
Then dotype(data)
# hopefully dict. If it's a str then follow @JonB advice:data = json.loads(data)
Then dodata.keys()
# hopefully 'content' but whatever it is, go with it...
Now walk down the tree...
Then dotype(data['content'])
# hopefully still dict
Keep repeating that loop until you get todata...['text']
and then
Follow @JonB advice anddata = json.loads(data[x][y][z]['text'])
# get rid of the junk you don't care aboutTell us where you get to.
-
Hi JonB and ccc, thanks for your post, two days ago I didn't know anything about json, now I know what it is but I can't work/play with it, sorry, I will study further. For now I have a working script that convert the full output into a array or matrix for numpy ( also scalars).
""" A small client illustrating how to interact with the Sage Cell Server, version 2 Requires the websocket-client package: http://pypi.python.org/pypi/websocket-client """ import websocket import json import requests import numpy as np import re class SageCell(object): def __init__(self, url, timeout=10): if not url.endswith('/'): url += '/' # POST or GET <url>/kernel # if there is a terms of service agreement, you need to # indicate acceptance in the data parameter below (see the API docs) response = requests.post( url + 'kernel', data={'accepted_tos': 'true'}, headers={'Accept': 'application/json'}).json() # RESPONSE: {"id": "ce20fada-f757-45e5-92fa-05e952dd9c87", "ws_url": "ws://localhost:8888/"} # construct the websocket channel url from that self.kernel_url = '{ws_url}kernel/{id}/'.format(**response) print self.kernel_url websocket.setdefaulttimeout(timeout) self._ws = websocket.create_connection( self.kernel_url + 'channels', header={'Jupyter-Kernel-ID': response['id']}) # initialize our list of messages self.shell_messages = [] self.iopub_messages = [] def execute_request(self, code): # zero out our list of messages, in case this is not the first request self.shell_messages = [] self.iopub_messages = [] # Send the JSON execute_request message string down the shell channel msg = self._make_execute_request(code) self._ws.send(msg) # Wait until we get both a kernel status idle message and an execute_reply message got_execute_reply = False got_idle_status = False while not (got_execute_reply and got_idle_status): msg = json.loads(self._ws.recv()) if msg['channel'] == 'shell': self.shell_messages.append(msg) # an execute_reply message signifies the computation is done if msg['header']['msg_type'] == 'execute_reply': got_execute_reply = True elif msg['channel'] == 'iopub': self.iopub_messages.append(msg) # the kernel status idle message signifies the kernel is done if (msg['header']['msg_type'] == 'status' and msg['content']['execution_state'] == 'idle'): got_idle_status = True return {'shell': self.shell_messages, 'iopub': self.iopub_messages} def _make_execute_request(self, code): from uuid import uuid4 session = str(uuid4()) # Here is the general form for an execute_request message execute_request = { 'channel': 'shell', 'header': { 'msg_type': 'execute_request', 'msg_id': str(uuid4()), 'username': '', 'session': session, }, 'parent_header':{}, 'metadata': {}, 'content': { 'code': code, 'silent': False, 'user_expressions': { '_sagecell_files': 'sys._sage_.new_files()', }, 'allow_stdin': False, } } return json.dumps(execute_request) def close(self): # If we define this, we can use the closing() context manager to automatically close the channels self._ws.close() if __name__ == "__main__": import sys if len(sys.argv) >= 2: # argv[1] is the web address url = sys.argv[1] else: url = 'https://sagecell.sagemath.org' a = SageCell(url) import pprint ############################################### string_for_SageMathCell = """ ### SageMath - ODE System Solver: import numpy as np # Parameters: t_begin = 0 t_end = 5 step = 0.05 h , g = var(' h , g ') t = var(' t ') # ODEs and ICs: # 'dhdt=g-t-2' et # 'dgdt=h+g+t' with # 'h(t=0)=1' et # 'g(t=0)=3' functions = [ h , g ] indep_var = t system = [ g-t-2 , h+g+t ] init_conds = [ 1 , 3 ] # Solver: time_interval = srange(t_begin, t_end+step, step) solution = desolve_odeint(system, init_conds, time_interval, functions, indep_var) # Output matrix: number_of_steps = Integer((t_end-t_begin)/step)+1 time_interval_Matrix = np.reshape(time_interval, (1, number_of_steps)) solution_t_functions = np.concatenate((time_interval_Matrix.T, solution), axis=1) print(solution_t_functions) """ ############################################### #pprint.pprint(a.execute_request(string_for_SageMathCell)) #### use of regex to parse fullstring data = a.execute_request(string_for_SageMathCell) fulltext = str(data) pattern = "(?<={u'text': u').*?(?=', u'name': u'stdout'})" ls = re.findall(pattern, fulltext, re.DOTALL) sage_output = str(ls[0]) sage_output_1 = sage_output.replace(r'\n', ',') pattern = "(?<=\d)\s?(?=\s)" sage_output_2 = re.sub(pattern, ',', sage_output_1) sage_output_3 = eval(sage_output_2) var1 = np.array(sage_output_3) print(var1+1) # this is only to prove that var1 is ok for numpy
I've used regex, and massive stackoverflow.com surfing (without it I cant write more than 2-3 rows of code in python when I must work with strings and other python objects different from numbers and arrays).
I have some little ideas to increase the use of the script. I will write here in the next days.
Thanks for your help
Regards
Matteo -
Hi, some ideas for the script above that could help people in math calculations using Pythonista linked to Internet:
-
goal: to give more power to Pythonista about math coding with python (built-in mpmath, sympy, numpy and matplotlib are ok but not updated and too poor for the following numerical computations with existing and tested algorithms: ODE, PDE, System of big linear and nonlinear equations, numerical fitting of ndimensional functions with data, etc...).
-
problems with the above script:
- it must be improved to parse different types of output objects (strings) provided by SageMathCell : working on it.
- it must be improved to return, as text, any errors found by SageMathCell (for example if in the variable 'string_for_SageMathCell' there are some syntax errors, the script should return, in Pythonista shell , the same error that user can read when he uses the https://sagecell.sagemath.org/ service with any internet browsers): working on it.
- a big limit of the above script with Pythonista, in my opinion, is the lack of syntax checking and powerful word completion provided by the IDE inside the code between """ """, that is the code that is send to SageMathCell server: for now I have no idea to solve this.
-
as an example, the kind of math environment I'd like to have with Pythonista and SageMathCell (that could become a CoCalc account for more power) is the hypothetical following python script:
#!python2 print('Lets try a SageMathCell session with Pythonista') ##comment: with '#!sage' user could start a SageMath session inside the powerful Pythonista Editor, in the same way '#!python2' forces the interpreter to run with python 2.x instead of 3.x. #!sage import numpy as np # Parameters: t_begin = 0 t_end = 5 step = 0.05 h , g = var(' h , g ') t = var(' t ') # ODEs and ICs: # 'dhdt=g-t-2' et # 'dgdt=h+g+t' with # 'h(t=0)=1' et # 'g(t=0)=3' functions = [ h , g ] indep_var = t system = [ g-t-2 , h+g+t ] init_conds = [ 1 , 3 ] # Solver: time_interval = srange(t_begin, t_end+step, step) solution = desolve_odeint(system, init_conds, time_interval, functions, indep_var) # Output matrix: number_of_steps = Integer((t_end-t_begin)/step)+1 time_interval_Matrix = np.reshape(time_interval, (1, number_of_steps)) solution_t_functions = np.concatenate((time_interval_Matrix.T, solution), axis=1) ##comment: 'savesage' could be a new command to save into an internal Pythonista variable the output of the server to use it later for plotting or for other further analysis; the 'variable_for_Pythonista' should be any object known by Pythonista and by all its built-in libraries (numpy arrays, numbers, etc...). 'savesage' could extract the output of the calculation provided by SageMathCell and save it in the 'variable_for_Pythonista' variable. savesage(solution_t_functions, variable_for_Pythonista) ##comment: with '#!endsage' user could go out from SageMath syntax (that is very similar to python syntax) and could return to Pythonista environment. #!endsage print(variable_for_Pythonista) #The variable 'variable_for_Pythonista' now is supposed to be fully readable by Pythonista in order to use it with any its built-in libraries.
As you have seen, my programming approach is very basic and procedural, I have very little knowledges in Object-oriented programming; for this reason, the script enhancements could be very little.
Thanks
Matteo -
-
It seems to me that to truly handle variable saving, you need to use someing like pickle or, yaml, etc that can serialize arbitrary objects (say, a dict of results), which then gets printed, along with some tag that can be easily searched, and then unpickled to current workspace. I am not sure how robust pickle will be to unpickling objects from a different version of python, environment, etc. numpy.save to a StringIO could also work.
As for syntax checking, etc, I wonder if what you want is a wrench action which takes the current script in the pythonista editor, sends it to sage, and shows the output in a textview in a panel. That way you just write normal python, then hit the wrench button, and at least can see the output.
-
okay, i have not converted to a wrench script yet, but here is something that:
- lets you save variables using np.savez, and these are transferred directly to globals locally.
- images are transferred and shown in console.
errors and text are split up --probably need to grab traceback too, and maybe raise error directly.
https://gist.github.com/b72236f268f706e84d493cb67944e0e7
no.savez does support regular python variables, however some of these seem to end up as sage types (sage.rings.integer), so they are not directly importable. So, if that is desirable, it may be necessary to use a different mechanism, like casting to standard types first.
-
https://gist.github.com/dae885cd334bca0bd80ee0132978fc50
and now as a wrench action -- add this to your wrench actions, then run wrench on a script.
errors get highlighted in the script, and traceback printed to console.
printed text is printed to console, and images are shown in console.
variables saved as npz files are loaded into globals() of the 2.7 interpreter . This works for numpy arrays, and probably would work for raw literals (i.e x=1r instead of x=1, which is how sage denotes raw literals instead of sage rings) -
@JonB Hi JonB, your code is simply Amazing! Thank you very much for sharing it! I will use it for any calculations to learn advanced math coding with Python+Sage.
Best regards,
Matteo -
Hi JonB, feel free to answer: do you think it would be too hard to modify your script in order to use it not only like a wrench but also like a function inside any Pythonista script that can be executed with the classical |> (run) button?
For example, let's imagine I have a folder inside Pythonista in which I have my script for SageMath server, named it 'ODE Solver.py' (the one between """ """ in the above posts) and a different script that could be as follow:#!python2 import numpy as np print('this print is performed by Pythonista built-in core ...') print('... and now we use the SageMath server to run a script (ODE Solver.py) that performs advanced math calculations and that could be written/coded programmatically and/or automatically by a suitable script that uses only Pythonista built-in libraries') variable_1 = sage_interface('ODE Solver.py') print('now we have created a new variable that we can use inside this Pythonista script') print(variable_1) print('if variable_1 is a numpy object, NxM array or real/complex number, we could perform further calculations on it with Pythonista or yet with SageMath server, if variable_1 is a string, that is any error output by SageMath, any further calculation on the variable could return the SageMath error message.')
I don' know if your 'sage_interface.py' should be placed inside any reserved folder of Pythonista in order to use it like a function (sage_interface('any_sagemathcell_script.py')).
What do you think?
Thanks again!
Best regardsMatteo
-
Hi, the following is a script that tries to perform some tasks of my previous post (Sorry for the long post, I will create a cloud link for the scripts):
""" A small client illustrating how to interact with the Sage Cell Server, version 2 Requires the websocket-client package: http://pypi.python.org/pypi/websocket-client """ import websocket import json import requests import re import numpy as np np.set_printoptions(threshold=np.inf) global server_timeout server_timeout = None class SageCell(object): def __init__(self, url, timeout=server_timeout): if not url.endswith('/'): url += '/' ## POST or GET <url>/kernel ## if there is a terms of service agreement, you need to ## indicate acceptance in the data parameter below (see the API docs) response = requests.post( url + 'kernel', data={'accepted_tos': 'true'}, headers={'Accept': 'application/json'}).json() ## RESPONSE: {"id": "ce20fada-f757-45e5-92fa-05e952dd9c87", "ws_url": "ws://localhost:8888/"} ## construct the websocket channel url from that self.kernel_url = '{ws_url}kernel/{id}/'.format(**response) #print self.kernel_url websocket.setdefaulttimeout(timeout) self._ws = websocket.create_connection( self.kernel_url + 'channels', header={'Jupyter-Kernel-ID': response['id']}) ## initialize our list of messages self.shell_messages = [] self.iopub_messages = [] def execute_request(self, code): ## zero out our list of messages, in case this is not the first request self.shell_messages = [] self.iopub_messages = [] ## Send the JSON execute_request message string down the shell channel msg = self._make_execute_request(code) self._ws.send(msg) ## Wait until we get both a kernel status idle message and an execute_reply message got_execute_reply = False got_idle_status = False while not (got_execute_reply and got_idle_status): msg = json.loads(self._ws.recv()) if msg['channel'] == 'shell': self.shell_messages.append(msg) ## an execute_reply message signifies the computation is done if msg['header']['msg_type'] == 'execute_reply': got_execute_reply = True elif msg['channel'] == 'iopub': self.iopub_messages.append(msg) ## the kernel status idle message signifies the kernel is done if (msg['header']['msg_type'] == 'status' and msg['content']['execution_state'] == 'idle'): got_idle_status = True return {'shell': self.shell_messages, 'iopub': self.iopub_messages} def _make_execute_request(self, code): from uuid import uuid4 session = str(uuid4()) ## Here is the general form for an execute_request message execute_request = { 'channel': 'shell', 'header': { 'msg_type': 'execute_request', 'msg_id': str(uuid4()), 'username': '', 'session': session, }, 'parent_header':{}, 'metadata': {}, 'content': { 'code': code, 'silent': False, 'user_expressions': { '_sagecell_files': 'sys._sage_.new_files()', }, 'allow_stdin': False, } } return json.dumps(execute_request) def close(self): ## If we define this, we can use the closing() context manager to automatically close the channels self._ws.close() ## Function that we can use inside any Pythonista script with the import 'from sage_interface import *': def execute_sage(filename, server_timeout): import sys if len(sys.argv) >= 2: ## argv[1] is the web address url = sys.argv[1] else: url = 'https://sagecell.sagemath.org' file = open(filename, 'r') string_to_sage = file.read() file.close() try: a = SageCell(url, server_timeout) data = a.execute_request(string_to_sage) except: print("Can't finish the calculation. Try to increase the timeout parameter of the function 'execute_sage(filename, timeout)' for the script to be processed.") sys.exit() ## let's prettyprint the full output by SageMathCell server: #import pprint #file_output = open("sage_interface.out", 'w') #file_output.write(pprint.pformat(data)) #file_output.close() data_string = str(data) ## let's search for 'stdout' in 'data_string' (to find SageMathCell errors): ls = data_string.find('stdout') if ls == -1: print("There are some syntax errors in the sourcecodes passed to SageMathCell server: check them.") sys.exit() ## if 'ls = -1' it means that there is an error in the source passed to remote server, in all other cases let's look for a warning message, if it exists, inside 'data_string' (to find SageMathCell warning): ls = data_string.find('stderr') if ls == -1: ## there is not warning by SageMathCell, so let's find the numerical output inside 'data_string' (what we need, that is any real or complex NxM array or number): pattern = "(?<={u'text': u').*?(?=', u'name': u'stdout'})" ls = re.findall(pattern, data_string, re.DOTALL)[0] ls = ls.replace(r'\n', ',') pattern = "(?<=\d)\s?(?=\s)" ls = re.sub(pattern, ',', ls) ## let's add parsing for complex numbers and arrays: pattern = "(?<=j)\s?(?=\s)" ls = re.sub(pattern, ',', ls) ## let's solve error parsing for arrays numbers like 1.: pattern = "(?<=\.)\s?(?=\s)" ls = re.sub(pattern, ',', ls) try: output = np.array(eval(ls)) except: print('Output unrecognized by Pythonista numpy. For now it is possible to use/view only numpy real/complex NxM arrays or numbers.') sys.exit() else: ## there is a sage warning, let's find and print it in the console: pattern = "(?<={u'text': u').*?(?=, u'name': u'stderr'})" ls = re.findall(pattern, data_string, re.DOTALL)[0] ls = ls.replace(r"\n'",'') ls = ls.replace(r'\n','\n') print(ls) ## now find the numerical output (any real or complex NxM array or number): pattern = "(?<={u'text': u').*?(?=', u'name': u'stdout'})" ls = re.findall(pattern, data_string, re.DOTALL)[0] ls = ls.replace(r'\n', ',') pattern = "(?<=\d)\s?(?=\s)" ls = re.sub(pattern, ',', ls) pattern = "(?<={u'text': u').*(?=)" ls = re.findall(pattern, ls, re.DOTALL)[0] ## let's add parsing for complex numbers and arrays: pattern = "(?<=j)\s?(?=\s)" ls = re.sub(pattern, ',', ls) ## let's solve error parsing for arrays numbers like 1.: pattern = "(?<=\.)\s?(?=\s)" ls = re.sub(pattern, ',', ls) try: output = np.array(eval(ls)) except: print('Output unrecognized by Pythonista numpy. For now it is possible to use/view only numpy real/complex NxM arrays or numbers.') sys.exit() return output
If you want, try to name it 'sage_interface.py' and save it in a folder (let's name the folder 'sage_test').
Then, in folder 'sage_test', save the following script (the usual), giving it a name, let's say 'input_sage_03.py' (it is the full script that is passed to SageMathCell):
### SageMath - ODE System Solver: import numpy as np np.set_printoptions(threshold=np.inf) # Parameters: t_begin = 0 t_end = 10 step = 0.1 h , g = var(' h , g ') t = var(' t ') # ODEs and ICs: # 'dhdt=g' et # 'dgdt=-h' with # 'h(t=0)=1' et # 'g(t=0)=1' functions = [ h , g ] indep_var = t system = [ g , -h ] init_conds = [ 1 , 1 ] # Solver: time_interval = srange(t_begin, t_end+step, step) solution = desolve_odeint(system, init_conds, time_interval, functions, indep_var) # Output matrix: number_of_steps = Integer((t_end-t_begin)/step)+1 time_interval_Matrix = np.reshape(time_interval, (1, number_of_steps)) solution_t_functions = np.concatenate((time_interval_Matrix.T, solution), axis=1) print(solution_t_functions)
Finally, save, always in folder 'sage_test', the following main script with a name, let's say 'sage_test_03.py':
## For now this script can execute only independent scripts, that is: the output of a script executed by SageMathCell through the function 'execute_sage(filename, timeout)' can't be passed to the input for the next script executed by SageMathCell. ## Working on it to pass, easly, input-output from/for Sage server and Pythonista. from sage_interface import * import time import numpy as np np.set_printoptions(threshold=np.inf) import matplotlib.pyplot as plt ## third script = 'input_sage_03.py' : execute_sage(full-filename, timeout_in_seconds) output_03 = execute_sage('input_sage_03.py', 1) ## script executed by Sage remote server print(output_03+100) ## command executed by Pythonista (local Python core) print(output_03+(120+180.j)) ## to check the compatibility with the built-in Pythonista numpy v1.8.0 print('-----------------------') output_03_mod = output_03 * (1) ## command executed by Pythonista (local Python core), like the following ones: print(output_03_mod) plt.suptitle("Solutions for h (green) and g (red) by processed script 'input_sage_03.py'", fontsize=14, fontweight='bold') plt.plot(output_03_mod[:,0], output_03_mod[:,1], color='g') ## plot h(t) plt.plot(output_03_mod[:,0], output_03_mod[:,2], color='r') ## plot g(t) plt.xlabel('time (s)', fontsize=12, fontweight='bold') plt.ylabel('h, g', fontsize=12, fontweight='bold') plt.show() print('-----------------------')
If you execute the script 'sage_test_03.py' with Pythonista, it should return the wanted output (some arrays and a plot created with built-in matplotlib inside Pythonista).
Working to improve 'sage_interface.py'.
Thanks to Andrey Novoseltsev and co for SageMathCell, JonB and ccc for suggestions, JonB for the wrench version of sage_interface for Pythonista, omz for Pythonista.
I hope this could help people with math.
Regards
Matteo -
Hi, some news about 'sage_interface.py' (useful for me, I don't know if for you too):
-
here you find version dated 15-09-2017 (previous post) of 'sage_interface.py';
-
here you find version 29-09-2017, the new one, with some improvements.
Modify the scripts as you want to perform your calculations.
The script 'sage_interface.py' can be greatly improved to facilitate the use of the server.Bye
MatteoPS: if you want, use this script 'file_downloader.py' to download any zipped folder from any shared Google Drive folder with specific ID (this script is adapted for the two folders I shared in this post):
#!python2 import urllib url = "https://drive.google.com/uc?export=download&id=11vXOY7o3YPK2DkH6_p9e0F94LSiylydO" filename = "sage_interface 2017-09-15.zip" urllib.urlretrieve(url, filename) url = "https://drive.google.com/uc?export=download&id=1bdEHQjYEkxEtwgrhpSiUnOfOBItbtE3-" filename = "sage_interface 2017-09-29.zip" urllib.urlretrieve(url, filename)
-
-
@Matteo I noticed that the plt commands in the first input file are commented out. I tried uncommenting but it did not seem to have any effect. Is plotting working?
-
@ihf Hi. There are two version of sage_interface:
- JonB wrench version: it works with plot via the command 'plt.savefig('filename.png')' instead of the common 'plt.show()', but this trick works with the very first plot call. Try to change 'plt.show()' with 'plt.savefig(filename.png)' and uncomment it. Run the script with the wrench version of sage_interface: it should return the plot and the very first printed array. But about the wrench version you should ask to JonB because it uses some Python/Pythonista pieces of code that I don't know (sorry).
- Function version (my simple implementation for my own purposes): it works with several scripts and with a very simple variables input from Pythonista to server (see function 'execute_script_w_inputs') but the output is a list of strings (a string for each print command in the script passed to server) and you need to evaluate what you want, no plot for now, sorry.
Feel free to try to improve the code in order to obtain several plots with the sage_interface version that you use.
Thanks
Bye