Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Data analysis workflow
-
It seems to me that to truly handle variable saving, you need to use someing like pickle or, yaml, etc that can serialize arbitrary objects (say, a dict of results), which then gets printed, along with some tag that can be easily searched, and then unpickled to current workspace. I am not sure how robust pickle will be to unpickling objects from a different version of python, environment, etc. numpy.save to a StringIO could also work.
As for syntax checking, etc, I wonder if what you want is a wrench action which takes the current script in the pythonista editor, sends it to sage, and shows the output in a textview in a panel. That way you just write normal python, then hit the wrench button, and at least can see the output.
-
okay, i have not converted to a wrench script yet, but here is something that:
- lets you save variables using np.savez, and these are transferred directly to globals locally.
- images are transferred and shown in console.
errors and text are split up --probably need to grab traceback too, and maybe raise error directly.
https://gist.github.com/b72236f268f706e84d493cb67944e0e7
no.savez does support regular python variables, however some of these seem to end up as sage types (sage.rings.integer), so they are not directly importable. So, if that is desirable, it may be necessary to use a different mechanism, like casting to standard types first.
-
https://gist.github.com/dae885cd334bca0bd80ee0132978fc50
and now as a wrench action -- add this to your wrench actions, then run wrench on a script.
errors get highlighted in the script, and traceback printed to console.
printed text is printed to console, and images are shown in console.
variables saved as npz files are loaded into globals() of the 2.7 interpreter . This works for numpy arrays, and probably would work for raw literals (i.e x=1r instead of x=1, which is how sage denotes raw literals instead of sage rings) -
@JonB Hi JonB, your code is simply Amazing! Thank you very much for sharing it! I will use it for any calculations to learn advanced math coding with Python+Sage.
Best regards,
Matteo -
Hi JonB, feel free to answer: do you think it would be too hard to modify your script in order to use it not only like a wrench but also like a function inside any Pythonista script that can be executed with the classical |> (run) button?
For example, let's imagine I have a folder inside Pythonista in which I have my script for SageMath server, named it 'ODE Solver.py' (the one between """ """ in the above posts) and a different script that could be as follow:#!python2 import numpy as np print('this print is performed by Pythonista built-in core ...') print('... and now we use the SageMath server to run a script (ODE Solver.py) that performs advanced math calculations and that could be written/coded programmatically and/or automatically by a suitable script that uses only Pythonista built-in libraries') variable_1 = sage_interface('ODE Solver.py') print('now we have created a new variable that we can use inside this Pythonista script') print(variable_1) print('if variable_1 is a numpy object, NxM array or real/complex number, we could perform further calculations on it with Pythonista or yet with SageMath server, if variable_1 is a string, that is any error output by SageMath, any further calculation on the variable could return the SageMath error message.')
I don' know if your 'sage_interface.py' should be placed inside any reserved folder of Pythonista in order to use it like a function (sage_interface('any_sagemathcell_script.py')).
What do you think?
Thanks again!
Best regardsMatteo
-
Hi, the following is a script that tries to perform some tasks of my previous post (Sorry for the long post, I will create a cloud link for the scripts):
""" A small client illustrating how to interact with the Sage Cell Server, version 2 Requires the websocket-client package: http://pypi.python.org/pypi/websocket-client """ import websocket import json import requests import re import numpy as np np.set_printoptions(threshold=np.inf) global server_timeout server_timeout = None class SageCell(object): def __init__(self, url, timeout=server_timeout): if not url.endswith('/'): url += '/' ## POST or GET <url>/kernel ## if there is a terms of service agreement, you need to ## indicate acceptance in the data parameter below (see the API docs) response = requests.post( url + 'kernel', data={'accepted_tos': 'true'}, headers={'Accept': 'application/json'}).json() ## RESPONSE: {"id": "ce20fada-f757-45e5-92fa-05e952dd9c87", "ws_url": "ws://localhost:8888/"} ## construct the websocket channel url from that self.kernel_url = '{ws_url}kernel/{id}/'.format(**response) #print self.kernel_url websocket.setdefaulttimeout(timeout) self._ws = websocket.create_connection( self.kernel_url + 'channels', header={'Jupyter-Kernel-ID': response['id']}) ## initialize our list of messages self.shell_messages = [] self.iopub_messages = [] def execute_request(self, code): ## zero out our list of messages, in case this is not the first request self.shell_messages = [] self.iopub_messages = [] ## Send the JSON execute_request message string down the shell channel msg = self._make_execute_request(code) self._ws.send(msg) ## Wait until we get both a kernel status idle message and an execute_reply message got_execute_reply = False got_idle_status = False while not (got_execute_reply and got_idle_status): msg = json.loads(self._ws.recv()) if msg['channel'] == 'shell': self.shell_messages.append(msg) ## an execute_reply message signifies the computation is done if msg['header']['msg_type'] == 'execute_reply': got_execute_reply = True elif msg['channel'] == 'iopub': self.iopub_messages.append(msg) ## the kernel status idle message signifies the kernel is done if (msg['header']['msg_type'] == 'status' and msg['content']['execution_state'] == 'idle'): got_idle_status = True return {'shell': self.shell_messages, 'iopub': self.iopub_messages} def _make_execute_request(self, code): from uuid import uuid4 session = str(uuid4()) ## Here is the general form for an execute_request message execute_request = { 'channel': 'shell', 'header': { 'msg_type': 'execute_request', 'msg_id': str(uuid4()), 'username': '', 'session': session, }, 'parent_header':{}, 'metadata': {}, 'content': { 'code': code, 'silent': False, 'user_expressions': { '_sagecell_files': 'sys._sage_.new_files()', }, 'allow_stdin': False, } } return json.dumps(execute_request) def close(self): ## If we define this, we can use the closing() context manager to automatically close the channels self._ws.close() ## Function that we can use inside any Pythonista script with the import 'from sage_interface import *': def execute_sage(filename, server_timeout): import sys if len(sys.argv) >= 2: ## argv[1] is the web address url = sys.argv[1] else: url = 'https://sagecell.sagemath.org' file = open(filename, 'r') string_to_sage = file.read() file.close() try: a = SageCell(url, server_timeout) data = a.execute_request(string_to_sage) except: print("Can't finish the calculation. Try to increase the timeout parameter of the function 'execute_sage(filename, timeout)' for the script to be processed.") sys.exit() ## let's prettyprint the full output by SageMathCell server: #import pprint #file_output = open("sage_interface.out", 'w') #file_output.write(pprint.pformat(data)) #file_output.close() data_string = str(data) ## let's search for 'stdout' in 'data_string' (to find SageMathCell errors): ls = data_string.find('stdout') if ls == -1: print("There are some syntax errors in the sourcecodes passed to SageMathCell server: check them.") sys.exit() ## if 'ls = -1' it means that there is an error in the source passed to remote server, in all other cases let's look for a warning message, if it exists, inside 'data_string' (to find SageMathCell warning): ls = data_string.find('stderr') if ls == -1: ## there is not warning by SageMathCell, so let's find the numerical output inside 'data_string' (what we need, that is any real or complex NxM array or number): pattern = "(?<={u'text': u').*?(?=', u'name': u'stdout'})" ls = re.findall(pattern, data_string, re.DOTALL)[0] ls = ls.replace(r'\n', ',') pattern = "(?<=\d)\s?(?=\s)" ls = re.sub(pattern, ',', ls) ## let's add parsing for complex numbers and arrays: pattern = "(?<=j)\s?(?=\s)" ls = re.sub(pattern, ',', ls) ## let's solve error parsing for arrays numbers like 1.: pattern = "(?<=\.)\s?(?=\s)" ls = re.sub(pattern, ',', ls) try: output = np.array(eval(ls)) except: print('Output unrecognized by Pythonista numpy. For now it is possible to use/view only numpy real/complex NxM arrays or numbers.') sys.exit() else: ## there is a sage warning, let's find and print it in the console: pattern = "(?<={u'text': u').*?(?=, u'name': u'stderr'})" ls = re.findall(pattern, data_string, re.DOTALL)[0] ls = ls.replace(r"\n'",'') ls = ls.replace(r'\n','\n') print(ls) ## now find the numerical output (any real or complex NxM array or number): pattern = "(?<={u'text': u').*?(?=', u'name': u'stdout'})" ls = re.findall(pattern, data_string, re.DOTALL)[0] ls = ls.replace(r'\n', ',') pattern = "(?<=\d)\s?(?=\s)" ls = re.sub(pattern, ',', ls) pattern = "(?<={u'text': u').*(?=)" ls = re.findall(pattern, ls, re.DOTALL)[0] ## let's add parsing for complex numbers and arrays: pattern = "(?<=j)\s?(?=\s)" ls = re.sub(pattern, ',', ls) ## let's solve error parsing for arrays numbers like 1.: pattern = "(?<=\.)\s?(?=\s)" ls = re.sub(pattern, ',', ls) try: output = np.array(eval(ls)) except: print('Output unrecognized by Pythonista numpy. For now it is possible to use/view only numpy real/complex NxM arrays or numbers.') sys.exit() return output
If you want, try to name it 'sage_interface.py' and save it in a folder (let's name the folder 'sage_test').
Then, in folder 'sage_test', save the following script (the usual), giving it a name, let's say 'input_sage_03.py' (it is the full script that is passed to SageMathCell):
### SageMath - ODE System Solver: import numpy as np np.set_printoptions(threshold=np.inf) # Parameters: t_begin = 0 t_end = 10 step = 0.1 h , g = var(' h , g ') t = var(' t ') # ODEs and ICs: # 'dhdt=g' et # 'dgdt=-h' with # 'h(t=0)=1' et # 'g(t=0)=1' functions = [ h , g ] indep_var = t system = [ g , -h ] init_conds = [ 1 , 1 ] # Solver: time_interval = srange(t_begin, t_end+step, step) solution = desolve_odeint(system, init_conds, time_interval, functions, indep_var) # Output matrix: number_of_steps = Integer((t_end-t_begin)/step)+1 time_interval_Matrix = np.reshape(time_interval, (1, number_of_steps)) solution_t_functions = np.concatenate((time_interval_Matrix.T, solution), axis=1) print(solution_t_functions)
Finally, save, always in folder 'sage_test', the following main script with a name, let's say 'sage_test_03.py':
## For now this script can execute only independent scripts, that is: the output of a script executed by SageMathCell through the function 'execute_sage(filename, timeout)' can't be passed to the input for the next script executed by SageMathCell. ## Working on it to pass, easly, input-output from/for Sage server and Pythonista. from sage_interface import * import time import numpy as np np.set_printoptions(threshold=np.inf) import matplotlib.pyplot as plt ## third script = 'input_sage_03.py' : execute_sage(full-filename, timeout_in_seconds) output_03 = execute_sage('input_sage_03.py', 1) ## script executed by Sage remote server print(output_03+100) ## command executed by Pythonista (local Python core) print(output_03+(120+180.j)) ## to check the compatibility with the built-in Pythonista numpy v1.8.0 print('-----------------------') output_03_mod = output_03 * (1) ## command executed by Pythonista (local Python core), like the following ones: print(output_03_mod) plt.suptitle("Solutions for h (green) and g (red) by processed script 'input_sage_03.py'", fontsize=14, fontweight='bold') plt.plot(output_03_mod[:,0], output_03_mod[:,1], color='g') ## plot h(t) plt.plot(output_03_mod[:,0], output_03_mod[:,2], color='r') ## plot g(t) plt.xlabel('time (s)', fontsize=12, fontweight='bold') plt.ylabel('h, g', fontsize=12, fontweight='bold') plt.show() print('-----------------------')
If you execute the script 'sage_test_03.py' with Pythonista, it should return the wanted output (some arrays and a plot created with built-in matplotlib inside Pythonista).
Working to improve 'sage_interface.py'.
Thanks to Andrey Novoseltsev and co for SageMathCell, JonB and ccc for suggestions, JonB for the wrench version of sage_interface for Pythonista, omz for Pythonista.
I hope this could help people with math.
Regards
Matteo -
Hi, some news about 'sage_interface.py' (useful for me, I don't know if for you too):
-
here you find version dated 15-09-2017 (previous post) of 'sage_interface.py';
-
here you find version 29-09-2017, the new one, with some improvements.
Modify the scripts as you want to perform your calculations.
The script 'sage_interface.py' can be greatly improved to facilitate the use of the server.Bye
MatteoPS: if you want, use this script 'file_downloader.py' to download any zipped folder from any shared Google Drive folder with specific ID (this script is adapted for the two folders I shared in this post):
#!python2 import urllib url = "https://drive.google.com/uc?export=download&id=11vXOY7o3YPK2DkH6_p9e0F94LSiylydO" filename = "sage_interface 2017-09-15.zip" urllib.urlretrieve(url, filename) url = "https://drive.google.com/uc?export=download&id=1bdEHQjYEkxEtwgrhpSiUnOfOBItbtE3-" filename = "sage_interface 2017-09-29.zip" urllib.urlretrieve(url, filename)
-
-
@Matteo I noticed that the plt commands in the first input file are commented out. I tried uncommenting but it did not seem to have any effect. Is plotting working?
-
@ihf Hi. There are two version of sage_interface:
- JonB wrench version: it works with plot via the command 'plt.savefig('filename.png')' instead of the common 'plt.show()', but this trick works with the very first plot call. Try to change 'plt.show()' with 'plt.savefig(filename.png)' and uncomment it. Run the script with the wrench version of sage_interface: it should return the plot and the very first printed array. But about the wrench version you should ask to JonB because it uses some Python/Pythonista pieces of code that I don't know (sorry).
- Function version (my simple implementation for my own purposes): it works with several scripts and with a very simple variables input from Pythonista to server (see function 'execute_script_w_inputs') but the output is a list of strings (a string for each print command in the script passed to server) and you need to evaluate what you want, no plot for now, sorry.
Feel free to try to improve the code in order to obtain several plots with the sage_interface version that you use.
Thanks
Bye -
Hi, some news about 'sage_interface.py':
- here you find version dated 25-10-2017 of 'sage_interface.py'.
Added plot capability.
Bye -
Hi, some news about 'sage_interface.py':
- here you find version dated 20-11-2017 of 'sage_interface.py'.
Added basic error checking and device online checking.
Bye
Edit: I've reload the scripts because I've modified 'sage_interface.py' a little: I've changed a regex pattern in 'process_data'.
Future possible improvements:
- to write a robust routine to convert string type arrays to numpy type arrays ([ [ 1 2 ] [ 3 4 ] ] ===> [ [ 1, 2], [3, 4 ] ])
- to create a set of scripts (recipes) to perform easly some math numerical computations with scipy library (only the inputs in the main script need to be modified, the scripts like 'input_sage_XX.py' don't need to be modified.
Thanks
Bye
-
Hi, here you find version 27-12-2017 of 'sage_interface'.
Added string to numpy array conversion and some little improvements.Feel free to propose improvements or changes to the code.
Have you good end and good beginning for the new year!
Bye