Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Binary files read and write
-
My sense from my emails from the author of SPLnFFT is that the newer files have NO Infs and NO NaNs. I do not find either in the files that I generate with SPLnFFT. I will verify this with the author.
If you know how to create a matplotlib plot that looks like what Matlab generates, I would be happy to accept the pull request ;-).
-
pull request on the way. it is not clear what defines the time scale -- does the file always contain a full 24 hours (hence the msny zeros if it wasnt run for that long)?
-
Before the cleansing step, each file is 8 readings per second times 24 hours.
-
I'm using and observing SPLnFFT since more than one year ago. If you have the App with the 24 hours export option installed, please, start a recording, pause and go to the HISTO screen. You'll see in the lower right corner a camera icon. Pulse it and next pulse the icon below the 24 HOURS option. This action exports to Photos a 24 hours plot. If you start recording again, let's say after one hour, repeat the same process and observe the differences between plots. By observing this facts repeatedly, I came to the conclusion that when you start and pause the recording, the App must save in a memory buffer the previous SPL values until a 24 hours cycle is completed. It also explains the chunks of SPL values alternating with zero values chunks if you analyze the first of the two 5.2 MB files exported, I sent you a sample in a former comment. THEREFORE, YOU MUST OMITE THE CLEANSE PART OF YOUR CODE IF YOU WANT TO OBTAIN A SIMILAR PLOT with the first Python script you released. I reported this findings to the App developer. Hope this help.
-
I accepted the pull request from @JonB and then added more labels so the graph now looks quite presentable. There is also a
remove_zero_readings
flag to easily remove the cleansing step but if you set it to False, the graph can take more than a minute to appear. Be patient, it will appear. -
Hello,
I am new to coding and am going through Zed Shaw's Learning Python The Hard Way. I would like to practice the exercises using Pythonista but am a bit confused on how to proceed. In Zed's course we write the scripts in Text Wrangler then run them in the shell. Could someone point me in the right direction on how to do that but using Pythonista?
Thank you for any advice you could offer.
Mark Connors -
In Pythonista, you'd obviously use the built-in editor instead of e. g. TextWrangler as you would on a computer. However Pythonista doesn't include any kind of shell. To run scripts you just use the "play" button in the toolbar.
I don't know how much the book uses the shell - if it's only used to run the Python scripts you wrote in the editor, then it shouldn't be a problem. If it's used to pass runtime arguments to your script (e. g.
python myscript.py arguments et cetera
, where "arguments et cetera" are the arguments), you can tap and hold the play button. This will bring up a popup window where you can enter the arguments you want to pass to the script.If the book uses the shell for more things than that, you should do those parts on a real computer. There is StaSh, which is a shell written purely in Python for Pythonista. It is far from complete and only has basic features (plus a few things useful when working in Pythonista, such as a minimal
git
implementation) so you might not be able to run some of the shell commands from the book. -
CCC. The graph looks fine now.
- Why don't you use the 0:24 hours time axis, as the one I sent you and Is used by the Matlab-Octave script?
- Where are those Faulty readings, Infinites?
- Why don't you try to make intermittent recordings in a 24 hours periods and watch the result?.
As the App works like a noise Dosimeter, its purpose is to record the exposition to noise for a given time period during 24 hours, like at Job, when driving, at the disco, even while sleeping, -(maybe you snore and you are not aware)-so that it can be correlated with pulse, blood pressure, ECG and other biometric parameters.
If you give me an email address I'll send you another approach to graph plotting that I programmed with the basic interpreter. The graph plots can be zoomed in and out, stretched,expanded moved and rotated by simple finger gestures on the screen with a short code.Most of the source code is for the GUI.
-
I can not figure out the syntax for getting the x axis labels to go from 0 to 23.999.
Help if you can... https://github.com/cclauss/uncategorized_hacks/blob/master/SPLnFFT_Reader_numpy.py
-
count=2436008*2;
TabTime=[0:count/2-1]/(count/2)*24;This is the code of the Matlab-octave script related to Xaxis values. The graph that generates is for a full 24 hours recording, irrespective of the recording time for a given interval. If it's short, all values are so cluttered that the visual effect is of an aggregate of consecutive lines.
You can download the full script from the Facebook account of the author. If you have gnu.Octave try it an print the generated values if you have Microsoft Excel above 10, download Macro and analyze the code in Visual Basic.
-
I know all that you have written... My trouble is that I can't figure out the right matplotlib syntax.
-
In your last script when the data cleanse is set to False the App crashes. By the way, your former script with array and structure options read all 5.2 MB without problem. I think that the problem is related to the numpy library and the memory usage of float data type, since by default it uses only two 8 bytes
-
This version of your script works fine
#coding: utf-8 # cleanse import numpy data = numpy.fromfile('SPLnFFT_2015_07_23.bin', dtype=numpy.float32).reshape(-1, 2) #data = data[numpy.all(data > 0, axis=1)] # cleanse print(type(data), len(data)) # numpy.ndarray, 2786 print(data[:20]) # print first 20 fast, slow pairs # HELP: A scatter plot is all wrong for this dataset import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(1, 1, 1) x = data[:,0] y = data[:,1] ax.scatter(x, y) plt.title('SPLnFFT Noise data') plt.show()
-
the problem with the matlab example is that it doesnt seem to have the right data size relative to the actual file structure.
if you know that the datafile always contains exactly 24 hours worth of samples, the numpy equivalent to the matlab is
t=numpy.linspace(0.0,24.0, N)
and of course you would not cleanse!
you would want the ability to zoom, however, which now requires a little more trickery, and probably you would only plot a given time range to keep the number of points down. while matlab can plot a million points, matplotlib on ios is strained!
-
I'm not quiet sure, but I think that what MATLB script makes is simple artihmetic;
Suppose that the file has a total 635 pairs of data points;
635 × 24 = 15240, hence
634 ÷ 15240 = 0.041601049869
Therfore the point intervals woul be = 0.041601049869 times 0,1,2,3,4,5,,..............635After plotting the SLOW and FAST point values just need to put a label in the Time axis from 0 to 24 hours with one hour interval
Hope it helps -
Thanks again JonB ... I am learning.
I updated the code to remove all the cleansing, add the zero hour to 24h x axis label as you suggested, and add elapsed_time(). The script takes about 2min 20sec to produce a full 24h day plot on my iPad.
I am still not satisfied with the x axis that currently starts at -5h (!) and ends at 30h with ticks every 5 hours. My goal is to have it start at 0h and ends at 24h with a tick every hour. My attempts to do so result in crashes so they are commented out.
There is yet another new release (v5.6) of the SPLnFFT app.
-
This link shows how to compute aggregated data (LEQ) from non-zero SPL values for a given time period
-
I sent an algorithm 2 hours ago to compute the X axis values for a full SPLnFTT binary file with no data cleansing I. Try it, but using the values that come in the MATLAB script. It should work. Regards
-
ManuelU, you need to be a little more specific as to what you want in the end. Do you want a single number(average SPL) over 24 hours... ? or plot of 24 hours where the resolution is dropped by, using a moving average, say dropping the resolution down to once per minute, or 10 minutes, and showing average and peak over that 10 minutes?
note that periods where spl is 0 will stay 0 after averaging!
or do you want the original resolution, but the ability to zoom the plot on each region where you actually have data?
or, detect how many recording sessions there were, and show N subplots, with only the data from that session, but showing the correct original timestamp in each subplot?ccc. you want plt.xlim(0,24), and plt.xticks(numpy.arange(0,24))
-
JonB, In my program I have an option to process the whole data file in 1 hour or more chunks or even arbitrary chunks. The processing power for graphics is amazing. You just need to build up a pair of X,Y (SLOW-FAST) arrays and a few instructions to render a graphic output that response to finger gestures like any other professional graphic App for iOS.
A 24 hours overview is crucial to observe the segments that need to be analyzed in detail, for the search of pikes or a repeating pattern through time. By using this program and the SPLnFFT In an iPhone I detected some asymptomatic people with dangerous periods of sleep apnea, that otherwise couldn't have been detected. Many people work without any protection in noisy environments, like ambulance workers that are submitted to dangerous dB acoustic levels with the risk of permanent ear damage.
Respect to average values during a time period, the most common is the LEQ, that is a logarithmic average and you need to filter out zero values. That's easy done just by reading the array that hold the time values and SPL values for a given time period. For serious epidemiological investigation you need to sincronize with other biometric values.
My intention is to use a Holter-like ECG recorder. There are evidences of the relationship between noise and heart coronary disease, but few with the simultaneous recording of noise and ECG. To handle statistics I developed some years ago an App for the Mac OSX that uses Binary Logistic Regression and Survival analysis with parametric regression models for assessing the risk of diseases that might be related to noise. Statistics render numerical data, but sometimes I used the common sense instead of hypothesis contrast methods to take decisions. As you know, no hypothesis can be demonstrated; the most you obtain are only evidences. That can only be achieved with a team and the necessary tools.
One idea in mind is to analyze the noise data in the frequency domain with the FFT methods of the numpy library. This feature and the option to directly import the binary file from Dropbox was what made me chose Pythonista as a supporting App. Thanks for the valuable support from all people in this excellent Forum