Binary files read and write

ManuelU

With the former BASIC code I only generate the starting and ending data points for a given time interval. Suppose you observe the relevant data is in the time interval that goes from 8 to 14 hours. Then your loop would go from start(8) to end(14). I only save one binary file for that interval to compute aggregated data and graphics, that look Iess cluttered in a shorter interval tha in a full 24 graph plot.

I wish I knew how to send images to the forum, so you would appreciate differences in quality. "limit" is an integer variable of the 57600 data points per hour. Here are the values for 1 to 8 hours, startling in the left and end time in the right

1 55760
55761 111520
111521 167280
167281 223040
223041 278800
278801 334560
334561 390320
390321 446080

ccc

The numbers at the end of your post are not correct. The correct (zero-based) numbers are generated by:

floats_per_file = 1382400 / 24
print('{} floats per file'.format(floats_per_file))  #  55760
for i in xrange(24):
    print(i*floats_per_file, (i+1)*floats_per_file - 1)

This code is similar to the code in SPLnFFT_hourly_split.py. Python's span syntax my_list[start_index : end_index] means that we do not need to loop over all elements but instead can just directly grab a block of data elements (such as an hour's worth of data) in a single operation. This helps to explain how we can break one binary file of 1m+ floats into 24 binary files in 1/3 of a second.

How did you know that 08h00 to 14h00 were the hours of interest?

Is there a mathematical test that I could run to determine which hours have useful data and which do not? Or must this decision be made by a human looking at a graph of the full day. Numpy supports a dizzying number of operations that can be applied to a ndarray if you can tell me what to look for. Alternatively, a human can tell the program which hours (minutes, seconds) are of interest and the file can be divided that way and rewritten as a smaller binary file.

The way to post an image to the forum is to put the image on an accessible web page (GitHub, Dropbox, etc.) and then add ?raw=1 to the end of the publicly accessible URL. Like ![](https://www.dropbox.com/s/00e5iyfealnuzxs/my_image.png?raw=1).

ManuelU

When you save raw data from th SPLnFFT app, you can save a 24 hours plot as well. You see the chunks of data as vertical cluttered lines, no vallies at all. This graph can be used as a hint about what time interval deserves to be analyzed in detail. The arrays are one based. Thanks for the info about sending images through the forum. I'll try it.

Regards.

By the way the time used for loading, saving and processing data is about 2 minutes per hour. I have a 10 MB Wifi, but a fiber 300 MB is coming.

ccc

OK. So there could be a Pythonista UI where the user would specify a start time and an end time. The script could use those times to determine which exactly which floats to copy from the original, full size binary file to the new, smaller binary file. Is that what you want? Do you want the Matplotlib graph too? Is start hour and end hour good enough or would you want to be able to specify minutes too?

I am still unclear why you transfer the files from one iOS device to another iOS device on the local WiFi network. Why not just do all the data capture and visualization on the first iOS device. The speed of the local WiFi network will not be improved by your move to fiber. Unless a new WiFi hub is included in the fiber upgrade.

ManuelU

Please try this. I hope I understood your instructions
https://www.dropbox.com/s/00e5iyfealnuzxs/spl_graphplot_test1.png?raw=1

ManuelU

It didn't work out. It gives a 404 error message. I wonder if I should save in a specific folder in Dropbox.
Respect to your question, it's simple. The Pythonista interpreter is the only App that can import from Dropbox the Binary Files saved by the SPLnFFT App. To avoid the obsolete iTunes File Sharing, that needs a desktop computer, cable conextions, etc., I use your FTP server to upload the files from the Pythonista''s sandbox to my BASIC interpreter sandbox. I'm pretty stone-headed and y don't understand why Apple don't allow the easiest way as the Open in ... Option available in other programs.

ccc

What is the name of the Basic Interpreter app that you use?

ManuelU

TechBASIC

ccc

SPLnFFT_strip.py removes any hours which only contain (0, 0) values from the start and end of a SPLnFFT.bin file. Much like ' 1 2 3 '.strip() returns '1 2 3'. It finds the first hour that has sound and the last hour that has sound and writes a new file that only has the data between those hours. This will result in smaller file sizes which should reduce the FTP transfer time and provide more focused plots/graphs.

ManuelU

OPEN "test_tr.bin" FOR OUTPUT AS #2
REM BEGIN THE FOR - NEXT LOOP TO READ HOUR CHUNKS
count = 1
startm = timeSTART(1)
endtm = timeEND(1)
PRINT "START DATA POINT "startm
PRINT "END DATA POINT "endtm
REM LOOPS UNTIL THE START DATAPOINT IS FOUND. NO WAY TO ACCESS THERE DIRECTLY
FOR k = 1 to endtm
GET #1,,a
IF k < startm THEN
    GOTO 200 REM LOOP
END IF
REM ***LOOK FOR INFINITE AN NAN SPL VALUES
isinf# = a
isnan# = a
i = math.isInf(isinf#)
j = math.IsNaN(isnan#)
IF (i <> 0) OR (j = 1) THEN
    PUT #2,,v REM dB VALUE*v = 33.33333 WHEN NAN OR INFINITES ARE DETECTED
badspl = badspl  + 1
GOTO 100

This is the BASIC code I use To clean up NAN and Infinites. This prevents you, if you clean up in block, to change the order of SLOW and FAST values when you read data in one column and then you transform to a 2D matrix
In en second time and before transforming the sequential values to a 2D matrix I look for the flag value 33.333333 and replace it with the logarithm average of the preceding 10 SPL values if n > 10 or the next when n is <= 10

ccc

Did you run SPLnFFT_strip.py? Does that do something useful for you or not?

Do you still get INFs and NaNs in files generated by the current SPLnFFT app? Can you please add an INF_counter and a NaN_counter to your basic program and tell me how many of each that you are seeing in files generated by the current SPLnFFT app?

JonB

Do you have the beta?

i created SPLView which is an interactive plot for this data.
https://github.com/jsbain/objc_hacks.git

In order to handle the problems with a million points in a plot, the data is resampled, so that there are only ever 500 points in a plot. When zoomed way out, each point of the fast represents the peak of the fast signal over about ~3 minutes, while the slow is resampled using a log average (this seems appropriate for what you are doing, since i think you are interested in how far the fast peaks are above slow average).

As you zoom, the data is continually resampled, to always keep 500 points onscreen, meaning as you zoom, each point starts represnting less and less time. This way, you can first find interesting peaks from the wide view, then zoom to an area of interest, and eventually get the full resolution data.

The code that does the resample is
here.

If you dont have the beta, let me know, it would be easy to convert the touch handling to a non-beta interface.

ManuelU

SPL DATA START READING AT 19:14:53 CEST
START DATA POINT 1
END DATA POINT 55760
NaN
SPL DATA ENDED READING AT19:18:27 CEST
TOTAL CHUNK LENGTH 2788
SOURCE FILE LENGTH 5306560
VALID DATA SAVED IN THE SANDBOX : 5576
BAD VALUES (NAN OR INFINITE) DETECTED : 1
FILE LOAD ENDED AT 19:18:27 CEST

This is the output OF a SPLNFFT FILE analyzed with my program . The wifi speed was AT THIS TIME OF 5.8 MB

ccc

Did you run SPLnFFT_strip.py? Does that do something useful for you or not?

ManuelU

Not yet, but of course is useful to process SPL data in the Pytonista enviroment. I'll try it with a full 24 record of a MD that works transporting critical patients in an Ambulance. I sent him the SPLnFFT App. The record will be done with an iPhone 6. I'll send you the results when they are available. Another practical application is to monitor at bed time people that snores since they're prone to sleep apnea that may be dangerous for health. The measures of the app are accurate and render the same SPL values of that registered with a professional class 2 SPL meter that I use to calibrate it. By the way there are a free sample for evaluation of techBasic, the techSampler. There you can observe how easy is to plot high quality graphics with pinch gestures.

ManuelU

JonB, If you're talking about a new beta update of phytonista the answer is no. If is about the template to compile the scripts into a stand alone app with Xcode, the answer is yes; I downloaded yesterday the Zip file that will be installed in a Mac mini with Yosemite when I solve the problem of repeated phishing attempts that my antivirus blocks. It seems it's related to the Apple Store ID . I also have read the code for resampling the SPLnFFT data to plot graphics with pinch gestures. Since I'm a beginner in Phytonista, It's complicated for me to understand how you link the SPLnFFT data file to this script.

ManuelU

CCC, this is going to make you happy. This is the processing time of a 15 hours continuos recording with SPLnWATCH, a child App of SPLnFFT. It records in the background with a minimal battery usage (about 13%)
Elapsed time (3 Starting scatter...): 0:00:01.277414
Elapsed time (2 Scatter): 0:00:19.745673
Elapsed time (1 Adornments): 0:00:19.818221
Elapsed time (0 plt.show() Done.): 0:02:09.151619

The graph output is nice an fit well to the recording tines. I wish I new how to send it to you because I'll try to make it with your latest option by hours chunks.
CONGRATULATIONS FOR YOUR GREAT SCRIPT.

ccc

I wish I [k]new how to send it to you

If you have a free login to GitHub, you should be able to go to https://github.com/cclauss/SPLnFFT_tools and click the "+" to add a new file to the repository. Provide a reasonable name with the proper extension (.png, .jpg, etc.). You will then have to click the "Propose new file" and then "Create pull request" button a few times.

ManuelU

I made a screenshot and sent the image to the Twitter account @olemoritz to send it to you. Do you have a Twitter account?. It's simpler for me

ccc

This is the image that I got from @ManuelU via Twitter.

It shows why we want to preprocess our SPLnFTT.bin data files with SPLnFFT_strip.py. The hours from 14h thru 24h contain all zeros. That script would remove those null hours which would

decrease file size
reduce upload time, and
give us higher resolution plots.

And here is a plot from his TechBasic program: