Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
ObjCInstance eat memory
-
In a audio tap out program,just like this,but 1s eat 1M memory. when you disable buf = ObjCInstance(buffer), memory don't lost.
who can tell me how to solve this problem?
def processBuffer(self,buffer,when,cmd) buf = ObjCInstance(buffer) ... process buf process_block=ObjCBlock(processBuffer,restype=None,argtypes=[c_void_p,c_void_p,c_void_p,c_void_p]) engine.inputNode().installTapOnBus(0,bufferSize=4410,format=None,block=process_block)
-
Can you post the rest of the code for
processBuffer
?ObjCInstance
alone uses almost no memory, because a singleObjCInstance
only holds ac_void_p
and not much else. It's more likely that the memory leak is in the rest of your processing code. -
You will probably need to post a more complete example, and what you think the problem is. Note that there is no guarantee that you get the buffersize you ask for, and in fact anything under 0.375 seconds is probably ignored. Apparently some people have had luck modifying the bufferLength on the buffer object.. your results may vary.
My foray into AVAudioEngine was here
https://gist.github.com/jsbain/2cf4998949f49b58ff284239784e1561
and here
https://gist.github.com/jsbain/33d90edb232d8e825a8461c7aba23b95You need to carefully manage memory such that you create no reference loops or strong references that are going to hang around or require gc to untangle. Also, you need to finish in less time than the engine runs (on my machine it was about 2.666 Hz), otherwise your function might be called multiple times. If you are seeing a memory leak, one of the above assumptions is probably violated.
Are you trying to modify the samples and write them back out? Or just play with the samples?
-
@JonB test code
# -*- coding: utf-8 -*- from objc_util import * import ctypes,time,os,struct,array,console bufsize = 4410 # Audio Tap out input_type = 2 play_filename ='Daydream3.zip' Srv_Ctime,dWhen = 0,0 # # Get iOS System Memory Info # NSProcessInfo = ObjCClass('NSProcessInfo') NSByteCountFormatter = ObjCClass('NSByteCountFormatter') class c_vm_statistics(Structure): _fields_ = [('free_count', c_uint), ('active_count', c_uint), ('inactive_count', c_uint), ('wire_count', c_uint), ('zero_fill_count', c_uint), ('reactivations', c_uint), ('pageins', c_uint), ('pageouts', c_uint), ('faults', c_uint), ('cow_faults', c_uint), ('lookups', c_uint), ('hits', c_uint), ('purgeable_count', c_uint), ('purges', c_uint), ('speculative_count', c_uint)] c = ctypes.cdll.LoadLibrary(None) mach_host_self = c.mach_host_self mach_host_self.restype = c_uint mach_host_self.argtypes = [c_void_p] host_page_size = c.host_page_size host_page_size.restype = c_int host_page_size.argtypes = [c_uint, POINTER(c_uint)] host_statistics = c.host_statistics host_statistics.restype = c_int host_statistics.argtypes = [c_uint, c_int, POINTER(c_int), POINTER(c_uint)] host_port = c_uint() host_size = c_uint() page_size = c_uint() host_port = mach_host_self(None) host_size = c_uint(int(sizeof(c_vm_statistics) / sizeof(c_int))) host_page_size(host_port, byref(page_size)) vm_stat = c_vm_statistics() HOST_VM_INFO = c_int(2) # This is a c macro # Return System Used/Free memory (bytes) def Get_mem(): get_host_statistics = host_statistics(host_port, HOST_VM_INFO, ctypes.cast(byref(vm_stat),ctypes.POINTER(c_int)), ctypes.byref(host_size)) mem_used = (vm_stat.active_count + vm_stat.inactive_count + vm_stat.wire_count) * int(page_size.value) mem_free = vm_stat.free_count * int(page_size.value) return mem_used,mem_free # Get System Time curTime = ctypes.cdll.LoadLibrary(None) curTime.CACurrentMediaTime.restype=c_double # AVAudio Define AVAudioEngine=ObjCClass('AVAudioEngine') AVAudioSession=ObjCClass('AVAudioSession') AVAudioPlayerNode=ObjCClass('AVAudioPlayerNode') AVAudioFile=ObjCClass('AVAudioFile') AVAudioUnitEQ=ObjCClass('AVAudioUnitEQ') AVAudioMixerNode=ObjCClass('AVAudioMixerNode') AVAudioPCMBuffer=ObjCClass('AVAudioPCMBuffer') AVAudioFormat=ObjCClass('AVAudioFormat') AVAudioUnitEQFilterParameters=ObjCClass('AVAudioUnitEQFilterParameters') AVAudioSessionPortDescription=ObjCClass('AVAudioSessionPortDescription') AVAudioCompressedBuffer=ObjCClass('AVAudioCompressedBuffer') AVAudioConverter=ObjCClass('AVAudioConverter') AVAudioTime=ObjCClass('AVAudioTime') class AudioStreamBasicDescription(ctypes.Structure): _fields_=[('mSampleRate',ctypes.c_double),('mFormatID',ctypes.c_uint32),('mFormatFlags',ctypes.c_uint32),('mBytesPerPacket',ctypes.c_uint32),('mFramesPerPacket',ctypes.c_uint32),('mBytesPerFrame',ctypes.c_uint32),('mChannelsPerFrame',ctypes.c_uint32),('mBitsPerChannel',ctypes.c_uint32),('mReserved',ctypes.c_uint32)] # create AVAudio engine def setup(): error=ctypes.c_void_p(0) session=AVAudioSession.sharedInstance() session.setCategory('AVAudioSessionCategoryPlayAndRecord',error=ctypes.pointer(error)) if error: raise Exception('error setting up category') session.setActive(True, error=ctypes.pointer(error)) if error: raise Exception('error setting up session active') engine=AVAudioEngine.new() return engine # # Audio Tap # def processBuffer(self,buffer,when,cmd): global Srv_Ctime,dWhen # Record audio when time t_when = AVAudioTime.secondsForHostTime(ObjCInstance(when).hostTime()) # Record audio tap out time,and conver to 8 bytes Srv_Ctime = curTime.CACurrentMediaTime() dWhen = Srv_Ctime - t_when # get buffer cbuf = ObjCInstance(buffer) # ios 9 need this line,because installTapOnBus buffsize not effect,but ios 10 it's work. cbuf.frameLength = bufsize process_block=ObjCBlock(processBuffer,restype=None,argtypes=[c_void_p,c_void_p,c_void_p,c_void_p]) # # Audio Init # # open a mp3 player fileurl = nsurl(os.path.abspath(play_filename)) file = AVAudioFile.alloc().initForReading_error_(fileurl,None) bd=file.processingFormat().streamDescription() # AVAudioFormat: 0-other,1-PCM float32,2 PCM float64,3-PCM int16,4-PCM int32 #audioFormat=AVAudioFormat.alloc().initWithCommonFormat_sampleRate_channels_interleaved_(1,44100,2,False) audioFormat = file.processingFormat() # save file to buffer for loop player audioFrameCount = file.length() audioFileBuffer = AVAudioPCMBuffer.alloc().initWithPCMFormat_frameCapacity_(audioFormat,audioFrameCount) file.readIntoBuffer_error_(audioFileBuffer,None) # init Audio Engine engine=setup() # Mic ainput = engine.inputNode() # Player for mp3 player = AVAudioPlayerNode.new() engine.attachNode(player) # loop play audio files (so need using buff) #player.scheduleFile_atTime_completionHandler_(file,None,None) player.scheduleBuffer_atTime_options_completionHandler_(audioFileBuffer,None,1,None) # Mixer mixer = engine.mainMixerNode() # Connect devices if input_type == 1 or input_type == 3: engine.connect_to_format_(ainput,mixer,ainput.inputFormatForBus_(0)) if input_type == 2 or input_type == 3: engine.connect_to_format_(player,mixer,audioFormat) # install Tap on mixer mixer.installTapOnBus(0,bufferSize=bufsize,format=audioFormat,block=process_block) # start audio engin engine.prepare() engine.startAndReturnError_(None) # play mp3 music if input_type == 2 or input_type ==3: player.play() # UI Display Loop ui_cnt = 0 while 1: time.sleep(0.05) ui_cnt += 1 mem_used,mem_free = Get_mem() if (ui_cnt % 10) == 0: print 'Memory used:','{:,} K'.format(mem_used/1024),dWhen
-
@wolf71 And is your problem that you see a one time increase? Or that it continually increases? (Maybe post some sample output?). Keep in mind this way of getting memory just gets the amount allocated to the app, and I think it will not usually go down, even when you free memory - think of it as the peak heap size since launch. One-time increases each time you run a program are not that surprising, since the global clearing often leaves some garbage behind.
Try with a larger buffer, and without setting the buffer inside the processBuffer. The default would be approx. 16384. Also, delete the
cbuf.frameLength = bufsize
line, in case that is creating a strong reference.Are you on the beta, or app store version?
-
-
App store version.
-
when change tap out
cbuf = ObjCInstance(buffer) to
#cbuf = ObjCInstance(buffer)
the memory eat is stop.
- Run about 30min,the pythonista will crash.
-
-
What about if you set the buffer size to 16384, and try this for your processBuffer:
def processBuffer(self,buffer,when,cmd): cbuf=ObjCInstance(buffer) del cbuf
Also, what does the faulthandler show as the reason for crash? (see @dgelessus's fault handler and uncaught exception handler module) in his pythonista_startup repo. I have found that running out of memory results in an unceremonious crash without any uncaught exceptions, but there could be other reasons as well.
When you say the memory is getting eaten, can you be more specific? i.e. how fast is it increasing?
-
- add del cbuf and change bufsize = 4096*4, but not change.still eat memory.
2.memory lost speed: about 1M Bytes/s, very fast.
-
Ok, this is similar to the problems I was havinig trying to process images in real time.
I think the issue is that this code in
ObjCInstance
creates a strong reference to the object in all of the ObjCInstanceMethods.def __getattr__(self, attr): cached_method = self._cached_methods.get(attr, None) if not cached_method: if '_' in attr: # Old method call syntax cached_method = ObjCInstanceMethod(self, attr) else: # New syntax, actual method resolution is done at call time cached_method = ObjCInstanceMethodProxy(self, attr) self._cached_methods[attr] = cached_method return cached_method
I believe this should be referring to the weakref instead.
@omz, @dgelessus ... thoughts?I noiced if you manually run gc, the memory does go down...
-
Ah, that explains that. As a temporary hack, you could add something like
cbuf._cached_methods.clear() del cbuf
to manually break the reference cycles between the ObjCInstance and its methods. If this callback is time-sensitive like @JonB says, then this kind of hack might even be necessary, because a full
gc.collect()
run can take much longer than this manual cleanup.@JonB The method objects should probably keep a weakref to their owner, instead of a normal reference. (Making
self._cached_methods
aWeakValueDict
isn't an option, that would make the cache useless.) -
@dgelessus
Add this linecbuf._cached_methods.clear() del cbuf
but the memory keep lost.just like before.
-
@dgelessus
ObjCInstanceMethodProxy's do store a weakref, but ObjCinstanceMethods store a strong reference. Not sure if that was intentional. I tried fixing this in ObjCInstanceMethod, but then my block never was called, so perhaps i just missed something.What I found digging a bit deeper:
cbuf._cached_methods
here contains a single ObjCinstanceMethodProxy... ironically the one method that was called:retain
. : The method_cache does have a strong reference back to cbuf. clearing the cache does break the ref cycle, as evidenced by checking c.CFGetRefCount before and after del cbuf ( without clearing the cache, the objc refcount never decreases when deleting cbuf, since .release doesn't get called except when then b. Doing so, for me anyway, keeps the virtual_size from the kernel task info pretty constant. I have not tried the method wolf is using. -
@wolf71
https://gist.github.com/b926b5cbc1b7c7d5b2a94cf41dd8ec4c
I made a slight mod to your test code -- I download and use an mp3 file (since I don't know what Daydream.zip is), and I added the clear code which @dgelessus suggested.I also catch a KeyboardInterrupt and stop the engine, otherwise you will crash when running the script again.
I also added a delay after starting to report memory before playing the mp3, so you can get a "baesline".
The result was a few MB increase the first time I ran the script, eventually levelling off, and no increase the second. i included a dump of e console in the gist so you can see....
It is also possible that there were some bug fixes in the beta version that prevent the problem you are seeing.
-
@JonB Thanks you very much.
Are you using Pythonista 3 new Beta version?
I run on my iPad Pro 9.7 and Pythonista 3 (App Store Version).
cbuf._cached_methods.clear()
cbuf=[]can slow down memory lost speed, total 240s audio file play,eat about 20M memory.