Load PIL image in scene with retina resolution

upwart

@mikael
Thank you so much for the code you provided.
Based on that I have made a small proof of concept in which I let your rectangle shrink with one point on every draw().
I do that by just changing the texture of self.bg.
That works excellent.
I suppose due to the overhead of

        im.save(fp, 'PNG')
        img = ui.Image.from_data(fp.getvalue(),scale)

, I can reach just 4 frames per second.
Not ideal, but enough for me to try and implement it in my salabim package.

Thanks a lot.

upwart

@mikael
I just found out that if change the file format from PNG to BMP speeds up the animation significantly: from 4 to > 16 fps.

mikael

@upwart, excellent, thank you for the info. Do you know if the BMP format handles transparency?

JonB

How are you generating the image?

See https://forum.omz-software.com/topic/5155/real-time-audio-buffer-synth-real-time-image-smudge-tool

This has the fastest methods to update images although in a ui.View. you can get 60 fps or higher with a recent device.

Not sure if this would work with scene.

upwart

@JonB
The PIL image is created for each frame by combining several (sometimes hundreds or thousands) small PIL images. For non-retina resolution it is easy to show them in a scene event loop. For retina resolution I have to escape to a SpriteNode which is filled with a texture for each and every frame.

upwart

Now that I can display the built up picture (as a SpriteNode), I run into another problem. In my scene.draw method, I now update the SpriteNode's texture and then I want to draw some rectangles and texts with the scene_drawing primitives rect and text.
Unfortunately these do not show up at all, as if they are hidden behind the SpriteNode. I tried to set the SpriteNode's z_position to -1, but that doesn't help.
So my concrete question is: "How can I show the scene_drawing results in front of the SpriteNode?".

cvp

@upwart As I'm very far to be a Scene specialist, I'm not sure that scene_drawing is compatible with SpriteNode.
Perhaps, you could try path and ShapeNode.

Or perhaps a shader with SpriteNode, like

        img = ui.Image.named('test:Peppers')
        self.bg = scene.SpriteNode(scene.Texture(img))
        rect_shader = '''
          precision highp float;
          uniform sampler2D texture;
          varying vec2 v_tex_coord;
          void main( void ) {
           vec2 uv = v_tex_coord;
           vec4 rect = vec4(0.2, 0.3, 0.4, 0.5);
           vec2 hv = step(rect.xy, uv) * step(uv, rect.zw); 
           float onOff = hv.x * hv.y;	// off if in rect
           vec4 color = texture2D(texture, vec2(uv.x, uv.y)); // original color
           gl_FragColor = mix(color, color+vec4(1,0,0,0), onOff);texture2D(texture, vec2(uv.x, uv.y));
           }
           '''
        self.bg.shader = scene.Shader(rect_shader)

cvp

@upwart and this?

        self.bg = scene.SpriteNode(scene.Texture(img))
        self.red = scene.ShapeNode(ui.Path.rect(0, 0, 150, 150), 
                             parent=self.bg,
                             fill_color = (1,0,0,0.5),
                             position=(10,10))

upwart

@cvp
Well, that's maybe the way to go, although I prefer to stick with my old fashioned scene_drawing code.

Thanks just the same.

cvp

@upwart said:

Thanks just the same

What do you want to say?

mikael

@upwart, I experimented with subclassing Node with scene_drawing functions in the draw method, and putting that on top. No errors, no visible results. I guess the two modes do not mix.

mikael

@upwart, if you really really want to do it, you can create another Scene with the scene_drawing functions in the draw method, then include that as the scene of a SceneView that you place as a subview of of the main Scene’s view. Then all you need is some funky objc to make the upper Scene’s background transparent (thread in this forum).

Awful, but works.

These definitions at the top:

import objc_util

glClearColor = objc_util.c.glClearColor
glClearColor.restype = None
glClearColor.argtypes = [objc_util.c_float, objc_util.c_float, objc_util.c_float, objc_util.c_float]
glClear = objc_util.c.glClear
glClear.restype = None
glClear.argtypes = [objc_util.c_uint]
GL_COLOR_BUFFER_BIT = 0x00004000

This in the main Scene's setup method:

spc = scene.SceneView()
spc.scene = OntopScene()
spc.background_color = 'transparent'
spc.frame = self.view.frame.inset(100,100)
self.view.add_subview(spc)

Inset by 100 is so that I can still close the scene with the 'X'...

And your drawing code here:

class OntopScene(scene.Scene):
  def setup(self):
    self.view.objc_instance.glkView().setOpaque_(False)
  def draw(self):
    glClearColor(0, 0, 0, 0)
    glClear(GL_COLOR_BUFFER_BIT)
    scene_drawing.stroke(1,0,0)
    scene_drawing.rect(-50,-50,100,100)

JonB

@upwart so you are combining many PIL images into a single SpriteNode because thousands of spritenodes have a performance penalty?

Are all of the subimages continuously changing? Are these mini plots of some sort, or status icons? In other words is there a lot of repeated content? (Can you post a screenshot?)

There may be some low level ways to get from something like an io surface into a texture. Or, there may be ways to create your ui.Images or scene.Textures such that the PIL image can write directly into the buffer, which minimizes the amount of conversions needed.