Load PIL image in scene with retina resolution

brumm

look for screen scale

upwart

I can see the screen scale.
But how do I put a Pillow image to a scene in pixels and not points?

mikael

@upwart, sorry, moving slowly, but just to confirm:

If you create and display the image as you show, image quality is not sufficient
If you use a full-scale image, only one quarter of the image is shown?
You are committed to using scene
Do you have some fps requirement?

brumm

PhotoTextV2 Class

upwart

@mikael
Here are the answers to your questions:

If you create and display the image as you show, image quality is not sufficient
Sometimes it is not sufficient and then double resolution would be helpful

If you use a full-scale image, only one quarter of the image is shown?
Yes, that's true.

You are committed to using scene
Yes, because I use the same code base for tkinter I have to rely on a PIL images that are shown in a loop.

Do you have some fps requirement?
Not really, because my package automatically corrects for missed frames, but as it is an animation, there are some performance requirements, for sure.

mikael

@upwart:

Here is one way. You can update the texture of self.bg in an update method.

Note that since we go via ui.Image, we could do this all in the ui module as well, and I am not sure if the performance meets your requirements.

This is not perfect (you see me fudging with the value 102), let me know how it works for you.

import scene
import Image, ImageDraw
import ui, io

class MyScene (scene.Scene):
    ims = None
    def setup(self):
      scale = scene.get_screen_scale()
      full_width = int(self.size[0]*scale)
      full_height = int(self.size[1]*scale)
      full_size = (full_width,full_height)
      im = Image.new("RGB", full_size, (0, 0, 0))
      draw = ImageDraw.Draw(im)
      
      # Draw lines to show we have every pixel
      for y_block in range(int(full_height/scale)):
        for y_in_block in range(int(scale)):
          colors = ('white','black','black')
          y = int(y_block*scale+y_in_block)
          draw.line(((0, y), (full_width,y)), fill=colors[y_in_block])
          
      # Draw lines to show we are seeing the whole picture
      draw.line(((102, 102),(full_width-102,102),(full_width-102,full_height-102),(102,full_height-102),(102,102)), fill=128)
      del draw
      
      with io.BytesIO() as fp:
        im.save(fp, 'PNG')
        img = ui.Image.from_data(fp.getvalue(),scale)
        self.bg = scene.SpriteNode(scene.Texture(img))
        self.add_child(self.bg)
        self.bg.position = self.size / 2

scene.run(MyScene())

upwart

@mikael
Thank you so much for the code you provided.
Based on that I have made a small proof of concept in which I let your rectangle shrink with one point on every draw().
I do that by just changing the texture of self.bg.
That works excellent.
I suppose due to the overhead of

        im.save(fp, 'PNG')
        img = ui.Image.from_data(fp.getvalue(),scale)

, I can reach just 4 frames per second.
Not ideal, but enough for me to try and implement it in my salabim package.

Thanks a lot.

upwart

@mikael
I just found out that if change the file format from PNG to BMP speeds up the animation significantly: from 4 to > 16 fps.

mikael

@upwart, excellent, thank you for the info. Do you know if the BMP format handles transparency?

JonB

How are you generating the image?

See https://forum.omz-software.com/topic/5155/real-time-audio-buffer-synth-real-time-image-smudge-tool

This has the fastest methods to update images although in a ui.View. you can get 60 fps or higher with a recent device.

Not sure if this would work with scene.

upwart

@JonB
The PIL image is created for each frame by combining several (sometimes hundreds or thousands) small PIL images. For non-retina resolution it is easy to show them in a scene event loop. For retina resolution I have to escape to a SpriteNode which is filled with a texture for each and every frame.

upwart

Now that I can display the built up picture (as a SpriteNode), I run into another problem. In my scene.draw method, I now update the SpriteNode's texture and then I want to draw some rectangles and texts with the scene_drawing primitives rect and text.
Unfortunately these do not show up at all, as if they are hidden behind the SpriteNode. I tried to set the SpriteNode's z_position to -1, but that doesn't help.
So my concrete question is: "How can I show the scene_drawing results in front of the SpriteNode?".

cvp

@upwart As I'm very far to be a Scene specialist, I'm not sure that scene_drawing is compatible with SpriteNode.
Perhaps, you could try path and ShapeNode.

Or perhaps a shader with SpriteNode, like

        img = ui.Image.named('test:Peppers')
        self.bg = scene.SpriteNode(scene.Texture(img))
        rect_shader = '''
          precision highp float;
          uniform sampler2D texture;
          varying vec2 v_tex_coord;
          void main( void ) {
           vec2 uv = v_tex_coord;
           vec4 rect = vec4(0.2, 0.3, 0.4, 0.5);
           vec2 hv = step(rect.xy, uv) * step(uv, rect.zw); 
           float onOff = hv.x * hv.y;	// off if in rect
           vec4 color = texture2D(texture, vec2(uv.x, uv.y)); // original color
           gl_FragColor = mix(color, color+vec4(1,0,0,0), onOff);texture2D(texture, vec2(uv.x, uv.y));
           }
           '''
        self.bg.shader = scene.Shader(rect_shader)

cvp

@upwart and this?

        self.bg = scene.SpriteNode(scene.Texture(img))
        self.red = scene.ShapeNode(ui.Path.rect(0, 0, 150, 150), 
                             parent=self.bg,
                             fill_color = (1,0,0,0.5),
                             position=(10,10))

upwart

@cvp
Well, that's maybe the way to go, although I prefer to stick with my old fashioned scene_drawing code.

Thanks just the same.

cvp

@upwart said:

Thanks just the same

What do you want to say?

mikael

@upwart, I experimented with subclassing Node with scene_drawing functions in the draw method, and putting that on top. No errors, no visible results. I guess the two modes do not mix.

mikael

@upwart, if you really really want to do it, you can create another Scene with the scene_drawing functions in the draw method, then include that as the scene of a SceneView that you place as a subview of of the main Scene’s view. Then all you need is some funky objc to make the upper Scene’s background transparent (thread in this forum).

Awful, but works.

These definitions at the top:

import objc_util

glClearColor = objc_util.c.glClearColor
glClearColor.restype = None
glClearColor.argtypes = [objc_util.c_float, objc_util.c_float, objc_util.c_float, objc_util.c_float]
glClear = objc_util.c.glClear
glClear.restype = None
glClear.argtypes = [objc_util.c_uint]
GL_COLOR_BUFFER_BIT = 0x00004000

This in the main Scene's setup method:

spc = scene.SceneView()
spc.scene = OntopScene()
spc.background_color = 'transparent'
spc.frame = self.view.frame.inset(100,100)
self.view.add_subview(spc)

Inset by 100 is so that I can still close the scene with the 'X'...

And your drawing code here:

class OntopScene(scene.Scene):
  def setup(self):
    self.view.objc_instance.glkView().setOpaque_(False)
  def draw(self):
    glClearColor(0, 0, 0, 0)
    glClear(GL_COLOR_BUFFER_BIT)
    scene_drawing.stroke(1,0,0)
    scene_drawing.rect(-50,-50,100,100)

JonB

@upwart so you are combining many PIL images into a single SpriteNode because thousands of spritenodes have a performance penalty?

Are all of the subimages continuously changing? Are these mini plots of some sort, or status icons? In other words is there a lot of repeated content? (Can you post a screenshot?)

There may be some low level ways to get from something like an io surface into a texture. Or, there may be ways to create your ui.Images or scene.Textures such that the PIL image can write directly into the buffer, which minimizes the amount of conversions needed.