Getting pixel data from a Scene?

mkeywood

Hi,

Is there an equivalent of JavaScript getImageData to be able to get pixel data from a Scene?

In Javascript I would use something like:

var c = document.getElementById("myCanvas");
var ctx = c.getContext("2d");
var imgData = ctx.getImageData(10, 10, 50, 50);

And I’d have a lovely array of pixel data I could use.

Is there anything similar in Pythonista for data in a Scene?

Thanks

Martin

JonB

See
https://forum.omz-software.com/topic/5155/real-time-audio-buffer-synth-real-time-image-smudge-tool

This is a long thread dealing with both real time audio, and also image pixel buffer applications, but see the summary by @Mederic at the start. Then see his smidge tool, which makes use of an IOSurfaceWrapper, thar let's you use a numpy array to draw "directly" to the screen. It is actually not directly, but uses some low level magic to make screen operations relatively fast.

This would be used in a UI.View type application, but won't really work in scene.

In addition, scene does let you implement opengl shaders, so depending on what you are trying to do, that might get an option. (There may be a fast way to tie an opengl uniform to an IOSurface, but Im not certain)

mikael

You can snapshot scene.view. I think this will give you the visible area only.

def snapshot(view):
  with ui.ImageContext(view.width, view.height) as ctx:
    view.draw_snapshot()
    return ctx.get_image()

Then you can turn the image to PIL image for x,y access. I can dig out the code for this as well if this option looks appealing to you.

mikael

@mkeywood, found it:

import ui, io
from PIL import Image as PILImage

def snapshot(view):
  with ui.ImageContext(view.width, view.height) as ctx:
    view.draw_snapshot()
    return ctx.get_image()
    
def ui2pil(ui_img):
  return PILImage.open(io.BytesIO(ui_img.to_png()))
  
def pil2ui(pil_image):
  buffer = io.BytesIO()
  pil_image.save(buffer, format='PNG')
  return ui.Image.from_data(buffer.getvalue())
  
pil_image = ui2pil(snapshot(scene.view)).load()

print(pil_image[x,y])

crop_area = (left, upper, right, lower)
cropped_pil = pil_image.crop(crop_area).load()

ui_image = pil2ui(cropped_pil)

JonB

One thing about using PIL ... This probably won't support real time /60fps type stuff.

mkeywood

Thank you so much. This is excellent.
Really appreciate the help and the time you’ve taken to respond.
Thanks 😄😄