Here is a way to capture all notifications:
This could be modified to observe a specific object, selector, etc. I believe there is a notification when a view is shown, but i didnt see an obvious one when the X is pushed.
The confusing thing about what you are asking is that sheet is already basically modal, since it blocks anything else happening. in which case wait_modal might be what you want.
For truly non-modal "windows", you might consider the Overlay, found here. This lets you create resizable views that float, and you can use connect to register actions that get called when the view is closed.
Finally, it would be possible to delete the standard X, and replace it with your own button.
gives me the X button on ipad when presenting as sheet. This might be different on iPhone, and might be different if you have leftbuttonitems or rightbuttonitems defined.
But this lets you, for instance, hide the default X, and define your own using right_button_items.