Custom keyboard support

[Webmaster4o]

Dear @omz ,

It occurred to me recently how powerful it would be to have a module for building custom keyboards, introduced in iOS 8. I assume that this cannot be accomplished with objc_util and must be changed with the info.plist. However, judging by this guide it IS possible to have a keyboard in a containing app.

I was using reboard today, and it got me thinking. Reboard allows some simple actions to be performed on the text you type (searching for images, translating, etc) by typing a ⌘ character in front of the word you wish to perform an action on. This is useful at first, but suffers from the same problem as apps like Workflow. The developer has to put together actions individually over many updates, and the functionality is very limited at first.

I think it'd be great if pythonista provided some kind of framework for building custom keyboards. There could be a keyboard module that encompassed all the functions. You could build the interfaces with the ui module. The way I envision such a thing working is having a default keyboard that can be set by the user. Some of the functions from editor would be supported, for example getting selected text and changing text.

It would look something like this:

#Simple "keyboard" that capitalizes all text in the current field. 
import keyboard, ui

def capitalize(self, sender):
	text = keyboard.get_text()
	keyboard.set_text(text.upper())

kb = ui.View()
b = ui.Button()
b.action = capitalize
b.frame = (0, 0, 1024,200)
kb.add_subview(b)

keyboard.present(kb)

I could also envision a solution that used subclassing of a keyboard.Keyboard class.

Since (I think) an app can only have one keyboard at a time, there would have to be a possibility to have multiple pythonista keyboards available. This could be tackled as in the appex module, with a script picker, but you would have to switch to the pythonista keyboard and then to your desired keyboard, adding an extra step. It would be better to add a programmatic way of changing keyboards, so a user could build a keyboard that allows for the selection of other keyboards, if they so desired.

I think building your own custom keyboards would be an incredible feature for the app, and I'd like you to please consider adding it as a feature.

[omz]

Hey @Webmaster4o, sorry about the late response!

When iOS 8 came out, I tinkered a little bit (quite a lot actually) with custom keyboards, but my experiments never really left the rough prototype phase for a couple of reasons.

My idea was basically to make a "BYOK" ("build your own keyboard") app, separate from Pythonista, in which you could completely customize key layout, and trigger script actions, Editorial-like workflows, or text snippets via special keys or maybe abbreviations (similar to TextExpander)... By default, it would look similar to the standard keyboard, but you'd be able to rearrange all the keys, and change their mapping. I basically wanted the thing to be usable as a full replacement for the system keyboard... Unfortunately, it turns out that replicating the system keyboard's functionality is extremely difficult, in particular getting auto-correction reliable enough for an even halfway decent typing experience. Before you even get to stuff like predictive text, the system keyboard does quite a bit of magic that you don't really notice until it's gone, e.g. invisibly adjusting the touch targets of keys based on statistical language data (e.g. the "U" key invisibly gets larger after you type "Q", but that's just the simplest example – it also uses individual typing speed/accuracy data etc.). The kind of research that goes into touchscreen keyboards is actually quite fascinating, but the point is, the app that I had in mind turned out to be not realistically feasible for me.

Another source of frustration was that the custom keyboard APIs in iOS 8 were incredibly buggy and limited. I'm not quite sure about the bugginess in iOS 9 (I think the situation may have improved a bit), but as far as I know, the limitations are largely still the same, which means that the system keyboard can do things that custom keyboards just can't implement at all. Since the system keyboard gained new capabilities in IOS 9, these limitations are actually even more frustrating. For example, the new trackpad gestures are simply impossible to implement in a custom keyboard because you can only move the cursor left or right, not up or down... The thing I find most annoying is that switching from one keyboard to another takes far too long if you have a couple of keyboards installed, because from a custom keyboard, you can only switch to the next one, whereas on the system keyboard, you can hold the 'globe' key to switch to any keyboard directly (there's still no API to do that in a custom keyboard).

This last point was actually the major reason that I wanted to implement a "full" keyboard, not just a special-purpose one (like the various alternative Emoji/Unicode pickers). I just find it way too annoying to switch between keyboards, and while it's gotten a little better simply because the devices are faster now, I found that I just don't like even having third-party keyboards installed at all because once I switch to one of them accidentally, it takes forever to get back to the standard keyboard (made worse by the fact that third-party keyboards often have their "next keyboard" buttons in different places).

Aaaanyway, if I understood you correctly, the thing you have in mind would actually be closer to a "blank canvas" keyboard that you could program yourself, using the ui module etc. This would require a couple of changes in my Python modules because some system capabilities aren't available in a keyboard (app extension), but some of that is actually the same as for the action extension in the current beta... There are lots of details that I'm not sure about, e.g. what would happen with console output, how would text entry be handled (I guess there would need to be a "keyboard within the keyboard"?) etc... Also, the capabilities of Apple's APIs for custom keyboards (that I'd expose in a "keyboard" module or something like that) are probably more limited than you think. For example, it is not possible at all to get the currently selected text, you can basically only get the previous/next word (sometimes not even that, depending on the cursor position). You also cannot trigger actions like "copy", "cut", or "undo" in the edited document (pasting is possible though, by simply "typing" the text in the clipboard). Cursor movement is limited to left/right, and there isn't a way to know where the cursor currently is, relative to the current line or the whole document (so you couldn't implement things like "jump to the end of the line")...

Given all that, I do wonder how useful a thing like this would actually be. I admittedly have a hard time coming up with concrete use cases that aren't basically gimmicks, but I imagine you have some ideas there?

Wrapping up, I do think that a "blank canvas" custom keyboard, as part of Pythonista, would be feasible – I definitely don't think it's something I'd start working on before 1.6 is out (as you might have guessed), and I'm honestly not sure at this point if it would be useful for enough people to make it actually worth doing.

[Webmaster4o]

Thanks for the thorough response. The limitations were more than what I'd realized, and the predictive key-adjustment sounds like it'd be nightmarish to implement. That's why I think it'd be better to have a blank canvas, and let the user do much of the work. Some ideas I had were a keyboard that can search Wolfram|Alpha as you type, or that can search images or gifs.

Maybe it wouldn't be practical to have a replacement for the existing keyboard, maybe something that can't even type would be practical. You could build small things, like a keyboard that allowed you to browse your photos, pick one, and then paste an imgur link to that image that it created with requests.

I didn't think it'd be practical for 1.6, but it would be fun to have eventually.

[blmacbeth]

@omz I would very much like that keyboard app, if you ever get back around to it.