Get Safari page source code in share extension.

lukaskollmer

Is there a way to access the HTML source code displayed in safari in the Pythonista share extension?

I know that I can get the url via the appex module and then use requests.get() to load the source code, but that won't work properly when the website requires a user to be logged in to display certain content.

Apparently, there is a way to do this in Objcective-C, by accessing the NSExtensionContext property of the share extension, but how can I accomplish that in Python?

ccc

https://forum.omz-software.com/topic/2582/get-web-source-code

lukaskollmer

@ccc that's not what I want. The problem with getting the url via appex.get_url() and loading the page content in the extension via some Python module is that it won't result in the same source code that s displayed in Safari.

EG: I'm on a website that requires the user to be logged in to view data. When I load the HTML source using the requests or urllib module, the returned page source will not contain the data you can only view when you're logged in. It will just contain the websites login mask asking you to enter your credentials to view your data

ccc

Requests allows you enter userid and password credentials via multiple authentication schemes http://docs.python-requests.org/en/master/user/advanced/#custom-authentication

Webmaster4o

@ccc I'm pretty sure that's still not what he's asking for. He's asking about how to use NSExtensionContent to reliably view the exact content that is loaded into Safari. As I understand it, this would allow:

Viewing content that is only available through authentication. I think he's talking about form-based authentication, not HTTP auth, which I think is possible through mechanize but not requests.
Viewing DOM nodes that have been manipulated by JavaScript through user interaction. Visiting this page anew would not include these modifications, because they resulted from explicit user actions

I'm sure there are other advantages to seeing the exact HTML that Safari is seeing.

JonB

Of course form based authorization can be done with requests! You just have to formulate the request correctly, which sometimes might be tricky to figure out. WebView is another way.

I have not tried the NSExtensionContext, but reading the docs you need
NSExtensionJavaScriptPreprocessingFile in the info.plist, and maybe one other related to weburls. Here is what is currently in to plist

NSExtension =     {
        NSExtensionAttributes =         {
            NSExtensionActivationRule = "SUBQUERY(extensionItems, $extensionItem, SUBQUERY($extensionItem.attachments, $attachment, (ANY $attachment.registeredTypeIdentifiers UTI-CONFORMS-TO \"public.url\" OR ANY $attachment.registeredTypeIdentifiers UTI-CONFORMS-TO \"public.image\" OR ANY  $attachment.registeredTypeIdentifiers UTI-CONFORMS-TO \"public.audio\" OR ANY $attachment.registeredTypeIdentifiers UTI-CONFORMS-TO \"public.text\")).@count > 0).@count > 0";
        };
        NSExtensionPointIdentifier = "com.apple.ui-services";
        NSExtensionPrincipalClass = PAESlidingContainerViewController;
    };

So, this won't work unless @omz adds this permission. I suspect that is a step apple would not be comfortable with.

omz

@JonB I've tried using the JS preprocessing when I originally built the app extension (that was still on iOS 8), but for some reason, I couldn't get it to work reliably. I'm not sure why exactly, but in some cases, having this key in the Info.plist resulted in getting no data at all in the extension. Given that there are app extensions that work with this pretty reliably (1Password...), it's probably either something I did wrong, or something that was fixed in the meantime. I might give it another shot sometime.

lukaskollmer

I tried to work around that by loading the page in a offscreen WKWebView, waiting for the delegate callback and then using the -evaluateJavascript:completionHandler method to fetch the html.

WKWebView has one major advantage: it saves cookies. This means that I only need to log in once and I don't need to display the webView for all following requests.

Problem: The -evaluateJavaScript:completionHandler method is block based and I can't get it to work (@omz @JonB do you have any idea what I'm doing wrong?)

@on_main_thread
def js_eval_completionHandler(_cmd, _obj_ptr, err_ptr):
    print('Did call completionHandler')

def webView_didFinishNavigation_(_self, _cmd, _webView, _navigation):
    webView = ObjcInstance(_webView)
    snippet_content_load_block = ObjCBlock(js_eval_completionHandler, restype=None, argtypes=[c_void_p, c_void_p, c_void_p])
    
    webView.evaluateJavaScript_completionHandler_('document.documentElement.outerHTML.toString()', snippet_content_load_block)

The js_eval_completionHandler gets never called

omz

@lukaskollmer Why not just use ui.WebView? It should save cookies too.

lukaskollmer

@omz I didn't know that ui.WebView can evaluate JavaScript as well

lukaskollmer

@omz is there a way to have ui.WebView load a URL without presenting the webView?

JonB

Webview also lets you use a few delegate methods which can be useful.

If you start mucking about with javascript/webviews, I suggest implementing a window.onerror, some console.log functions, and a corresponding delegate method to print to the console. This makes javascript work MUCH less mysterious. See for example here, the lines through the end of the delegate are my standard starting place for touching any javascript.
(edit: pointed to correct message)

JonB

w.load_url

the webview never needs to be presented to load.

lukaskollmer

@JonB I can't get it to load the page until I called present()

webView = ui.WebView()
webView.delegate = WebViewDelegate()
webView.load_url(url)
webView.present()

If I remove present(), the delegate won't get called

JonB

This works for me:

w=ui.WebView()
w.load_url('http://www.google.com')
src=w.eval_js('document.documentElement.outerHTML')

Possibly the delegate only gets called after it is presented? I seem to recall something like that. If so, you should be able to present it once, then close it, or present in a panel.

JonB

Actually this works fine for me. Perhaps your delegate or url has some issues. You also need to wait until,the page loads, either via javascript or using the delegate.

See for example:
https://gist.github.com/jsbain/80cdc7dd82da23cbe16c9befef91d707

This shows the delegate getting called without being presented.
When I tried to grab outerHTML directly at the end of the script, it has no data, because the page had not yet loaded. In the delegate webview_did_finish_load, it works fine.

Also, you could have a for loop that checks for document.readyState, to allow a more script-y sequence of commands. though you have to be a little careful because the readyState may go through complete two times initially.

lukaskollmer

@omz does ui.WebView share cookies between the main app and the extension?

JonB

does not look like it.
you can get cookies this way ( it would be possible to save this to a file which could be in a shared spot)

from objc_util import *
storage=ObjCClass('NSHTTPCookieStorage').sharedHTTPCookieStorage()
print(storage.cookies())

I have seen some stackoverflow code using NSKeyedArchiver to turn this into NSData

cookieData= ObjCClass('NSKeyedArchiver').archivedDataWithRootObject_(storage.cookies())

though i have not yet had luck in using NSKeyedUnarchiver to go from data back to an cookie storage object (just have not tried hard yet)

You can also get the binary cookie file from

cookiefile=re.findall(r'/private[^,]*',str(ObjCInstance(storage._cookieStorage()).description()))[0]
cookiedata=open(cookiefile).read()

jumpbeen

From Apple Developer website .

Accessing a Webpage

In Share extensions (on both platforms) and Action extensions (iOS only), you can give users access to web content by asking Safari to run a JavaScript file and return the results to the extension. You can also use the JavaScript file to access a webpage before your extension runs (on both platforms), or to access or modify the webpage after your extension completes its task (iOS only). For example, a Share extension can help users share content from a webpage, or an Action extension in iOS might display a translation of the user’s current webpage.

To add webpage access and manipulation to your app extension, perform the following steps:

Create a JavaScript file that includes a global object named ExtensionPreprocessingJS. Assign a new instance of your custom JavaScript class to this object.
In the NSExtensionActivationRule dictionary in your app extension’s Info.plist file, give the NSExtensionActivationSupportsWebPageWithMaxCount key a nonzero value. (To learn more about the activation rule dictionary, see Declaring Supported Data Types for a Share or Action Extension.)
When your app extension starts, use the NSItemProvider class to get the results returned by the execution of the JavaScript file.
In an iOS app extension, pass values to the JavaScript file if you want Safari to modify the webpage when your extension completes its task. (You use the NSItemProvider class in this step, too.)
To tell Safari that your app extension includes a JavaScript file, add the NSExtensionJavaScriptPreprocessingFile key to the NSExtensionAttributes dictionary. The value of the key should be the file that you want Safari to load before your extension starts. For example:

<key>NSExtensionAttributes</key>
        <dict>
            <key>NSExtensionJavaScriptPreprocessingFile</key>
            <string>MyJavaScriptFile</string> <!-- Do not include the ".js" filename extension -->
        </dict>

jaimeesc

I was able to get the source code of the current Safari web page using the following code:

req = appex.get_web_page_info()
print(req['html'])

From there I was able to parse through the HTML using BeautifulSoup.