Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Scraping Hidden Website Data
-
This could be general to any python API but I want to scrape data (numbers) from a betting site. Requesting an URL displays those numbers in Safari but when I inspect the HTML the data packages and their headers are not always there.
Some websites display the top item with its header and data. Then it seems the following items follow suit but without their header or data as though there is a general element within the string that infills what is wanted. I don't actually see the data.
Other websites hide all headers and data at all times
I see price comparison websites that seem to get around this but how is it done?
v
-
I find it is helpful to use developer tools on your desktop (F12 on chrome or IE/edge, I suspect safari has something similar), then select the element, and check the XPath.
Sometimes, sites may have elements populated via JavaScript, dynamically. In that case, you need to use a webview. @michael's wkwebview wrapper should let you do this easily.