Welcome!
This is the community forum for my apps Pythonista and Editorial.
For individual support questions, you can also send an email. If you have a very short question or just want to say hello — I'm @olemoritz on Twitter.
Scrapy
-
Has anyone tried to run Scrapy with Pythonista?
-
hi,i'm reviving this old post. i'm thinking about purchasing pythonista and wondering if it supports the scrapy lib?
curious if editorial's python interpretter would be able to do the same too?
if not scrapy, what other similar libs outside of bs4 would be available to me in either product?
thx!
-
@robopu You may be able to get it working with some modifications. The library itself is pure-Python, however some of its dependencies are not, and this might make it impossible. The dependencies:
- Twisted ✅- I was able to install and import this sucessfully. First, I had to manually install zope.interface after downloading it from here. Although this module contains native C code, which Pythonista can't handle due to Apple's restrictions, the only C code is labeled as "optimizations" so I'm hopeful that the library runs without it. After installing zope, I could install Twisted from here and import it without error.
lxml
⚠️ - I've found in the past that you can sometimes cheat the system on this one, simply by replacinglxml
with its slower cousin,xml
, in the library's code. So you may be able to make this work.- PyOpenSSL ❌ - This is not included in Pythonista and you won't have much luck installing it. It depends on the
cryptography
module for all of its functionality. Although @scj643 has been lobbying @omz to includecryptography
in Pythonista, we haven't seen it yet. cssselect
,queuelib
, andw3lib
✅ - these are pure-Python with no dependencies. They were all easy to install successfully.service_identity
❌ - this builds on PyOpenSSL, so if you can't run that you can't run this.
So, in short, you can almost install Scrapy, and it will likely be possible in the near future. Don't let this put you off buying Pythonista, it's awesome and incredibly valuable 👍
-
hey thx webmaster.. fwiw, i went ahead and used the natively supplied beautifulsoup and urllib instead. it worked well enough and meant i didn't have to mess around with dependencies.
thx
-
hey webmaster one quick followup. i'm new to python console in editorial as well as anything at the system level in iOS.
if a module isn't already included in editorial's stock python implementation, is it ok to just use pip to pull down whatever you need?
if not what should you use to get it locally?
also, do i have to worry about mucking up my local environment at all if i start downloading other modules to my device? i don't know if editorial's python console is sandboxed from the rest of iOS in any way in terms of these libraries.
thx
-
@robopu I'm not sure about Editorial. In Pythonista, you can use Pip through StaSh or put modules in the site-packages folder. I'll take a look for you, though.
-
@robopu Putting Python modules in Editorial is certainly difficult. Pythonista is easier. However, if you really want to:
Create a "site-packages" folder inside your local documents. This is already included in
sys.path
, so modules inside it can be imported.However, you can't create Python files inside Editorial. I used the Python console to create a "test.py" file inside "site-packages", just saying
print("Hello!")
.I had to restart editorial for the changes to take effect, but once I did I could
import test
.For real modules, you'd have to use
requests
to download it and possiblyzipfile
to decompress.The reason this is so hard to do is Apple's restrictions. It's much easier in Pythonista, though, since that's designed as a coding app.
-
@Webmaster4o you might want to check out https://forum.omz-software.com/topic/1438/own-python-modules
-
thx for the info / links. i'll refer to it in the future