Scrapy

ihf

Has anyone tried to run Scrapy with Pythonista?

robopu

hi,i'm reviving this old post. i'm thinking about purchasing pythonista and wondering if it supports the scrapy lib?

curious if editorial's python interpretter would be able to do the same too?

if not scrapy, what other similar libs outside of bs4 would be available to me in either product?

thx!

Webmaster4o

@robopu You may be able to get it working with some modifications. The library itself is pure-Python, however some of its dependencies are not, and this might make it impossible. The dependencies:

Twisted ✅- I was able to install and import this sucessfully. First, I had to manually install zope.interface after downloading it from here. Although this module contains native C code, which Pythonista can't handle due to Apple's restrictions, the only C code is labeled as "optimizations" so I'm hopeful that the library runs without it. After installing zope, I could install Twisted from here and import it without error.
lxml ⚠️ - I've found in the past that you can sometimes cheat the system on this one, simply by replacing lxml with its slower cousin, xml, in the library's code. So you may be able to make this work.
PyOpenSSL ❌ - This is not included in Pythonista and you won't have much luck installing it. It depends on the cryptography module for all of its functionality. Although @scj643 has been lobbying @omz to include cryptography in Pythonista, we haven't seen it yet.
cssselect, queuelib, and w3lib ✅ - these are pure-Python with no dependencies. They were all easy to install successfully.
service_identity ❌ - this builds on PyOpenSSL, so if you can't run that you can't run this.

So, in short, you can almost install Scrapy, and it will likely be possible in the near future. Don't let this put you off buying Pythonista, it's awesome and incredibly valuable 👍

robopu

hey thx webmaster.. fwiw, i went ahead and used the natively supplied beautifulsoup and urllib instead. it worked well enough and meant i didn't have to mess around with dependencies.

thx

robopu

hey webmaster one quick followup. i'm new to python console in editorial as well as anything at the system level in iOS.

if a module isn't already included in editorial's stock python implementation, is it ok to just use pip to pull down whatever you need?

if not what should you use to get it locally?

also, do i have to worry about mucking up my local environment at all if i start downloading other modules to my device? i don't know if editorial's python console is sandboxed from the rest of iOS in any way in terms of these libraries.

thx

Webmaster4o

@robopu I'm not sure about Editorial. In Pythonista, you can use Pip through StaSh or put modules in the site-packages folder. I'll take a look for you, though.

Webmaster4o

@robopu Putting Python modules in Editorial is certainly difficult. Pythonista is easier. However, if you really want to:

Create a "site-packages" folder inside your local documents. This is already included in sys.path, so modules inside it can be imported.

However, you can't create Python files inside Editorial. I used the Python console to create a "test.py" file inside "site-packages", just saying print("Hello!").

I had to restart editorial for the changes to take effect, but once I did I could import test.

For real modules, you'd have to use requests to download it and possibly zipfile to decompress.

The reason this is so hard to do is Apple's restrictions. It's much easier in Pythonista, though, since that's designed as a coding app.

ccc

@Webmaster4o you might want to check out https://forum.omz-software.com/topic/1438/own-python-modules

robopu

thx for the info / links. i'll refer to it in the future