Wednesday, 22 October 2014

Getting scrapy to work on ubuntu

This post is for those who are trying to install scrapy on their ubuntu machines.

In order to install scrapy, the following components have to be installed
1)python 2.7
2)pip and setuptools
3)lxml
4)OpenSSl

When you install ubuntu, it also installs python 2.7.X,lxml and OpenSSL along with it (i've tested it atleast on ubuntu 12.04 to 14.04). In certain cases pip is not preinstalled , so you can install it like this:
sudo apt-get install python-pip

One thing that the docs do not talk about is installing the python development package. Trying to install scrapy leads to errors. You must install it prior to installing scrapy by:
sudo apt-get install python-dev


After this you just need to do a
 sudo pip install scrapy

One important point that I would like to mention is that people who connect to Internet by a proxy can still face some problem installing python packages.This problem has an answer here .
TL;DR append the E flag to sudo like sudo -E install scrapy. This makes use of the proxy settings to connect to the package library.

No comments:

Post a Comment