How do you install using Scrapy?

Installing Scrapy¶

You can install Scrapy using easy_install or pip (which is the canonical way to distribute and install Python packages).

To install using pip:

To install using easy_install:

Platform specific installation notes¶

Windows¶

After installing Python, follow these steps before installing Scrapy:

• add the C:\python27\Scripts and C:\python27 folders to the system path by adding those directories to the PATH environment variable from the Control Panel.
• install OpenSSL by following these steps:
  1. go to Win32 OpenSSL page
  2. download Visual C++ 2008 redistributables for your Windows and architecture
  3. download OpenSSL for your Windows and architecture (the regular version, not the light one)
  4. add the c:\openssl-win32\bin (or similar) directory to your PATH, the same way you added python27 in the first step`` in the first step
• some binary packages that Scrapy depends on (like Twisted, lxml and pyOpenSSL) require a compiler available to install, and fail if you don’t have Visual Studio installed. You can find Windows installers for those in the following links. Make sure you respect your Python version and Windows architecture.
  • pywin32: http://sourceforge.net/projects/pywin32/files/
  • Twisted: http://twistedmatrix.com/trac/wiki/Downloads
  • zope.interface: download the egg from zope.interface pypi page and install it by running easy_install file.egg
  • lxml: http://pypi.python.org/pypi/lxml/
  • pyOpenSSL: https://launchpad.net/pyopenssl

Finally, this page contains many precompiled Python binary libraries, which may come handy to fulfill Scrapy dependencies:

Ubuntu 9.10 or above¶

Don’t use the python-scrapy package provided by Ubuntu, they are typically too old and slow to catch up with latest Scrapy.

Instead, use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.

• Project description
• Project details
• Release history
• Download files

Project description

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Built Distribution

Close

Hashes for Scrapy-2.7.0.tar.gz

Close

Hashes for Scrapy-2.7.0-py2.py3-none-any.whl

Scrapyis an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

To put it in a more simpler term, Scrapy is used to perform web scraping and web scraping is the process of data extraction from websites.

Data here can be anything ranging from text, images, videos, emails, phone numbers etc…(You get what I mean)

Take Google as an example: Everytime when you search something on Google search engine, under the hood it searches and scraps data from the almost the entire websites in the World Wide Web and returns you the search results.

All these happened in seconds. Amazing right?

I’ve always been wanting to learn web scraping and recently I have a project that requires this technique. Upon the recommendation by my close friend — Low Wei Hong and reading through his article — Scrapy or Selenium?, I started learning Scrapy.

And it’s fun!

In this article, I’ll share with you some simple yet practical guides on how to install Scrapy to Windows OS after struggling with some technicality. At the end of this article, I hope you’d find that helpful before stepping in to learn Scrapy.

Let’s get started!

How to Install Scrapy to Windows OS

1. Create a virtual environment

First thing first, it is highly recommended to create a virtual environment and install Scrapy in the virtual environment created. This is to avoid conflict with already-installed Python system packages (which could break some of your system tools and scripts)

In the terminal above, I created a virtual environment named virtualenv_scapy using conda create — name virtualenv_scrapy. Once you’re done with this step, you can use conda to activate the virtual environment.

But before we move to the next step, it is always a good idea to check which conda environment that you’re currently using by typing conda info --envs

Great. Now that I knew that I was using the original conda environment. We can verify if we have successfully changed to the virtual environment created later by using the same method.

2. Activate the virtual environment

To activate the virtual environment created, you just type conda activate virtualenv_scrapy. And we’ve also checked that we’re now using the virtual environment.

The next thing to do is to create a new folder, in this case I created a new folder called virtualenv_scrapy and changed my directory to the folder as shown below.

3. Install Scrapy via conda-forge channel

Though it’s possible to install Scrapy on Windows using pip, it is recommended to install Anaconda or Miniconda and use the package from the conda-forge channel, which will avoid most installation issues.

Since we already have Anaconda installed, we can directly install Scrapy with the code: conda install -c conda-forge scrapy

Once you’ve typed yto proceed with the installation of all the necessary packages you’ll start downloading and extracting all the packages as below.

4. Use Scrapy to create a new project

Bamm!

If you’ve reached this stage. Congratulations and you’ve successfully installed Scrapy to your local machine (Windows OS)!

And guess what? You can create a new project using Scrapy in no time by typing scrapy startproject demo_project as below.

Up to this point, you can definitely start playing around with Scrapy. Enjoy and have fun!

Final Thoughts

Thank you for reading.

I hope this article gave you some simple guides on how to install Scrapy to your local machine (if you’re using Windows OS) in the quickest way possible.

As always, if you have any questions or comments feel free to leave your feedback below or you can always reach me on LinkedIn. Till then, see you in the next post! 😄

About the Author

Admond Lee is currently the Co-Founder/CTO of Staq — the #1 business banking API platform for Southeast Asia.

Want to get free weekly data science and startup insights?

Join Admond’s email newsletter — Hustle Hub, where every week he shares actionable data science career tips, mistakes & learnings from building his startup — Staq.

You can connect with him on LinkedIn, Medium, Twitter, and Facebook.

Where is Scrapy installed?

How do you use Scrapy in Python?

How do you install Scrapy shells?

How do you use Scrapy in Linux?