How to Scrape Reddit with Python and Scrapy

Reddit is a living, breathing forum of ideas, opinions, and trends. Every subreddit is a goldmine of posts, comments, and votes. If you can extract this data efficiently, you gain insights that can power analytics, market research, or content strategy. Today, we'll show you how to harness Python and Scrapy to scrape Reddit like a pro—step by step, with real-world applicability.

Why Scrape Reddit

Sometimes APIs don't exist—or they limit the data you need. That's where web scraping comes in. Reddit, for example, lets you view tons of content in your browser—but not everything is neatly packaged for download. By scraping:

You can collect post titles, upvotes, images, and timestamps.

Track trending topics or user sentiment.

Feed this data into analysis tools to generate actionable insights.

Python is perfect for this. Its simplicity, combined with libraries like Scrapy, allows you to scrape, store, and analyze data efficiently.

How Scrapy Works

Scrapy doesn't wait for each request to finish—it sends multiple requests asynchronously. If one request fails, the others continue. You can control concurrent requests per domain, implement auto-throttling, and set a depth limit to avoid endless scraping. This makes it fast and efficient, even for large subreddits.

Setting Up Your Environment

Before we dive in, let's make sure your system is ready.

Installing Scrapy

Option 1: Using pip

Scrapy requires Python 2.7 or 3.4+. Install it with:

pip install Scrapy

Scrapy relies on several dependencies:

lxml: Fast XML and HTML parsing

Parsel: Built on lxml, simplifies data extraction

w3lib: Handles URLs and encoding

Twisted: Enables asynchronous network programming

Cryptography and pyOpenSSL: Secure network connections

Option 2: Using Anaconda

Anaconda simplifies dependency management. Install Scrapy with:

conda install -c conda-forge scrapy

Anaconda simplifies the setup, avoids installation headaches, and ensures all dependencies work correctly out of the box.

Building Your Scrapy Project

Create a directory for your project:

cd <your-project-path>
scrapy startproject scraping_reddit

Explore the folder structure:

scraping_reddit/
├── scrapy.cfg
└── scraping_reddit/
    ├── __init__.py
    ├── items.py
    ├── middlewares.py
    ├── pipelines.py
    ├── settings.py
    └── spiders/
        ├── __init__.py

For this project, the key work happens in the spiders folder.

Building a Spider

A spider is your crawler—a Python script that navigates pages and extracts data.

cd scraping_reddit/scraping_reddit
scrapy genspider redditSpider https://www.reddit.com/r/cats/

This generates a template spider called redditSpider.

Analyzing the Reddit Page

Before coding, inspect the elements you want to scrape:

Title: //*[@class="_eYtD2XCVieq6emjKBH3m"]/text()

Images: //img[@alt="Post image"]/@src

Upvotes: //*[@class="_1rZYMD_4xY3gRcSS3p8ODO"]/text()

Date/Time: //*[@data-click-id="timestamp"]/text()

Scrapy's XPath selectors help you target exactly what you need.

Writing the Spider

Here's a fully working example:

# -*- coding: utf-8 -*-
import scrapy

class RedditspiderSpider(scrapy.Spider):
    name = 'redditSpider'
    allowed_domains = ['www.reddit.com']
    start_urls = ['https://www.reddit.com/r/cats/']

    custom_settings = {
        'DEPTH_LIMIT': 10  # limits recursive scraping
    }

    def parse(self, response):
        titles = response.xpath('//*[@class="_eYtD2XCVieq6emjKBH3m"]/text()').extract()
        imgs = response.xpath('//img[@alt="Post image"]/@src').extract()
        upVotesList = response.xpath('//*[@class="_1rZYMD_4xY3gRcSS3p8ODO"]/text()').extract()
        datetimes = response.xpath('//*[@data-click-id="timestamp"]/text()').extract()

        for (title, img, upVotes, datetime) in zip(titles, imgs, upVotesList, datetimes):
            yield {
                'Title': title.encode('utf-8'),
                'Image': img,
                'Up Votes': upVotes,
                'Date Time': datetime
            }

        next_page = response.xpath('//link[@rel="next"]/@href').extract_first()
        if next_page:
            yield response.follow(next_page, self.parse)

Running the Spider

Set the output format in settings.py:

FEED_FORMAT = "csv"
FEED_URI = "reddit.csv"

Run the spider:

scrapy runspider spiders/redditSpider.py

The CSV file will contain titles, images, upvotes, and timestamps, ready for analysis.

Data Analysis

Once the data is in CSV:

Use pandas to clean and structure it.

Generate charts of upvotes vs. post type.

Track trends over time with timestamps.

Build dashboards or reports for business insights.

Scrapy + Python transforms Reddit from a noisy forum into structured, actionable data.

In Summary

Scraping Reddit transforms scattered posts, upvotes, and timestamps into structured insights that drive smarter decisions. With Python and Scrapy, data can be collected efficiently, analyzed effectively, and visualized clearly—turning a noisy forum into a powerful source of trends, sentiment, and market intelligence. Whether for analytics, research, or strategy, mastering this workflow unlocks the true potential hidden within Reddit.