How to Scrape Tweets from Twitter Using Python

Scraping tweets from X (formerly Twitter) using Python unlocks a goldmine of real-time data for market research, sentiment analysis, and content strategy. This guide breaks down exactly how to pull public tweet data into a format you can actually use, walking you through every step from setting up your environment to exporting your findings to a CSV file.

Why Scrape Tweets Anyway? The Marketing Advantage

Before writing a single line of code, it’s worth understanding why this is such a powerful technique for marketers, brand managers, and content creators. Raw data from X gives you a direct, unfiltered look into public conversations. You can use it to:

• Perform Competitor Analysis: What are people saying about your competitors? Scrape tweets mentioning their brand handle to find out what customers love, what they complain about, and where there might be gaps in their service you can exploit.
• Monitor Brand Sentiment: Track mentions of your brand, products, or campaigns. Are the conversations positive, negative, or neutral? This information is invaluable for managing your reputation and responding to customer feedback.
• Find User-Generated Content (UGC): Discover authentic posts from customers who are talking about your product. Scraping for your brand name or a specific campaign hashtag can help you find amazing content to reshare (with permission, of course).
• Identify Content Ideas: What questions is your target audience asking? By scraping tweets containing keywords or questions related to your industry (e.g., "how to do social media marketing?" or "best tool for content planning"), you can uncover topics your audience is eager to learn more about.
• Spot Emerging Trends: Track keywords and hashtags relevant to your niche to see what’s gaining traction. This helps you jump on trending topics and create resonant content before the conversation moves on.

In short, scraping tweets moves you from guessing what your audience wants to know to understanding what they’re actively talking about.

A Quick Word on Ethics and X's API

Things have changed on X. The once easily-accessible free API is now far more restrictive, making official access for large-scale data collection a costly endeavor. This has led many to use scraping libraries that access the same publicly available information you can see in a web browser, just programmatically.

When you're doing this, always remember to be a good digital citizen:

• Only Scrape Public Data: Never attempt to access private accounts or direct messages. Stick to what's visible to everyone.
• Don't Be a Nuisance: Avoid sending too many requests in a short period (mass scraping) as this can overload servers and get your IP address blocked. Be respectful of rate limits.
• Follow the Rules: X's Terms of Service have rules about data scraping. Be sure to review them and understand the guidelines. This tutorial is for educational purposes and responsible use.

With that ground rule set, let's get into the technical side of things.

Your Toolkit: Snscrape + Pandas

To pull this off, we'll use a fantastic Python library called snscrape. It's a scraper for social networking services that is particularly effective for pulling historical and recent public data from platforms like X without needing official API keys. It’s a great starting point for anyone new to data collection.

We'll also use pandas, the go-to library in Python for data manipulation and analysis. It allows us to neatly organize the scraped data into a structured format, like a spreadsheet, ready for you to analyze Twitter data, and easily save it to a file.

Step 1: Setting Up Your Python Environment

First, you need to get your computer ready. This involves installing Python and the necessary libraries. If you already have Python set up, you can skip to installing the libraries.

We’ll use pip, Python's package manager, to install snscrape and pandas. It’s highly recommended to do this within a virtual environment to keep your project dependencies separate from your system's Python installation.

Create and Activate a Virtual Environment (Optional but Recommended)

On macOS/Linux:

python3 -m venv twitter-scraper
source twitter-scraper/bin/activate

On Windows:

python -m venv twitter-scraper
.\\twitter-scraper\\Scripts\\activate

Install the Required Libraries

Now, with your environment active, run this command in your terminal:

pip install snscrape pandas

This command downloads and installs the versions of snscrape and pandas we need. You’re now ready to start scraping.

Step 2: Writing Your First Scraper

Let's start with a simple script. Our goal is to scrape the 10 most recent tweets that contain the hashtag "#SocialMediaMarketing". Create a new Python file (e.g., scraper.py) and add the following code:

import snscrape.modules.twitter as sntwitter
import pandas as pd

# Define the search query and the number of tweets to scrape
query = "#SocialMediaMarketing"
limit = 10
tweets = []

# Use the TwitterSearchScraper to get the tweets
for tweet in sntwitter.TwitterSearchScraper(query).get_items():
if len(tweets) == limit:
break
else:
tweets.append([tweet.date, tweet.user.username, tweet.rawContent])

# Create a DataFrame from the tweets list
df = pd.DataFrame(tweets, columns=['Date', 'User', 'Tweet'])

# Print the DataFrame to see the results
print(df)

Let's review what this script does:

1. We import the necessary modules from snscrape and pandas.
2. We set our search query (#SocialMediaMarketing) and a limit of 10 tweets.
3. We create an empty list called tweets to store our data.
4. The for loop iterates through the results from sntwitter.TwitterSearchScraper(query).get_items(). This is the part that does the actual scraping.
5. Inside the loop, we check if we've reached our limit. If not, we append the tweet's date, username, and raw content to our list.
6. Finally, we convert our list into a pandas DataFrame for a clean, table-like view and print it.

Run this script from your terminal (python scraper.py), and you should see the 10 most recent tweets related to social media marketing neatly printed out!

Step 3: Supercharging Your Search Queries

A simple keyword search is just the beginning. The real power comes from using X's advanced search operators within your snscrape query string. This lets you drill down to find exactly the information you need.

Here are some of the most useful operators for marketers:

Scrape Tweets from a Specific User

Want to see what a competitor or industry leader has been tweeting about?

query = "from:postbase"

Scrape Tweets Sent to a Specific User

Find public replies directed at a particular account.

query = "to:garyvee"

Scrape Tweets Within a Date Range

This is extremely useful for analyzing sentiment during a specific campaign or event.

query = "#productlaunch since:2023-10-01 until:2023-10-31"

Combine Operators for Powerful Filtering

Now, let's put it all together. Imagine you want to find all tweets from a specific marketing influencer about "AI" since the start of the year that received at least 50 likes.

query = "(from:username) AI since:2024-01-01 min_faves:50"

A handful of other useful operators include:

• min_retweets:10 (tweets with at least 10 retweets)
• min_replies:5 (tweets with at least 5 replies)
• lang:en (tweets in English)

Experiment by swapping these into your query variable to see how much control you have over the data you collect.

Step 4: Collecting More Data and Saving It to CSV

Printing data to the console is fine for a quick test, but it's not practical for analysis. The final step is to refine our script to collect a wider range of data points and save everything to a CSV file, which you can then open in Excel or Google Sheets.

Here’s the complete, final script that pulls more information (like replies, retweets, and likes) and saves it neatly.

import snscrape.modules.twitter as sntwitter
import pandas as pd
import time

# Create a list to store tweets
tweets_list = []

# Define search query and number of tweets to fetch
# Let's find tweets about 'content strategy' from the past year
query = "content strategy since:2023-01-01"
limit = 500

print(f"Starting to scrape for '{query}'...")

# Use TwitterSearchScraper to get our tweets
for i, tweet in enumerate(sntwitter.TwitterSearchScraper(query).get_items()):
if i >,= limit:
break

# Appending relevant information
tweets_list.append([
tweet.date,
tweet.id,
tweet.rawContent,
tweet.user.username,
tweet.likeCount,
tweet.retweetCount,
tweet.replyCount,
tweet.lang,
tweet.sourceLabel,
tweet.url
])

# Simple progress indicator
if (i+1) % 50 == 0:
print(f"Scraped {i+1} tweets...")

# Creating a DataFrame from the tweets list above
tweets_df = pd.DataFrame(tweets_list, columns=[
'Datetime',
'Tweet Id',
'Text',
'Username',
'Likes',
'Retweets',
'Replies',
'Language',
'Source',
'URL'
])

# Saving the DataFrame to a CSV file
file_name = "content_strategy_tweets.csv"
tweets_df.to_csv(file_name, sep=',', index=False)

print(f"\nScraping finished! Data saved to {file_name}.")


When you run this code, it will:

1. Scrape up to 500 tweets about "content strategy" from the last year.
2. Extract valuable metadata for each tweet, including engagement metrics and a direct URL.
3. Provide updates in the terminal as it works.
4. Save all the data into a CSV file named content_strategy_tweets.csv in the same directory as your script.

You now have a structured dataset ready for analysis. You can sort it by the number of likes to find the most popular content ideas, filter it by keywords to gauge sentiment, or identify influential users in the conversation.

Final Thoughts

Scraping public tweets with Python and snscrape is an accessible yet powerful way to gather real-time data for market research and content strategy inspiration. By using advanced search operators and organizing the output with pandas, you transform raw conversation into structured, actionable insights that can give you a genuine competitive edge.

Of course, collecting the data is just the beginning. The real work starts when you turn those insights into compelling content and manage its delivery across all your platforms. That’s an area we found particularly chaotic and frustrating, which is why we built Postbase. We designed it to be a clean, modern hub for social media management, focusing on today's reality of short-form video and multi-platform scheduling, without the clunky interfaces of older tools. Once you have your content ideas from your scraped data, our visual calendar and reliable scheduling help you execute that strategy flawlessly.