How to Get Data from Twitter Using Python

Extracting data from Twitter is a genuine goldmine for understanding public conversation, tracking trends, or seeing how people feel about your brand. With a few lines of Python, you can tap into this real-time stream of information and pull out valuable insights. This guide will walk you through the entire process, step-by-step, from getting your developer access to fetching your first set of tweets and organizing them for analysis.

Why Bother with Twitter Data Anyway?

Before jumping into the code, it's worth seeing what's possible. Accessing X data isn't just a technical exercise, it’s a powerful tool for marketing, research, and strategy. For social media marketers and brand builders, the applications are immediate:

• Brand Monitoring: See what people are saying about your company or products in real time. Are they happy? Are they running into issues? This is your unfiltered feedback channel.
• Sentiment Analysis: Automatically gauge the overall feeling (positive, negative, neutral) surrounding a topic, campaign launch, or event.
• Competitive Analysis: Keep an eye on your competitors. See how well their campaigns are performing, how customers are reacting to their announcements, and find gaps in their strategy.
• Market Research: Identify emerging trends, discover customer pain points, and hear what your ideal audience is talking about. It’s like an always-on focus group.
• Find User-Generated Content: Discover amazing posts from customers using your product and get permission to share them as social proof.

In short, the data gives you an edge. It helps you move from guessing what your audience wants to knowing what they want because you can listen directly to their conversations.

Setting Up: What You’ll Need Before You Write a Line of Code

Getting set up is the most tedious part of the process, but once it's done, it's done. You just need three things: Python installed, a library to talk to the X API, and your unique API keys.

1. Get API Keys from the X Developer Portal

This is the biggest hurdle. X requires you to apply for a developer account to access its data. While they've changed the process and tiers over the years, the core steps remain similar. You'll need credentials to prove to the platform that your script is authorized to make requests.

Steps to Get Your Keys:

1. Apply for a Developer Account: Go to the X Developer Portal and sign in with your X account. You will need to apply for a developer account. This usually involves answering a few questions about how you intend to use the API. Be clear and honest - for example, state that you're using it for "analyzing brand mentions" or "academic research on public sentiment."
2. Create a Project and an App: Once approved, the portal will guide you to create a "Project." Think of a project as a container for your work. Inside that project, you'll create an "App." Name it something memorable, like "MyBrandMonitor."
3. Generate Your Authentication Credentials: This is the prize. Once your app is created, you can generate your authentication tokens. You'll need three specific tokens for a simple "read-only" application that just fetches data:
   • API Key: This is like your username.
   • API Key Secret: This is like your password.
   • Bearer Token: This is a special key for authenticating certain kinds of "app-only" requests, which is perfect for searching tweets. It's the simplest way to get started.

Super Important: Treat these keys like your passwords. Do not share them publicly or commit them to a public code repository like GitHub. Store them securely on your machine.

2. Install Python and the Tweepy Library

If you don't already have Python, head over to the official Python website to download the latest version for your operating system.

Next, you'll need a library to make working with the X API much, much easier. Instead of building HTTP requests from scratch, a library called Tweepy handles all the messy details for you. It's the community standard for this kind of work.

To install it, open your terminal or command prompt and run this simple command:

pip install tweepy

And that's it for the setup! With Python, Tweepy, and your API keys in hand, you’re ready to start pulling data.

Your First Script: Finding Recent Tweets About a Topic

Let's write a simple script to find the 100 most recent tweets that mention "social media marketing." The goal here is simple: authenticate your script, make a request, and print the text of the tweets it finds.

Step 1: Authenticate with Your Bearer Token

Create a new Python file (e.g., `twitter_search.py`). In this file, you'll import Tweepy and set up the client using the Bearer Token you generated.

import tweepy

# Replace this with your own bearer token
bearer_token = "YOUR_BEARER_TOKEN_HERE"

# Create a client instance
client = tweepy.Client(bearer_token)

# Quick check to see if it works
if client:
print("Authentication successful!")
else:
print("Authentication failed.")


Replace `YOUR_BEARER_TOKEN_HERE` with your actual token. If you run this file and it prints "Authentication successful!", you're officially connected to the API.

Step 2: Define Your Search Query and Fetch Tweets

Now for the fun part. The X API has a powerful query language. You can search for keywords, hashtags, mentions, and even filter by language or exclude retweets.

Let's build a query that looks for tweets containing the phrase "social media marketing," are in English, and are not retweets.

Here's the full script:

import tweepy

# Replace with your bearer token
bearer_token = "YOUR_BEARER_TOKEN_HERE"

client = tweepy.Client(bearer_token)

# Define our search query
# The query filters for English tweets ('lang:en') and excludes retweets ('-is:retweet')
query = '"social media marketing" lang:en -is:retweet'

# Use the search_recent_tweets method to get tweets from the last 7 days
# tweet_fields lets us request extra information about each tweet
# max_results controls how many tweets to return (max 100 per request)
response = client.search_recent_tweets(query=query, tweet_fields=["created_at", "public_metrics"], max_results=100)

# Check if the response contains any data
if response.data:
# Loop through the list of tweets
for tweet in response.data:
print(f"ID: {tweet.id}")
print(f"Text: {tweet.text}")
print(f"Created at: {tweet.created_at}")
print(f"Likes: {tweet.public_metrics['like_count']}")
print("-" * 20) # Separator for readability
else:
print("No tweets found for the query.")



When you run this, you'll see a stream of recent tweets related to social media marketing, complete with their text, creation time, and like count. You just successfully pulled live data from X!

Getting More Complex: Fetching a User's Timeline

Searching is useful, but sometimes you want to analyze the activity of a specific account - maybe a competitor or an industry influencer. This is also straightforward with Tweepy.

To do this, you first need the user's ID, not their handle (e.g., an ID like `2244994945` instead of `@twitterdev`). Thankfully, Tweepy can get the ID for you.

import tweepy

bearer_token = "YOUR_BEARER_TOKEN_HERE"
client = tweepy.Client(bearer_token)

# The username of the account you want to target
username = "twitterdev"

# Get the user's information, including their ID
user_response = client.get_user(username=username)
user_id = user_response.data.id

print(f"Found user ID for {username}: {user_id}")

# Now, fetch the user's most recent tweets using their ID
timeline_response = client.get_users_tweets(id=user_id, tweet_fields=["created_at", "public_metrics"], max_results=10)

# Print the tweets from their timeline
if timeline_response.data:
for tweet in timeline_response.data:
print(f"Tweet: {tweet.text}")
print(f"Retweets: {tweet.public_metrics['retweet_count']}")
print(f"Replies: {tweet.public_metrics['reply_count']}")
print("-" * 20)
else:
print(f"No tweets found for user {username}.")


This script gets the X Dev account's user ID and then pulls their 10 most recent tweets, showing metrics for each one. This method is incredibly useful for competitor monitoring or finding an influencer's most popular posts.

Organizing Your Data with Pandas

Printing data to your terminal is fine for a quick check, but for any real analysis, you need to structure it. This is where the Pandas library comes in. It’s the standard for data manipulation in Python and lets you organize your tweet data into a clean table (called a DataFrame) that you can easily save to a CSV file.

First, install Pandas if you don't have it:

pip install pandas

Now, let’s adapt our previous search script to save the results to a CSV file.

import tweepy
import pandas as pd

bearer_token = "YOUR_BEARER_TOKEN_HERE"
client = tweepy.Client(bearer_token)

query = '"social media marketing" lang:en -is:retweet'

response = client.search_recent_tweets(query=query, tweet_fields=["created_at", "public_metrics"], max_results=100)

# Create an empty list to store our processed tweet data
tweet_data = []

if response.data:
for tweet in response.data:
# For each tweet, create a dictionary of the interesting bits
tweet_info = {
'id': tweet.id,
'created_at': tweet.created_at,
'text': tweet.text,
'like_count': tweet.public_metrics['like_count'],
'retweet_count': tweet.public_metrics['retweet_count'],
'reply_count': tweet.public_metrics['reply_count'],
}
tweet_data.append(tweet_info)

# Create a pandas DataFrame from our list of dictionaries
df = pd.DataFrame(tweet_data)

# Save the DataFrame to a CSV file
df.to_csv("marketing_tweets.csv", index=False)

print("Successfully saved tweets to marketing_tweets.csv")


Run this script, and instead of printing to the console, it will create a file named `marketing_tweets.csv` in the same directory. You can open this file in Excel, Google Sheets, or any data analysis tool to sort, filter, and visualize your findings.

API Rate Limits and Other Reminders

As you build more advanced tools, keep a couple of things in mind:

• Rate Limits: The X API prevents you from making too many requests in a short period. For instance, you might only be able to make 15 search requests every 15 minutes. Tweepy can handle these limits gracefully in some cases, but if you're pulling huge amounts of data, you need to be aware of them. Read the API documentation for the specific limits of the endpoints you're using.
• Be Respectful: Always follow the X Developer Policy. Don't spam, respect user privacy, and be transparent about what your application does.
• Keep Your Keys Secret: It's worth repeating. Never put your API keys directly in a script you share. A common practice for more advanced projects is to use environment variables to store them securely.

Final Thoughts

This guide walked you through the full journey of connecting to the X API with Python, fetching tweets based on searches and user timelines, and structuring that data into a usable CSV file. Armed with these skills, you have a potent toolkit for gathering marketing intel, understanding customer sentiment, and tracking conversations on a massive scale.

Manually analyzing tweet data is incredible for deep insights, but managing your own social presence day-to-day requires dedicated tools for planning and execution. We found that most schedulers felt designed for a different era of social media, struggling with the short-form video and multi-platform reality marketers now live in. That's why we built Postbase. It's a clean, modern social media management tool that streamlines your content planning, scheduling, analytics, and community engagement across all major platforms, helping you get back to creating great content instead of wrestling with outdated software.