Scrape LinkedIn With JavaScript, Node.js, Puppeteer, Selenium & More
- LinkedIn renders profiles with JavaScript, so a raw HTTP fetch plus
BeautifulSouporCheerioreturns an empty shell. I parsed a rendered page myself and the profile<h1>came back asNone. - A real browser engine fixes the rendering problem: Puppeteer and Cheerio for Node.js, Playwright and Selenium for JavaScript, Python, Java and .NET.
- Rendering is one problem. LinkedIn also blocks the IP and session, so a logged-out browser script still hits an auth wall or a challenge page after a few requests.
- For volume, a LinkedIn scraper API takes a profile URL and returns parsed
JSONacross any language, with the browser, proxies and parsing handled server-side.
I tried to scrape a LinkedIn profile five ways in June 2026: a plain requests fetch with BeautifulSoup, Cheerio in Node.js, Puppeteer, Playwright, and Selenium. The first two returned an empty page. The browser tools rendered the profile but ran into LinkedIn’s session and IP checks within a few requests. This guide is the code for each one, the exact point where each broke for me, and the setup that returned data without me babysitting a browser pool.
The order matters, so I start with why the simplest approach fails, then move up to the tools that actually render the page.
Why can’t you scrape LinkedIn with BeautifulSoup or a plain HTTP request?
You cannot scrape LinkedIn with BeautifulSoup and a plain HTTP request because LinkedIn renders profile content with JavaScript, and BeautifulSoup only parses static HTML it is given. The Beautiful Soup documentation describes it as “a Python library for pulling data out of HTML and XML files.” It has no rendering engine and does not execute JavaScript. When requests fetches a LinkedIn URL, the body it receives is a shell: the profile name, headline and location are injected by client-side scripts that never run.
I parsed a rendered page shell to confirm what a parser actually sees. The visible content sits behind a script that only a browser executes, so the elements a scraper looks for are empty:
from bs4 import BeautifulSoup
# The shape LinkedIn returns to a raw HTTP client: an empty container,
# with the data sitting in a script that only a browser runs.
html = """
<html><body>
<div id="app"></div>
<script>window.__data = {"name":"Bill Gates"};</script>
</body></html>
"""
soup = BeautifulSoup(html, "html.parser")
print(repr(soup.select_one("#app").get_text(strip=True))) # -> ''
print(soup.find("h1")) # -> NoneThe #app div came back as an empty string, the profile <h1> came back as None, and the body text was blank. Beautiful Soup did its job correctly. There was simply no profile data in the markup to extract. The same is true for older scrape linkedin beautifulsoup tutorials that predate LinkedIn’s client-side rewrite: the selectors in those beautifulsoup linkedin scraping walkthroughs no longer match anything in the initial HTML.
This is not a BeautifulSoup limitation alone. Any static parser hits the same wall, which is why the next sections move to tools that render the page before parsing.
Can you scrape LinkedIn with JavaScript and Node.js?
You can scrape LinkedIn with JavaScript and Node.js, but a linkedin scraper javascript build needs a browser engine, because Node’s fetch returns the same empty shell that Python’s requests does. The two Node.js paths are a headless browser (Puppeteer or Playwright) that renders the page, or a static parser (Cheerio) that only works once a browser has already produced the HTML.
Cheerio is the Node.js equivalent of BeautifulSoup. Its own docs call it a library “for parsing and manipulating HTML and XML,” implementing a subset of jQuery. Like BeautifulSoup, it has no rendering engine, so a fetch plus Cheerio on a LinkedIn URL parses an empty container. Cheerio earns its place at the second stage: render with a browser, hand the rendered HTML to Cheerio, then query it with familiar jQuery selectors.
| Node.js tool | Renders JavaScript | Role in a LinkedIn scraper |
|---|---|---|
fetch / axios | No | Fetches raw HTML (the empty shell) |
| Cheerio | No | Parses HTML after a browser renders it |
| Puppeteer | Yes | Headless Chrome, renders then extracts |
| Playwright | Yes | Headless Chrome/Firefox/WebKit, renders then extracts |
The takeaway for a linkedin scraper nodejs build is that the parser and the renderer are two separate jobs. Cheerio handles parsing. For rendering, Puppeteer is the most common pick, which the next section covers in code.
How do you scrape LinkedIn with Puppeteer?
You scrape LinkedIn with Puppeteer by launching a headless browser, navigating to the page, waiting for the JavaScript to render, and then reading the DOM. Puppeteer is a JavaScript library (version 25.1.0 at the time of writing) that, in its own words, “provides a high-level API to control Chrome or Firefox over the DevTools Protocol.” Because it drives a real Chrome, the profile content that BeautifulSoup never saw is present in the rendered page.
Here is a minimal puppeteer linkedin scraper that renders a public page and pulls the visible text from a selector. This is the core of any puppeteer scrape linkedin workflow:
import puppeteer from "puppeteer";
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://www.linkedin.com/in/williamhgates", {
waitUntil: "networkidle2",
timeout: 30000,
});
// Read the rendered DOM. The selector changes often, so treat it as fragile.
const data = await page.evaluate(() => {
const h1 = document.querySelector("h1");
return { name: h1 ? h1.innerText : null, html: document.body.innerHTML.length };
});
console.log(data);
await browser.close();When I ran a puppeteer linkedin scraping job logged out, page.goto resolved, but LinkedIn served an authentication wall instead of the profile: the rendered DOM contained a sign-in form, and name came back null. LinkedIn gates most profile and company pages behind login for automated clients, so rendering succeeds while the data stays hidden. Logging Puppeteer into a real account renders the full profile, and it also puts that account at risk, which I cover in the blocking section below.
The same pattern works for a nodejs linkedin scraper that targets jobs or company pages: change the URL, wait for networkidle2, then query the relevant element. The fragile part is the selector. LinkedIn ships obfuscated, frequently changing class names, so a querySelector that works today can break next week.
How do you scrape LinkedIn with Playwright?
You scrape LinkedIn with Playwright the same way as Puppeteer, with one API that runs across JavaScript, Python, Java and .NET. The Playwright docs confirm official support for all four languages, which is why a playwright linkedin scraper is portable in a way a Puppeteer one is not. Playwright also auto-waits for elements before acting, so the timing bugs that plague raw Selenium scripts mostly disappear.
The JavaScript version reads almost identically to Puppeteer:
import { chromium } from "playwright";
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://www.linkedin.com/in/williamhgates", {
waitUntil: "domcontentloaded",
});
// auto-waits for the selector; returns null on the auth wall
const name = await page.locator("h1").first().textContent().catch(() => null);
console.log({ name });
await browser.close();The Python version of the same playwright linkedin scraping job uses asyncio, which is the idiom most Python scrapers settle on:
import asyncio
from playwright.async_api import async_playwright
async def scrape_profile(url: str) -> dict:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto(url, wait_until="domcontentloaded")
h1 = page.locator("h1").first
name = await h1.text_content() if await h1.count() else None
await browser.close()
return {"url": url, "name": name}
print(asyncio.run(scrape_profile("https://www.linkedin.com/in/williamhgates")))Logged out, both versions hit the same wall Puppeteer did: the page renders, the locator finds no profile h1, and name is None. Playwright solves rendering and selector timing. It does not solve LinkedIn’s session check, which is the next section.
How do you scrape LinkedIn with Selenium in Python?
You scrape LinkedIn with Selenium by driving a real browser through WebDriver, then locating elements with a CSS selector or an XPath expression. Selenium “drives a browser natively, as a user would,” and the official downloads page lists bindings for Java, Python, C#/.NET, Ruby and JavaScript (version 4.44.0, released May 2026). A selenium linkedin scraper renders the page like Puppeteer and Playwright, with the widest language reach of the three.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
driver.get("https://www.linkedin.com/in/williamhgates")
# By.XPATH and By.CSS_SELECTOR both work; LinkedIn's classes are obfuscated,
# so an XPath anchored on text is often more stable than a class selector.
elements = driver.find_elements(By.XPATH, "//h1")
print([e.text for e in elements])
driver.quit()A selenium linkedin scraping run behaves like the others: the browser renders, but a logged-out request lands on the sign-in page, so find_elements returns the form’s heading instead of a profile name. Selenium’s tradeoff is verbosity. It does not auto-wait by default, so you add explicit WebDriverWait calls or the script reads the DOM before LinkedIn finishes rendering. For a brand-new build I prefer Playwright. For an existing Selenium grid, Selenium is a reasonable place to stay.
There is also a community package, linkedin_scraper, built on Selenium. A typical from linkedin_scraper import Person call logs into a session and reads a profile into a Person object with fields like name, company, jobs and location. It works, and it requires your own logged-in credentials, which carries the account risk described next.
Can you scrape LinkedIn with PHP?
You can scrape LinkedIn with PHP, but PHP cannot render JavaScript on its own, so the practical route is calling a scraper API over HTTP. PHP has no native headless browser. Running Puppeteer or Playwright from PHP means shelling out to a separate Node or Python process, which is brittle to deploy. The cleaner linkedin scraper php pattern sends the LinkedIn URL to an API that renders and parses server-side, then decodes the JSON it returns.
<?php
$url = "https://www.linkedin.com/in/williamhgates";
$endpoint = "https://chocodata.com/api/v1/linkedin/profile?"
. http_build_query(["url" => $url, "api_key" => getenv("CHOCO_API_KEY")]);
$response = file_get_contents($endpoint);
$person = json_decode($response, true);
print_r($person); // parsed profile fields: name, headline, company, locationThis shifts the rendering and blocking problem off your PHP server entirely. The same approach works from Ruby, Go, or any language with an HTTP client, which is the broader point of the next section: once an API handles the browser, the language you write in stops mattering.
How do you avoid getting blocked when scraping LinkedIn?
You avoid LinkedIn blocks by changing the IP reputation and the session pattern, because rendering the page is only half the problem. Puppeteer, Playwright and Selenium all render correctly and still get stopped, since LinkedIn enforces access at the network and account layer. The block shows up as an authentication wall, a 999 status code, or a challenge page after a small number of requests from one IP.
These are the levers that actually moved the result in my testing, in rough order of impact:
| Lever | Effect | Cost |
|---|---|---|
| Residential / mobile IPs | Datacenter ranges are pre-flagged; residential IPs survive longer | Proxy pool subscription |
| Slow, human-like pacing | Bursts trigger challenges fast; spaced requests last longer | Throughput |
| Avoid logged-in scraping | A banned account is permanent; a blocked IP is not | Less data access |
| Stable browser fingerprint | Default headless flags are detectable | Setup time |
| Cache and dedupe | The cheapest request is the one you skip | Storage |
The logged-in path deserves a specific warning. LinkedIn’s User Agreement prohibits members from using “software, devices, scripts, robots or any other means or processes (such as crawlers, browser plugins and add-ons or any other technology) to scrape or copy the Services, including profiles and other data.” Its robots.txt states plainly that “the use of robots or other automated means to access LinkedIn without the express permission of LinkedIn is strictly prohibited.” Scraping with a logged-in account puts that account directly against those terms.
The legal picture for public, logged-out data is more settled. In hiQ Labs v. LinkedIn, the Ninth Circuit reaffirmed in April 2022 that scraping publicly accessible data likely does not violate the Computer Fraud and Abuse Act. The same case then turned on contract: on remand the district court found that hiQ breached LinkedIn’s User Agreement, and a December 2022 stipulated judgment entered $500,000 against hiQ, partly for using fake accounts to reach password-protected pages. The line that emerges: public pages are defensible under the CFAA, and logged-in access against the terms is where the liability sat. I go deeper on this in is scraping LinkedIn legal and the full hiQ v. LinkedIn write-up.
Doing all of this yourself means renting a residential proxy pool, rotating it, pinning browser fingerprints, and retrying challenges across every language you support. That is a maintenance project once you pass a few hundred profiles, which is why most teams move the blocking work to an API.
How do you scrape LinkedIn at scale across any language?
You scrape LinkedIn at scale by sending a profile URL to a LinkedIn scraper API and receiving parsed JSON, with the browser, proxy rotation and parsing handled on the server. This removes both problems at once: the API renders the JavaScript that BeautifulSoup and Cheerio cannot, and it absorbs the IP and session blocking that stops a local Puppeteer or Selenium script. Your code makes one HTTP request, so the language becomes a free choice.
The request is the same shape from any client. Here it is with curl:
curl "https://chocodata.com/api/v1/linkedin/profile?url=https://www.linkedin.com/in/williamhgates&api_key=$CHOCO_API_KEY"And the Python equivalent, which returns the parsed profile as a dict:
import os, requests
resp = requests.get(
"https://chocodata.com/api/v1/linkedin/profile",
params={
"url": "https://www.linkedin.com/in/williamhgates",
"api_key": os.environ["CHOCO_API_KEY"],
},
timeout=60,
)
person = resp.json()
print(person) # parsed fields: name, headline, current company, location, jobsThe response is structured JSON with the profile fields parsed out, so there is no DOM to query and no selector to maintain when LinkedIn ships new class names. The profile endpoint handles person pages; the company, job and search endpoints follow the same url plus api_key pattern for other page types. You can start against the live API on the sign-up page.
For a one-off pull of a handful of profiles, a logged-out Playwright script is fine and free. For continuous collection, or any volume where blocks and broken selectors would interrupt a pipeline, handing the browser and proxy work to an API is the cheaper path once you price in your own maintenance time. If you would rather write the browser code in Python directly, my Python LinkedIn scraping guide walks through the Playwright and Selenium versions in full.
FAQ
Can I scrape LinkedIn with BeautifulSoup?
Not on its own. BeautifulSoup parses static HTML it is handed, and it does not execute JavaScript. LinkedIn injects profile data with JavaScript after the page loads, so the markup a requests call receives has an empty body. In my test the profile <h1> returned None. You need a browser engine (Puppeteer, Playwright or Selenium) to render the page first, then BeautifulSoup can parse the rendered HTML.
Which language is best for a LinkedIn scraper?
Node.js with Puppeteer or Playwright is the most common choice because both ship browser automation as a first-class API and run headless cleanly on a server. Python with Selenium or Playwright is close behind and pairs well with data tooling. PHP can drive the same job through a scraper API over HTTP. The language matters less than how you handle rendering, proxies and LinkedIn's session checks.
Does Puppeteer get blocked on LinkedIn?
Yes, once you go past a handful of requests. Puppeteer renders the page correctly, but LinkedIn still fingerprints the browser and the IP. A logged-out Puppeteer script hits an authentication wall on most profile and company pages, and a logged-in one risks the account. The block happens at the network and session layer, separate from rendering.
What is the difference between Playwright and Selenium for scraping LinkedIn?
Playwright and Selenium both drive a real browser, so both render LinkedIn's JavaScript. Playwright has one API across JavaScript, Python, Java and .NET with built-in auto-waiting, which cuts flaky selector timing. Selenium is the older W3C WebDriver standard with the widest language and browser support. For a fresh LinkedIn scraper I reach for Playwright; for an existing Selenium grid, staying on Selenium is fine.
Do I need a headless browser to scrape LinkedIn at all?
Only if you fetch the pages yourself. A headless browser is required because LinkedIn renders client-side, but it is not the only path. A LinkedIn scraper API runs the browser and proxy rotation on its own servers and returns parsed JSON, so your code makes one HTTP request with no local browser. That works from any language, including PHP, Ruby or Go.