What Scrapewright's exported scrape.py actually contains

Where the file comes from.

You pick fields in the side panel; the panel infers a CSS selector for each one and finds the repeating item they all live inside. Export Python turns that recipe into a Playwright script and saves it. The filename is built from the host — this one arrives as scrape_quotes_toscrape_com.py — and the date it was generated is written into the first line, so a file you find in a downloads folder in six months can still tell you what it is.

Code generation happens in the extension, in your browser. The recipe is not uploaded anywhere to be compiled, because there is nowhere for it to go: the only host the extension ever contacts is the licence server, and the only thing it sends there is a key. The privacy page lists every request in the codebase.

Picking and previewing are free. Export Python is the one thing the £29 Pro licence unlocks.

What it needs, and how to run it.

Two things, neither of them ours: Python 3, and Playwright with a browser downloaded. The script's own docstring carries the commands, so the file explains itself to whoever opens it next:

Three details in those commands are deliberate, and worth knowing if you are typing them from memory instead:

• python3 and pip3, not python and pip. macOS ships no python at all, and Debian and Ubuntu have shipped the suffixed names without the plain aliases for years.
• python3 -m playwright rather than a bare playwright. pip drops that console script into whichever bin directory it installed to, which on a --user or Homebrew install is routinely not on your PATH. Going through the interpreter always finds the module you just installed.
• On Windows the launcher is py: py -m pip install playwright, then py -m playwright install chromium, then py scrape_quotes_toscrape_com.py.

If pip3 answers externally-managed-environment — Homebrew Python, or Debian and Ubuntu 23.04 and later — that is PEP 668 refusing to install into the system interpreter. Make a virtual environment and run all three commands inside it:

Every new terminal needs that activate line again before the script will run. On Windows it is .venv/Scripts/activate in place of the source line.

The run writes scrape_output.csv and scrape_output.json into the working directory and prints one line to standard output saying how many rows it got. Anything that went wrong on the way — a detail page that 404ed, a pagination click that a cookie banner ate — goes to standard error instead, so piping the run somewhere does not mix warnings into the result.

The file, block by block.

Printed in the order it is written. Line numbers refer to this example; a script with different picks has the same shape with different constants, and a different flow mode swaps one function.

The docstring and the constants

Everything you picked arrives here as a plain module-level constant. Changing the start URL, widening the page cap or fixing a selector after a site redesign is an edit to this block — not a re-pick, not a new export.

"""Scrape quotes.toscrape.com — generated by Scrapewright on 2026-09-09.

Setup (one time):
    pip3 install playwright
    python3 -m playwright install chromium

Run:
    python3 scrape_quotes_toscrape_com.py

If pip3 says "externally-managed-environment" (Homebrew, Debian, Ubuntu 23.04+),
create a virtual environment and run all three lines inside it:
    python3 -m venv .venv && source .venv/bin/activate
Every new terminal needs that activate line again before the script will run.

On Windows the launcher is py: py -m pip install playwright, then
py -m playwright install chromium, then py scrape_quotes_toscrape_com.py.
A venv activates there with .venv/Scripts/activate in place of the source line.

Output: scrape_output.csv and scrape_output.json in the working directory.
This file is yours — edit it freely. The constants below are the usual knobs.
"""

import csv
import json
import sys
import time

from playwright.sync_api import sync_playwright

START_URL = "https://quotes.toscrape.com/"
HEADLESS = True
OUTPUT_BASE = "scrape_output"
DELAY_SECONDS = 1.0  # pause between page loads; raise to be gentler
ITEM_SELECTOR = "div.quote"
FIELDS = ["quote", "author", "tags"]
NEXT_SELECTOR = "li.next a"
MAX_PAGES = 5  # set in the panel; change to None to scrape every page

HEADLESS = True is the one to reach for first when a run comes back empty: set it to False and the browser opens where you can watch it, which usually turns "no rows" into "oh, a cookie banner".

Reading one row

extract_item is your picks, one dictionary key per column, in the order you picked them. The helper functions under it are emitted only if the file uses them — a script with no text columns has no text() in it at all.

def extract_item(item):
    """Pull one row's fields from a single item element."""
    return {
        "quote": text(item, "span.text"),
        "author": text(item, "small.author"),
        "tags": text(item, "div.tags"),
    }


def rendered_text(node):
    """The text a node renders. inner_text() is HTML-only and raises on SVG and other
    non-HTML nodes, where text_content() asks the same question and answers it.
    """
    try:
        return node.inner_text().strip()
    except Exception:
        pass
    try:
        return (node.text_content() or "").strip()
    except Exception:
        return ""


def text(el, selector):
    node = el.query_selector(selector)
    return rendered_text(node) if node else ""

rendered_text exists because Playwright's inner_text() is HTML-only and raises on an SVG node, where text_content() answers the same question perfectly well. A column that reads an attribute instead gets an attr() helper; an image column gets one that walks src, then data-src, then srcset, because a lazy-loading site parks a placeholder in src for everything below the fold.

Turning the item selector into rows

def rows_on(page):
    """Every real item on the page in front of us.

    A row whose every column came back empty means the item selector matched something that is
    not an item — a page section sitting beside the real cards — so it is dropped rather than
    padding the output with blank lines.
    """
    rows = []
    for item in page.query_selector_all(ITEM_SELECTOR):
        row = extract_item(item)
        if any(row.values()):
            rows.append(row)
    return rows

One function that every flow mode calls, so they all drop junk identically. The any(row.values()) test is doing real work: item selectors routinely match one extra thing that looks structurally like a card and is actually a filter panel, and an all-empty row is how that announces itself.

Opening the start page

This is the block written for the failure you are most likely to meet, and it is longer than the scraping itself.

# The tail every start-page failure shares. A wall served to a fresh profile explains a
# navigation that hangs, a 403, and a document with no items in it equally well.
BLOCKED_ADVICE = (
    "A site will often serve a bot check or a cookie wall to a fresh headless profile "
    "instead of the page. Set HEADLESS = False to see what came back, and if it is a "
    "banner or a login, record a session with the storage_state note in main()."
)


def open_start_page(page):
    """Load the start page and wait for the items, explaining what went wrong before it bites."""
    try:
        # "domcontentloaded", not Playwright's default "load". The default waits for every
        # subresource the page pulls in, so one hanging tracker or ad pixel times the navigation
        # out thirty seconds after the markup — all of it, items included — already arrived.
        # Readiness here is "are the items there", and the wait below asks exactly that.
        response = page.goto(START_URL, wait_until="domcontentloaded")
    except Exception:
        print(f"Could not load {START_URL}: the document never arrived.", file=sys.stderr)
        print(BLOCKED_ADVICE, file=sys.stderr)
        raise
    # goto() raises on a transport failure but not on an HTTP one: 404, 403 and 500 are all
    # successful navigations to a page that says no. Unchecked, they fall through to the wait
    # below, cost its full timeout, and are then reported as something they are not.
    if response is not None and not response.ok:
        print(
            f"{START_URL} answered HTTP {response.status}, so the items never loaded.",
            file=sys.stderr,
        )
        print(
            "A 404 or 410 means START_URL is wrong or the listing has moved — open it in a "
            "normal browser and check. A 401, 403 or 429 is usually a wall rather than a "
            "verdict on the URL:",
            file=sys.stderr,
        )
        print(BLOCKED_ADVICE, file=sys.stderr)
        raise RuntimeError(f"HTTP {response.status} at {START_URL}")
    try:
        page.wait_for_selector(ITEM_SELECTOR)
    except Exception:
        print(f"Nothing matched ITEM_SELECTOR on {START_URL}.", file=sys.stderr)
        print(BLOCKED_ADVICE, file=sys.stderr)
        raise

A headless browser is a brand-new profile that has agreed to nothing and looks like a bot, so a real site will often answer the very first request with a consent wall or a soft block instead of the page. Without this block, what reaches you is a bare Playwright timeout with nothing in it that names a cause. Three different things get three different first lines — the document never arrived, the server answered with an error, or the document arrived without the items in it — because a message that fits all three fits none of them.

Note wait_until="domcontentloaded" rather than Playwright's default. The default waits for every subresource, so one hanging ad pixel can time the navigation out thirty seconds after the markup you actually wanted has already arrived.

Walking the pages

This is the only function that differs between flow modes. This one is pagination.

def scrape(page):
    """Walk every page via the next link, collecting items as we go."""
    open_start_page(page)
    rows = []
    pages_done = 0
    while True:
        rows += rows_on(page)
        pages_done += 1
        if MAX_PAGES is not None and pages_done >= MAX_PAGES:
            break
        next_link = page.query_selector(NEXT_SELECTOR)
        if next_link is None:
            break
        previous_url = page.url
        try:
            next_link.click()
            page.wait_for_load_state()
            page.wait_for_selector(ITEM_SELECTOR)
        except Exception as error:
            # Usually a consent banner over the link: a fresh profile has agreed to nothing, so
            # the overlay is there and it eats the click (see the storage_state note in main()).
            # The pages already walked are still yours; losing them to the last click is not.
            print(
                f"Stopped paginating: {error} — keeping the {len(rows)} rows collected so far.",
                file=sys.stderr,
            )
            break
        time.sleep(DELAY_SECONDS)
        # A next arrow that never disappears would loop forever, so stop if the URL
        # did not move. Site paginates without changing it? Delete this, set MAX_PAGES.
        if page.url == previous_url:
            break
    return rows

Two stopping conditions beyond the obvious one. MAX_PAGES is the cap you set in the panel, and setting it to None scrapes every page. The page.url == previous_url check is the guard against a next arrow that never disappears, which would otherwise loop forever on the last page. If your site paginates without changing its URL, delete that check and rely on the cap — the comment in the file says so, so you do not have to guess.

A click that fails does not lose the run. It prints what happened to standard error and returns the rows collected so far. The flow modes page prints the other three versions of this function.

Running it, and writing the files

def main():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=HEADLESS)
        page = browser.new_page()
        # Cookie banner or login wall? This browser starts with a clean profile that has
        # agreed to nothing, so plenty of sites — most UK and EU ones — cover the page with a
        # consent overlay that swallows every click, and a login-gated site never shows you
        # the content at all. Fix both the same way: record a session once with
        #     python3 -m playwright codegen --save-storage=auth.json "https://quotes.toscrape.com/"
        # dismiss the banner in the window that opens (and sign in, if you need to), close it,
        # then replace the new_page() line above with:
        #     page = browser.new_context(storage_state="auth.json").new_page()
        rows = scrape(page)
        browser.close()
    write_output(rows)
    print(f"Scraped {len(rows)} rows -> {OUTPUT_BASE}.csv and {OUTPUT_BASE}.json")


def write_output(rows):
    with open(f"{OUTPUT_BASE}.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)
    with open(f"{OUTPUT_BASE}.json", "w", encoding="utf-8") as f:
        json.dump(rows, f, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    main()

The comment inside main is the escape hatch for cookie walls and logins, and it is the same fix for both. Record a session once with playwright codegen --save-storage=auth.json, dismiss the banner (or sign in) in the window that opens, then swap the new_page() line for the new_context(storage_state=...) one. The generated file writes both the command and the replacement line for you, with your own start URL already in it.

Editing the selectors when the site changes.

Sites redesign, and a scraper that was working stops. There are exactly three places a selector can live in this file, and all of them are near the top:

• ITEM_SELECTOR — the repeating element one row is read from. If the run returns zero rows and the page clearly has items on it, this is almost always the one.
• The selector strings inside extract_item — one per column. If you get the right number of rows with one column empty, it is that column's selector.
• NEXT_SELECTOR — the next link or the load-more button, for the modes that have one. If page one comes back and nothing after it does, start here.

To find the replacement: open the page in Chrome, right-click the element, Inspect, then right-click the highlighted node in the elements panel and copy a selector. Prefer something stable and short — div.quote, h3 a — over the long auto-generated path, which encodes positions that will move again at the next redesign. You can test a candidate before editing anything by typing document.querySelectorAll("your.selector").length into the browser console; if that number matches the number of items you can see, the selector is right.

Re-picking in the panel and exporting again is also a perfectly good answer, and it is faster when several things moved at once. The point of the constants is that you are not forced to.

Putting it on a schedule with cron.

Scrapewright does not schedule anything — there are no scheduled runs in any tier, and no cloud for them to run in. What you have is an ordinary Python file, which your operating system already knows how to run on a timer.

On macOS or Linux, crontab -e and a line like this runs it every morning at six:

Four things in that line are there for a reason, and each is a common way to spend an evening debugging a cron job that works fine by hand:

• The absolute path to the interpreter. cron runs with a minimal environment and does not source your shell profile, so python3 may not resolve and an activated virtual environment certainly will not be. Pointing at .venv/bin/python directly sidesteps both.
• The cd first. The script writes its output into the working directory, which for cron is your home directory unless you say otherwise.
• The redirect. Both output files are opened in write mode, so each run replaces the last one's CSV and JSON; the log is where the row count and any warnings end up. If you want to keep history, either move the output aside after each run or set OUTPUT_BASE to something with a date in it.
• The browser download belongs to a user. playwright install chromium installs into that user's home directory, so run the cron job as the same user who installed it.

On Windows, Task Scheduler does the same job: a basic task on a daily trigger, the action being py with the script as its argument and "start in" set to the folder the script lives in.

Be a considerate scheduler. DELAY_SECONDS is the pause between page loads and raising it is the single politest thing you can do to a site you are reading regularly; hourly runs against a small site are usually a request nobody asked you to make.

What is not in the file.

Stated here so it is not a discovery you make at run time:

• No proxy rotation, no user-agent rotation and no CAPTCHA solving. If a site is actively refusing automated traffic, this script will be refused too, and it will tell you it was.
• No retries around a failed page. A detail page that errors is skipped with a note on standard error and the row keeps its listing fields; a failed pagination click ends the walk and keeps what it has.
• No concurrency. Pages are visited one at a time, on purpose.
• Nothing of ours. The imports are the standard library and playwright.sync_api. If Scrapewright disappeared tomorrow, the file would still run.

All four are editable, because it is a Python file and you have it. That is rather the point.