curl command

Options Cookies as cookies={} dict Drop headers Scrapy manages (Host, Connection, Accept-Encoding) Also support Scrapy < 2.13 (adds start_requests()) Add dont_filter=True

Browser impersonation Impersonate Use scrapy-impersonate in the spider (TLS fingerprint via curl_cffi)

The curl_cffi snippet always impersonates the browser picked here.

Scrapy shell

standalone test

How to convert a curl command to Scrapy

Open the page you want to scrape and open DevTools (F12), then the Network tab.

Reload the page and find the request that returns the data. Filtering by Fetch/XHR often finds the JSON API behind the page.

Right-click the request and choose Copy → Copy as cURL. Both the bash and Windows (cmd) formats work.

Paste it above. Run the Scrapy shell snippet to try the request, then copy the Spider into your project.

What you get

Scrapy shell snippet: builds the scrapy.Request and calls fetch(), so you can inspect the response straight away.

Runnable spider: a complete spider with a parse callback, plus an optional start_requests() for Scrapy older than 2.13.

curl_cffi script: replays the request with a real browser TLS fingerprint, to check whether the site blocks on fingerprinting rather than headers.

The right request class: FormRequest with a formdata dict for form posts, JsonRequest for JSON bodies, or a plain Request that sends the body byte for byte.

Browser impersonation: optional scrapy-impersonate settings, with a warning when the copied User-Agent doesn't match the browser you impersonate.

Honest warnings: curl features that can't be reproduced exactly, such as @file bodies, multipart uploads and digest or NTLM auth, are flagged instead of silently dropped.

FAQ

Is it safe to paste curl commands with cookies or tokens?

Yes. The converter runs entirely in your browser and nothing you paste is sent anywhere. The code is open source, so you can check, or download the repo and open index.html offline.

How is this different from scrapy.Request.from_curl()?

Request.from_curl() builds a request object at runtime and ignores options it doesn't understand. This tool gives you readable code to edit (headers, a cookies dict, form or JSON payloads), a full spider, a curl_cffi check script and warnings about anything that won't translate exactly.

Why does the request work in my browser but get a 403 in Scrapy?

Often the site checks the TLS or HTTP/2 fingerprint, not just headers. Run the generated curl_cffi script: if it succeeds while plain Scrapy fails, enable the scrapy-impersonate option for the spider.

Which Scrapy versions are supported?

The spider targets current Scrapy and its start() method. Tick “Also support Scrapy < 2.13” to add start_requests() for older versions. The generated code is tested against the real Scrapy, scrapy-impersonate and curl_cffi libraries.

Scrapy shell

Spider

curl_cffi standalone test

How to convert a curl command to Scrapy

What you get

FAQ

standalone test