FAQ

Why does the JA3 fingerprints change for Chrome 110+ impersonation?

This is intended.

Chrome introduces ClientHello permutation in version 110, which means the order of extensions will be random, thus JA3 fingerprints will be random. So, when comparing JA3 fingerprints of curl_cffi and a browser, they may differ. However, this does not mean that TLS fingerprints will not be a problem, ClientHello extension order is just one factor of how servers can tell automated requests from browsers.

Roughly, this can be mitigated like:

ja3 = md5(list(extensions), ...other arguments)
ja3n = md5(set(extensions), ...other arguments)

See more from this article and curl-impersonate notes.

Can I bypass Cloudflare with this project? or any other specific site.

Short answer is: it depends.

TLS and http2 fingerprints are just one of the many factors Cloudflare considers. Other factors include but are not limited to: IP quality, request rate, JS fingerprints, etc.

There are different protection levels for website owners to choose. For the most basic ones, TLS fingerprints alone maybe enough, but for higher levels, you may need to find a better proxy IP provider and use browser automation tools like playwright.

If you are in a hurry or just want the professionals to take care of the hard parts, you can consider the commercial solutions from our sponsors:

  • Scraply, Cloud-based scraping platform.

  • Yescaptcha, captcha resolver and proxy service for bypassing Cloudflare.

  • ScrapeNinja, Managed web scraping API.

For details, see the Sponsor section on front page.

I’m getting certs errors

The simplest way is to turn off cert verification by verify=False:

r = requests.get("https://example.com", verify=False)

ErrCode: 77, Reason: error setting certificate verify locations

Install curl_cffi and its dependencies in a pure-ASCII path, i.e. no Chinese or any other characters out of the ASCII table in the path.

How to use with fiddler/charles to intercept content

Fiddler and Charles uses man-in-the-middle self-signed certs to intercept TLS traffic, to use with them, simply set verify=False.

ErrCode: 92, Reason: ‘HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)’

This error(http/2 stream 0) has been reported many times ever since curl_cffi was published, but I still can not find a reproducible way to trigger it. Given that the majority users are behind proxies, the situation is even more difficult to deal with.

I’m even not sure it’s a bug introduced in libcurl, curl-impersonate or curl_cffi, or it’s just a server error. Depending on your context, here are some general suggestions for users:

  • First, try removing the Content-Length header from you request.

  • Try to see if this error was caused by proxies, if so, use better proxies.

  • If it stops working after a while, maybe you’re just being blocked by, such as, Akamai.

  • Force http/1.1 mode. Some websites’ h2 implemetation is simple broken.

  • See if the url works in your real browser.

  • Find a stable way to reproduce it, so we can finally fix, or at least bypass it.

To force curl to use http 1.1 only.

from curl_cffi import requests, CurlHttpVersion

r = requests.get("https://postman-echo.com", http_version=CurlHttpVersion.V1_1)

Related issues:

Packaging with PyInstaller

If you encountered any issue with PyInstaller, here are a list of options provided by the community:

Add the --hidden-import option.

pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi

Add other paths:

pyinstaller --noconfirm --onefile --console \
    --paths "C:/Users/Administrator/AppData/Local/Programs/Python/Python39" \
    --add-data "C:/Users/Administrator/AppData/Local/Programs/Python/Python39/Lib/site-packages/curl_cffi.libs/libcurl-cbb416caa1dd01638554eab3f38d682d.dll;." \
    --collect-data "curl_cffi" \
    "C:/Users/Administrator/Desktop/test_script.py"

See also:

How to set proxy?

You can use the proxy parameter:

from curl_cffi import requests

requests.get(url, proxy="http://user:pass@example.com:3128")

You can also use the http_proxy, https_proxy, and ws_proxy, wss_proxy environment variables, respectively.

For explanation of differences between http_proxy and https_proxy, please see #6.

How to change the order of headers?

By default, setting impersonate parameter will bring the corresponding headers. If you want to change the order or use your own headers, you need to turn off that and bring your own headers.

requests.get(url, impersonate="chrome", default_headers=False, headers=...)