FAQ

What does the pro version offer? Is the open source project still maintained?

Yes, the open source project is maintained as before.

In the pro version, we provide:

  • weekly update of targets

  • profiles for mobile browsers and apps

  • some private detection fields

  • http/3 fingerprints and proxy support

And, a better financial situation will help the open source version better maintained.

Why does the JA3 fingerprints change for Chrome 110+ impersonation?

This is intended.

Chrome introduces ClientHello permutation in version 110, which means the order of extensions will be random, thus JA3 fingerprints will be random. So, when comparing JA3 fingerprints of curl_cffi and a browser, they may differ. However, this does not mean that TLS fingerprints will not be a problem, ClientHello extension order is just one factor of how servers can tell automated requests from browsers.

Roughly, this can be mitigated like:

ja3 = md5(list(extensions), ...other arguments)
ja3n = md5(set(extensions), ...other arguments)

See more from this article and curl-impersonate notes.

Can I bypass Cloudflare with this project? or any other specific site.

Short answer is: it depends.

TLS and http2 fingerprints are just one of the many factors Cloudflare considers. Other factors include but are not limited to: IP quality, request rate, JS fingerprints, etc.

There are different protection levels for website owners to choose. For the most basic ones, TLS fingerprints alone maybe enough, but for higher levels, you may need to find a better proxy IP provider and use browser automation tools like playwright.

If you are in a hurry or just want the professionals to take care of the hard parts, you can consider the commercial solutions from our sponsors:

  • Yescaptcha, captcha resolver and proxy service for bypassing Cloudflare.

  • ScrapeNinja, Managed web scraping API.

For details, see the Sponsor section on front page.

I’m getting certs errors

The simplest way is to turn off cert verification by verify=False:

r = curl_cffi.get("https://example.com", verify=False)

ErrCode: 77, Reason: error setting certificate verify locations

On Windows, if your Python environment or CA bundle path contains non-ASCII characters (e.g. accents), libcurl may fail to open the CA file when passed as a narrow char*. curl_cffi now encodes file-path options (e.g. CAINFO, PROXY_CAINFO, SSLCERT) using the system’s preferred ANSI code page on Windows to ensure correct file access. This fixes most occurrences of error 77.

How to use with fiddler/charles to intercept content

Fiddler and Charles uses man-in-the-middle self-signed certs to intercept TLS traffic, to use with them, simply set verify=False.

ErrCode: 92, Reason: ‘HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)’

This error(http/2 stream 0) has been reported many times ever since curl_cffi was published, but I still can not find a reproducible way to trigger it. Given that the majority users are behind proxies, the situation is even more difficult to deal with.

I’m even not sure it’s a bug introduced in libcurl, curl-impersonate or curl_cffi, or it’s just a server error. Depending on your context, here are some general suggestions for you:

  • First, try removing the Content-Length header from you request.

  • Try to see if this error was caused by proxies, if so, use better proxies.

  • If it stops working after a while, maybe you’re just being blocked by, such as, Akamai.

  • Force http/1.1 mode. Some websites’ h2 implementation is simply broken.

  • See if the url works in your real browser.

  • Find a stable way to reproduce it, so we can finally fix, or at least bypass it.

To force curl to use http 1.1 only.

import curl_cffi

r = curl_cffi.get("https://postman-echo.com", http_version=curl_cffi.CurlHttpVersion.V1_1)

Related issues:

Packaging with PyInstaller

If you encountered any issue with PyInstaller, here are a list of options provided by the community:

Add the --hidden-import option.

pyinstaller -F .\example.py --hidden-import=_cffi_backend --collect-all curl_cffi

Add other paths:

pyinstaller --noconfirm --onefile --console \
    --paths "C:/Users/Administrator/AppData/Local/Programs/Python/Python39" \
    --add-data "C:/Users/Administrator/AppData/Local/Programs/Python/Python39/Lib/site-packages/curl_cffi.libs/libcurl-cbb416caa1dd01638554eab3f38d682d.dll;." \
    --collect-data "curl_cffi" \
    "C:/Users/Administrator/Desktop/test_script.py"

See also:

How to change the order of headers?

By default, setting impersonate parameter will bring the corresponding headers. If you want to change the order or use your own headers, you need to turn off that and bring your own headers. For stored fingerprint targets, you can also load and edit a Fingerprint object directly.

fingerprint = curl_cffi.get_fingerprint("edge_146_macos_26")
fingerprint.headers["User-Agent"] = "..."
requests.get(url, impersonate=fingerprint)

How to deal with encoding/decoding errors?

Use chardet or cchardet

>>> import curl_cffi
>>> r = curl_cffi.get("https://example.com/messy_codec.html")
>>> import chardet
>>> chardet.detect(r.content)
{'encoding': 'GB2312', 'confidence': 0.99, 'language': 'Chinese'}

Or use regex or lxml to parse the meta header:

<meta http-equiv="Content-Type" content="text/html; charset=gbk" />