fix(xhs): httpx 请求使用浏览器真实的 User-Agent 和 Client Hints - #986
Open
ChrisDoufu wants to merge 1 commit into
Open
ChrisDoufu wants to merge 1 commit into
ChrisDoufu wants to merge 1 commit into
Conversation
The API client and the media downloader used hardcoded headers: the API client sent a macOS Chrome 137 user agent with Windows Chromium 136 client hints, and media downloads sent a macOS Chrome 126 user agent. In CDP mode the page is the user's own Chrome, so one session reached Xiaohongshu with three different browser identities on the same cookies. Read navigator.userAgent and navigator.userAgentData from the page after it loads, and use them for both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
小红书的 API 客户端和媒体下载器用的是写死的请求头，和浏览器本身不一致：
create_xhs_client发送user-agent: ... Macintosh ... Chrome/137，同时发送sec-ch-ua: ... v="136"和sec-ch-ua-platform: "Windows"。这两组值本身就互相矛盾。_media_headers使用self.user_agent，即Macintosh ... Chrome/126。默认的 CDP 模式（
ENABLE_CDP_MODE = True，CDP_CONNECT_EXISTING = True）会复用用户自己的 Chrome。CDPBrowserManager._create_browser_context直接使用已有的 context，所以页面不会用self.user_agent，发出的是 Chrome 的真实 UA。结果是同一个会话、同一组 Cookie，以三种不同的浏览器身份访问小红书：页面、API 请求、媒体下载各不相同。这是风控很容易识别的特征。改动
新增
XiaoHongShuCrawler.read_browser_identity()。页面加载后，从页面读取navigator.userAgent和navigator.userAgentData，然后：self.user_agent，API 客户端和媒体下载器都使用它；sec-ch-ua、sec-ch-ua-mobile、sec-ch-ua-platform。如果浏览器没有navigator.userAgentData，就不发送这三个头。标准模式（非 CDP）不受影响：页面仍使用
launch_browser设置的 UA，httpx 请求现在也和它保持一致。测试
read_browser_identity()，确认 UA 和 Client Hints 与页面报告的值一致。tests/test_xhs_core_access_error.py、tests/test_xhs_raw_response_errors.py通过。🤖 Generated with Claude Code