[BUG] 爬取用户主页作品问题 #509

qingmou · 2024-11-22T12:33:06Z

发生错误的平台？

抖音

发生错误的端点？

Web APP

提交的输入值？

如：短视频链接

是否有再次尝试？

如：是，发生错误后X时间后错误依旧存在。

你有查看本项目的自述文件或接口文档吗？

如：有，并且很确定该问题是程序导致的。

爬取用户主页视频，web api是：/api/douyin/web/fetch_user_post_videos
例如：https://www.douyin.com/user/MS4wLjABAAAA3dRAmxkZfpdvYENgqTA0ewccirDkNEM4kBbUfqetEh8
爬取到的视频数量比实际数量少了，我检查找到了问题所在，如下：
问题出现文件在Douyin_TikTok_Download_API-main\crawlers\douyin\web\models.py的类class BaseRequestModel(BaseModel)中，
将

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "190500"
    version_name: str = "19.5.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Firefox"
    browser_version: str = "124.0"
    browser_online: str = "true"
    engine_name: str = "Gecko"
    engine_version: str = "122.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    # webid: str = "7388296161008862738"
    # downlink: int = 10
    # effective_type: str = "4g"
    # round_trip_time: int = 100
    msToken: str = TokenManager.gen_real_msToken()

修改为

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "290100"
    version_name: str = "29.1.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Chrome"
    browser_version: str = "130.0.0.0"
    browser_online: str = "true"
    engine_name: str = "Blink"
    engine_version: str = "130.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    downlink: str = "10"
    effective_type: str = "4g"
    from_user_page: str = "1"
    locate_query: str = "false"
    need_time_list: str = "1"
    pc_libra_divert: str = "Windows"
    publish_video_strategy_type: str = "2"
    round_trip_time: str = "0"
    show_live_replay_strategy: str = "1"
    time_list_query: str = "0"
    whale_cut_token: str = ""
    update_version_code: str = "170400"
    msToken: str = TokenManager.gen_real_msToken()

修改前获取到80+，修改后获取到110+，这样修改解决了问题。
以上在电脑测试。

Evil0ctal · 2024-11-22T18:34:48Z

嗯嗯，了解，我晚点去更新一下这个，感谢反馈！

9ihbd2DZSMjtsf7vecXjz · 2024-12-02T10:07:35Z

发生错误的平台？

抖音

发生错误的端点？

Web APP

提交的输入值？

如：短视频链接

是否有再次尝试？

如：是，发生错误后X时间后错误依旧存在。

你有查看本项目的自述文件或接口文档吗？

如：有，并且很确定该问题是程序导致的。

爬取用户主页视频，web api是：/api/douyin/web/fetch_user_post_videos 例如：https://www.douyin.com/user/MS4wLjABAAAA3dRAmxkZfpdvYENgqTA0ewccirDkNEM4kBbUfqetEh8 爬取到的视频数量比实际数量少了，我检查找到了问题所在，如下：问题出现文件在Douyin_TikTok_Download_API-main\crawlers\douyin\web\models.py的类class BaseRequestModel(BaseModel)中，将

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "190500"
    version_name: str = "19.5.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Firefox"
    browser_version: str = "124.0"
    browser_online: str = "true"
    engine_name: str = "Gecko"
    engine_version: str = "122.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    # webid: str = "7388296161008862738"
    # downlink: int = 10
    # effective_type: str = "4g"
    # round_trip_time: int = 100
    msToken: str = TokenManager.gen_real_msToken()

修改为

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "290100"
    version_name: str = "29.1.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Chrome"
    browser_version: str = "130.0.0.0"
    browser_online: str = "true"
    engine_name: str = "Blink"
    engine_version: str = "130.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    downlink: str = "10"
    effective_type: str = "4g"
    from_user_page: str = "1"
    locate_query: str = "false"
    need_time_list: str = "1"
    pc_libra_divert: str = "Windows"
    publish_video_strategy_type: str = "2"
    round_trip_time: str = "0"
    show_live_replay_strategy: str = "1"
    time_list_query: str = "0"
    whale_cut_token: str = ""
    update_version_code: str = "170400"
    msToken: str = TokenManager.gen_real_msToken()

修改前获取到80+，修改后获取到110+，这样修改解决了问题。以上在电脑测试。

求大佬指点如何用python请求这个api, 我需要通过作者主页地址让程序抓取作者最近更新作品的url

9ihbd2DZSMjtsf7vecXjz · 2024-12-02T10:38:11Z

发生错误的平台？

抖音

发生错误的端点？

Web APP

提交的输入值？

如：短视频链接

是否有再次尝试？

如：是，发生错误后X时间后错误依旧存在。

你有查看本项目的自述文件或接口文档吗？

如：有，并且很确定该问题是程序导致的。

爬取用户主页视频，web api是：/api/douyin/web/fetch_user_post_videos 例如：https://www.douyin.com/user/MS4wLjABAAAA3dRAmxkZfpdvYENgqTA0ewccirDkNEM4kBbUfqetEh8 爬取到的视频数量比实际数量少了，我检查找到了问题所在，如下：问题出现文件在Douyin_TikTok_Download_API-main\crawlers\douyin\web\models.py的类class BaseRequestModel(BaseModel)中，将

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "190500"
    version_name: str = "19.5.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Firefox"
    browser_version: str = "124.0"
    browser_online: str = "true"
    engine_name: str = "Gecko"
    engine_version: str = "122.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    # webid: str = "7388296161008862738"
    # downlink: int = 10
    # effective_type: str = "4g"
    # round_trip_time: int = 100
    msToken: str = TokenManager.gen_real_msToken()

修改为

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "290100"
    version_name: str = "29.1.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Chrome"
    browser_version: str = "130.0.0.0"
    browser_online: str = "true"
    engine_name: str = "Blink"
    engine_version: str = "130.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    downlink: str = "10"
    effective_type: str = "4g"
    from_user_page: str = "1"
    locate_query: str = "false"
    need_time_list: str = "1"
    pc_libra_divert: str = "Windows"
    publish_video_strategy_type: str = "2"
    round_trip_time: str = "0"
    show_live_replay_strategy: str = "1"
    time_list_query: str = "0"
    whale_cut_token: str = ""
    update_version_code: str = "170400"
    msToken: str = TokenManager.gen_real_msToken()

修改前获取到80+，修改后获取到110+，这样修改解决了问题。以上在电脑测试。

求指点怎么使用这个接口

qingmou · 2024-12-03T09:22:50Z

发生错误的平台？
抖音
发生错误的端点？
Web APP
提交的输入值？
如：短视频链接
是否有再次尝试？
如：是，发生错误后X时间后错误依旧存在。
你有查看本项目的自述文件或接口文档吗？
如：有，并且很确定该问题是程序导致的。
爬取用户主页视频，web api是：/api/douyin/web/fetch_user_post_videos 例如：https://www.douyin.com/user/MS4wLjABAAAA3dRAmxkZfpdvYENgqTA0ewccirDkNEM4kBbUfqetEh8 爬取到的视频数量比实际数量少了，我检查找到了问题所在，如下：问题出现文件在Douyin_TikTok_Download_API-main\crawlers\douyin\web\models.py的类class BaseRequestModel(BaseModel)中，将

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "190500"
    version_name: str = "19.5.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Firefox"
    browser_version: str = "124.0"
    browser_online: str = "true"
    engine_name: str = "Gecko"
    engine_version: str = "122.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    # webid: str = "7388296161008862738"
    # downlink: int = 10
    # effective_type: str = "4g"
    # round_trip_time: int = 100
    msToken: str = TokenManager.gen_real_msToken()

修改为

class BaseRequestModel(BaseModel):
    device_platform: str = "webapp"
    aid: str = "6383"
    channel: str = "channel_pc_web"
    pc_client_type: int = 1
    version_code: str = "290100"
    version_name: str = "29.1.0"
    cookie_enabled: str = "true"
    screen_width: int = 1920
    screen_height: int = 1080
    browser_language: str = "zh-CN"
    browser_platform: str = "Win32"
    browser_name: str = "Chrome"
    browser_version: str = "130.0.0.0"
    browser_online: str = "true"
    engine_name: str = "Blink"
    engine_version: str = "130.0.0.0"
    os_name: str = "Windows"
    os_version: str = "10"
    cpu_core_num: int = 12
    device_memory: int = 8
    platform: str = "PC"
    downlink: str = "10"
    effective_type: str = "4g"
    from_user_page: str = "1"
    locate_query: str = "false"
    need_time_list: str = "1"
    pc_libra_divert: str = "Windows"
    publish_video_strategy_type: str = "2"
    round_trip_time: str = "0"
    show_live_replay_strategy: str = "1"
    time_list_query: str = "0"
    whale_cut_token: str = ""
    update_version_code: str = "170400"
    msToken: str = TokenManager.gen_real_msToken()

修改前获取到80+，修改后获取到110+，这样修改解决了问题。以上在电脑测试。

求指点怎么使用这个接口

我只调用api接口，下载代码自己写的

qingmou · 2024-12-03T09:46:50Z

有api调用文档，根据文档写

jiucheng-kj · 2024-12-08T09:26:23Z

curl -X 'GET'
'https://api.douyin.wtf/api/douyin/web/fetch_user_post_videos?sec_user_id=MS4wLjABAAAANXSltcLCzDGmdNFI2Q_QixVTr67NiYzjKOIP5s03CAE&max_cursor=0&count=20'
-H 'accept: application/json'

{
"detail": {
"code": 400,
"message": "An error occurred.",
"support": "Please contact us on Github: https://github.com/Evil0ctal/Douyin_TikTok_Download_API",
"time": "2024-10-17 06:42:01",
"router": "/api/douyin/web/fetch_user_post_videos",
"params": {
"sec_user_id": "MS4wLjABAAAANXSltcLCzDGmdNFI2Q_QixVTr67NiYzjKOIP5s03CAE",
"max_cursor": "0",
"count": "20"
}
}
}
这种是什么问题

qingmou · 2024-12-11T10:06:13Z

curl -X 'GET' 'https://api.douyin.wtf/api/douyin/web/fetch_user_post_videos?sec_user_id=MS4wLjABAAAANXSltcLCzDGmdNFI2Q_QixVTr67NiYzjKOIP5s03CAE&max_cursor=0&count=20' -H 'accept: application/json'

{ "detail": { "code": 400, "message": "An error occurred.", "support": "Please contact us on Github: https://github.com/Evil0ctal/Douyin_TikTok_Download_API", "time": "2024-10-17 06:42:01", "router": "/api/douyin/web/fetch_user_post_videos", "params": { "sec_user_id": "MS4wLjABAAAANXSltcLCzDGmdNFI2Q_QixVTr67NiYzjKOIP5s03CAE", "max_cursor": "0", "count": "20" } } } 这种是什么问题

大概率是cookie问题，建议自己本地部署，使用自己的cookie

qingmou added the BUG Something isn't working label Nov 22, 2024

qingmou assigned Evil0ctal Nov 22, 2024

Evil0ctal added the enhancement New feature or request label Nov 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] 爬取用户主页作品问题 #509

[BUG] 爬取用户主页作品问题 #509

qingmou commented Nov 22, 2024 •

edited

Loading

Evil0ctal commented Nov 22, 2024

9ihbd2DZSMjtsf7vecXjz commented Dec 2, 2024

9ihbd2DZSMjtsf7vecXjz commented Dec 2, 2024

qingmou commented Dec 3, 2024

qingmou commented Dec 3, 2024

jiucheng-kj commented Dec 8, 2024 •

edited

Loading

qingmou commented Dec 11, 2024

[BUG] 爬取用户主页作品问题 #509

[BUG] 爬取用户主页作品问题 #509

Comments

qingmou commented Nov 22, 2024 • edited Loading

Evil0ctal commented Nov 22, 2024

9ihbd2DZSMjtsf7vecXjz commented Dec 2, 2024

9ihbd2DZSMjtsf7vecXjz commented Dec 2, 2024

qingmou commented Dec 3, 2024

qingmou commented Dec 3, 2024

jiucheng-kj commented Dec 8, 2024 • edited Loading

qingmou commented Dec 11, 2024

qingmou commented Nov 22, 2024 •

edited

Loading

jiucheng-kj commented Dec 8, 2024 •

edited

Loading