Searching for Code, Web and API differ

[FHantke]

Searching for Code, Web and API differ

Hi all,

I am not sure if this is the correct repository to open this issue, but I did not find any other better fitting one.
I started playing around with the Git rest API today to search for code snippets. However, after a few searches, I realized that the API finds less files than the normal web search. Is there an option that prevents the search from showing all results?

My code is simply a request to the /search/code endpoint.

    headers = {
        'Accept': 'application/vnd.github+json',
        'Authorization': f'Bearer {GIT_API_KEY}',
        'X-GitHub-Api-Version': '2022-11-28',
    }

    params = { 'q': '"Hello World"' }

    res = requests.get('https://api.github.com/search/code', params=params, headers=headers)
    json_data = res.json()
    print(f"Total count: {json_data['total_count']}")

The API returns 817152 results, while using the GitHub web search return 4.6M.

I would expect to find the same number of files with both methods.

Thank you
Florian

[shiftkey]

@FHantke thanks for the feedback!

Unfortunately the new web search uses Blackbird (more details in this post), and because of changes to the search syntax and search results we have not yet upgraded the REST API to use this engine.

I can't be more specific about when this will occur, but keep an eye on the issue tracker in https://github.com/github/roadmap for a preview of this new API...

[FHantke]

Thank you for the quick answer 👍
I will keep an eye on it.

[joac]

I'm here 8 month later, having the same issue...
Will clone the repos locally and search them using grep...

[paulslaby]

So you say those search syntax is different. Do you have a documentation for API search syntax? I used https://docs.github.com/en/search-github/github-code-search/understanding-github-code-search-syntax#path-qualifier which is linked from here: https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28#constructing-a-search-query and it's not working.

I can live with syntax being different, but I need to know, what's possible.

Currently I am looking for query for finding files by names like: path:**/*.Dockerfile repo:XXX