Limit + Offset cannot exceed 10000 for discover.py getHosts

[LarsenEric]

I am trying to determine the correct way to get all hosts when you have more then 10000.

Originally I started with https://github.com/CrowdStrike/falconpy/blob/main/src/falconpy/discover.py#L114 and was planning to use limit and offset to get all of the hosts back with paging. However, if we have 10000 hosts then I can't seem to figure out the proper way to get them all back.

I see the response here, #536, I am just trying to see if this would be an equivalent solution for this endpoint.

Thanks.

[crowdstrikedcs]

Hi @LarsenEric thanks for the question! Yes the refrenced sample here shows the usage of the query_devices_by_filter_scroll operationthis will loop theough each host in the environment.

Let us know if you have any questions on it's usage.!

[LarsenEric]

Okay, I am just trying to understand what each of these returns.
When I use query_hosts I get more back then when I use query_devices_by_filter_scroll

[crowdstrikedcs]

The differences lie in how pagination is managed. With query_hosts for each call you optionally provide a limit and offset as query paramaters. The sum of which cannot exceed 10000. For query_devices_by_filter_scroll you optionally provide a limit and offset as well however the offset is no longer a numeric value but instead a pagination token. In the referenced sample you can see for the first call to query_devices_by_filter_scroll we set the limit to 5000 (the max records to return per API call) and the offset value to an empty string. The code then loops through the avalaible resouces while counter is less than total indicating we still have more pages to retreive. Let's say for example we have 12000 devices, since we can get 5000 per API call it would take us 3 calls to get all the devices IDs.

def get_query_results(max_rows: int):
    """
    Performs a QueryDevicesByFilterScroll operation.
    Loops thru all results available.
    """
    # Higher than counter so our loop starts
    total = 1
    # Running counter
    counter = 0
    # List to hold all of the IDs returned by calls
    returning = []
    # Since the scrolling offset value is a string,
    # we will need to use a second variable to track
    # it's value when using this method
    offset_value = ""
    # A counter is used to keep track of our position in the
    # entire resultset.
    while counter < total:
        # Scrolling offset token, no maximum record count
        # We use the offset token string, not the offset integer
        result = falcon.query_devices_by_filter_scroll(
            sort="hostname|asc",
            limit=max_rows,
            offset=offset_value,
            # filter="platform_name:'Linux'"   # Uncomment to add filter
        )["body"]

        # Increment our counter by the value of our limit
        counter += max_rows

        # Retrieve the value of the offset. This will be a token string.
        offset_value = resultresult["meta"]["pagination"]["offset"]

        # This will be the same every time, overrides our init value of 1
        total = result["meta"]["pagination"]["total"]

        # Retrieve the list of IDs returned
        id_list = result["resources"]

        # Append this list to our running list of all IDs
        returning.extend(id_list)

    return returning

# Connect to the Hosts API
falcon = Hosts(client_id="API_CLIENT_ID_HERE",
               client_secret="API_CLIENT_SECRET_HERE"
               )

# Number of records to return per call, max is 5000
LIMIT = 5000

all_results_returned = get_query_results(LIMIT)

print(all_results_returned)

With either operation the the number of devices that match the provided filter is stored at ["meta"]["pagination"]["total"] when you use query_hosts how are you checking that the number of records you get back is greater than query_devices_by_filter_scroll perhaps you are not specifying a limit paramater?

[LarsenEric]

Could the difference be related to wether or not they are managed assets?

[crowdstrikedcs]

With these operations we only return data for managed assets. What sort of differences are you seeing in the operations?

[LarsenEric]

So in my little test environment I have 10 total assets:

• 3 managed assets
• 4 unmanaged assets
• 3 unsupported assets

When I use /discover/queries/hosts/v1 I get 10 returns
When I use /devices/queries/devices-scroll/v1 I get 3

[crowdstrikedcs]

Ah my apoligies, misread your original question you are using the Discover operations, ignore all I said about query_devices_by_filter_scroll 😄

With the Discover operations you'll use limit+offset as you mentioned and the sum of which cannot be greater than 10000 records. So we have to get a bit creative to pull all the records.

To get around this limitation I reccomend using a filter to reduce the total count below 10000. Based on your environment there may be a few different options for this filter, these are outlined here

For my usage I used a last_seen_timestamp and the polling logic goes something like this

1. Make an initial call with no filter and paginate through all records. Additionally sort the records using last_seen_timestamp.asc which will return records sorted in ascending order by when they were last seen.
2. For each page of 100 use the get_hosts operation to gather asset details.
3. On the last page (Limit 100 and offset 9900) for the last resource ID store it's last_seen_timestamp.
4. Make a new call to query_hosts now using the stored filter
5. Repeat this process until your count matches the toal.

It would be another option to use a filter that would break assets into groups based on their tags, product, or platform depending on your needs. I have found using last_seen_timestamp is a good way to get delta changes between runs.

[LarsenEric]

Thank you very much, I will give this a try.