Need help with retrieving html data with VBA

[brooksr1]

Body

This webpage lists events for an organization:
https://www.meetup.com/cflfreethought/events/
Currently, there are 33 events, but only 10 are displayed initially when the page is loaded.
The remaining 23 events are retrieved when downward scrolling occurs. This may be called dynamic loading of the data or webpage.
Data is needed for all 33 events.
This VB code returns the data for the first 10 events which can be parsed out using strJson.
Set ie = CreateObject("InternetExplorer.Application")
ie.Navigate html_webpage
strJson = ie.Document.body.innertext

The difficulty is trying to obtain data for events 11 to 33.

Some things that have been tried:

Me!EdgeBrowser1.ExecuteJavascript "window.scrollBy(0, 5000);"
Me!EdgeBrowser1.ExecuteJavascript "window.scrollTo(0, document.body.scrollHeight);"

Neither of these have worked. An html expert is needed to advise how to retrieve the missing data. Thanks!

Guidelines

• I have read and understood this category's guidelines before making this post.

[thedarckxx]

Updated VBA Code to Retrieve All Events

Sub RetrieveAllEvents()
    Dim ie As Object
    Dim strJson As String
    Dim totalEvents As Integer
    Dim lastHeight As Long
    Dim currentHeight As Long
    Dim scrollAttempts As Integer
    Dim maxScrollAttempts As Integer
    
    ' Create a new Internet Explorer application
    Set ie = CreateObject("InternetExplorer.Application")
    ie.Visible = True
    
    ' Navigate to the webpage
    ie.Navigate "https://www.meetup.com/cflfreethought/events/"
    
    ' Wait for the page to load completely
    Do While ie.Busy Or ie.ReadyState <> 4
        DoEvents
    Loop
    
    ' Initialize variables
    totalEvents = 0
    lastHeight = 0
    scrollAttempts = 0
    maxScrollAttempts = 10 ' Set a limit to avoid infinite loops
    
    ' Scroll until no new content is loaded
    Do While scrollAttempts < maxScrollAttempts
        ' Scroll down the page
        ie.Document.parentWindow.scrollBy 0, 1000
        
        ' Wait for new content to load
        Application.Wait Now + TimeValue("00:00:02") ' Adjust wait time if necessary
        
        ' Get the current height of the document
        currentHeight = ie.Document.body.scrollHeight
        
        ' Check if the height has changed
        If currentHeight = lastHeight Then
            ' No new content loaded, exit the loop
            Exit Do
        End If
        
        ' Update lastHeight for the next iteration
        lastHeight = currentHeight
        
        ' Increment the scroll attempt counter
        scrollAttempts = scrollAttempts + 1
    Loop
    
    ' Now retrieve the HTML content
    strJson = ie.Document.body.innerText
    
    ' Output the result to the Immediate Window (Ctrl + G to view)
    Debug.Print strJson
    
    ' Optionally, parse strJson to extract the event data
    ' Your parsing logic goes here
    
    ' Clean up
    ie.Quit
    Set ie = Nothing
End Sub

• Initialize Internet Explorer: The code creates an instance of Internet Explorer and navigates to the specified URL.

• Wait for Page Load: The code waits for the page to fully load before proceeding.

• Scroll Loop: The code uses a loop to scroll down the page by a fixed amount (1000 pixels).
  After each scroll, it waits for 2 seconds to allow new content to load. You can adjust this wait time based on your internet speed and the responsiveness of the website. It checks the height of the document. If the height does not change after a scroll, it indicates that no new content was loaded, and the loop exits.

• Retrieve HTML Content: After scrolling, the code retrieves the complete HTML content of the page.

• Output the Result: The HTML content is printed to the Immediate Window for inspection. You can implement your parsing logic to extract event details from strJson.

• Clean Up: Finally, the code quits the Internet Explorer instance and cleans up the object.

[brooksr1]

I appreciate the post but it is only brings in the first 10 events. Events 11 to 30 are missing in the JSON results. I ran the code, then searched the JSON data. It only contains the events 1 to 10. I tried the currentHeight = ie.Document.body.scrollHeight type code multiple times in the past and it hasn't worked. I don't know why.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐