This Go project performs the following tasks:
- Takes in a list of company ticker symbols (e.g. AMZN for Amazon.com, Inc.)
- Sends HTTP requests to retrieve HTML from target web pages
- Extracts data on stock prices by specifying HTML elements
- Writes out the retrieved data to a .csv file
To run this project, you will need to install Go
Once you have Go installed on your local machine, run the following commands in the terminal:
git clone https://github.com/shosukenoma/stock-price-scraper.git
go get -u github.com/gocolly/colly/...
Run the following command in the terminal:
go run main.go
Once the program finishes running, you will see a stocks.csv file generated in the same directory.
This project is inspired by Web Scraping in Golang: Complete Guide 2023
This tutorial article walks us through how to scrape a mock e-commerce website that consists of a well-structured HTML.
Web Scraping for Stock Prices in Python
This article is helpful in getting a general sense of the project structure.
Building a web scraper in Go with Colly
This article accurately shows that c.Visit() should come after all of the callback functions, not before.
Programming in Go for Web Scraping
This article is helpful in understanding how to retrieve data from HTML elements in varying cases not mentioned in the Colly documentation.
Introduction to bufio package in Golang
This article is helpful in understanding the bufio package and the role of defer writer.Flush().
Banner Image
COPYRIGHT (C) 2021 BIDAALA STUDIO/SHUTTERSTOCK