Added new use case docs for Web Scraping, Chromium loader, BS4 transformer #8732
Conversation
trancethehuman
commented
Aug 4, 2023
trancethehuman
commented
- Description: Added a new use case category called "Web Scraping", and a tutorial to scrape websites using OpenAI Functions Extraction chain to the docs.
- Tag maintainer:@baskaryan @hwchase17 ,
- Twitter handle: https://www.linkedin.com/in/haiphunghiem/ (I'm on LinkedIn mostly)
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
the
🤖:docs
trancethehuman
changed the title
I guess I don't need to run |
| "source": [ | ||
| "# Web scraping using OpenAI Functions Extraction chain\n", | ||
| "\n", | ||
| "Web scraping is challenging for many reasons; one of them is the changing nature of modern websites' layouts and content, which requires modifying scraping scripts to accommodate the changes.\n", |
| "id": "5ef7f514", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Create a simple scraper function\n", |
There was a problem hiding this comment.
OK, this is cool.
1/ Let's create a new web loader for Chromium that wraps this logic.
2/ It can follow what we did here:
In particular, see.
Simply move this code to create a new loader (e.g., chromium_loader.py or similar).
Will launch a headless instance of Chromium to scrape.
I ran what you have and compared to this. (See docs here):
loader = AsyncHtmlLoader(url)
docs = loader.load()
html2text = Html2TextTransformer()
docs = html2text.transform_documents(docs)
html_content=docs[0].page_content
I found found that Chromium is better in this case.
For some reason, html2text is loosing the news article summaries.
We should add it as a new web loader and simply import here.
There was a problem hiding this comment.
Nice. Will change mine
There was a problem hiding this comment.
Just took care of this for you!
| "\n", | ||
| "openai_api_key = \"OPENAI_API_KEY\"\n", | ||
| "\n", | ||
| "llm = ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-0613\", openai_api_key=openai_api_key)\n", |
There was a problem hiding this comment.
Functions work w/ default gpt3.5 / 4 now, AFAIK.
| } | ||
| ], | ||
| "source": [ | ||
| "pip install -q openai langchain playwright beautifulsoup4" |
There was a problem hiding this comment.
Also need to add:
! playwright install
to download the necessary browser binaries (Chromium, Firefox, WebKit).
|
I cleaned this up a bit more. Main issue: the extraction is sensitive to the transformation of raw HTML (HTML2Text vs BS4). Have a look at the ntbk. Also title / summary extraction doesn't look quite right. |
| "id": "97f7de42", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Run the web scraper w/ BeautifulSoup\n", |
There was a problem hiding this comment.
Can you update the notebook to use the BeautifulSoupTransformer here?
rlancemartin
changed the title
rlancemartin
force-pushed
the
add-web-scraper-to-usecases
branch
from
2edafe6
to
e266287
Compare
rlancemartin
force-pushed
the
add-web-scraper-to-usecases
branch
from
e266287
to
3553df1
Compare
rlancemartin
force-pushed
the
add-web-scraper-to-usecases
branch
2 times, most recently
from
8577adb
to
976bd1c
Compare
rlancemartin
force-pushed
the
add-web-scraper-to-usecases
branch
from
976bd1c
to
7a96ae4
Compare
…ormer (langchain-ai#8732) - Description: Added a new use case category called "Web Scraping", and a tutorial to scrape websites using OpenAI Functions Extraction chain to the docs. - Tag maintainer:@baskaryan @hwchase17 , - Twitter handle: https://www.linkedin.com/in/haiphunghiem/ (I'm on LinkedIn mostly) --------- Co-authored-by: Lance Martin <[email protected]>
