- Released on CRAN. Initial release v0.1.2
- Added
politely
adverb - Fixed bug related to incorrect calling of bow() inside the nod() function #31
- Added
times
argument tobow()
to control default number of retries #36 - Fixed a bug on.exit() restoring the user-agent #29
- Switched to webmockr for testing infrastructure
- Replaced httr::GET with httr::RETRY (closing #24)
- Removed tests base on Wikipedia due to changed routing
- Removed dependency on
here
- Released on CRAN. Initial release v0.1.1
- submitted to CRAN
- the
param
argument ofscrape()
is now softly deprecated. New argumentquery
is introduced. (closing #16) - dependency on
urltools
is removed in favor of native functions inhttr
- new
usethis
-like functionuse_manners()
for producing own polite scraping infrastructure is included
- added
set_scrape_delay()
andset_rip_delay()
to adjust default scraping rate limit - took
httr_rate_ltd()
out ofbow()
(closing #9)
- added
html_attrs_dfr()
- function for tidying html_attrs() - added polite download function called
rip()
- rate-limited
GET
function is now a method instantiated insidebow
to allow for setting persistentdelay
argument - messages (but not warnings) from robotstxt are suppressed for cleaner console
- added
content
parameter toscrape()
for overriding default data type and encoding
- Argument for setting up crawl delay has been renamed from
period
todelay
and moved fromscrape
tobow
. - Print method and warning messages reflect
delay
argument set for the session.
- Added a
NEWS.md
file to track changes to the package. - Added custom print method for
polite session
usingcrayon
bow
andnod
now warn if the current path is not permitted to be scrapedpolite
has gotten a sticker!png
is stored indata-raw
- Implemented bow(), nod() and scrape()
- Added documentation and examples
- Updated README to include examples
- Polite is born!