Rejects javascript href + simplifications + tests #3

matstc · 2015-01-13T23:11:48Z

No description provided.

samnissen · 2015-12-07T13:21:54Z

+1

matstc added 6 commits January 13, 2015 23:49

rejects href that contain javascript code

ed065c8

simplifies a few lines

253f69c

adds a few tests

3d57caf

rejects hrefs starting with '(' or mailto

dee37ba

upgrades typhoeus dependency

9d7facc

makes sanitization more robust

d3e56c8

matstc force-pushed the master branch 3 times, most recently from 3bf7d66 to d1e6f75 Compare January 17, 2015 01:14

prevents explosion of global queue

25aafc1

matstc force-pushed the master branch from d1e6f75 to 25aafc1 Compare January 17, 2015 01:33

matstc added 2 commits January 17, 2015 11:56

fixes bugs + refactors

591cf54

allows enabling of cookies + more robust link rejection

73eb8ac

matstc force-pushed the master branch from ff83ed9 to 73eb8ac Compare January 17, 2015 13:26

removes the parsing note in debug mode

0c28d98

matstc force-pushed the master branch from fe104a7 to 27e1940 Compare January 17, 2015 13:50

matstc added 2 commits January 18, 2015 16:43

allows crawling from multiple entry points

9d1e364

adds a url filter

4344d23

matstc force-pushed the master branch from 27e1940 to 4344d23 Compare January 18, 2015 15:43

matstc force-pushed the master branch 3 times, most recently from 7e17704 to 5fb7a0a Compare January 27, 2015 09:00

better escapes the URLs before parsing them

fd3998d

matstc force-pushed the master branch from 5fb7a0a to fd3998d Compare January 27, 2015 09:04

matstc added 5 commits January 27, 2015 13:51

uses addressable to make urls absolute

07882c8

adds more info to invalid URI error

e0aa176

skips about urls and urls outside of domain

3d09fcb

only prints invalid uris in debug mode

cef7c0b

makes parse_domain a class method

c5baee9

Provide feedback