Skip to content

0.2.0

Pre-release
Pre-release
Compare
Choose a tag to compare
@Shardj Shardj released this 05 Jul 11:59
· 2 commits to master since this release
  • Added regex fitler on url.
  • Fixed issues with multiple h1 tags.
  • Added checks for fragments and query strings in urls so we don't get duplicate pages infinitely.
  • Now randomises user-agent on every request (why did I do this? Guess it'll become profiles with proxies later).
  • Better formatting and validation on strings.
  • Files returned are now raw text instead of holding tags.
  • Handles redirects
  • improved some attribute setting and error logging