forked from luyishisi/Anti-Anti-Spider
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
14 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,31 +1,33 @@ | ||
# Anti-Anti-Spider | ||
# 2016-10-24 | ||
# 2016-10-24 begin | ||
|
||
仓库网址位于https://github.com/luyishisi/Anti-Anti-Spider 欢迎stat | ||
|
||
本项目由URLTEAM维护 | ||
|
||
作者博客 https://www.urlteam.org | ||
作者博客 https://www.urlteam.org | ||
|
||
项目简介: | ||
运用请求伪造,浏览器伪造,浏览器自动化,图像处理,ip处理等方式进行反爬虫技术的通用化使用。 | ||
为以后的采集任务快速开展留下基础代码。 | ||
如今项目会包含多项技术的样例代码. | ||
|
||
|
||
项目起因 | ||
|
||
本身是想做一个反爬虫的技术攻关站点,如果在总结诸多技术中发觉可以将反反爬虫技术直接保留与代码中。 | ||
在之后采集需要时能快速有效的测试该站点具有怎样的反爬特性,并且可以快速的进行代码复用 | ||
|
||
|
||
|
||
|
||
你可以做什么: | ||
2:提交你觉得难以采集的网站 | ||
联系方式: [email protected] | ||
|
||
|
||
项目结构树: | ||
|
||
项目结构树:(有待更新) | ||
https://github.com/luyishisi/Anti-Anti-Spider/blob/master/tree.txt | ||
|
||
重点项目: | ||
1:验证码 {亚马逊验证码破解,knn,svm,Tensorflow自动生成验证码并大量训练从而破解--98%成功率} | ||
2:代理 {抓取西刺代理,以及一个高可用的国外代理网站,并存入数据库,从而随时调用} | ||
3:代码模板 {多线程优化,百度地图可视化采集,聚焦爬虫,selenium模拟登陆,域名爬虫} | ||
5:爬虫项目源码 {优酷网,腾讯视频,推特,拉钩网,百度地图,妹子图网,百家号,百度百科,csdn,新浪微博} | ||
6:ip跟换技术 {代理,tor,adsl} | ||
7:请求伪造 {phantomjs,requests,selenium} | ||
8:phantomjs {伪造请求头,获取页面截图,获取页面源码,设置超时} | ||
9:selenium {伪造请求头,支付宝模拟登陆} | ||
UrlSpider {项目中常用的采集代码样本,经过多线程数据库操作优化,最高速度6kw/d} |