Skip to content

Commit

Permalink
update changes
Browse files Browse the repository at this point in the history
  • Loading branch information
Miniast committed Jun 25, 2024
1 parent 2106ac1 commit c193aa6
Show file tree
Hide file tree
Showing 20 changed files with 143 additions and 124 deletions.
21 changes: 17 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
node-crawler ChangeLog
-------------------------

2.0.2
- Add **Global Only Options : `silence`** to mute all crawler messages, including warnings and errors.
- Change default error message when request failed in **Crawler.add**
- Update dependencies

2.0.1
- [#474](https://github.com/bda-research/node-crawler/issues/474) Solve the 'userAgents' is not working
- Add some unit tests
- Migration from Mocha to AVA
- Use c8 to add code coverage report
- Update dependencies
- Others

2.0.0
- Crawler V2 has been released, which is a complete rewrite of the original crawler.
- see [README.md](https://github.com/bda-research/node-crawler/blob/master/README.md) for details.
Expand Down Expand Up @@ -90,7 +103,7 @@ node-crawler ChangeLog
* add event `schedule` which is emitted when a task is being added to scheduler
* in callback, move $ into `res` because of weird API
* change rateLimits to rateLimit

0.7.5
* delete entity in options before copy, and assgin after, `jar` is one of the typical properties which is an `Entity` wich functions [#177](https://github.com/bda-research/node-crawler/issues/177)
* upgrade `request` to version 2.74.0
Expand Down Expand Up @@ -119,11 +132,11 @@ node-crawler ChangeLog

0.6.0
* add `bottleneck` to implement rate limit, one can set limit for each connection at same time.

0.5.2
* you can manually terminate all the resources in your pool, when `onDrain` called, before their timeouts have been reached
* add a read-only property `queueSize` to crawler [#148](https://github.com/bda-research/node-crawler/issues/148) [#76](https://github.com/bda-research/node-crawler/issues/76) [#107](https://github.com/bda-research/node-crawler/issues/107)

0.5.1
* remove cache feature, it's useless
* add `localAddress`, `time`, `tunnel`, `proxyHeaderWhiteList`, `proxyHeaderExclusiveList` properties to pass to `request` [#155](https://github.com/bda-research/node-crawler/issues/155)
Expand All @@ -137,4 +150,4 @@ node-crawler ChangeLog
* remove cache and skip duplicate `request` for `GET`, `POST`(only for type `urlencode`), `HEAD`
* add log feature, you can use `winston` to set `logger:winston`, or crawler will output to console
* rotate user-agent in case some sites ban your requests

5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -428,6 +428,11 @@ items in the **crawler.add()** calls if you want them to be specific to that ite

### Global only options

#### `silence`
- **Type:** `boolean`
- **Default** : false
- If true, the crawler will mute all warning and error messages. The request error will be still reported.

#### `maxConnections`

- **Type:** `number`
Expand Down
11 changes: 5 additions & 6 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
"exports": "./dist/index.js",
"scripts": {
"build": "tsc",
"test": "ava",
"cover": "c8 ava"
"test": "NODE_ENV=test ava",
"cover": "NODE_ENV=test c8 ava"
},
"engines": {
"node": ">=18"
Expand Down Expand Up @@ -46,7 +46,7 @@
"devDependencies": {
"@eslint/js": "^9.5.0",
"@types/got": "^9.6.12",
"@types/node": "^20.14.7",
"@types/node": "^20.14.8",
"ava": "^6.1.3",
"c8": "^10.1.2",
"eslint": "~9.4.0",
Expand All @@ -55,8 +55,8 @@
"sinon": "^18.0.0",
"tough-cookie": "^4.1.4",
"tsx": "^4.15.7",
"typescript": "^5.5.2",
"typescript-eslint": "8.0.0-alpha.27"
"typescript": "^5.4.5",
"typescript-eslint": "8.0.0-alpha.30"
},
"ava": {
"files": [
Expand All @@ -67,7 +67,6 @@
"extensions": {
"js": true
},
"failFast": true,
"verbose": true
},
"c8": {
Expand Down
Loading

0 comments on commit c193aa6

Please sign in to comment.