Merge branch 'master' into fix-dnt-header-test

Conflicts: src/js/background.js tests/selenium/dnt_test.py
EFForg · Sep 7, 2018 · 79452a6 · 79452a6
2 parents 130977b + d308e64
commit 79452a6
Show file tree

Hide file tree

Showing 87 changed files with 22,888 additions and 5,211 deletions.
diff --git a/.eslintignore b/.eslintignore
@@ -1,3 +1,5 @@
 src/lib/vendor/
 src/tests/lib/vendor/
 node_modules/
+**/selenium/
+!tests/selenium/
diff --git a/.eslintrc.yml b/.eslintrc.yml
@@ -13,44 +13,45 @@ parserOptions:
   sourceType: module
 rules:
   brace-style:
-    - warn
+    - error
     - 1tbs
     - allowSingleLine: true
   consistent-this:
     - error
     - self
   curly: error
-  dot-notation: warn
+  dot-notation: error
+  eol-last: error
   indent:
     - error
     - 2
     - outerIIFEBody: 0
       SwitchCase: 1
-  keyword-spacing: warn
+  keyword-spacing: error
   linebreak-style:
     - error
     - unix
   new-cap: error
   no-array-constructor: error
-  no-bitwise: warn
+  no-bitwise: error
   no-caller: error
   no-console: off
-  no-eval: warn
-  no-implied-eval: warn
+  no-eval: error
+  no-implied-eval: error
   no-iterator: error
   no-loop-func: error
-  no-multi-spaces: warn
-  no-multi-str: warn
+  no-multi-spaces: error
+  no-multi-str: error
   no-new: error
-  no-new-func: warn
+  no-new-func: error
   no-new-object: error
   no-new-wrappers: error
   no-proto: error
-  no-script-url: warn
-  no-shadow: warn
+  no-script-url: error
+  no-shadow: error
   no-shadow-restricted-names: error
-  no-trailing-spaces: warn
-  no-unused-expressions: warn
+  no-trailing-spaces: error
+  no-unused-expressions: error
   semi: error
-  space-before-blocks: warn
-  space-in-parens: warn
+  space-before-blocks: error
+  space-in-parens: error
diff --git a/.github/ISSUE_TEMPLATE/broken-site-report.md b/.github/ISSUE_TEMPLATE/broken-site-report.md
@@ -0,0 +1,20 @@
+---
+name: Broken Site Report
+about: Report a broken site or site service
+
+---
+
+<!--
+Is Privacy Badger breaking something somewhere? We'd like to get it fixed! Let's get some debugging information first.
+-->
+
+### What is your browser and browser version?
+
+### What is broken and where?
+
+### What is the domain that causes breakage when blocked? What is your debug output for this domain?
+```
+Please follow the instructions on
+https://github.com/EFForg/privacybadger/wiki/Find-out-why-Privacy-Badger-is-blocking-a-domain
+and paste the output here.
+```
diff --git a/.gitignore b/.gitignore
@@ -41,14 +41,19 @@ buildtools/jshydra/mozilla/js
 
 node_modules
 
-tests/selenium/.cache
 tests/.cache
+tests/.pytest_cache
+.pytest_cache
 
 *.xpi
 .chrome-profile
 
 .idea
 
+#Eclipse project stuff
+.settings/
+.project
+
 web-ext-artifacts
 pkg
 release-utils/config.sh

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -16,16 +16,14 @@ Here is a quick orientation to guide your adventure:
   and
   ["help wanted"](https://github.com/EFForg/privacybadger/labels/help%20wanted)
   labels.
-* When working on Privacy Badger's codebase,
-  please consult our [guide to setting up automated tests](/doc/tests.md).
+* When making changes to Privacy Badger's code, please consult our [developer](/doc/develop.md) and [automated testing](/doc/tests.md) guides.
 * Security vulnerabilities can be reported privately to
   [[email protected]](mailto:[email protected]). Please see our
   [Security Vulnerability Disclosure Program](https://www.eff.org/security)
   for the GPG key and more information.
 <!---
 * This short overview of Privacy Badger's architecture can help put things in perspective.
 * Developer setup and debugging.
-* Documentation brought you this document.
 -->
 
 If you find something confusing or frustrating, please let us know!

diff --git a/Makefile b/Makefile
@@ -1,11 +1,14 @@
-build: updatepsl zip crx
+build: updatepsl updateseed zip crx
 
 travisbuild: zip crx
 	ls -1tr *.crx | tail -n 1
 
 updatepsl:
 	scripts/updatepsl.sh
 
+updateseed:
+	scripts/updateseeddata.sh
+
 zip:
 	scripts/makezip.sh 
 
@@ -27,4 +30,14 @@ upload:
 lint:
 	./node_modules/.bin/eslint .
 
-.PHONY: build travisbuild updatepsl zip crx todo logging lint
+tx:
+	tx pull -f
+	scripts/fix_placeholders.py
+
+runff:
+	./node_modules/.bin/web-ext run --start-url "about:debugging" -s src/
+
+runfn:
+	./node_modules/.bin/web-ext run --start-url "about:debugging" -s src/ -f /opt/firefox/nightly/firefox
+
+.PHONY: build travisbuild updatepsl updateseed zip crx todo logging lint tx runff runfn
diff --git a/README.md b/README.md
@@ -6,17 +6,13 @@ Privacy Badger sends the [Do Not Track signal](https://www.eff.org/issues/do-not
 
 Besides automatic tracker blocking, Privacy Badger removes outgoing link click tracking on [Facebook](https://www.eff.org/deeplinks/2018/05/privacy-badger-rolls-out-new-ways-fight-facebook-tracking) and [Twitter](https://www.eff.org/deeplinks/2017/08/privacy-badger-makes-twitter-little-less-creepy), with more privacy protections on the way.
 
-To learn more, see [the FAQ on Privacy Badger's homepage](https://www.eff.org/privacybadger#badger-faq-header).
+To learn more, see [the FAQ on Privacy Badger's homepage](https://www.eff.org/privacybadger/faq).
 
 
 ## Contributing
 
 We're glad you want to help! Please see [our contributor guide](/CONTRIBUTING.md).
 
-To install Privacy Badger from source in Chrome, visit `chrome://extensions`, enable "Developer mode", click "Load unpacked extension..." and select the [`src`](src/) subdirectory inside your copy of the Privacy Badger source code.
-
-In Firefox, visit `about:debugging`, click "Load Temporary Add-on" and select the [`src/manifest.json`](src/manifest.json) file. Note that this only installs the extension temporarily; it will be removed when you close Firefox.
-
 
 ## Getting in touch
 

diff --git a/doc/Changelog b/doc/Changelog
@@ -1,5 +1,35 @@
 Changes for Privacy Badger 
 ====================================
+## 2018.8.22
+* Added pre-trained tracker data for new Privacy Badger installations.
+Visit www.eff.org/badger-pretraining to learn more.
+* Added reset/clear tracker data buttons to the Manage Data options page tab
+* Fixed various site breakages
+* Added Persian and Brazilian Portuguese translations
+* Improved translations (Simplified Chinese, Danish, Esperanto, French,
+German, Italian, Norwegian Bokmål, Spanish, Swedish)
+
+## 2018.8.1
+* Fixed security issues with HTML5 local storage tracking detection as well as
+SoundCloud widget replacement. Thanks again to Cure53 for discovering and
+reporting these vulnerabilities.
+* Improved Facebook link unwrapping; now enabled on the Facebook Onion domain
+* Improved translations (Italian, Norwegian Bokmål, Swedish, Ukrainian)
+
+## 2018.7.18.1
+* Added setting to disable sending Do Not Track to websites
+* Fixed security issue with link unwrapping on Facebook. Thanks to Cure53 for
+discovering and reporting this vulnerability.
+* Improved ordering of domain names in the popup and on the options page
+* Improved handling of disabled sites with wildcards
+* Added t.co link replacement to user profiles on Twitter
+* Linked to EFF software privacy policy from the new user welcome page
+* Updated to latest dummy Google Tag Manager script from uBlock Origin
+to avoid "failed to redirect a network request" warnings in Chrome
+* Fixed various site breakages
+* Improved translations (Simplified Chinese, Dutch, Esperanto, German,
+Russian, Swedish, Turkish)
+
 ## 2018.5.10
 * Added protection against outgoing link click tracking on Facebook
 * Updated WebRTC protection to revert to browser default (off) when disabled

diff --git a/doc/Translation.md b/doc/Translation.md
@@ -12,6 +12,7 @@ reference. You can also use existing translations from other languages to help
 you in case of doubt, but you should always consider the English version as the
 correct one.
 
+
 ## A note about adding translation strings in PRs
 
 While working on a Privacy Badger enhancement, you might find yourself needing
@@ -21,6 +22,7 @@ messages to all other locales. This will happen when a Privacy Badger
 maintainer syncs translations with Transifex after your pull request passes
 review.
 
+
 ## GitHub
 
 Translations on GitHub are done with JSON files.
@@ -72,6 +74,7 @@ To correct errors in an existing translation:
 Here you can find an example translation pull request:
 [#1270](https://github.com/EFForg/privacybadger/pull/1270).
 
+
 ## Transifex
 
 To contribute to translations through Transifex you should have
@@ -82,20 +85,26 @@ To translate Privacy Badger with Transifex, go to the
 [Privacy Badger Transifex project](https://www.transifex.com/eff/privacy-badger/dashboard/),
 log in to your account, and click on the Translate button.
 
-Before starting your work, send a message to [ghostwords](https://www.transifex.com/user/profile/ghostwords/),
-who will check if Transifex and GitHub translations are correctly
-synced. To send him a message, go to [his profile](https://www.transifex.com/user/profile/ghostwords/)
-and click on "Send message". This will help avoid translating
-something that has already been translated or that is currently being
-translated. Also, send ghostwords a message when you have finished your work
-so that he will upload your translation to GitHub.
-
 The Transifex platform is quite intuitive to use. For any questions about
 Transifex refer to its [documentation](https://docs.transifex.com/).
 
 In case of questions or concerns with translations, you can leave comments
 for individual strings in the Transifex translations editor.
 
+
+## Testing your translations
+
+To see your (in-progress) translations in the actual Privacy Badger UI, you should first [load Privacy Badger from source code](/doc/develop.md#install-from-source).
+
+A quick/hacky way to change Privacy Badger's locale is to temporarily copy the locale you want to use to your default (OS) locale's folder in `src/_locales/` and reload Privacy Badger.
+
+The proper way would be to launch the browser in your desired locale.
+
+For Chrome, it might be as easy as [launching it from the command line with `LANGUAGE=fr` (for example) in front of the executable](https://stackoverflow.com/questions/24992240/start-google-chrome-with-a-specific-locale-using-a-command-line-argument).
+
+[Firefox seems to require more effort](https://askubuntu.com/questions/63724/how-do-i-start-firefox-in-another-language-than-the-default).
+
+
 ## Other information
 
 To learn about outstanding translations-related issues, and to

diff --git a/doc/develop.md b/doc/develop.md
@@ -0,0 +1,25 @@
+# Working with Privacy Badger's code
+
+To make changes to Privacy Badger, you have to first load the extension from a source code checkout.
+
+
+## Install from source
+
+To install Privacy Badger from source in Chrome, visit `chrome://extensions`, enable "Developer mode", click "Load unpacked" and select the [`src`](src/) subdirectory inside your copy of the Privacy Badger source code.
+
+In Firefox, visit `about:debugging`, click "Load Temporary Add-on" and select the [`src/manifest.json`](src/manifest.json) file. Note that this only installs the extension temporarily; it will be removed when you close Firefox.
+
+
+## Send a pull request
+
+Before submitting a pull request (PR), please check your changes using [ESLint](https://eslint.org), our preferred automated static analysis ("lint") tool.
+
+### Lint your changes
+
+First, install the exact expected version of ESLint by running `npm install` in your Privacy Badger source code checkout directory. You should then be able to produce a lint report by running `make lint` in the same directory.
+
+You can review our set of ESLint rules in [`.eslintrc.yml`](/.eslintrc.yml). Files we want ESLint to ignore are specified in [`.eslintignore`](/.eslintignore).
+
+### Writing good commit messages
+
+I highly suggest reviewing the suggestions in this [excellent guide to writing commit messages](https://chris.beams.io/posts/git-commit/).
diff --git a/doc/fixing-broken-sites.md b/doc/fixing-broken-sites.md
@@ -35,9 +35,11 @@ Let's look at some common kinds of breakages and see how they relate to the appr
 
 ### Domains that are part of the site but don't look like it
 
-Does the blocked domain actually belong to the site, but Privacy Badger doesn't know that and so treats the domain as an external tracker? Sounds like a job for [multi-domain first parties](https://github.com/EFForg/privacybadger/issues/781).
+Does the blocked domain actually belong to the site, but Privacy Badger doesn't know that and so treats the domain as an external tracker? Sounds like a job for [multi-domain first parties](https://github.com/EFForg/privacybadger/issues/781) (MDFP).
 
-For example, see [this "Disney–ABC Television Group" pull request](https://github.com/EFForg/privacybadger/pull/1526).
+When adding domains to the MDFP list, please add base ([eTLD](https://en.wikipedia.org/wiki/Public_Suffix_List)+1) domains only. For example, there is no need to add `api.example.net` when adding `example.com` and `example.net`.
+
+For past examples, you could browse [the list of merged pull requests with the "MDFP" label](https://github.com/EFForg/privacybadger/issues?q=label%3AMDFP+is%3Amerged).
 
 
 ### JavaScript errors

diff --git a/doc/tests.md b/doc/tests.md
@@ -12,39 +12,46 @@ Every pull request runs the full suite of functional and unit tests on [Travis C
 
 See [`.travis.yml`](/.travis.yml) for Travis configuration, [`scripts/setup_travis.sh`](/scripts/setup_travis.sh) for test setup, and [`scripts/run_travis.sh`](/scripts/run_travis.sh) for test execution procedures.
 
-We use [ESLint](https://eslint.org) to flag potential JavaScript errors and style issues. See [`.eslintrc.yml`](/.eslintrc.yml) and [`.eslintignore`](/.eslintignore) for specifics.
+We use [ESLint](https://eslint.org) to flag potential JavaScript errors and style issues. Please see our [developer guide](/doc/develop.md#lint-your-changes) for setup instructions.
 
 ## Unit tests
 
-We use [QUnit](https://qunitjs.com/) for unit tests. To run them, click on the badger icon next to the URL bar to open the popup.
+We use [QUnit](https://qunitjs.com/) for unit tests.
+Unit tests are defined in [`/src/tests/tests`](/src/tests/tests). Unit test dependencies live in [`/src/tests/lib`](/src/tests/lib).
+
+To run unit tests, first [load Privacy Badger from source code](/doc/develop.md#install-from-source) (as we don't ship unit tests with published versions).
+Once you loaded Badger from source, click on its button in your browser toolbar to open Badger's popup.
 Then in the popup, click on the gear icon (⚙) to open the options page.
 Your browser should navigate to an internal URL that starts with `chrome-extension://` or `moz-extension://` and ends with `/skin/options.html`.
-Replace `/skin/options.html` with `/tests/index.html` and hit enter.
-This will open the unit tests and run them.
-
-Unit tests are located in [`/src/tests/tests`](/src/tests/tests). The unit test dependencies are in [`/src/tests/lib`](/src/tests/lib). Please add unit tests whenever possible.
+Replace `/skin/options.html` with `/tests/index.html` and hit <kbd>Enter</kbd>.
+This will open the unit test suite and run the tests.
 
 ## Functional tests
 
 Our functional tests are written in [Python](https://www.python.org/) and driven by
 [Selenium](https://selenium-python.readthedocs.io/) and [pytest](https://docs.pytest.org/en/latest/).
-To run them in Chrome, you'll need to [install `chromedriver`](https://github.com/EFForg/privacybadger/blob/f780438ca146e932ebc80fa163ddcfdae4e56c25/scripts/setup_travis.sh#L4-L12). In Firefox, you need to [install `geckodriver`](https://github.com/EFForg/privacybadger/blob/f780438ca146e932ebc80fa163ddcfdae4e56c25/scripts/setup_travis.sh#L14-L28).
+
+To run them in Chrome, you need to [install `chromedriver`](http://chromedriver.chromium.org/getting-started). In Firefox, you need to [install `geckodriver`](https://github.com/EFForg/privacybadger/blob/547b19a8c3eddf60eed03aed3f60f252506490b7/scripts/setup_travis.sh#L21-L56).
+
 You also need some Python packages that can be installed by running:
 ```bash
 $ pip install -r tests/requirements.txt
 ```
 
-Now you should be able to run the Selenium tests!
-Try them out by running the code below.
-This should take several minutes.
+You should now be able to run the Selenium tests. Try them out by running
+the code below. This should take several minutes.
 ```bash
 $ BROWSER=chrome pytest -v
 ```
 
-The `BROWSER` environment variable must be set. It must be one of:
-* `BROWSER=/path/to/a/browser`
-* the name of a browser executable that can be found like `which $BROWSER`
-* or simply `BROWSER=chrome` or `BROWSER=firefox` if you have them installed
+macOS users may need to provide the full path to the browser application folder. For example, to run tests on macOS in Firefox:
+```bash
+$ BROWSER=/Applications/Firefox.app/Contents/MacOS/firefox-bin pytest -v
+```
+
+For more information, see our Travis CI [setup](/scripts/setup_travis.sh) and
+[run](/scripts/run_travis.sh) scripts.
+
 
 ### Examples
 
@@ -70,8 +77,3 @@ Like this:
 ```bash
 $ BROWSER=~/Downloads/firefox/firefox ENABLE_XVFB=1 pytest -s -v -k pbtest_org
 ```
-
-Refer to the our Travis CI scripts for more information:
-[`scripts/setup_travis.sh`](/scripts/setup_travis.sh) and
-[`scripts/run_travis.sh`](/scripts/run_travis.sh).
-