add playground environment, inspired heavily from israeli-ynab-updater #116

esakal · 2018-04-15T07:13:18Z

An integrated playground environment that can be used to maintain/extend scrapers:

Simplifies the development workflow by providing a safe way to run the scrapers.
Enables using IDE debugging tools.

eshaham

@esakal 👍 left some comments

eshaham · 2018-04-15T17:40:08Z

CONTRIBUTING.md

+## Using the playground 
+> With the integrated playground support, you can easily extend the scrapers and also enjoy the IDE debug feature if needed.
+
+Once you prepare the playground environment you will be able to set scraping options, debug using the IDE and get the scraped transactions as generated csv file.


@esakal files*

eshaham · 2018-04-15T17:44:56Z

package.json

+    "lint": "eslint src playground",
+    "playground:prepare": "npm run prepublishOnly && babel playground --out-dir .playground-lib",
+    "playground:scrape": "node .playground-lib/scrape.js",
+    "playground:setup": "node .playground-lib/setup.js",


@esakal why not simply run the playground:scrape and playground:setup scripts using babel-cli to avoid the need to prepare stuff every time?

Adding babel-cli make sense, but I didn't want to do major changes to the current project structure. If you want I can change the workflow to use babel-cli. let me know.

A prerequisite will be to make sure that we can add breakpoints somewhere when using babel-cli, otherwise we are missing the point of having playground support

@esakal I don't think this is a major change, we don't even have to add another dependency to use it (it's included under babel module).

A prerequisite will be to make sure that we can add breakpoints somewhere when using babel-cli, otherwise we are missing the point of having playground support

Good point, but I'm sure this is doable.

eshaham · 2018-04-15T17:46:24Z

CONTRIBUTING.md

+### Run a scraper
+To run a scraper you should:
+1. Provide the scraper credentials by running `npm run playground:setup`.
+1. Set scraper options in file  `.playground-lib/scrape-options.js`. Make sure you are not modifying the original file under `playground` folder.


@esakal I would personally much rather providing the option in the command line than saving them under a temp file in the playground lib folder

it is not practical providing it in command line:

many IDEs like webstorm doesn't play nice with command line prompt in debug mode. I tried to debug ynab-updater that is using inquirer and gave up after an hour or so.

setup the IDE is easier, just mention the path to the scrape.js and you are all set

it is more practical defining them once and then re-running the same preset with few clicks as possible.

@esakal got it.
Then, I would rather save the options under ${os.homedir()}/.bank-scrapers-playground, than under the repos folders.
What do you think?

eshaham · 2018-04-15T17:48:41Z

CONTRIBUTING.md

+
+> To keep your credentials safe, they will be stored in a separated folder under your os home directory as defined in file `playground/definitions.js`. The default value is set to `{homedir}/.bank-scrapers-playground`.
+
+> The generated report will be added under `.playground-lib/output` folder. This folder is being ignored by github and npm to prevent exposing your data. 


@esakal Again, I would prefer setting this using command line parameter, and would definitely not output into the lib folder. I prefer to create as little personal / local files under the repo folder, even if they are excluded using .gitignore

The motivation to put it under playground-lib was to create an isolated folder underwith we have all the playground assets. To remove the playground with all its related files all you need to do is to delete this folders. We can consider changing the folder names so:
./playground -> ./playground-src or ./src-playground
./playground-lib -> ./playground
./playground-lib/output -> ./playground/output

wdyt?

@esakal I really hate mixing code and output under the repo.
Why can't the output go somewhere else, like in the ynab-updater repo?

eshaham · 2018-04-15T17:51:05Z

CONTRIBUTING.md

+> Before your push your changes to github, it is your responsibility to review changes and make sure that only desired information are being commited. 
+
+### Make changes to scrapers
+The playground is using the transpiled files created in `lib` folder. 


@esakal what's the benefit of using the playground if it doesn't import the original files directly?
If you're importing the lib files, you can just as well create this playground in a different repo.
Or am I missing the motivation for this?

writing a playground in a different repo takes time, you can see the amount of code I needed to include here just to support the core debugging flows.

Also you cannot debug using the IDE between two projects, having the playground internally allow you to add breakpoints which is what was the original motivation for having the playground.

@esakal understood.
But why not import the source files directly, instead of the transpiled files?

eshaham · 2018-04-15T17:51:51Z

CONTRIBUTING.md

+### Debug your changes using the IDE
+To run the playground scripts within your IDE debugger, make sure you use file `.playground-lib/scrape.js` 
+
+If you want to set breakpoints, you should add them to the transpiled code under `lib` folders. Any breakpoints added to the `src` folder will be ignored by the IDE. 


@esakal which is why we should import the source files and not the lib folder, see comment above.

agreed, quoted from my previous reply:

A prerequisite will be to make sure that we can add breakpoints somewhere when using babel-cli, otherwise we are missing the point of having playground support

@esakal 👍

eshaham · 2018-04-15T17:54:14Z

playground/.eslintrc

+    "Headers": true
+  },
+  "extends": "airbnb-base"
+}


@esakal this file should only specify the differences from the .eslintrc file in the root directory, not repeat all the settings. See here.

eshaham · 2018-04-15T17:56:11Z

playground/helpers/crypto.js

+import crypto from 'crypto';
+
+const ALGORITHM = 'aes-256-ctr';
+const SALT = '8cs+8Y(nxDLY';


@esakal perhaps use a different salt than what we use in https://github.com/eshaham/israeli-ynab-updater

I think having the salt in source-version which means exposing the salt value misses the point of having encryptions at all.

I think we should create an issue addressing the current behavior. wdyt?

@esakal agree

@esakal eshaham/israeli-ynab-updater#34

eshaham · 2018-04-16T06:28:37Z

playground/scrape-options.js

+  startDate: moment().add(-2, 'months'),
+  combineInstallments: false,
+  saveLocation: `${__dirname}/output`,
+};


@esakal as mentioned above, I would not want this to be setup under the repo.
My favorite option would be to supply these as command line parameters (we could provide sample bash scripts).
The other option would be to instruct the user to save these settings under the config folder.

My favorite option would be to supply these as command line parameters

It is possible, yet I don't see the big advantage of having them as a cli arguments unless you want this feature to be added to ynab-updater as an alternative to the existing flow which requires user interactions

The other option would be to instruct the user to save these settings under the config folder

I personally try to avoid forcing the user to perform manual step if possible as I find them error prone and confusing.

@esakal ok.
But can we save it for the user under ${os.homedir()}/.bank-scrapers-playground?

eshaham · 2018-04-16T06:29:48Z

playground/setup.js

+import { encryptCredentials } from './helpers/credentials';
+import CONFIG_FOLDER from './definitions';
+
+


@esakal redundant line break

esakal · 2018-04-16T07:16:49Z

@eshaham see my replies, I will push fixes once we resolve all the open issues.

esakal · 2018-05-17T17:54:59Z

@eshaham I think you are going to love the commit I just pushed.

Interactive setup

Setup of playground credentials and options is done with interactive menus. No need to change sources as it was before.

Executing scraping without any cli interactions/prompts

I moved the selection of the scraper into the setup process so we can run the scraping process without prompting for information, this allow us to utilize the IDE debugger.

Debug the source files

I'm using babel-register to transpile and run the sources without creating them. This allow us to set breakpoints in the sources directly.

External folders for settings and output files

Per your request I moved both the setting file and the output files outside from the library folder as you did in israeli-ynab-updater.

esakal · 2018-05-23T13:27:45Z

@eshaham let me know if you have any further thoughts/comments

esakal · 2018-06-07T10:57:37Z

@eshaham Hi,
Can I assist somehow with the pending PRs?

eshaham

@esakal really sorry, busy couple of weeks!

eshaham · 2018-06-10T08:05:15Z

package.json

@@ -9,7 +9,9 @@
  "main": "lib/index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
-    "lint": "eslint src",
+    "lint": "eslint src playground",
+    "start": "node -r babel-register playground/scrape.js",


@esakal never tried using babel-register in the past, looks interesting... 👍

eshaham requested changes Apr 16, 2018

View reviewed changes

add playground environment, inspired heavily from israeli-ynab-updater

b700acf

update instruction to user when saving playground options

e1d9b73

eshaham approved these changes Jun 10, 2018

View reviewed changes

eshaham merged commit 169fbf8 into eshaham:master Jun 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add playground environment, inspired heavily from israeli-ynab-updater #116

add playground environment, inspired heavily from israeli-ynab-updater #116

esakal commented Apr 15, 2018

eshaham left a comment

eshaham Apr 15, 2018

esakal May 17, 2018

eshaham Apr 15, 2018

esakal Apr 16, 2018 •

edited

Loading

eshaham Apr 22, 2018

eshaham Apr 15, 2018

esakal Apr 16, 2018

eshaham Apr 22, 2018 •

edited

Loading

eshaham Apr 15, 2018

esakal Apr 16, 2018

eshaham Apr 22, 2018

eshaham Apr 15, 2018

esakal Apr 16, 2018

eshaham Apr 22, 2018

eshaham Apr 15, 2018

esakal Apr 16, 2018

eshaham Apr 22, 2018

eshaham Apr 15, 2018

esakal May 17, 2018

eshaham Apr 15, 2018

esakal Apr 16, 2018

eshaham Apr 22, 2018

eshaham Apr 22, 2018

eshaham Apr 16, 2018

esakal Apr 16, 2018

eshaham Apr 22, 2018

eshaham Apr 16, 2018

esakal commented Apr 16, 2018

esakal commented May 17, 2018

esakal commented May 23, 2018

esakal commented Jun 7, 2018

eshaham left a comment

eshaham Jun 10, 2018


		> To keep your credentials safe, they will be stored in a separated folder under your os home directory as defined in file `playground/definitions.js`. The default value is set to `{homedir}/.bank-scrapers-playground`.

		> The generated report will be added under `.playground-lib/output` folder. This folder is being ignored by github and npm to prevent exposing your data.

		import { encryptCredentials } from './helpers/credentials';
		import CONFIG_FOLDER from './definitions';

add playground environment, inspired heavily from israeli-ynab-updater #116

add playground environment, inspired heavily from israeli-ynab-updater #116

Conversation

esakal commented Apr 15, 2018

eshaham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

esakal Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eshaham Apr 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

esakal commented Apr 16, 2018

esakal commented May 17, 2018

Interactive setup

Executing scraping without any cli interactions/prompts

Debug the source files

External folders for settings and output files

esakal commented May 23, 2018

esakal commented Jun 7, 2018

eshaham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

esakal Apr 16, 2018 •

edited

Loading

eshaham Apr 22, 2018 •

edited

Loading