Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CFP crawler #2

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

CFP crawler #2

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
74c2fbd
added cfp crawler
Niveshkrishna Jun 27, 2018
1fc026c
added readme file
Niveshkrishna Jun 27, 2018
a3238b2
Delete README.md
Niveshkrishna Jun 27, 2018
88a4f1a
Create README.md
Niveshkrishna Jun 27, 2018
b00db07
Update crawler.py
Niveshkrishna Jul 11, 2018
1a2a1dd
Delete proposals.json
Niveshkrishna Jul 13, 2018
fc64312
latest cfp json
Niveshkrishna Jul 13, 2018
ce13792
removed redudant json file
Niveshkrishna Jul 13, 2018
2a2dc64
removed unnecessary files
Niveshkrishna Jul 13, 2018
123d323
made changes as requested by @ananyo2012
Niveshkrishna Jul 22, 2018
26632bf
Delete logs.log
Niveshkrishna Jul 22, 2018
fc5efc4
Delete __init__.pyc
Niveshkrishna Jul 22, 2018
6213d08
Delete settings.pyc
Niveshkrishna Jul 22, 2018
e5abb38
Delete __init__.pyc
Niveshkrishna Jul 22, 2018
08f965a
Delete crawler.pyc
Niveshkrishna Jul 22, 2018
2f281f8
update proposal.json file to contain only one proposal
Niveshkrishna Jul 22, 2018
f2cf342
Create .gitignore
Niveshkrishna Jul 22, 2018
e4d2f60
closed proposal.json file
Niveshkrishna Jul 22, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cfp_crawler/proposal/spiders/crawler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import json
from scrapy import signals

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be 2 empty lines after import statements.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that can be done


class CrawlerSpider(scrapy.Spider):
name = 'crawler'
Copy link

@aaqaishtyaq aaqaishtyaq Jul 22, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also can you change the name of the spider, something like cfpcrawler or cfp.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you confirm about this @ananyo2012 ?

allowed_domains = ['in.pycon.org']
Expand Down
Empty file removed cfp_crawler/proposal/spiders/logs.log
View file
Open in desktop
Empty file.
Loading