Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Saarland #119

Open
robbi5 opened this issue Jan 8, 2019 · 2 comments
Open

Update Saarland #119

robbi5 opened this issue Jan 8, 2019 · 2 comments

Comments

@robbi5
Copy link
Owner

robbi5 commented Jan 8, 2019

They changed their website again, this time it even doesn't look like Sharepoint anymore.

Overview:

POST https://www.landtag-saar.de/umbraco/aawSearchSurfaceController/SearchSurface/GetSearchResults/
Accept application/json, text/javascript, */*; q=0.01

{"Filter":{"Periods":[]},"Pageination":{"Skip":0,"Take":10},"Sections":{"Print":true},"Sort":{"SortType":0,"SortValue":0},"OnlyTitle":false,"Value":"","CurrentSearchTab":0}

Detail:

POST https://www.landtag-saar.de/umbraco/aawSearchSurfaceController/SearchSurface/GetSearchResults/
Accept application/json, text/javascript, */*; q=0.01

{"Filter":{"Periods":[]},"Pageination":{"Skip":0,"Take":10},"Sections":{"Print":true},"Sort":{"SortType":0,"SortValue":0},"OnlyTitle":false,"Value":"","CurrentSearchTab":1,"KendoFilter":{"filters":[{"field":"DocumentNumber","operator":"startswith","value":"16/1234"}],"logic":"and"}}
enricogenauck added a commit to enricogenauck/kleineanfragen that referenced this issue Jan 30, 2019
This is my first point of contact with this projects code.
Upon inspecting the saarland_scraper test cases I found mostly tests of internal
and potentially private methods. I didn't feel confident enough to touch the
scrapers implementation without a high level test perspective of:

- Giving real world derived server response to scraper
- Checking for extracted entries and their properties

This commit adds webmock to stub the scrapers network calls. That way the used
fixtures can be more easily checked against the real world responses or updated
with current data in case the website changes their format again.

references robbi5#119
enricogenauck added a commit to enricogenauck/kleineanfragen that referenced this issue Jan 30, 2019
- Input data was updated
- Test expectations were updated

TODO: Update DetailScraper

references robbi5#119
@enricogenauck
Copy link
Contributor

Hi @robbi5,

I tried to take a quick stab at fixing the saarland scraper without touching to much of the existing code. I'm pretty confident about the overview_scraper part. But since this is my first contact with the code base I'm not sure how to proceed any further. Are pull requests your preferred method of contribution?

Unfortunately I'm not able to test the scraper and the changes in more depth with my current knowledge of the code base.

@robbi5
Copy link
Owner Author

robbi5 commented Jan 31, 2019

Hi @enricogenauck,
wow. thank you very much for the contribution - yes, a pull request would be great. Don't worry about not being able to test the scraper, i'll try it out and also have a look at the detail scraper. btw, thanks for introducing webmock into the project :)

enricogenauck added a commit to enricogenauck/kleineanfragen that referenced this issue Feb 1, 2019
enricogenauck added a commit to enricogenauck/kleineanfragen that referenced this issue Feb 1, 2019
This is my first point of contact with this projects code.
Upon inspecting the saarland_scraper test cases I found mostly tests of internal
and potentially private methods. I didn't feel confident enough to touch the
scrapers implementation without a high level test perspective of:

- Giving real world derived server response to scraper
- Checking for extracted entries and their properties

This commit adds webmock to stub the scrapers network calls. That way the used
fixtures can be more easily checked against the real world responses or updated
with current data in case the website changes their format again.

references robbi5#119
enricogenauck added a commit to enricogenauck/kleineanfragen that referenced this issue Feb 1, 2019
- Input data was updated
- Test expectations were updated

TODO: Update DetailScraper

references robbi5#119
enricogenauck added a commit to enricogenauck/kleineanfragen that referenced this issue Feb 1, 2019
enricogenauck added a commit to enricogenauck/kleineanfragen that referenced this issue Feb 1, 2019
robbi5 added a commit to enricogenauck/kleineanfragen that referenced this issue Apr 4, 2019
Had failing tests because of daylight saving time

references robbi5#119
robbi5 added a commit to enricogenauck/kleineanfragen that referenced this issue Apr 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants