Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy Rotation #778

Open
wants to merge 243 commits into
base: alpha
Choose a base branch
from
Open

Proxy Rotation #778

wants to merge 243 commits into from

Conversation

hwamil
Copy link

@hwamil hwamil commented May 14, 2021

I cloned a fresh repo and am re-making the pull request.

connector = None
if self.proxies and idx < len(self.proxies):
connector = ProxyConnector.from_url(self.proxies[idx]["https"])
for idx in range(len(self.proxies)):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't be suitable for merging unless it also supports the "default" configuration of no proxies.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But since you're cycling each AmazonMonitor through all the items, it seems like that shouldn't be hard to add, you'd just need to make sure that if there are no proxies, to instantiate a single AmazonMonitor that uses the default connector.

This PR changes the meaning of the delay parameter from a per-item delay to a per-connection/proxy delay. That still seems totally workable, though, as long as people running the code are aware of it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I wrote it so that the delay parameter still works as intended - it's still the delay for check per session/proxy which should keep them getting dogg'd (503). I kinda figured if you're not using proxies alpha is kinda useless; might as well use master branch.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree pretty strongly that alpha is useless without proxies. I was getting softbanned using master because it was doing stock checks while logged in. Alpha uses anonymous sessions for stock checking, which is a huge improvement. I successfully bought what I wanted (granted, an AMD CPU, not a GPU) using alpha with no proxies.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented a default mode with no proxies, though it looks ugly rn. Gonna try to clean it up once I figure out how.

stores/amazon_monitoring.py Outdated Show resolved Hide resolved
stores/amazon_monitoring.py Outdated Show resolved Hide resolved
stores/amazon_monitoring.py Outdated Show resolved Hide resolved
stores/amazon_monitoring.py Outdated Show resolved Hide resolved
stores/amazon_monitoring.py Outdated Show resolved Hide resolved
@mbbush
Copy link

mbbush commented May 15, 2021

I should have put all those comments in a review, so they'd go together, and then I'd be able to provide general feedback like I'm doing now. Oh well, this works too.

I think in general I like this approach. One thing that made this difficult was the way that each AmazonMonitor instance was bound to both a ClientSession (and therefore a single proxy) and a single item. When I was thinking about how to do this, I was assuming I'd need to keep each AmazonMonitor per-item, and rotate the proxies. You've done the opposite, which works just as well and is much easier.

I think the biggest thing that needs fixing is the implementation of next_item(). Maybe there's some python mutability magic that makes this code work, but I don't see it.

I didn't see anything that removed the proxies that returned 503 from the rotation. Is your plan to just do that manually by stopping, editing config, and restarting?

@hwamil
Copy link
Author

hwamil commented May 15, 2021

Yeah it was mostly Dakk's idea. I have 500 proxies running and there doesn't seem to be a problem creating and handling all the sessions. I originally switched out the connectors but someone suggested that it may be a bad idea because the sessions would appear suspicious since all session params would stay the same except for the ip address. It was also a lot more complicated/messy to implement as it needed a separate delay that had to be hardcoded, and having to randomly select a proxy that has cooled down seemed inefficient (although if I used itertools.cycle would have worked well here as well.)

Overall I'm content with what it's doing currently. I wish I knew more about backend and how to deal with interwoven modules. I've only written stuff with like two scripts at most so looking over FG code and figuring out the flow is a bit overwhelming especially because of async lmao. I'll need to watch/read more on asyncio in order to optimize further.

Thanks for the feedback. Appreciate it.

hwamil added 18 commits May 15, 2021 23:05
… is technically an iteration but I thought it made more intuitive sense to call it pop
… instead of saving n times where n amounts to number of sessions as it would occasionally crash because of too many open files error
…d, the last session's issaver attr is changed to True, making it the only session that saves to bad_proxies.json
…ions appear like the same browser/ip every time FG restarts.
hwamil added 28 commits June 5, 2021 15:42
…ly slows down stock check so decreased to 10 tries
…d about program hanging. added OS platform check for uvloop so even if windwos users use --uv flag the program won't crash
…s 503. instead when check_fail returns -1 we revalidate session since it only counts 503s from json method
@EEMike82
Copy link

Pip install is still trying to pull in uvloop and failing on Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants