Respect _MAX_RECORDS_LIMIT

[qbatten]

Since we're overriding get_records from SQLStream, let's add in these two lines so we maintain same behavior as SQLStream.

brief convo about this here

[tayloramurphy]

@qbatten any chance you want to make a PR for this?

[techtangents]

Ability to limit the number of rows would be very useful for my team. We need to ingest some large tables and can't read them all in one go due to our postgres replication timeout.

[tayloramurphy]

cc @visch @pnadolny13

[visch]

@tayloramurphy It's a quick one, I can take this one.

SQLAlchemy and pre_pool should theoretically already handle this without a MAX_RECORDS_LIMIT and it should be automatic, but I can make a seperate issue for that and handle it after we get this one in place.

[visch]

@techtangents This issue doesn't address the issue you're after. MAX_RECORDS_LIMIT was put together for running pytest not for running taps.

I ended up implementing what you were after here anyways. Here's the PR #393

I think the "right" way to approach this issue is to look at the connection pool #394 , it is harder to do this though as we'd still want to fail eventually if postgres times out, and it'd require more effort to get in place

[techtangents]

@visch apologies and thank you.

To be clear, we'd just want it to read, say, 500k records then stop. Then the next time we run the tap, it'd read another 500k records then stop. This is what the "limit" parameter on the pipelinewise-tap-postgres does.

[techtangents]

@visch I'm not a Python guy, but, that PR looks like exactly what we want. Really appreciate you jumping in and helping with this.