Domain Status False Positives

[robindimyan]

Tool informs me that some domains are available even when they're not. See the screenshots.

[picatz]

How dare it say such lies.

I'm looking into this problem now.

[picatz]

@robindimyan Ooooh, so. This is actually quite interesting!

Facebook seems to have registered www-facebook.com but there's no actual DNS related things to actually resolv that address. Meaning, they can own that address but obviously don't have to force any sort of resolution of the address to an IP.

No Dice

Which means when, as catphish is currently implemented, there's no secondary check for validity.

require 'resolv'
Resolv.getaddress "www-facebook.com"

Will error out like so:

Resolv::ResolvError: no address for www-facebook.com

Whois Tell Us Who Is Plz

But, if we ran a whois on that address:

> whois www-facebook.com

We will see the whois information for facebook.

Ping'em

If we ping the address, just cuz:

> ping www-facebook.com

We will see:

ping: cannot resolve www-facebook.com: Unknown host

[picatz]

Well, I guess it's not that interesting. But, it means something.

How To Fix

Honestly, after you pointing out this issue, I am seeing that catphish is using the whois gem -- but, like, not in the code at all. CORRECTION: It's only being used for puny code resolutions. Perhaps a whois call could be made for all of the failed resolv methods that error out.

OR this could be implemented as a second option. But, there's also the case of why it's still saying it's available for some reason. CORRECTION: Because it doesn't work properly, obviously. Which I'll look into now.

[picatz]

I wonder if the resolv option should just always be a whois option. Or to separate the two functionalities into their own command-line arguments. There's lots of options with that.

[picatz]

Yay Redundancies

Since this is just one of those fun teaching moments: let's do that.

Catphish, as it is currently implemented, has the following gems:

require 'resolv'
require 'getoptlong'
require 'simpleidn'
require 'whois'
require 'whois-parser'

Let's narrow in on just whois and whois-parser for a second.

This is how that happens in, let's say, a pry REPL:

[1] pry(main)> require 'whois'
=> true
[2] pry(main)> require 'whois-parser'
=> true

Now, what if we reversed that order?

[1] pry(main)> require 'whois-parser'
=> true
[2] pry(main)> require 'whois'
=> false

Redundant Gems are Fun

You actually could just require the whois-parser gem instead of requiring the whois gem and the whois-parser gem. This is because the whois-parser gem is actually also requiring the whois gem for you. Because it's actually extending it's functionality (if I'm not mistaken about the extending part).

So, anyway.

Still working on fixing some stuff.

[picatz]

Whois Parser Makes Me Feels Better

require 'whois-parser'
# This is almost kinda, like, ugly tho, but works:
Whois.whois("www-facebook.com").parser.available?
# => false

I'm pretty sure just relying on the whois information is going to be more reliable.

So, perhaps if there was a resolv command-line option. It could be for speed, maybe. Depends on the speed of getting/parsing whois information. There's some design thoughts to be had for sure. And benchmarks might help.

[ring0lab]

The problem with 'whois' its limited to <= 200 - 300 queries before it gets timeout, which is not going to work for a large number of domain queries. On the other hand, 'resolve' gives a faster response and can query a large number domains without any issues. However, like in this case 'www-facebook.com'', it gives a false positive because the domain can't be resolved, which is not the case for 'whois'. We are aware of this issue and it will be addressed soon in the near future. Any suggestions would be appreciated.

[dgadelha]

What about dig for NS entries on NIC?