Drop in Snowflake users from Russia and evidence of blocking starting 2024-11

[cohosh]

We've noticed a significant drop in Snowflake users from Russia, starting earlier this month, along with some vantage point testing that suggests a different kind of block than what we've seen before.

At first we thought this was due to several recent certificate renewals of Fastly front domains that were used for Snowflake, but we've since ruled out blocking of rendezvous channel. The timing of the drop in users also doesn't quite match up with the domain renewals.

From our vantage point tests, it does not look like the DTLS fingerprinting of Snowflake that happened in Russia in 2021. The DTLS handshake completes and several bytes of application data are received from the proxy before the client suddenly stops receiving data. This doesn't rule out fingerprinting. There are a few proxies that do not appear to be blocked and Tor can be fully bootstrapped through these proxies, but we haven't noticed any difference in the DTLS fingerprint between working proxies and proxies that go stale. It's also possible that our vantage point tests aren't offering an accurate view of why there is a drop in users.

Related Links:

• Snowflake GitLab issue: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40407
• Full analysis of previous DTLS fingerprinting: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40014#note_2765061
• ntc.party topic: https://ntc.party/t/drop-in-snowflake-users-from-russia-and-evidence-of-blocking-starting-2024-11/13083

[PricacyPro0]

on my Android Orbot Snowflake Relay I saw a recent sudden increase in twice as much russian connections as usual per day. I already thought, that something must be going on.
Just spread that more people start installing the snowflake browser extension.

[IrradiatedKiwi]

I find the following result interesting. Because earlier this year in January I experience the similiar situation in China.

What's weird is that data channel connections to proxies appear to be opening successfully, and some data is being sent bidirectionally before the connections become stale. From one of the snowflake probe logs (edited to remove extraneous messages)

Here is the post I made at that time:

#325

and @wkrp wrote this in that post:

The anti-censorship team looked into this report a little. The cause is uncertain. The logs show STUN, rendezvous, and DTLS connection establishment working correctly, but apparently not much data is transferred over the DTLS connection before it is closed.

My experience at that time was that the snowflake bootstrap could not even finish most of time.
But in rare case they did finish and wireshark would start to show some DTLS Application Data transfer. But no websites could be opened.
And in even rarer case website could work, It would be very slow and the connection would only last for a really short time (no more than 1 to 3 minutes). Then the connection would be blocked and wireshark would show Encrypted Alert messages.

The problem lasted for more than one month then gone, We couldn't figure out the exact reason back then. At that time we even couldn't figure out the width of the blocking.
I am not sure whether that case is related with this one. But still maybe the experience is kinda similar?

[cohosh]

I've written a patch for snowflake to more easily test Snowflake connections from vantage points. Manual WebRTC signalling can be used to test client connections to a specific Snowflake proxy under your control that also has manual signalling enabled. Hopefully this is useful to anyone who wants to try out theories on how it is being blocked or test fixes.

You will need the following patch for both the client and the proxy: https://gitlab.torproject.org/cohosh/snowflake/-/commit/9523d9bbed0baf44e8779c1c78bb7ebd4e501a8a

Proxy setup

To run the proxy, you will need 2 terminals:

• Terminal A: Run the proxy with ./proxy --copy-paste --verbose --unsafe-logging
• Terminal B: Run cat > signal

Client setup

To run the client, you will need 3 terminals, and the following torrc file:

UseBridges 1
DataDirectory datadir

ClientTransportPlugin snowflake exec ./client -copy-paste -log snowflake.log --unsafe-logging

Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn

SocksPort auto

• Terminal A: Run the client with tor -f torrc
• Terminal B: Run tail -f snowflake.log
• Terminal C: Run cat > signal

Manual signalling

Once the client has started, you will see something like the following output in the client's Terminal B.

2024/11/20 00:45:58 Please Copy & Paste the following to the peer:
2024/11/20 00:45:58 ----------------
2024/11/20 00:45:58 

{"type":"offer","sdp":"v=0\r\no=- 7792309864421436583 1732063557 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 82:33:BE:0E:A0:DC:FC:24:85:B5:7B:D5:9D:64:32:C9:A4:2F:56:B0:A6:47:35:B0:A4:4F:5D:96:E9:A6:7F:DA\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:vggBpjGDVijaxuQY\r\na=ice-pwd:skiZVUcgjPZpSyENHMHFBtRGUszeInMU\r\na=candidate:1408731006 1 udp 1694498815 XX.XX.XX.XX 40705 typ srflx raddr 0.0.0.0 rport 40705\r\na=end-of-candidates\r\n"}

2024/11/20 00:45:58 ----------------

Copy that, and paste it into the proxy's Terminal B. Soon after, you should see the following output in the proxy's Terminal A:

2024/11/20 00:46:05 Please Copy & Paste the following to the peer:
2024/11/20 00:46:05 ----------------
2024/11/20 00:46:05 

{"type":"answer","sdp":"v=0\r\no=- 415248651781399192 1732218246 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 25:C6:51:5D:35:61:F6:21:40:39:60:DB:47:85:9F:78:EB:A6:8E:B5:27:91:8C:AA:34:4E:73:DD:A4:1B:65:85\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:active\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:zyyLdoWNpFiIRUXn\r\na=ice-pwd:AzLMxJJrliwDFEHzpSwODjFrqlCtYsbv\r\na=candidate:1408731006 1 udp 1694498815 XX.XX.XX.XX 52584 typ srflx raddr 0.0.0.0 rport 52584\r\na=candidate:1408731006 2 udp 1694498815 XX.XX.XX.XX 52584 typ srflx raddr 0.0.0.0 rport 52584\r\na=end-of-candidates\r\n"}

2024/11/20 00:46:05 ----------------

Copy that and paste it into the client's Terminal C. If everything goes right, the WebRTC datachannel should open at both the client and the proxy.

[cohosh]

It looks like the blocking has at least temporarily stopped since Thursday. Most snowflake connections from our vantage point have been succeeding for the last few days. The number of snowflake users from Russia seems to be climbing again.

[kad09]

Nobody seems to mention one thing outside of russian-speaking forums. There's the thread on NTC discussing seemingly random inaccessibility of Hetzner IPs from Russia. In post 435 one user found the culprit of immediate blocking of (all/most?) Hetzner IPs. It is enough to visit https://www.phpmyadmin.net in your browser to trigger block (no answer to TCP SYN to any Hetzner IP).

I didn't believe it at first, but decided to test. I'm an Arch Linux user, most of arch infra is hosted on Hetzner, and I've never had any problem with downloading from AUR, or reading archwiki from Russia (home, non-mobile internet). But as soon as I have visited the forementioned site, traffic to Hetzner got blocked, and I couldn't open neither AUR nor archwiki (no answer to TCP SYN). It lasted for at least 30 minutes, I had to go and turn off the computer so I didn't check for longer, but people on NTC told the block can last for up to 2 hours. The block didn't affect phpmyadmin site itself, it opened perfectly fine all that time.

phpmyadmin site is the default front of Tor Snowflake, so obviously the censor targets that site, as we can see, in interesting way. I wonder if other hoster's IP ranges are blocked too, or monitoring of user's traffic becomes stricter after that (maybe WebRTC connections succsess rate drops or something else, this requires additional study).

Accessing phpmyadmin's CNAME address (1115546720.rsc.cdn77.org) doesn't seem to trigger block.

Also in my opinion choosing phpmyadmin website as front is bad (or worse than previous domains). Unlike previous fronts like cdn.sstatic.net or ajax.aspnetcdn.net, there are less people to visit phpmyadmin site directly (installing from repos, using instructions in someone's blogs to set it up rather that reading official instruction on official website), and these people are very specific and narrow kind of people (developers), unlike regular people visiting regular sites with js cdns linked in them. But I guess Tor devs don't have many options since more and more cdn are blocking domain fronting. But you could have provided some address for tech-savvy users to set up fronting with user's own domain (domain borrowing or whatever it's called), so power user and tor broker can meet and just negotiate through some cdn that doesn't have blocking and without censor seeing tor broker domain. I wonder whether that would work, did someone propose that?

[ghost]

with user's own domain (domain borrowing or whatever it's called)

It's called "steal oneself," even if this is not quite correct English.

[meskio]

But I guess Tor devs don't have many options since more and more cdn are blocking domain fronting. But you could have provided some address for tech-savvy users to set up fronting with user's own domain (domain borrowing or whatever it's called), so power user and tor broker can meet and just negotiate through some cdn that doesn't have blocking and without censor seeing tor broker domain. I wonder whether that would work, did someone propose that?

You can do that easily just changing the "fronts=www.phpmyadmin.net" part of the bridgeline. You'll need use a domain pointing to cdn77.

If what you want is to host it in a different CDN that is a bit more of work, and you'll need to need to configure it to forward the traffic to the broker. We don't have a guide for that currently. But you can just use apmcache or sqs instead of domain fronting: https://forum.torproject.org/t/new-sqs-rendezvous-method-for-snowflake/11713

But I have the feeling that the hetzner block is not related to snowflake, as not so many snowflake proxies are in hetzner, but it might be an attempt to block bridges. As many bridges are hosted in hetzner and we use the same domain front for requesting bridges from Tor Browser.

[temhelk]

I had to go and turn off the computer so I didn't check for longer, but people on NTC told the block can last for up to 2 hours

From my experience the block is indefinite, I had it for 2 days until I disconnected my internet for 15 minutes. I assume it would not go away at all without that. That said I also have a static ip address which might be the reason.

I was reading different English speaking places that discuss that block and I'm surprised that there's not a lot of mentions of OVH ips being blocked as well, which happens together with Hetzner, which is the case for me and a lot of people on NTC.