Accounting

[pierrejoye]

Hello,

I made a POC for accounting, based on conntrack which is very likely already avaiable on any device using Luci (what is used for realtime connection tracking on Luci's front).

Before I go for a cleaner version, do you have it somehow already?

If not, I was pondering between create a separate service similar to uspotfilter, and a recurring call to read the conntracker nf_table, and feed uspotfilter, or inside uspotfilter. It can be done in uspotfilter directly but it is a bit out of uspotfilter scope. Thoughts?

It also sends the data to the uam (for stats uses and related, only the IP of the client in the uspot instance, not visited IPs).

[f00b4r0]

I have considered this option and dismissed it: constant polling is taxing on the CPU and will not provide sufficient precision for quota management (disconnecting a client which exceeds their quota) unless the polling interval is very short, at which point the CPU load will definitely be unacceptable.

You also can't use event-based accounting because conntrack events suitable for accounting are only sent on connection teardown, so long-standing (e.g. streaming) ones would not be properly accounted for.

Furthermore using conntrack means IP-based accounting, whereas uspot/uspotfilter are MAC-indexed, so there would be extra collation work to perform.

My current opinion is that the best tool to implement accounting is eBPF which would solve all these issues in a clean and CPU-friendly way: I have an idea of what's needed but no time to deal with it yet.

tl;dr: I wouldn't waste too much time with that approach.

[pierrejoye]

ah totally makes sense, thanks 😊

I tried for a couple of hours learning that part of the uspot code base in the process. eBPF is indeed significantly more effective.

if you like a hand on that, don't hesitate to ask me :)

[pierrejoye]

I was looking at what is available by default on recent openwrt and what tools doing this kind of things are using. fw4 and nftables are standard and seems to be used by similar services for openwrt recent versions.

ebpf is available but requires a custom kernel. That's something worrying me a tat bit and it will considerably complicate the release/deployments. I use some pretty standard routers, mid level type (Archer series AX1800 or similar, gl-inet). So far I was using opennds and it relies on nft for in and outgoing traffic. When it gets busy, there are maybe 100 clients and so far no slow down or heavy CPU load have been tracked.

I was wondering if that could be an approach too.

Something like on client_create:

const nft_add_counter_in = sprintf('nft add rule inet fw4 track_in ether saddr %s counter comment "%s-in"', mac, client_id);
const nft_add_counter_in = sprintf('nft add rule inet fw4 track_out ether daddr %s counter comment "%s-out"', mac, client_id);
let p_in =  popen(nft_add_counter_mac_in', 'r');
let p_out =  popen(nft_add_counter_mac_in', 'r');

and the same but to remove the rules on client_remove.

It could be done in a non intrusive way in the current uspot base, config based f.e. I am sure eBPF will be faster. At the same time, my personal biggest challenge is to avoid custom builds.

Just a thought as it does not look overly complex to implement (I will spend more time on the rules than anything else anyway). It could be worth a shot and would not step on a more effective solution based on eBPF. Thoughts?

[f00b4r0]

ebpf is available but requires a custom kernel.

No it doesn't. libbpf has been supported in OpenWrt for quite some time now.

[pierrejoye]

Correct, however the ebpf user mod part required compilation, which is not a big deal as uspot already has compiled services and tools in place.

What do you think about allowing either way by doing this separately and only send the per Mac data to uspotfilter?

A entry point in uspotfilter to add traffic data per Mac or mapped by MAC can be called with any implementation.

that would bring a solution to most medium users already while leaving the door open for an ebpf implementation.

I may be wrong here, I feel ebpf is perfect but benefits mostly for high traffic setup, where tbe hardware would be significantly more powerful, more storage as well.

I did use ebpf before but for system calls profiling and debugging, mainly for fs access, very powerful and little impact, but on servers, slightly easier to customize pkg or builds :)

[f00b4r0]

Correct, however the ebpf user mod part required compilation, which is not a big deal as uspot already has compiled services and tools in place.

There are already OpenWrt packages shipping eBPF modules, this is no problem at all, the OpenWrt build infrastructure can handle this.

What do you think about allowing either way by doing this separately and only send the per Mac data to uspotfilter?

I don't want to end up with having to maintain multiple redundant bits of code (or multiple interfaces for the same functionality). I want to keep things simple and uncluttered. Besides, you can't "send" data to uspotfilter, or you're back to the same CPU-hog type of load as you would with a polling system, with the extra downside of having uspotfilter processing this flow of data at times it wouldn't otherwise need to. While ucode is pretty fast and efficient for an interpreted language, it's not C-fast/efficient and it's single-threaded.

I believe eBPF is the perfect tool to extract just the information needed (packet size) from packets going to/from uspot clients in the least obtrusive and most efficient way.

My current idea is thus to have a very simple eBPF module handling tx/rx counters for each registered MAC address which uspot can then lookup as needed (which currently would be in the accounting() loop, although I also plan to eventually trim it down and do more things asynchronously to reduce the cpu load here as well). That's pretty much "all" that's needed to reliably implement accounting and quota limits IMO and that's what I plan to implement, eventually.

To summarize, this eBPF module:

• imparts the least amount of CPU and memory load for the given task
• doesn't care about connection states, it would sum all seen packets for a given MAC (if you want to do accounting via conntrack, you need to both poll and handle conntrack events to accurately account for all data)
• doesn't pollute firewall tables with numerous individual counters rules that themselves need to be managed (setup/teardown) and processed individually to extract the relevant data

tl:dr: a non-eBPF implementation will have a pretty high burden of proof to convince me to merge it.

HTH