Date: 2005-05-04 13:02:00
Tags: web, computers
per-process bandwidth accounting
Dear Lazyweb,

I want to track bandwidth used by processes on my colo server on a per-process basis. I'm running Apache, BitTorrent, Freenet, and miscellaneous other things like automated backups. I have a monthly bandwidth cap of 1800 GB. I'm running FreeBSD 5.3. What are my options?
Have a donut and coffee, and pay somebody else to do it? =oD

heh..sorry, that's all I've got. =oþ
I'll add greg to the list of people who will be helping to toss your butt out of a plane this summer. (Something he's already done.)
OH NO!! There are TWO of you?!?! =oO

I'm SURE to be killed.

I didn't realize you guys were "real life" type friends. You guys should post more so I can figure all this stuff out.

At the rate either of you post, and as much content as you put into your posts, I won't know your favorite food, drink or color until...oooh....maybe 2007...? ;oP
Well I first encountered your journal when I was over at [info]snaxxx 's place with [info]decibel45 and [info]_fool , and we were planning SXSW 2005 Nuclear Taco Night. I can't find it now, but you invited people to call your voice mail and leave a message. We did, collectively, and I recall [info]snaxxx mentioning that it made an impression on you. :)

You definitely should plan a trip to Austin sometime so we can throw you out of an airplane, feed you , have some homebrew (and make some more), and so on. And a new thing - last night we made fresh mozzarella cheese (first time for all of us) that was absolutely to die for.
Ooooh YOU were the Cartman??? xoD hahaha! That was great. My favorite message of the night. And it's still on my machine. I refuse to erase it.

I'll be making my annual summer road trip in a few months, and am actually in the process of planning out my locations I'll be going to right now, and Austin is already on the list!

But I can't make any promises about jumping out of planes. heh. However, I'm more than game for tacos drinking and all the cheeziness you people have to offer! =oD
[info]decibel45 is the loud one, he was cartman.

I don't think we've tried drinking the tacos before, though. You'll have to let us know how that goes.
[info]lucky_mcgee : 'scuse me for forgetting a comma.
tacos, drinking...

Better!? =oP
You know [info] has jumped out of a plane as well, right? :>
OH GOD NO. The pressure.

Can't we just eat nuclear tacos instead?!? =oD
Don't worry, the pressure isn't a problem. At roughly 1 inch of mercury per thousand feet, the pressure at exit altitude is just over half what it is on the ground. :)
You're not helping the fears here.
Sure, after we get back on the ground. :)
I can't believe for like 0.009 seconds, after I read that, I actually entertained the idea.

Thank the good lord for reality. heh.
As you will no doubt see as this project proceeds, that is so not my style. :)
[info]lucky_mcgee : So I'm not one of the cool kids...
heh...I'm sure it isn't, but like I said, it's the best I've got. :oþ
That is a cool idea. Track and limit bandwidth per process (or process type, all httpd clumped together in stats and control). Time for greggie to do some kernel hacking? :)
I don't think it's going to need kernel hacking. I'm pretty sure I can do this in all user mode code.
The only thought that comes to mind (and this is a bit ugly, so hold on to your hat) is this: NAT let you forward ports from the outside interface to a specific IP behind NAT. So, if you can collect SNMP data on aliased IP addresses, you can run NAT and have it point at a bunch of aliased IP's.

Another possibility is afaik ipfw will let you filter by user, so you might be able to do something with that.

Maybe your best bet would be to write some code that ties into tcpdump and tracks how much data went where.
Well see I've got this great colo server that's not behind nat or anything, it's plugged right into the internet. It would be a real shame to set up some kind of virtual nat layer on that machine!

I do run the major services on my machine under separate accounts, so perhaps I could look at the per-user accounting angle for this. Maybe a jail with the same root directory.

Using data from bpf (berkeley packet filter, which tcpdump uses on freebsd) wouldn't tell me where the data comes from, except by port number which is annoying for dynamic services like bittorrent and freenet.
I'm pretty sure jail will want you to setup seperate IPs for the jails.

If you use the per-user/group filtering in ipfw, you could then use tee to send it to a divert port, which could be a daemon that does nothing but count bytes (and packets). It might not be 100% accurate compared to SNMP because it would ignore lower layer overhead.
Greg Hewgill <>