PPPoE server performance / problem with uploads [long]

Hi all,

I have (about 500 users total and up to 200 pppoe online users at
one time) network based on Mikrotik bridging traffic that is
transported by PPPoE protocol. Uplink is about 30-40 mbits . Software
I use :
-pppoe-server is rp-pppoe 3.8 (compiled from source as well as from packages)
-ppp version :

beta skrypty # pppd --version
Plugin radius.so loaded.
RADIUS plugin initialized.
Plugin radattr.so loaded.
RADATTR plugin initialized.
pppd version 2.4.4

-freeradius 1.1.6
-mysql 5.0.22
-iptables v1.3.4 (from source with p-o-m)
-kernel 2.6.15 (vanilla from source with p-o-m)
-patch-o-matic with some patches (conntrack,multiport etc..)
-IMQ for kernel 2.6.15
-ipp2p (source)
-iproute2 (source)

Kernel configuration regarding PPP:

beta skrypty # grep PPP /usr/src/linux/.config
CONFIG_PPP=y
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_SYNC_TTY=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=y

* Server load is about 20-30% of system load with loadavg 0.0
* About Max 512 of 2048 MBram used
*there are 3 interfaces :
- eth0 - internet
- eth1 - dhcp users (very few - about 30 )
- eth2 - NIC dedicated for pppoe users and pppoe-server

Some graphs/benchmarks:

* there are ethernet errors that are probably caused by wifi nework (
not shure ) , the graph is here :
http://www.filegunner.net/uploadedimages/329868beta-if_err_eth2-day.png
*CPU load (2x P4 3.0Ghz )
http://www.filegunner.net/uploadedimages/140851cpu-day.png
*Entropy (month ) :
http://www.filegunner.net/uploadedimages/371789entropy-month.png
(probably something here is making the problem)
*Interrupts : http://www.filegunner.net/uploadedimages/491292interrupts-day.png
*IRQstats : http://www.filegunner.net/uploadedimages/433789irqstats-day.png
(eth2 is the one with pppoe-server on )

My problem is that i have extremely low transfer (upload from client
to the internet) when the client is connected to the internet by
pppoe-server . I also have some DHCP users and there is no problem .

Typical iptables rule for an user is :

beta skrypty # iptables-save | grep 10.100.2.250
-A FORWARD -s 10.100.2.250 -j ACCEPT
-A FORWARD -d 10.100.2.250 -j ACCEPT
-A PREROUTING -d 217.x.x.250 -i eth0 -j DNAT --to-destination 10.100.2.250
-A POSTROUTING -s 10.100.2.250 -o eth0 -j SNAT --to-source 217.x.x.250
-A UPLOAD -s 10.100.2.250 -o eth0 -j CLASSIFY --set-class 0002:4001

Typical traffic shaping rule of outgoing traffic for an user (here we
have 4 mbit rule)

beta skrypty # tc class show dev eth0 | grep 4001
class htb 2:4001 parent 2:4 leaf 4001: prio 5 rate 64000bit ceil
4096Kbit burst 4294950117b cburst 2112b

Typical rule for incoming traffic :

tc filter add dev imq0 parent 1: protocol ip u32 match ip dst
10.100.2.250 flowid 1:2001
tc qdisc add dev imq0 parent 1:2001 handle 2001:0 sfq perturb 10
tc class add dev eth0 parent 2:4 classid 2:4001 htb rate 64kbit ceil
4096kbit burst 4096kbit prio 5

And the route to IMQ interface :

beta skrypty # iptables-save | grep IMQ
-A POSTROUTING -o ppp+ -j IMQ --todev 0

What i've tried to do :

a) flushing and deleting the iproute2 from kernel table so the "tc
class show dev eth0" or "tc qdisc show dev eth0" show nothing. It
means that users had absolutely traffic_control-free access to the
internet. That eliminates problems with traffic shaping, CBQ, HTB ,
iproute2 or SFQ .
b) recompiling rp-pppoe and ppp with kernel for many times
c) flushing iptables leaving a big masquerade for all - that
eliminates problems with not working firewall rules.

Effect : dhcp users having normal transfers (up to 20,30 mbits ;) ) ,
pppoe users having problems with their uploads

Thanks for your time in advance.
Regards,

--
Wojciech Ziniewicz
Unix SEX :{look;gawk;find;sed;talk;grep;touch;finger;find;fl
ex;unzip;head;tail; mount;workbone;fsck;yes;gasp;fsck;more;yes;yes;eje
ct;umount;makeclean; zip;split;done;exit:xargs!!;)}

--