-ntop is great.
-Check for processes gone crazy.
-check logs for suspicious or large activity.
-assuming you're using iptables dump your per-rule packet/byte counts every 5 
minutes with a timestamp and see if there is a significant jump during the 
timeframe your ISP indicates. and add some LOG rules to iptables.

On Wednesday 09 August 2006 10:07, Dan Rue wrote:
> Whoa.  There could be a number of legitimate things wrong that could
> cause bandwidth to spike.
>
> Check the logs for all the services you're running.  Sometimes a poorly
> behaved web spider or some idiot doing a wget --mirror can really cause
> problems (ask me how I know).
>
> For the future, set up graphs using mrtg or rrdtool and friends so that
> you know what is going on.  Maybe the isp is just seeing a daily cron
> backup or something?  Hard to say without more details, or the graphs
> themselves.
>
> To monitor in real time, check out iftop - it's a top like interface
> that shows current connections and kbps.
>
> Unless you have further evidence of being hacked, the prudent thing to
> do is check the obvious and legitimate things first.
>
> Dan
>
> On Tue, Aug 08, 2006 at 11:39:31PM -0500, David Carlson wrote:
> > 0) Absolutely distrust the server in question.  If it appears that users
> > aren't logged in, don't believe it. That goes for most admin utilities
> > (w,users,uptime).  Don't think you can delete things and restart it - you
> > want to reimage the OS.
> >
> > 1) Ask the ISP if they detect promiscuous mode (meaning suspicious ARP)
> > coming from the server
> >
> > 2) nmap or have the ISP nmap the server (from a nearby host)
> >
> > 3) Check for strange traffic with tcpdump/tshark (exclude the login
> > traffic port with [(tshark) -f] 'not port 22', etc).  This is probably
> > only useful from another machine that sees all the traffic from that
> > machine.
> >
> > 4) Check for rootkits. http://www.chkrootkit.org
> > This isn't totally reliable though.
> >
> > 5) Sniff (or, better, have the ISP sniff and deliver) some outgoing
> > traffic and analyze it with wireshark GUI.
> >
> > If any of the tests show something wrong, have the ISP cut power (don't
> > run 'halt') forcefully.  Save the hard drive image somewhere for
> > forensics (don't boot off of it).  You will likely have to rebuild the
> > server - the only thing you should copy over are user files that have
> > been examined.
> >
> > -Dave
> >
> > On Tue, August 8, 2006 10:03 pm, Chris Schumann wrote:
> > > The ISP of my company's server called because our bandwidth was
> > > spiking. No
> > > one was logged in, and I'm not sure how to pinpoint what caused the
> > > traffic.
> > >
> > > Tips or pointers on where to track this down are most sincerely
> > > appreciated.
> > >
> > > Many thanks,
> > > Chris Schumann
> > >
> > >
> > > _______________________________________________
> > > TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> > > tclug-list at mn-linux.org
> > > http://mailman.mn-linux.org/mailman/listinfo/tclug-list
> >
> > -=-=-=-=-=-=-=-
> > David Carlson
> > thecubic at thecubic.net
> >
> > _______________________________________________
> > TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> > tclug-list at mn-linux.org
> > http://mailman.mn-linux.org/mailman/listinfo/tclug-list
>
> _______________________________________________
> TCLUG Mailing List - Minneapolis/St. Paul, Minnesota
> tclug-list at mn-linux.org
> http://mailman.mn-linux.org/mailman/listinfo/tclug-list