Home Application Network Apps Splunk Troubleshooting example using Splunk - Syslog

Splunk was configured as a Syslog server on my lab network for few weeks now. Since I logo_splunk.pngmodified few parameters on service side and configured network devices, firewall, switches and router, which sent syslog message to the Splunk server. Past few days it built pretty thick index database. Well my index size is way below than 500Mbyte daily limitation for free version of Splunk. I am not expecting huge errors and critical messages because it is small Lab and also I am only monitoring network devices.

Today, while I was playing and reviewing Splunk, I found some of device within network sending spoofing traffic which is sending packet to unknown network, 192.168.100.15(RFC1918). As far as I know I didn't set it up within the lab network.

Let's find out what's going on.

From main page, search '192.168.100.15'. Many packets destined to 192.168.100.15 are keeping forwarded to 192.168.77.1 which is my firewall. And it dropped those. The packets were origin from 192.168.77.9 that is my wireless access point at lab.

splunk_troublehoot1.png

 

Here is my lab network layout. As I mentioned, those ghost packets are form coming from access point (192.168.77.9). There are three candidates who might generate the packets. I got lucky. First one I checked is the one.

 lab_network_layout.png

 

From DOS prompt, type 'netstat -a' to see TCP session. That's it. Notebook/ibm is sending syn_sent to destination IP, 192.168.100.15:9100. Okay now we know which device is generating the useless traffic. How we can stop it? 

splunk_troublehoot2.png

 

In order to figure out what processor is involved with the behavior, type 'netstat -b' command.
PID stand for Processor Identification will be shown up. That is 324.

splunk_troublehoot3.png

 

Now, open 'Window Task Manager' and find a PID number that is associated with program or service. In this case, printer spooler service is the one. You can kill the processor, but I had to check one more think to kill the processor in permanent. 

splunk_troublehoot4.png

 

Looked Printer folder and figured which printer kept sending data to the network. "Dell Laser Printer 3000cn" is the one and I remembered I used the notebook for church event last year. I setup network printer which has 192.168.100.15. Wow, one of printing job is hanging on the printer queue since then. Finally it is removed.

splunk_troublehoot5.png

 

Nice Splunk! The traffic was not critically affected to my network, but it was good to be removed.

 

 

Last Updated (Thursday, 25 June 2009 20:23)

 
Smart Link
Search
Statistics
Content View Hits : 2219315
Polls
Highly recommended firewall vendor?
 
Google Translation
English Arabic Chinese (Simplified) Czech Dutch French German Italian Korean Portuguese Russian Spanish Filipino Vietnamese Thai Turkish
BGP routing issue?
Banner
World Route Servers
Banner
Who's Online
We have 87 guests online