ubnt case study

Views:
 
Category: Entertainment
     
 

Presentation Description

No description available.

Comments

Presentation Transcript

Troubleshooting SNMP and Telnet Failures With Wireshark:

Troubleshooting SNMP and Telnet Failures With Wireshark Tony Fortunato, Sr Network Specialist The Technology Firm

Problem Overview:

Problem Overview ISP reports Intermittent SNMP and Telnet failures with their Wireless Stations. They run a Perl script hourly for their “Fair Access Policy” that throttles the client to 300 kbps if they exceed their download limit. The client radio usage stats are gather by the script via SNMP If the client is throttled or being released from a throttle, telnet is used to change the bandwith settings ISP notices that occasionally the script records Telnet and SNMP timeouts Upon further analysis, it seems that telnet to the radios are not working as well as SNMP

Wireshark Setup:

Wireshark Setup To eliminate any background packets use the simple “ port telnet” protocol capture filter and another capture with just “port snmp” . These can be combined if I wanted them both, but I wanted to try and analyze one thing at one time. By using a port filter I can capture traffic to multiple radios in case some work and some don’t.

Results of Capture - SNMP:

Results of Capture - SNMP You can see the radio sending Port Unreachable Messages back Gratuitous ARPS, indicate the radio is rebooting and checking to ensure his IP is unique We verified that SNMP was failing, rebooted the radio and tried the same SNMP commands again After the reboot, the radio is responding to SNMP requests

Results of Capture - Telnet:

Results of Capture - Telnet Here I see Telnet data flow The FIN show me the script is done and it applied its changes I finally captured a radio where Telnet worked and then failed Now suddenly Telnet is refusing a connection.

Review:

Review Captured SNMP failing Radio’s SNMP service failed and was fixed by rebooting the radio Evidenced by ICMP Destination Unreachable/Port Unreachable We also manually reloaded the snmp daemon/service to fix it Captured Telnet failing Radio Telnet service also seemed to fail Evidenced by TCP RST packets Additional Observations Noticed SNMP fails after the telnet session/failure, not on its own Seems SNMP’s failure is directly related to Telnet script/failure Now What? Review Telnet script to see if there are any clues since this seems to be the only common event before the errors/failures

Script Analysis:

Script Analysis The main part of the script uses the Linux “ sed” command to search and replace a config file Then there is a command to commit the changes permanently The last command performs a ‘soft reset’ to apply the changes I took the commands and put them into notepad, then I pasted them into a command shell to see if I can reproduce the problem; Funny thing; I could NOT telnet back in after running all the commands.

Further Script Analysis:

Further Script Analysis So now I broke the script up; Firstly I ran the sed commands 10 times no issues Then I can the cfgmtd command to commit the changes permanently 10 times no issues Then I ran the softrestart command to apply the changes COULD NOT TELNET BACK IN – BINGO!!!!!

Proof Of Concept:

Proof Of Concept I asked the ISP to replace the softrestart with a simple reboot command We ran the script 10 times with out an issue Then we put the softreset back in and reproduced the problem 8 out of 10 times Then they asked if that is the issue why didn’t all the modems experience the problem I explained that firstly the problem wasn’t consistently reproduced 100% of the time even when we knew how to reproduce it. Secondly I told them that I doubted that all the radios exceed their download allotment. So the combination that not all radios had their configs modified and the softreset seems a bit unstable seems to be the issue. I have submitted my findings to the radio vendor and am awaiting their feedback/

Slide 10:

Tony Fortunato, Sr Network Specialist The Technology Firm Thank you Troubleshooting SNMP and Telnet Failures With Wireshark

authorStream Live Help