CSI: Engage
This article is based on a true story.
Troubleshooting intermittent technical issues in general is much like a criminal investigation. Interestingly enough both require careful observation, listening, experience, inquiry, etc.
1723 hours, 4/9/2010, Atlanta, GA, HQ
Received support request from on-site support personnel that a new installation was activated after testing but communications were severely hindered leaving civilians in the dark. Unfortunately, they could not turn back immediately and may be in this crippled state for over a week without outside assistance to troubleshoot the issue.
0900 hours, 4/10/2010, Suburb of Atlanta, GA, Scene of the crime
A seasoned investigator was dispatched to the scene. Upon arrival he was given a walk-through of the installation and the impact of the issue on the user community. The investigator listened carefully, not only to what the on-site support was saying but also to the secondary comments heard in the halls and around the office – more clues. Questions were asked carefully not to lead the witness, as much can be learned when recounted in one’s own words and personal experience.
Investigation ensued and after several hours of analyzing event logs, server configurations, traces, tests, more questions and answers many things were discovered but no smoking gun – yet.
1030 hours, 4/11/2010, Atlanta, GA, HQ
The lead Investigator was back at the desk working other cases when a phone call came in from the distressed customer (victim). This time things were worse as the situation had deteriorated. Quickly, the investigator summoned his trusty tool-of-the-trade, VPN access to the customer site accompanied by ‘old faithful’ – remote desktop. A few more questions about the circumstances leading up the new development quickly pointed to the physical layer and a few adjustments and some hands-on from the customer on site yielded fast results. Crisis averted. However, the original problem persisted, albeit, not as prolific.
After assessing the situation and considering the change of behavior from the previous discovery the physical layer required further scrutiny. After gaining access to the devices in question, probing them for evidence, researching their respective documentation, a few additional changes were advised for after-hours.
1735 hours, 4/11/2010, email from victim to investigator
“Advised changes were completed, systems were tested and appear functional. Please monitor and we’ll reconvene after the weekend.”
1130 hours, 4/12/2010, Atlanta, GA, HQ
Conference call to discuss actions and observations following advised changes. All system appear to be functional and issue appears to be resolved. Further monitoring required for assurance.
The investigator provided a few more suggestions based on evidence gathered during the investigation to prevent future issues. Customer agreed to pursue the suggestions and report any anomalies.
Case closed
No related posts.
