Introduction: The Calm Before the Storm

Supporting IT operations across multiple time zones comes with its own challenges. When a company operates around the clock, its infrastructure, support teams, and business operations don’t always follow the same schedule. A problem that appears during one shift may disappear before anyone has identified its cause, only to return when the next working day begins.

That’s exactly the situation Wolf O’Rourc found himself dealing with when a call center started experiencing serious network performance problems. What initially looked like an intermittent issue would eventually lead to an unexpected source of network congestion.

Chapter 1: Networking Issues

The company operated multiple data centers around the world, following a “follow the sun” approach to handling transactions. As business activity shifted between regions, the workload moved with it. Each location also had a customer service center supporting its operations. This meant that infrastructure and support teams had to coordinate across different time zones.

In the United States, much of the transaction activity followed East Coast business hours. Meanwhile, operations on the West Coast were still getting started. This created an additional challenge for IT support, as technical staff weren’t always available during periods of highest demand.

Different teams were responsible for different parts of the infrastructure, making network troubleshooting more complicated. Identifying and resolving problems became particularly difficult when symptoms appeared and disappeared throughout the working day.

Chapter 2: Communication Breakdown

Supporting operations across different time zones also meant working with separate IT teams. Each team was responsible for its own part of the infrastructure. When the call center began experiencing serious network performance problems, the help desk followed its usual troubleshooting procedures. However, they couldn’t identify the cause.

Then, at 5 pm, the problem simply disappeared.

“Help desk went through all the troubleshooting, didn’t find anything. And then suddenly at 5 o’ clock, the problem went away. So nobody cared.” -Wolf O’Rourc

With the systems apparently working again, there was little immediate pressure to continue investigating. But the underlying problem hadn’t been resolved. When employees returned the following morning, the same network issues returned, and what had initially appeared to be a temporary disruption was becoming a recurring operational problem.

Chapter 3: The Mystery of the Network Overload

By the following morning, the network problems had returned, leaving hundreds of call center employees struggling to do their jobs. Something was consuming an enormous amount of network capacity, but the source of the traffic remained a mystery.

As the situation escalated, the Vice President of Operations demanded an investigation and a solution, bringing Wolf O’Rourc into the picture. What initially seemed like an intermittent performance issue was now seriously affecting customer service operations.

The investigation revealed that excessive network traffic was overwhelming the Cisco Gigabit backbone. The traffic consumed so much capacity that even routine operations became difficult. The question was no longer whether the network was overloaded, but what was generating all that traffic in the first place.

Solving the Mystery

Further investigation revealed that the excessive traffic was coming from within the company’s own network. Using packet monitoring at headquarters, Wolf and the network team began examining what was consuming so much bandwidth.

The results pointed toward video traffic. Large amounts of video data were moving across the network, consuming capacity that the call center needed for its daily operations. But where was all this video coming from, and why was it appearing during business hours?

Chapter 4: The Unexpected Culprit

The source of the video traffic turned out to be the company’s own surveillance camera system. The IP cameras were configured to transmit video whenever they detected movement. This became a problem once hundreds of employees arrived at work.

During business hours, the constant activity triggered the cameras repeatedly. The resulting video traffic overwhelmed the network infrastructure shared with the call center. When employees left at 5 pm, movement decreased, and the video traffic subsided. The network returned to normal, only for the same cycle to start again the following morning.

The incident highlighted a problem that would become increasingly familiar as more devices were connected to corporate networks. Equipment installed for one purpose could have unexpected consequences for entirely unrelated systems. In this case, a security camera installation had become a serious operational problem for customer service.

---

“It goes back to know that the technology outruns the policies.” -Wolf O’Rourc

---

Resolving the Problem

Once the source of the excessive traffic had been identified, the cameras were moved to a separate VLAN. This isolated their video traffic from the network used by the call center.

The incident also prompted improvements to network monitoring and internal policies. Greater attention was given to devices that could generate unexpected traffic. What initially appeared to be a mysterious network performance problem demonstrated how even relatively simple equipment could have a significant impact on shared infrastructure.

Chapter 5: Could AI Have Spotted the Problem?

Looking back at the incident, Jack and Wolf discuss how modern network monitoring tools might have helped identify the problem much earlier. Rather than waiting for users to report performance issues, network traffic analysis can establish patterns of normal activity. Unusual changes can then be identified more easily.

AI-assisted monitoring could make that process easier by detecting unexpected traffic patterns. One example would be the sudden increase in video streams during business hours. In this case, the connection between employee activity, camera traffic, and network congestion might have provided an important clue before the situation escalated.

Of course, identifying unusual traffic is only part of the solution. Understanding where that traffic originates and why it affects other systems still requires knowledge of the underlying infrastructure. That knowledge is also essential to preventing similar problems in the future.

The discussion illustrates how network monitoring has evolved since the incident. It also demonstrates why visibility into network activity remains important, regardless of the technology used to analyze it.

Conclusion: A Lesson in Network Visibility

What started as an intermittent network performance problem turned out to be an unexpected interaction between two systems sharing the same infrastructure. The surveillance cameras were doing what they were configured to do, but the resulting traffic had consequences nobody had anticipated.

The incident highlights the importance of knowing what is connected to a corporate network and how those devices behave. It also demonstrates what can happen when their traffic competes with business-critical applications.

Proper network segmentation, monitoring, and communication between IT teams can help identify problems more quickly. Without them, troubleshooting can turn into a lengthy investigation of why everything slows down at 9 am and mysteriously recovers at 5 pm.

Sometimes the biggest IT headaches aren’t caused by sophisticated attacks or complicated software failures. Sometimes they’re caused by something as ordinary as a security camera watching people arrive at work.