Why My SIEM Missed an Intrusion on a Misconfigured Switch

From Wiki Spirit
Jump to navigationJump to search

The alert arrived after the damage was done. An endpoint detection platform had flagged unusual activity on a workstation, and the investigation found that an attacker had moved across parts of the network that should have been separated. We went looking in the SIEM for a clear trail: connections, device events, perhaps an obvious policy breach. There was no useful story there.

That did not mean the SIEM was broken. It meant the network was not giving it the evidence it needed. A switch had been misconfigured, a monitoring path was incomplete, and the intended segmentation existed more clearly in the diagram than on the floor. The intrusion was visible at the endpoint, but much less visible in the network logs.

This is one of the uncomfortable truths of security operations: SIEM blind spots can begin with a patch lead, a switch port or a design assumption. Physical infrastructure is not separate from cybersecurity. It is part of the system that determines what can communicate, what gets monitored and what investigators can reconstruct at 2am.

The incident was not just a logging problem

In this scenario, a device was connected to a port that had been intended for a restricted network. The switch port configuration did not match the design: the access VLAN was wrong, and the port was not locked down as expected. A small, unmanaged switch then made the situation harder to understand by connecting more than one device through a single outlet.

From the user’s point of view, everything appeared to work. From a security point of view, traffic was crossing a boundary that the team believed was in place. The firewall rules were designed around the expected network layout, but the traffic was not taking the path the design assumed. Meanwhile, the SIEM received only a partial account of what happened.

It is tempting to call this a monitoring failure. That is only part of it. Logging is downstream of the network: if the network does not send the right traffic to a sensor, generate useful events or preserve reliable device records, a SIEM cannot invent that evidence.

How switch misconfiguration creates SIEM blind spots

Switches are often treated as quiet plumbing: install them, connect the ports and move on. That is a mistake. Switch configuration determines where devices sit, which traffic can reach them, and whether network monitoring can see what is happening. A switch misconfiguration can therefore create both a security exposure and a visibility problem.

VLANs are only useful when they are implemented correctly

VLANs can provide logical separation, but naming a VLAN “OT”, “guest” or “restricted” does not make it secure. The design depends on correct access-port assignments, controlled trunks, sensible treatment of unused ports and deliberate routing between networks. A wrongly assigned access port can put a device in the wrong segment. An overly permissive trunk can carry VLANs where they are not needed.

Segmentation also needs enforcement at the points where traffic moves between zones. That may involve routing controls, access control lists and firewalls, depending on the architecture. If a device can bypass the expected path, or if a routing boundary is missing, firewall policy may not see the traffic it was meant to control.

Monitoring links can fail quietly

Many organisations use switched port analysis (SPAN) or network taps to feed traffic to monitoring tools. These are useful, but they are not magic. A SPAN session may be pointed at the wrong ports, omit a relevant VLAN, or be oversubscribed when more traffic is mirrored than the destination port can handle. Traffic can be dropped without an obvious alert in the SIEM.

That is a classic source of SIEM blind spots: the monitoring system is healthy, the dashboard is green, and the packets it never received are absent from the picture. Network visibility needs to be tested end to end, including under realistic traffic load. If a sensor only sees a sample or a subset of the paths, incident responders need to know that before an incident.

Physical faults change the network’s behaviour

A damaged cable, failed transceiver or poorly seated patch lead is usually thought of as an availability issue. It can be. It can also change where traffic flows, cause a link to flap, force a failover path or leave monitoring equipment disconnected. Even a simple patching error can put a device on a different switch or segment than the documentation says.

Physical-layer faults create both outages and security blind spots because they can break the assumptions behind segmentation and monitoring. A network may recover service through an alternate path while losing the expected inspection point. A monitoring link may quietly disappear during a maintenance change. The symptoms can look unrelated until someone checks the physical route.

This is why I keep a label maker in the laptop bag and treat mystery cables as a security smell. Unlabelled patch panels and undocumented links do not merely slow down troubleshooting; they make it harder to establish what is connected to what, and whether the connection is authorised.

What the SIEM could—and could not—tell us

A SIEM correlates the events and logs it receives. It does not automatically observe every packet, validate every switch port or know that a device is physically connected somewhere unexpected. In this case, the SIEM had firewall logs and selected infrastructure events, but not a complete view of east-west traffic inside the affected network.

That limited the investigation in several ways:

  • Incomplete traffic records: The SIEM could show some connections at the network boundary, but not all internal communication between devices.
  • Inconsistent switch logs: Event forwarding and time synchronisation were not uniform, making it difficult to line up port changes with endpoint activity.
  • Missing context: Asset records did not reliably identify which device was connected to the affected port at the time.
  • Unverified monitoring coverage: The team had not recently confirmed which VLANs and links the network sensor could actually see.

Firewalls remain essential controls, but they only inspect traffic that passes through them. Endpoint detection platforms can provide strong evidence about activity on managed devices, but they do not replace network visibility—especially for unmanaged equipment, embedded systems or traffic that does not cross an endpoint agent’s view.

The lesson is not to collect every possible log. It is to know what evidence each control can provide, where its blind spots are and how those controls fit together. A SIEM dashboard is not proof that the network is observable.

Segmentation depends on capable switching and correct design

Segmentation is a property of the whole design, not a tick-box feature on a switch. It depends on capable hardware, consistent configuration, controlled interconnections and operational discipline. If a switch cannot support the required controls or monitoring features, the design needs to account for that limitation rather than assume it away.

Vendors such as Cisco, Garland and MOXA serve different environments and use cases, from enterprise networks to industrial settings. The brand on the front is not a substitute for selecting equipment that suits the security and operational requirements. Check that the platform supports the needed VLAN, access control, management, logging and monitoring functions—and that the team can configure and maintain them safely.

In operational technology environments, switching decisions also have to respect availability, vendor support and the consequences of change. A rushed firmware update or an untested configuration can disrupt a production process. Conversely, leaving an unsupported or unmanaged device in place because it is “working” can create an enduring gap. The right approach is controlled change, documented dependencies and a recovery plan that has been tested.

ACSC guidance starts with solid network foundations

Australian Cyber Security Centre guidance on strengthening cyber security is most useful when treated as a set of operational practices, not a substitute for sound network engineering. Logging, segmentation, access control and incident response all assume that the underlying network is reasonably understood and configured.

If switch management is exposed, configurations are not backed up, network devices do not send logs, or physical connections are undocumented, higher-level controls have less to work with. Guidance cannot compensate for a network map that is years out of date or a production switch with default credentials. The foundations matter: know your assets, secure management access, apply changes deliberately, and verify that intended controls are actually operating.

How to reduce switch-related SIEM blind spots

Closing these gaps does not require turning every switch into a science project. It does require treating switching, cabling and monitoring as part of the security architecture.

  1. Validate the segmentation design. Document the intended zones, permitted flows and enforcement points. Confirm that access ports, trunks and inter-VLAN routing match the design.
  2. Establish a known-good switch baseline. Use approved configuration templates, restrict management access, remove default credentials, disable unused ports where practical and control who can make changes.
  3. Test monitoring coverage. Confirm that taps or SPAN sessions include the expected links and VLANs. Check for packet loss and oversubscription, and repeat the test after significant network changes.
  4. Forward and protect infrastructure logs. Send relevant switch and firewall events to central logging, use reliable time synchronisation, and alert on meaningful events such as configuration changes, link-state changes and authentication failures.
  5. Reconcile the physical and logical maps. Keep port schedules and patch-panel labels current. Record what connects to each port and investigate undocumented equipment rather than assuming it is harmless.
  6. Exercise the failure path. Ask what happens when a switch, monitoring link or uplink fails at 2am. Identify who gets called, which controls remain in place and how the team will know visibility has been lost.
  7. Correlate multiple evidence sources. Use firewall logs, endpoint detection platforms, network telemetry and asset records together. Make the gaps between sources explicit in incident playbooks.

A practical visibility check

A simple review can reveal whether the documented network and the observable network are the same thing. Pick a critical segment and answer the following questions with evidence, not assumptions:

Check What good looks like Warning sign Switch-port assignment Port configuration matches the approved design and connected asset. Unknown devices, unexpected VLANs or inconsistent port descriptions. Segmentation enforcement Permitted paths are documented and controls are verified at boundaries. Teams rely on VLAN names or diagrams without testing actual reachability. Network monitoring Required traffic reaches sensors, with coverage and capacity checked. SPAN sessions are assumed to work, but coverage or packet loss is unknown. Infrastructure logging Switch and firewall events arrive centrally with consistent timestamps. Logs are local only, incomplete or too inconsistent to correlate.

Make the network tell the truth

The SIEM did not miss this intrusion because it was useless. It missed the useful network evidence because a misconfigured switch and incomplete monitoring undermined the assumptions behind the https://www.cyberdaily.au/strategy/14271-why-physical-network-infrastructure-is-the-overlooked-foundation-of-your-cybersecurity-strategy security design. Endpoint alerts helped identify suspicious activity; firewall records provided part of the picture; neither could reconstruct traffic the network failed to expose.

Good security depends on the unglamorous work: capable switches, deliberate segmentation, clean patching, current diagrams, protected management access and monitoring that is tested rather than presumed. Physical infrastructure is part of cybersecurity because it decides how systems connect and what defenders can see.

So when the SIEM goes quiet, do not only ask whether the detection rule fired. Ask whether the switch carried the traffic where you expected, whether the sensor saw it, and whether the labels on the patch panel still match reality. Then ask the question that matters when the office is dark and the on-call phone rings: what happens at 2am when this fails?