Data center networks are divided into two distinct functions: Ethernet LAN, which carries applications, users, and IP traffic, and Fibre Channel SAN, which carries storage blocks with strict loss, latency, and ordering requirements. This guide walks through spine-leaf architecture, the Cisco ACI and Juniper EVPN-VXLAN approaches for the LAN, Fibre Channel with Cisco MDS and Brocade for the SAN, and how to size, secure, and operate both.
About this material: the capabilities cited vary by model, software version, license, optics, distance, and configuration. Always validate the compatibility matrices and current manufacturer documentation before purchasing or making changes.
LAN and SAN: different functions
The data center LAN uses 10/25/50/100/200/400/800G Ethernet depending on the platform, with VLAN/VRF for segmentation, VXLAN to scale the overlay, and BGP/OSPF/IS-IS in the underlay — carrying east-west traffic, north-south traffic, management, backup, IP storage, and clusters.
The Fibre Channel SAN is dedicated, with WWPN, zoning, name server, FSPF, ISL, and VSAN/Virtual Fabric, operating independent A/B paths and host-side multipathing; FC-NVMe can coexist with FCP when supported end to end.
RoCEv2, iSCSI, and NVMe/TCP use Ethernet to converge storage and network data, but require their own design for QoS, congestion, buffers, PFC/ECN where applicable, and operational isolation — convergence requires caution, it is not an automatic shortcut.
Spine, leaf, ToR, and border leaf
The Clos fabric reduces hops, distributes capacity, and allows predictable horizontal expansion. The spine is the high-capacity layer: each spine connects to all leafs and does not receive servers directly in the standard design.
Leaf is the function within the architecture; ToR describes the physical position in the rack — a leaf can be ToR, middle-of-row, or end-of-row. The border leaf handles the transition to WAN, Internet, firewalls, legacy networks, MPLS/EVPN, or another site, and must be sized for north-south traffic and external routes. The service leaf connects services such as ADC, firewall, IDS/IPS, DNS, and appliances — avoid complex service insertion without observability and symmetry testing.
Basic LAN topology: two spines and three leafs
The standard that eliminates bottlenecks starts simple: each leaf connects to all spines, with no direct connection between leafs.
East-west traffic (server to server) crosses leaf-spine-leaf; with ECMP, multiple active paths distribute the flows. North-south traffic exits through a border leaf to a firewall, WAN, Internet, load balancer, or legacy network. The loss of a link or spine reduces capacity, but should not interrupt connectivity — rule of thumb: each leaf should have at least two uplinks, one to each spine, with capacity calculated for the worst-case N-1 scenario and cabling, optics, and MTU standardized across the entire fabric.
Complex LAN topology: redundant enterprise fabric
From the edge to the workloads, a redundant enterprise fabric eliminates single points of failure at every layer.
In this design, the L3 underlay uses point-to-point adjacencies, ECMP, BFD, and /31 or unnumbered addressing. The VXLAN overlay uses a BGP EVPN control plane or, in ACI, policies distributed by the APIC. The services layer — border leaf, service leaf, firewall, load balancer, and external gateways — must be redundant at each of these points.
Underlay, overlay, and VXLAN
Separating physical connectivity from logical connectivity makes scaling and automation easier. The L3 underlay uses routed leaf-spine links with ECMP for multiple paths, BGP, OSPF, or IS-IS depending on the architecture, BFD for fast detection when supported, and consistent MTU, deterministic addressing, and summarization where possible.
In the VXLAN overlay, VTEPs encapsulate frames/routes in VXLAN; VNIs represent L2 segments or L3 VRFs, expanding the space beyond VLANs. BGP EVPN distributes MAC/IP information and prefixes, reduces flooding, and enables multihoming, anycast gateway, and controlled mobility. BUM traffic (broadcast, unknown unicast, and multicast) requires a replication policy — monitor the volume and avoid L2 domains larger than necessary.
Cisco ACI: architecture and objects
ACI transforms connectivity requirements into policies applied across the Nexus 9000 fabric and orchestrated by APIC controllers. The APIC is a cluster of controllers responsible for inventory, policies, APIs, health score, and integration; Nexus 9000 spines and leafs perform distributed forwarding in the VXLAN fabric; Nexus Dashboard adds operational services, visibility, and orchestration depending on licensing.
In the policy model, a tenant contains VRFs, bridge domains, and application profiles; EPGs group endpoints, and contracts define the relationships permitted between EPGs. VMM domains, bare metal, L3Out, L2Out, service graphs, and multisite connectivity must be treated as design building blocks, not merely GUI objects. ACI does not eliminate networking fundamentals: addressing, routing, MTU, multicast, QoS, capacity, and failure domains remain essential.
ACI: policy flow in plain language
Policy starts from application intent and ends in rules distributed across the leafs. A typical example: tenant ENQ-PROD, VRF VRF-ERP, bridge domains BD-APP and BD-DB; EPG-APP contains the application servers, EPG-DB contains the databases, and a contract ERP-SQL allows only TCP/1433 from APP to DB.
Endpoints are learned and associated with the EPG; the anycast gateway can exist on the leafs, and the contract is applied in a distributed manner. L3Out connects the VRF to external routers or firewalls via BGP/OSPF/static routes, with route control and import/export policies. Governance requires naming conventions, templates, peer review, versioning, and testing — avoid orphaned objects, any-any contracts, and dependency on manual configuration.
Cisco ACI: high availability and scale
Availability must consider controllers, forwarding, external services, and operations. Plan the number and distribution of the APIC cluster according to the supported scale and manufacturer design — an APIC outside the quorum should not interrupt forwarding that has already been programmed, but it limits changes.
Servers and appliances can use vPC for L2 redundancy; validate LACP, orphan ports, hashing, consistency, and the behavior of the connected equipment. Multi-Pod extends a single fabric with an IPN; Multi-Site coordinates independent fabrics — the choice changes blast radius, latency, operation, and recovery. For upgrades, read release notes, the compatibility matrix, and upgrade paths, and perform pre-checks, configuration backups, fault validation, and a rollback plan.
Juniper: EVPN-VXLAN fabric
In the Juniper approach, the QFX operates the IP fabric and BGP EVPN provides the overlay control plane; Apstra can add intent, validation, and automation. Fixed or modular QFX switches act as leaf, spine, border, and super-spine depending on scale; Junos OS provides routing policy, EVPN, VXLAN, telemetry, and automation; Apstra offers intent-based design, blueprints, assurance, and multivendor operation depending on support.
EVPN models include VLAN-based, VLAN-aware bundle, or VLAN bundle — the choice affects scale, isolation, and operation; Type 2 advertises MAC/IP, Type 5 advertises IP prefixes. EVPN multihoming uses Ethernet Segments and DF election, reducing dependency on MLAG in supported designs. IRB and anycast gateway bring routing closer to the workloads, reduce hairpinning, and support mobility within the limits of the design.
Juniper: design and operation
A standards-based fabric offers flexibility, but requires disciplined BGP policies and automation. eBGP underlay is common for its policy simplicity and isolation; iBGP with route reflectors is also possible — the EVPN overlay needs consistent address families and policies.
Type 5 routing allows IP prefixes to be advertised independently of MAC, which is useful for L3 services, border leaf, and DCI. For interconnecting data centers, avoid extending L2 unnecessarily — compare EVPN DCI, inter-VRF routing, firewalls, and application replication, since latency and failure domain matter more than convenience. Apstra's Assurance validates cabling, adjacencies, ASN, loopbacks, VNIs, VLANs, IRBs, policies, and endpoints against the blueprint before and after each change.
Cisco ACI x Juniper EVPN-VXLAN
These are two mature approaches with different operating models. ACI is a policy-based fabric with the APIC on top of Nexus 9000 hardware, where the APIC holds the intent and policies, segmentation uses tenant/VRF/BD/EPG/contract, the gateway is anycast/distributed at the leaf, operation is centered on GUI/API over ACI objects, and multisite goes through Nexus Dashboard Orchestrator — the choice favors those who want an ACI ecosystem and integrated governance.
Juniper EVPN-VXLAN is a standards-based fabric with BGP EVPN over QFX running Junos OS/Apstra, where BGP distributes MAC/IP and EVPN routes, segmentation uses VRF/VLAN/VNI and firewall policies, the gateway is anycast IRB at the leaf, operation is centered on CLI/API/automation over EVPN, and multisite goes through EVPN DCI, Apstra, and specific design — the choice favors adherence to standards and flexibility. There is no universal winner: the decision must weigh team skills, automation, support, interoperability, total cost, roadmap, and security requirements.
LAN sizing
Sizing means translating traffic profiles into ports, uplinks, buffers, optics, power, and growth margin. Oversubscription is the ratio between server port capacity and leaf uplinks — for example, 48 25G ports total 1.2 Tb/s, against 6 100G uplinks totaling 600 Gb/s, a nominal oversubscription of 2:1.
In the N-1 scenario, check whether, with the loss of a spine or uplink, the remaining capacity meets the acceptable peak — don't rely solely on the daily average. Traffic profiles differ: virtualization generates many flows and mobility, focused on east-west; IP storage/RoCE brings microbursts, loss, latency, and congestion; AI/HPC produces elephant flows, all-reduce, rail optimization, and job completion time as core metrics. For optics and distance, validate speed, breakout, FEC, fiber type, optical power, reach, temperature, and manufacturer compatibility.
Basic SAN topology: dual fabric A/B
Two independent networks running to each controller is the reference design for the enterprise SAN.
The isolation is real: fabric A and B have no ISL between them, so failures and changes remain contained within one fabric. Multipathing (MPIO) chooses paths and reacts to the loss of an HBA, link, switch, port, or controller. For zoning, prefer single-initiator/single-target or single-initiator/multi-target, according to the array's standard.
Fibre Channel: essential concepts
FC is a network oriented toward minimal loss and reliable delivery, with its own mechanisms for login, naming, routing, and flow control. In terms of identity, the WWNN identifies the node, the WWPN identifies the port, and the FCID is assigned after login to the fabric — document all three during troubleshooting.
In logins, FLOGI registers the port on the fabric, PLOGI creates the session between ports, and PRLI negotiates the upper-layer protocol, such as FCP or FC-NVMe. In routing and flow control, FSPF calculates paths, and buffer-to-buffer credits limit how many frames can be sent without confirmation of available buffer. ISLs join switches within the same fabric; trunking aggregates compatible links — distance and RTT may require additional credits.
Cisco MDS: VSAN, zoning, and operation
The Cisco MDS 9000 uses NX-OS and offers segmentation via VSAN, zoning, FC port-channels, and telemetry, depending on model and license. The VSAN creates independent logical fabrics on top of the shared physical infrastructure — each VSAN has its own services and FSPF, and should be used consciously for isolation.
In zoning, the mode can be basic or enhanced depending on governance; aliases/device aliases reduce errors, and zonesets need to be activated on the correct VSAN — Smart Zoning can reduce hardware entries in compatible scenarios. The FC port-channel aggregates compatible ISLs, improving utilization and resilience; check speed, trunking, allowed VSANs, and consistency. For O&M, Cisco SAN Analytics/telemetry, syslog, SNMP, Call Home, and dashboards help identify congestion, slow drain, and physical errors.
Brocade: Fabric OS and best practices
Brocade offers Fibre Channel switches and directors with Fabric OS, zoning, trunking, virtual fabrics, and Fabric Vision, depending on platform and license. Virtual Fabrics uses logical switches to separate environments and services — define FIDs, base switch, and inter-fabric routing only when the case requires it.
In zoning, use consistent aliases, versioned zone configurations, and peer zoning when appropriate; changes should be small, reviewed, and reversible. ISL Trunking aggregates compatible links into trunks for frame-level load balancing — confirm license, distance, optics, and port group requirements. Fabric Vision brings together metrics, MAPS, Flow Vision, and diagnostics to detect degradation, congestion, and slow-drain devices, depending on support/license.
Complex SAN topology: core-edge with directors
Scaling the SAN requires well-defined failure domains and calculated ISLs, not just more ports.
The core design rules: separate fabric A and B across switches, power, patch panels, physical routes, and administrative domains; size ISLs by simultaneous peak, distance, compression, buffer credits, and the N-1 scenario; avoid excessively large fabrics, limiting the blast radius with virtual fabrics/VSANs when necessary; and continuously monitor CRC, enc_out, loss of sync, congestion, slow drain, credit zero, and I/O latency.
Zoning and masking: defense in depth
Zoning controls communication within the fabric; LUN masking controls presentation at the array — both are necessary. The recommended model is one zone per initiator with one or a few targets, which reduces interference, facilitates auditing, and limits the impact of changes.
Aliases should carry environment, host, HBA/port, fabric, and function — for example, PRD-DB01-HBAA and AFF01-CTA-P01. The change process follows: gather WWPNs from a trusted source, validate login to the fabric before creating the zone, change only one fabric at a time when possible, and activate, test MPIO, observe errors, and log evidence. Avoid all-access zones, mixing initiators without justification, targets from different arrays in the same zone, and simultaneous removal in A/B.
Congestion and slow drain
A slow device can hold credits and spread congestion across the fabric, raising latency and affecting unrelated workloads. The signs include prolonged credit zero, timeouts, queueing, and I/O latency; CRC, invalid transmission word, loss of sync/signal, and enc_out; and persistent backpressure, dropped frames, or unstable MPIO paths.
In diagnosis, correlate host, HBA, driver/firmware, SFP, fiber, port, ISL, controller, and array — an isolated counter rarely tells the whole story. The fix involves replacing physical components with proven defects, correcting firmware/driver, redistributing load, and only increasing ISL/credits when the cause and sizing justify it. Prevention depends on per-port baselines, trend-based alerts, failover testing, capacity margins, and a rigorous compatibility process.
Automation and infrastructure as code
Automation should increase consistency and evidence, not accelerate errors. In terms of APIs and tools, Cisco ACI offers REST API, Cobra SDK, Ansible, and Terraform providers depending on support; Juniper offers NETCONF/YANG, PyEZ, gNMI, Ansible, Terraform, and Apstra; and Cisco MDS/Brocade offer APIs, Ansible, scripts, and management platforms depending on version.
A safe pipeline includes lint and schema validation, lab testing, diff, approval, change window, serial execution by failure domain, health checks, and rollback. The source of truth — NetBox/DCIM/CMDB or a versioned repository — should store IPAM, ASN, ports, optics, VLAN/VNI, VRF, zones, WWPN, and dependencies. Secrets should never be stored in code: use a vault, minimal RBAC, rotation, audit logs, and individual accounts.
Network infrastructure security
The management plane is a critical asset and must be separate, authenticated, monitored, and recoverable. In the management plane, use an independent OOB network, central AAA with controlled fallback, TACACS+/RADIUS, and MFA where supported; SSH, HTTPS, and SNMPv3, disabling legacy services and ciphers; authenticated NTP when available; and RBAC by role with immutable logs sent to a SIEM.
In the control plane, apply CoPP/policers, adjacency authentication, peer filters, TTL security where applicable, and protection against route leaks. In the data plane, use microsegmentation, minimal contracts/policies, VRFs, firewalls, and validation of allowed traffic — MACsec/IPsec/FC-SP encryption depends on risk and support. For resilience, maintain exported backups, golden configuration, spare parts, support contacts, and recovery drills.
Observability and SLOs
Availability isn't "ping responds": you need to monitor application experience, capacity, and health per domain. On the LAN, track utilization and headroom per link/queue, drops, ECN/PFC when used, latency, and jitter; BGP/EVPN/OSPF/IS-IS neighbors, MAC moves, endpoint churn, and inconsistencies; and optics — Rx/Tx power, temperature, FEC, and errors.
On the SAN, track IOPS, throughput, and latency per host/target/LUN; credits, congestion, CRC, link resets, and slow drain; and the state of zoneset/config, logins, and MPIO paths. Define measurable SLOs — availability, convergence time, loss, latency, headroom, correction deadline, and change success — and ensure that changes, alerts, flows, inventory, and tickets share timestamp, asset, and affected service to enable real correlation.
Eight-step deployment plan
Deploy in small, verifiable waves, always keeping a tested rollback path. In steps 1–2 (discovery and HLD), gather requirements, applications, traffic, latency, security, growth, RTO/RPO, sites, and integrations, producing an HLD with decisions and alternatives.
In steps 3–4 (LLD and staging), define port maps, IPAM, ASN, VLAN/VNI, VRF, policies, zoning, optics, cables, power, configuration, testing, and MOP, assembling and updating equipment outside production. In steps 5–6 (pilot and migration), start with a non-critical service, validate failures and observability, and migrate in batches with go/no-go criteria and checkpoints. In steps 7–8 (stabilization and handover), track baselines, correct deviations, update documentation, train operations, record the as-built, and formally accept the environment.
LAN acceptance checklist
Use this list as a starting point, adapting it to the environment's standards and risks. Physically: racks, RU, airflow, A/B power, grounding, cables, and labels checked; compatible optics, with Tx/Rx levels and FEC within expected range. In the underlay/overlay: all links and neighbors up, ECMP installed, MTU and BFD tested; VNIs/VLANs/VRFs/gateways/EPGs/contracts validated, with no abnormal flooding or MAC moves.
In resilience: link, leaf, spine, power supply, and peer failure tested under representative load; border/firewall/LB with proven symmetry and convergence. In operation: backup, OOB, AAA, NTP, DNS, syslog, SNMP/telemetry, and functional alerts; dashboards, runbooks, as-built, inventory, and support delivered.
SAN acceptance checklist
Each path must be tested in isolation and in plausible failure combinations. In the fabric: A/B physically independent, correct domain IDs/FIDs/VSANs, healthy ISLs and trunks, no CRC, loss of sync, credit starvation, or persistent congestion. In hosts and arrays: WWPNs, HBA firmware/driver, and array interoperability confirmed; zoning and masking reviewed, with all MPIO paths active/optimized according to policy.
In testing: controlled removal of one fabric without application interruption; failover of controller, port, HBA, and ISL according to the test matrix; performance and latency within baseline under normal and peak load. In governance: backups, aliases, zonesets/configs, port maps, and as-built versioned; runbook for slow drain, physical failure, and zoning change approved.
Conclusion
Well-designed data center networks separate responsibilities clearly: the LAN carries applications and users over a redundant spine-leaf fabric, with Cisco ACI or Juniper EVPN-VXLAN as mature and distinct approaches; the SAN carries storage blocks over Fibre Channel with Cisco MDS or Brocade, isolated in a dual fabric A/B. Convergence with Ethernet — RoCEv2, iSCSI, NVMe/TCP — is possible, but requires its own QoS and isolation design, it is not a shortcut.
The real value emerges when sizing, redundancy, automation, security, and observability are treated as a single system, tested under real failure before going into production — not as isolated choices made at different moments of the project.