Storage is not just capacity in terabytes. It is a risk discipline: availability, consistency, performance, recoverability, security and cost throughout the lifecycle need to be designed together. This guide covers the fundamentals of DAS, NAS and SAN, block protocols, high-availability architecture, TrueNAS and OpenZFS, sizing, security, continuity, hardware, operation, TCO and reference architectures.
About this material: performance figures are engineering references, not guarantees. The actual result depends on disks, controllers, network, I/O pattern, firmware, operating system and acceptance testing. TrueNAS features and licensing may change — confirm the documentation and support matrix before purchasing.
Executive summary
The storage decision should start from the impact of unavailability and data loss, not the price per terabyte. Leadership needs to decide which applications are critical and the cost per hour of downtime, which RPO (maximum data loss) and RTO (maximum recovery time) the business accepts, whether the operation requires redundant controllers, 24x7 support and parts under SLA, whether the workload is dominated by capacity, IOPS, throughput or latency, and whether growth will be vertical (pool expansion) or horizontal (new systems).
| Decision | Key question | Impact |
|---|---|---|
| Availability | Single controller or HA? | CAPEX versus operational risk |
| Media | HDD, SSD or NVMe? | Cost/TB, latency and endurance |
| Protocol | FC, iSCSI, NVMe-oF, NFS or SMB? | Integration and complexity |
| Protection | Snapshot, replication and backup? | RPO/RTO and ransomware resilience |
| Support | Community or Enterprise? | Responsibility and recovery time |
Executive recommendation: for critical production, treat HA, multipath, immutable or isolated backup, monitoring and restore testing as baseline requirements, not optional items.
Fundamentals: DAS, NAS and SAN
The three models can coexist. The main difference is where storage is presented and who controls the file system.
| Model | Presentation | Common protocols | Best use |
|---|---|---|---|
| DAS | Local disk/block | SAS, SATA, NVMe | Low complexity; single host |
| NAS | Shared file | SMB, NFS | Collaboration, home directories, repositories |
| SAN | Remote block | FC, iSCSI, NVMe-oF | Virtualization, databases, clusters |
The essential concepts: LUN/namespace is the logical unit presented to the host; initiator is the client that starts the session, target is the storage that delivers the block; fabric is the redundant set of switches and links in an FC SAN; zoning limits visibility on the fabric, masking limits which hosts access each LUN; and MPIO maintains multiple paths and can balance I/O according to policy.
Classic mistake: SAN does not replace backup. A replicated LUN can replicate corruption, deletion, malicious encryption and human error.
Block protocols in depth
Choosing the protocol means choosing an ecosystem of latency, network, operation and compatibility.
| Criterion | Fibre Channel | iSCSI | NVMe-oF |
|---|---|---|---|
| Transport | Dedicated FC fabric | Ethernet/TCP | TCP or RDMA |
| Latency | Low and predictable | Good; depends on the TCP stack | Very low, especially RDMA |
| Operation | Specialized | Familiar to IP teams | Requires maturity and compatibility |
| Cost | Higher | Lower/gradual | Variable; high-performance network |
| Typical use | Enterprise core | Virtualization and midmarket | All-flash and demanding workloads |
Fibre Channel uses WWPNs, HBAs, switches and independent A/B fabrics. The recommended design avoids any single component: each host has paths to both fabrics and each storage controller connects to both. Single-initiator/single-target zoning reduces the failure domain and simplifies diagnostics.
iSCSI transports SCSI commands over TCP/IP. Separate storage traffic into its own VLANs/subnets, distribute sessions across distinct switches and implement MPIO — LACP does not replace multipath for end-to-end path availability. NVMe over Fabrics carries NVMe's parallel queues over the network, potentially using TCP or RDMA; validate compatibility across the entire stack.
SAN architecture: high availability
Real availability depends on eliminating single points along the full path. Design principles: two fully independent fabrics, without interconnecting Fabric A and B; two HBAs or two independent ports per host, preferably on distinct buses; redundant storage controllers with tested active paths; power supplies, PDUs, UPS and A/B circuits, with monitoring of the electrical chain; zoning and LUN masking documented per host/cluster; and testing of cable, port, switch and controller failure before production.
| Simulated failure | Expected result | Evidence |
|---|---|---|
| Cable/HBA | I/O continues via alternate path | MPIO log + latency |
| Fabric switch | No volume loss | Alarm and failover |
| Controller | Trespass within SLA | Time and events |
| Disk/vdev | Pool remains online/degraded | Alert + resilver |
| Power A | Operation via power feed B | PDU/UPS telemetry |
Performance and sizing
Usable capacity and performance must be calculated separately — a pool can have many terabytes and still be inadequate for the I/O profile. Four central metrics: IOPS (operations per second, important for small random I/O), throughput (MB/s or GB/s, important for streaming, backup and media), latency (time per operation, evaluated as average and p95/p99 percentiles), and queue depth (number of simultaneous outstanding commands, which can raise throughput and also hide saturation).
Approximate relationship: IOPS = throughput / average block size. Example: 1 GB/s with 128 KiB blocks represents about 8,000 IOPS. For HDDs, mirrors deliver more random IOPS than RAIDZ with the same number of disks; for SSD/NVMe, controller, CPU, PCIe and network frequently become the bottleneck.
| Layer | Collect | Why |
|---|---|---|
| Application | IOPS, block size, read/write, sync | Defines actual behavior |
| Host | queue depth, latency, multipath | Detects path/bottleneck |
| Network | utilization, drops, retransmissions | Validates transport |
| Storage | ARC, disks, pool, CPU | Locates saturation |
| Protection | snapshots, replication, scrub | Includes maintenance load |
Capacity rule: plan for operational free space. Pools that are too full lose flexibility and can degrade; adopt alerts and an expansion policy before hitting the limit.
TrueNAS and its role
TrueNAS turns compatible hardware into a storage platform based on OpenZFS, with file, block, data protection, integration and automation services.
| Edition | Positioning | Recommended use |
|---|---|---|
| Community Edition | Open software, self-managed deployment | Lab, edge, backup and production with accepted risk |
| Enterprise | Validated appliances, enterprise features and support | Mission-critical, HA and vendor accountability |
Services and integrations include block (iSCSI; Fibre Channel and NVMe-oF depend on the edition, release, hardware and supported licensing), file (SMB and NFS, with ACLs and integration with directory services), object and applications (capabilities vary by version — separate app workloads from critical storage), protection (ZFS snapshots, replication, rsync and cloud sync tasks) and management (web interface, alerts, API and integration with TrueCommand depending on the scenario).
Governance decision: "works on generic hardware" is not the same as "has end-to-end support." For critical workloads, validate the HCL, firmware, controllers, NICs/HBAs, enclosure and SLA.
OpenZFS: integrity by design
ZFS combines a volume manager and a file system. Its copy-on-write model avoids overwriting active blocks and enables consistent snapshots at the storage level. Components: pool (a set of vdevs — the failure of a data vdev can compromise the entire pool), vdev (redundancy unit: mirror, RAIDZ1, RAIDZ2 or RAIDZ3), dataset (file system with its own properties), zvol (block volume, normally used for iSCSI/FC), checksum (detects corruption; redundancy enables self-healing) and scrub (periodic read/verification of data and metadata).
| Layout | Tolerance per vdev | Strength | Caution |
|---|---|---|---|
| Mirror | 1 disk per pair (typical) | IOPS and fast rebuild | 50% raw efficiency |
| RAIDZ1 | 1 disk | Capacity in smaller groups | Reduced margin on large disks |
| RAIDZ2 | 2 disks | Capacity/protection balance | Fewer random IOPS |
| RAIDZ3 | 3 disks | High protection | Higher overhead |
Capacity and performance aggregate per vdev, but redundancy is local to each vdev. Expansion should preserve symmetry and predictability. Avoid mixing disks of very different capacities, performance and endurance without a tested justification.
ARC, L2ARC, ZIL, SLOG and special vdev
ZFS's auxiliary devices solve specific problems. Adding them without measurement can reduce performance or create risk.
| Feature | Function | When it helps | Risk/note |
|---|---|---|---|
| ARC | Primary cache in RAM | Almost always | RAM is faster than L2ARC |
| L2ARC | Secondary read cache | Working set larger than ARC | Consumes RAM for metadata |
| ZIL | Synchronous write log | Sync semantics | Exists in the pool even without SLOG |
| SLOG | Separate device for the ZIL | Heavy synchronous writes | Requires low latency and PLP |
| Special vdev | Metadata and optionally small blocks | File/metadata-heavy pools | Must have robust redundancy |
SLOG is not a general write cache and does not accelerate asynchronous writes — the critical requirement is safe persistence under power loss, low latency and endurance. L2ARC should only be adopted after confirming relevant ARC misses and a reusable working set.
Critical caution: unrecoverable loss of a special vdev can compromise the pool. Mirror it appropriately and treat it as part of the data, not as disposable cache.
TrueNAS as an iSCSI SAN
To present block storage, TrueNAS creates a zvol or extent and associates it with a target, portal and authorized initiators.
The logical flow: create a pool and zvol with volblocksize aligned to the workload; configure the portal on storage interfaces/VLANs; define initiators and authentication where applicable; create the target, extent and the target-extent association; present multiple paths and configure MPIO on the host; and create the filesystem/datastore on the host — never mount the same LUN on multiple hosts without a cluster filesystem.
| Network | Subnet | Use |
|---|---|---|
| Storage A | 10.20.10.0/24 | iSCSI path A |
| Storage B | 10.20.20.0/24 | iSCSI path B |
| Management | 10.20.30.0/24 | GUI, API, monitoring |
| Replication | 10.20.40.0/24 | Traffic between storage systems |
Jumbo frames: MTU 9000 only brings a benefit when configured and validated end to end. MTU inconsistency causes hard-to-diagnose failures; a well-designed MTU 1500 is preferable to partial jumbo frames.
Ethernet network for storage
The network must be non-blocking for the expected traffic and predictable during failures and maintenance. Best practices: use redundant switches and distinct IP paths for MPIO; separate management, data, replication and client traffic when the risk justifies it; size buffers and uplinks for microbursts and oversubscription; monitor drops, errors, retransmissions, pause frames and latency; RSS/multiqueue and CPU affinity can matter at 25/40/100 GbE; and avoid changing flow control, offloads or congestion control without a baseline and testing.
| Nominal link | Approximate theoretical usable | Typical application |
|---|---|---|
| 10 GbE | ~1.1 GB/s | Hybrid HDD, backup, midmarket |
| 25 GbE | ~2.8 GB/s | All-flash and virtualization |
| 40 GbE | ~4.5 GB/s | Aggregation/legacy high bandwidth |
| 100 GbE | ~11 GB/s | NVMe, AI, consolidation |
Usable values are approximations and vary with overhead, MTU, CPU, protocol and I/O pattern. Two links do not automatically mean double per flow — validate MPIO behavior and the number of sessions.
Security, identity and ransomware
The strategy must prevent a compromised credential from destroying production, snapshots and backups at the same time. Priority controls: MFA for administration and nominative accounts with least privilege; restricted management network, with access via bastion/VPN and centralized logs; CHAP on iSCSI where applicable, strict zoning/masking on FC; encryption in transit and at rest according to risk and compliance; snapshots with defined retention, replication to a separate administrative domain and offline/immutable copy; tested backups of configuration and encryption keys; and windowed updates, advisory review and planned rollback.
| Layer | Control | Test |
|---|---|---|
| Prevention | segmentation, MFA, hardening | quarterly review |
| Detection | alerts, SIEM, anomalies | controlled simulation |
| Containment | separate credentials and network | tabletop exercise |
| Recovery | 3-2-1-1-0 and runbook | periodic restore |
3-2-1-1-0: three copies, two media types, one offsite, one offline/immutable and zero errors verified after testing. A local snapshot is one layer, not the whole strategy.
Continuity: RPO, RTO and DR
Recovery is a proven operational capability, not the existence of a replica.
| Mechanism | Protects against | Does not resolve alone |
|---|---|---|
| RAID/ZFS | media failure | deletion, ransomware, disaster |
| Snapshot | logical error and quick rollback | loss of the same system |
| Replica | site/storage failure | replicated corruption |
| Immutable backup | attack and retention | RTO without automation |
| Orchestrated DR | broad unavailability | data without adequate protection |
The minimum runbook: declare the event and freeze changes; confirm the last consistent point and the affected scope; isolate the source of the incident; promote the replica or restore in a clean environment; validate technical and application consistency; redirect clients and monitor; and log actual times, gaps and corrective actions.
An RPO of 15 minutes requires compatible frequency and transport. An RTO of 2 hours includes detection, decision, provisioning, restoration, validation and service return — not just copying bytes.
Hardware: platform design
Storage is an integrated system. CPU, memory, PCIe, HBA, backplane, enclosure, disks and firmware need to form a validated configuration.
| Component | Criteria |
|---|---|
| CPU | clock per thread, cores, PCIe lanes, encryption/compression |
| ECC RAM | ARC, metadata, dedup and operational margin |
| SAS HBA | proper/JBOD mode, compatible firmware, queues |
| NIC/FC HBA | speed, offloads, drivers, transceivers |
| HDD disks | CMR, vibration, URE, workload rating |
| SSD/NVMe | DWPD/TBW, sustained latency, PLP, thermal |
| Boot | dedicated device; mirroring according to criticality |
| Enclosure | dual-path SAS, expander, SES, cooling and A/B PSU |
Current TrueNAS documentation states a minimum of 8 GB of RAM, a 20 GB boot SSD and two devices of the same size for a basic pool — this is a functional minimum, not production sizing. The official guidance itself discourages SMR media for ZFS and highlights PLP for SLOG.
ECC: reduces the risk of silent memory corruption and is strongly recommended in enterprise storage. It does not replace checksum, redundancy or backup.
Operation, monitoring and maintenance
The goal is to detect degradation before it becomes unavailability and to perform maintenance without improvisation. Indicators: logical/physical capacity, trend and rate of change; average/p95/p99 latency, IOPS, throughput and queues; contextualized ARC hit ratio, RAM and CPU usage; SMART/NVMe health, temperature, wear, SAS errors and checksum; pool state, scrub, resilver and snapshots; sessions, MPIO paths, fabric/network errors and drops; and success, delay and duration of backup/replication.
| Frequency | Routine |
|---|---|
| Continuous | alerts, health, capacity, paths |
| Weekly | job failures, trends, tickets |
| Monthly | scrub per policy, capacity review |
| Quarterly | failover test and sample restore |
| Semiannual | DR, firmware, compatibility matrix |
| Annual | architecture and lifecycle review |
For changes, maintain inventory, diagram, host-LUN-WWPN/IQN matrix, baseline, configuration backup, rollback plan and success criteria. Storage updates must respect the combined matrix of firmware, system, HBA/NIC, switch and multipath.
TCO, licensing and decision model
The lowest CAPEX can produce a higher total cost when it shifts risk and effort onto the internal team.
| TCO component | Include |
|---|---|
| Acquisition | chassis, disks, NIC/HBA, switches, optics, licenses |
| Operation | power, cooling, rack, monitoring, staff |
| Protection | second storage system, backup, cloud, offline media |
| Support | SLA, parts, updates and assistance |
| Risk | downtime, data loss, reputation and penalties |
| Lifecycle | expansion, migration, disposal and renewal |
In the suggested scoring model, assign a weight from 1 to 5 for availability, performance, capacity, integration, security, support, expansion and TCO; score each alternative from 1 to 5, multiply by the weight and record evidence; disqualify beforehand any solution that fails to meet eliminatory requirements.
TrueNAS versus proprietary SAN: TrueNAS can offer excellent economy and flexibility; a vendor SAN can reduce integration, support and operational risk. The right comparison is by SLA and lifecycle, not just R$/TB.
Step-by-step deployment
A safe deployment advances through gates: requirements, design, build, test, migration and stabilization.
| Phase | Deliverables | Gate |
|---|---|---|
| Discovery | inventory, workload, RPO/RTO | requirements approved |
| HLD | topology, capacity, HA, security | architecture approved |
| LLD | ports, VLANs, zoning, LUNs, names | technical review |
| Build | firmware, pool, services, alerts | clean baseline |
| Test | FAT/SAT, performance, failures, restore | signed acceptance |
| Migration | waves, rollback, communication | go/no-go |
| Operation | runbook, training, as-built | handover |
Acceptance criteria: usable capacity and reserve confirmed; sustained performance with a representative profile and latency within the SLO; path, switch, disk and controller failures tested; data and configuration restoration proven; alerts received by the NOC/responsible team; and as-built documentation and responsibility matrix delivered.
Reference architectures
The examples below are starting points and should be adjusted to the workload and failure domain.
A. SMB / backup and files: TrueNAS with a single controller, mirrored boot, RAIDZ2 pool on CMR HDDs, ECC RAM, 10/25 GbE NICs, snapshots and replication to a second appliance. Suitable when interruption for maintenance is acceptable.
B. Critical virtualization: TrueNAS Enterprise HA or equivalent storage with redundant controllers, SSD/all-flash or appropriately sized mirrors, A/B 25 GbE iSCSI network, MPIO on all hosts, separate management and external replication. SLOG only if testing shows a benefit for synchronous writes.
C. Fibre Channel SAN: hosts with dual-port HBAs, two independent FC fabrics, single-initiator/single-target zoning, HA storage, LUN masking, certified multipath, LAN-free backup when justified, and fabric monitoring.
D. High-capacity repository: multiple RAIDZ2/3 vdevs on NL-SAS/CMR HDDs, metadata evaluated for a redundant special vdev, 25/100 GbE network, lifecycle policies and offsite copy. Prioritize sequential throughput and rebuild time.
Risks and anti-patterns
Most serious incidents arise from simple decisions that were never tested.
| Anti-pattern | Consequence | Fix |
|---|---|---|
| A single huge RAIDZ vdev | long rebuild and low flexibility | multiple balanced vdevs |
| RAIDZ1 with large critical disks | vulnerability window | RAIDZ2/3 or mirrors |
| LACP as "MPIO" | poorly covered end-to-end failure | distinct subnets/paths |
| SLOG without PLP | risk on power loss | enterprise device with PLP |
| Dedup by default | severe RAM/CPU pressure | measure first; compression first |
| Nearly full pool | fragmentation and performance drop | alert and early expansion |
| Snapshots on the same storage only | loss in disaster/attack | replica + isolated backup |
| No restore testing | fictitious RPO/RTO | periodic exercises |
Go/No-Go: do not go into production if failover, restore, alerts, rollback and documentation have not been validated.
Discovery checklist
Use this script in technical and commercial workshops. On business and SLA: applications, owners and criticality; RPO, RTO, maintenance window and cost of downtime; retention, compliance, sovereignty and audit. On workload: current capacity, monthly/annual growth and compressibility; IOPS, throughput, p95/p99 latency, block size and read/write; protocols, hosts, versions, clusters and multipath; peaks, backup, batch, scrub, replication and migration.
On infrastructure: rack, A/B power, kVA, cooling and weight; ports, speeds, optics, cables, VLANs and IPs; team, monitoring, support, parts and remote site. On acceptance: synthetic workload and real application; injected failures and recovery time; restore, DR and evidence; training and as-built documentation.
Commands and validations
Examples for diagnostics — execute changes only under an approved procedure. For pool health and events: zpool status -v, zpool list, zpool events -v, zpool iostat -v 2. For datasets, snapshots and properties: zfs list -o name,used,avail,refer,mountpoint,compression, zfs get -r recordsize,compression,sync,atime POOL, zfs list -t snapshot -o name,creation,used -s creation. For network and iSCSI on the Linux host: ip -s link, ss -ti, iscsiadm -m session -P 3, multipath -ll, iostat -x 2.
Best practices: collect a baseline before and after any adjustment. Change one variable at a time and log workload, time and result.
Glossary
- ARC: ZFS's adaptive in-memory cache.
- L2ARC: secondary read cache on a fast device.
- ZIL: log mechanism for synchronous writes.
- SLOG: separate device that stores the ZIL.
- Vdev: redundancy and performance unit of a pool.
- Zvol: block volume created on top of ZFS.
- LUN: logical unit presented over SCSI.
- WWPN: Fibre Channel port identifier.
- IQN: qualified name of an iSCSI initiator/target.
- MPIO: multipath I/O with redundant paths.
- RPO: maximum tolerated data loss.
- RTO: maximum time to restore service.
- PLP: power-loss protection on SSD/NVMe.
- DWPD: full drive writes per day during the warranty period.
Conclusion
A well-designed enterprise storage system treats availability, performance, data protection, security and cost as a single system of decisions, not as isolated choices. Fibre Channel, iSCSI and NVMe-oF each bring their own ecosystem of latency and complexity; TrueNAS on OpenZFS offers a flexible path from the lab to critical production, provided hardware, network and operations are validated with the same rigor as a proprietary SAN.
The final criterion is not the price per terabyte, but the question that must precede any architecture: which failure does the business need to survive, and within how much time — and that answer is only reliable once tested, not merely designed on paper.
This content is a consolidated technical summary based on official TrueNAS, OpenZFS, SNIA and NVM Express documentation, with a cutoff date of September 2026. Product documentation is dynamic — use the stable version applicable to your environment before any procurement or deployment project.