ONTAP

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) This article contains promotional content. Please help improve it by removing promotional language and inappropriate external links, and by adding encyclopedic text written from a neutral point of view. (November 2017) (Learn how and when to remove this message) Some of this article's listed sources may not be reliable. Please help improve this article by looking for better, more reliable sources. Unreliable citations may be challenged and removed. (November 2017) (Learn how and when to remove this message) This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "ONTAP" – news · newspapers · books · scholar · JSTOR (November 2017) (Learn how and when to remove this message) (Learn how and when to remove this message)

ONTAP

Developer	NetApp
OS family	Unix-like (BSD) (Data ONTAP GX, Data ONTAP 8, and later)
Working state	Active
Latest release	9.16.1[1] / November 2024; 1 month ago (2024-11)
Platforms	IA-32 (no longer supported), Alpha (no longer supported), MIPS (no longer supported), x86-64 with ONTAP 8 and higher
Kernel type	Monolithic with dynamically loadable modules
Userland	BSD
Default user interface	Command-line interface (PowerShell, SSH, Serial console) Graphical user interfaces over Web-based user interfaces, REST API
Official website	netapp.com/data-management/ontap-data-management-software/

ONTAP, Data ONTAP, Clustered Data ONTAP (cDOT), or Data ONTAP 7-Mode is NetApp's proprietary operating system used in storage disk arrays such as NetApp FAS and AFF, ONTAP Select, and Cloud Volumes ONTAP. With the release of version 9.0, NetApp decided to simplify the Data ONTAP name and removed the word "Data" from it, removed the 7-Mode image, therefore, ONTAP 9 is the successor of Clustered Data ONTAP 8.

ONTAP includes code from BSD Net/2 and 4.4BSD-Lite, Spinnaker Networks technology, and other operating systems.^[2] ONTAP originally only supported NFS, but later added support for SMB, iSCSI, and Fibre Channel Protocol (including Fibre Channel over Ethernet and FC-NVMe). On June 16, 2006,^[3] NetApp released two variants of Data ONTAP, namely Data ONTAP 7G and, with nearly a complete rewrite,^[2] Data ONTAP GX. Data ONTAP GX was based on grid technology acquired from Spinnaker Networks. In 2010 these software product lines merged into one OS - Data ONTAP 8, which folded Data ONTAP 7G onto the Data ONTAP GX cluster platform.

Data ONTAP 8 includes two distinct operating modes held on a single firmware image. The modes are called ONTAP 7-Mode and ONTAP Cluster-Mode. The last supported version of ONTAP 7-Mode issued by NetApp was version 8.2.5. All subsequent versions of ONTAP (version 8.3 and onwards) have only one operating mode - ONTAP Cluster-Mode.

NetApp storage arrays use highly customized hardware and the proprietary ONTAP operating system, both originally designed by NetApp founders David Hitz and James Lau specifically for storage-serving purposes. ONTAP is NetApp's internal operating system, specially optimized for storage functions at both high and low levels. The original version of ONTAP had a proprietary non-UNIX kernel and a TCP/IP stack, networking commands, and low-level startup code from BSD.^[4][2] The version descended from Data ONTAP GX boots from FreeBSD as a stand-alone kernel-space module and uses some functions of FreeBSD (for example, it uses a command interpreter and drivers stack).^[2] ONTAP is also used for virtual storage appliances (VSA), such as ONTAP Select and Cloud Volumes ONTAP, both of which are based on a previous product named Data ONTAP Edge.

All storage array hardware includes battery-backed non-volatile memory,^[5] which allows them to commit writes to stable storage quickly, without waiting on disks while virtual storage appliances use virtual nonvolatile memory.

Implementers often organize two storage systems in a high-availability cluster with a private high-speed link, either a Fibre Channel, InfiniBand, 10 Gigabit Ethernet, 40 Gigabit Ethernet, or 100 Gigabit Ethernet. One can additionally group such clusters under a single namespace when running in the "cluster mode" of the Data ONTAP 8 operating system or on ONTAP 9.

Data ONTAP was made available for commodity computing servers with x86 processors, running atop VMware vSphere hypervisor, under the name "ONTAP Edge".^[6] Later ONTAP Edge was renamed to ONTAP Select and KVM was added as a supported hypervisor.

Data ONTAP, including WAFL, was developed in 1992 by David Hitz, James Lau,^[7] and Michael Malcolm.^[8] Initially, it supported NFSv2; the CIFS protocol was introduced to Data ONTAP 4.0 in 1996.^[9] In April 2019, Octavian Tanase SVP ONTAP posted a preview photo on his Twitter of ONTAP running in Kubernetes as a container for a demonstration.

The Write Anywhere File Layout (WAFL) is a file layout used by ONTAP OS that supports large, high-performance RAID arrays, quick restarts without lengthy consistency checks in the event of a crash or power failure, and growing the size of the filesystems quickly.

ONTAP OS contains several storage efficiencies, which are based on WAFL functionalities. Supported by all protocols, does not require licenses. In February 2018^{[citation needed]} NetApp claims AFF systems for its clients gain an average of 4.72:1 Storage Efficiency from deduplication, compression, compaction, and clones savings. Starting with ONTAP 9.3 offline deduplication and compression scanners start automatically by default and based on a percentage of new data written instead of scheduling.

• Data Reduction efficiency is a summary of Volume and Aggregate Efficiencies and Zero-block deduplication:
  • Volume Efficiencies could be enabled/disabled individually and on a volume-by-volume basis:
    1. Offline Volume Deduplication, which works on a 4 KB block level
    2. Additional efficiency mechanisms were introduced later, such as Offline Volume Compression also known as Post-process (or Background) Compression, there are two types: Post-process secondary compression and Post-process adaptive compression
    3. Inline Volume Deduplication and Inline Volume Compression compress some of the data on the fly before it reaches the disks and designed to leave some of the data in the uncompressed form if it is considered by ONTAP to take a long time to process them on the fly, and to leverage other storage efficiency mechanisms for this uncompressed data later. There are two types of Inline Volume Compression: Inline adaptive compression and Inline secondary compression
  • Aggregate Level Storage Efficiencies include:
    1. Data Compaction is another mechanism used to compress many data blocks smaller than 4 KB to a single 4 KB block
    2. Inline Aggregate-wide data deduplication (IAD) and Post-process aggregate deduplication also known as Cross-Volume Deduplication^{[citation needed]} share common blocks between volumes on an aggregate. IAD can throttle itself when the storage system crosses a certain threshold. The current limit of physical space of a single SSD aggregate is 800 TiB
  • Inline Zero-block Deduplication^[10] deduplicate zeroes on the fly before they reach disks
• Snapshots and FlexClones are also considered efficiency mechanisms. Starting with 9.4 ONTAP by default deduplicate data across the active file system and all the snapshots on the volume, saving from snapshot sharing is a magnitude of the number of snapshots, the more snapshots the more savings will be, therefore snapshot sharing gives more savings on SnapMirror destination systems.
• Thin Provisioning

Cross-Volume Deduplication storage efficiency features work only for SSD media. Inline and Offline Deduplication mechanisms that leverage databases consist of links of data blocks and checksums, for those data blocks that have been handled by the deduplication process. Each deduplication database is located on each volume and aggregates where deduplication is enabled. All Flash FAS systems do not support Post-process Compression.

Order of Storage Efficiencies execution is as follows:

1. Inline Zero-Block deduplication
2. Inline Compression: for files that could be compressed to the 8 KB adaptive compression used, for files more than 32 KB secondary compression used
3. Inline Deduplication: Volume first, then Aggregate
4. Inline Adaptive Data Compaction
5. Post-process Compression
6. Post-process Deduplication: Volume first, then Aggregate

One or multiple RAID groups form an "aggregate", and within aggregates ONTAP operating system sets up "flexible volumes" (FlexVol) to store data that users can access. Similar to RAID 0, each aggregate consolidates space from underlying protected RAID groups to provide one logical piece of storage for flexible volumes. Alongside aggregates of NetApp's disks and RAID groups aggregates could consist of LUNs already protected with third-party storage systems with FlexArray, ONTAP Select, or Cloud Volumes ONTAP. Each aggregate could consist of either LUNs or NetApp's RAID groups. An alternative is "traditional volumes" where one or more RAID groups form a single static volume. Flexible volumes offer the advantage that many of them can be created on a single aggregate and resized at any time. Smaller volumes can then share all of the spindles available to the underlying aggregate and with a combination of storage, QoS allows the performance of flexible volumes to be changed on the fly while traditional volumes do not. However, traditional volumes can (theoretically) handle slightly higher I/O throughput than flexible volumes (with the same number of spindles), as they do not have to go through an additional virtualization layer to talk to the underlying disk. Aggregates and traditional volumes can only be expanded, never contracted. The current maximum aggregate physical useful space size is 800 TiB for All-Flash FAS Systems.^{[citation needed]}

This section needs additional citations for verification. Please help improve this article by adding citations to reliable sources in this section. Unsourced material may be challenged and removed. (November 2017) (Learn how and when to remove this message)

The first form of redundancy added to ONTAP was the ability to organize pairs of NetApp storage systems into a high-availability cluster (HA-Pair);^[11] an HA-Pair could scale capacity by adding disk shelves. When the performance maximum was reached with an HA-Pair, there were two ways to proceed: one was to buy another storage system and divide the workload between them, another was to buy a new, more powerful storage system and migrate all workload to it. All the AFF and FAS storage systems were usually able to connect old disk shelves from previous models—this process is called head-swap. Head-swap requires downtime for re-cabling operations and provides access to old data with new controller without system re-configuration. From Data ONTAP 8, each firmware image contains two operating systems, named "Modes": 7-Mode and Cluster-Mode.^[12] Both modes could be used on the same FAS platform, one at a time. However, data from each of the modes wasn't compatible with the other, in case of a FAS conversion from one mode to another, or in case of re-cabling disk shelves from 7-Mode to Cluster-Mode and vice versa.

Later, NetApp released the 7-Mode transition Tool (7MTT), which is able to convert data on old disk shelves from 7-Mode to Cluster-Mode. It is named Copy-Free Transition,^[13] a process which required downtime. With version 8.3, 7-Mode was removed from the Data ONTAP firmware image.^[14]

Clustered ONTAP is a new, more advanced OS, compared to its predecessor Data ONTAP (version 7 and version 8 in 7-Mode), which is able to scale out by adding new HA-pairs to a single namespace cluster with transparent data migration across the entire cluster. In version 8.0, a new aggregate type was introduced, with a size threshold larger than the 16-terabyte (TB) aggregate size threshold that was supported in previous releases of Data ONTAP, also named the 64-bit aggregate.^[15]

In version 9.0, nearly all of the features from 7-mode were successfully implemented in ONTAP (Clustered) including SnapLock,^[16] while many new features that were not available in 7-Mode were introduced, including features such as FlexGroup, FabricPool, and new capabilities such as fast-provisioning workloads and Flash optimization.^[17]

The uniqueness of NetApp's Clustered ONTAP is in the ability to add heterogeneous systems (where all systems in a single cluster do not have to be of the same model or generation) to a single cluster. This provides a single pane of glass for managing all the nodes in a cluster, and non-disruptive operations such as adding new models to a cluster, removing old nodes, online migration of volumes, and LUNs while data is contiguously available to its clients.^[18] In version 9.0, NetApp renamed Data ONTAP to ONTAP.

ONTAP is considered to be a unified storage system, meaning that it supports both block-level (FC, FCoE, NVMeoF and iSCSI) & file-level (NFS, pNFS, CIFS/SMB) protocols for its clients. SDS versions of ONTAP (ONTAP Select & Cloud Volumes ONTAP) do not support FC, FCoE or NVMeoF protocols due to their software-defined nature.

NFS was the first protocol available in ONTAP. The latest versions of ONTAP 9 support NFSv2, NFSv3, NFSv4 (4.0 and 4.1) and pNFS. Starting with ONTAP 9.5, 4-byte UTF-8 sequences, for characters outside the Basic Multilingual Plane, are supported in names for files and directories.^[19]

ONTAP supports CIFS 2.0 and higher up to SMB 3.1. Starting with ONTAP 9.4 SMB Multichannel, which provides functionality similar to multipathing in SAN protocols, is supported. Starting with ONTAP 8.2 CIFS protocol supports Continuous Availability (CA) with SMB 3.0 for Microsoft Hyper-V over SMB and SQL Server over SMB. ONTAP supports SMB encryption, which is also known as sealing. Accelerated AES instructions (Intel AES NI) encryption is supported in SMB 3.0 and later.

ONTAP on physical appliances supports FCoE as well as FC protocol, depending on HBA port speed.

iSCSI Data Center Bridging (DCB) protocol supported with A220/FAS2700 systems.

NVMe over Fabrics (NVMeoF) refers to the ability to utilize NVMe protocol over existing network infrastructure like Ethernet (Converged or traditional), TCP, Fibre Channel or InfiniBand for transport (as opposite to run NVMe over PCI). NVMe is SAN block level data storage protocol. NVMeoF supported only on All-Flash A-Systems and not supported for low-end A200 and A220 systems. Starting with ONTAP 9.5 ANA protocol supported which provide, similarly to ALUA multi-pathing functionality to NVMe. ANA for NVMe currently supported only with SUSE Enterprise Linux 15. FC-NVMe without ANA supported with SUSE Enterprise Linux 12 SP3 and RedHat Enterprise Linux 7.6.

FC-NVMe Supported on systems with 32 Gbit/s FC ports or higher speeds. The supported Operation Systems with FC-NVMe are: Oracle Linux, VMware, Windows Server, SUSE Linux, RedHat Linux.

S3 (object)

ONTAP supports limited functionality for serving data via the S3 protocol for object access (see product documentation for details on what is and is not supported). S3 buckets leverage FlexGroup volume technology and ONTAP 9.12.1 has announced support for presenting existing NAS volumes as S3-accessible buckets.

High Availability (HA) is clustered configuration of a storage system with two nodes or HA pairs, which aims to ensure an agreed level of operational during expected and unexpected events like reboots, software or firmware updates.

Even though a single HA pair consists of two nodes (or controllers), NetApp has designed it in such a way that it behaves as a single storage system. HA configurations in ONTAP employ a number of techniques to present the two nodes of the pair as a single system. This allows the storage system to provide its clients with nearly-uninterruptible access to their data should a node either fail unexpectedly or need to be rebooted in an operation known as a "takeover."

For example: on the network level, ONTAP will temporarily migrate the IP address of the downed node to the surviving node, and where applicable it will also temporarily switch ownership of FC WWPNs from the downed node to the surviving node. On the data level, the contents of the disks that are assigned to the downed node will automatically be available for use via the surviving node.

FAS and AFF storage systems use enterprise level HDD and SSD drives that are housed within disk shelves that have two bus ports, with one port connected to each controller. All of ONTAP's disks have an ownership marker written to them to reflect which controller in the HA pair owns and serves each individual disk. An Aggregate can include only disks owned by a single node, therefore each aggregate owned by a node and any upper objects such as FlexVol volumes, LUNs, File Shares are served with a single controller. Each controller can have its own disks and aggregates and serve them, therefore such HA pair configurations are called Active/Active where both nodes are utilized simultaneously even though they are not serving the same data.

Once the downed node of the HA pair has been repaired, or whatever maintenance window that necessitated a takeover has been completed, and the downed node is up and running without issue, a "giveback" command can be issued to bring the HA pair back to "Active/Active" status.

High-availability clusters (HA clusters) are the first type of clusterization introduced in ONTAP systems. It aimed to ensure an agreed level of operation. It is often confused with the horizontal scaling ONTAP clusterization that came from the Spinnaker acquisition; therefore, NetApp, in its documentation, refers to an HA configuration as an HA pair rather than as an HA cluster.

An HA pair uses some form of network connectivity (often direct connectivity) for communication between the servers in the pair; this is called an HA interconnect (HA-IC). The HA interconnect can use Ethernet or InfiniBand as the communication medium. The HA interconnect is used for non-volatile memory log (NVLOG) replication using RDMA technology and for some other purposes only to ensure an agreed level of operational during events like reboots always between two nodes in a HA pair configuration. ONTAP assigns dedicated, non-sharable HA ports for HA interconnect which could be external or build in chassis (and not visible from the outside). The HA-IC should not be confused with the intercluster or intracluster interconnect that is used for SnapMirror and that can coexist with data protocols on data ports or with Cluster Interconnect ports used for horizontal scaling & online data migration across the multi-node cluster. HA-IC interfaces are visible only on the node shell level. Starting with A320 HA-IC and Cluster interconnect traffic start to use the same ports.

MetroCluster (MC) is an additional level of data availability to HA configurations and supported only with FAS and AFF storage systems, later SDS version of MetroCluster was introduced with ONTAP Select & Cloud Volumes ONTAP products. In MC configuration two storage systems (each system can be single node or HA pair) form MetroCluster, often two systems located on two sites with the distance between them up to 300 km therefore called geo-distributed system. Plex is the key underlying technology which synchronizes data between two sites in MetroCluster. In MC configurations NVLOG also replicated between storage systems between sites but uses dedicated ports for that purpose, in addition to HA interconnect. Starting with ONTAP 9.5 SVM-DR supported in MetroCluster configurations.

Is a feature of ONTAP Select software, similarly to MetroCluster on FAS/AFF systems MetroCluster SDS (MC SDS) allows to synchronously replicate data between two sites using SyncMirror and automatically switch to survived node transparently to its users and applications. MetroCluster SDS work as ordinary HA pare so data volumes, LUNs and LIFs could be moved online between aggregates and controllers on both sites, which is slightly different than traditional MetroCluster on FAS/AFF systems where data cloud be moved across storage cluster only within site where data originally located. In traditional MetroCluster the only way for applications to access data locally on remote site is to disable one entire site, this process called switchover where in MC SDS ordinary HA process occurs. MetroCluster SDS uses ONTAP Deploy as the mediator (in FAS and AFF world this functionality known as MetroCluster tiebreaker) which came with ONTAP Select as a bundle and generally used for deploying clusters, installing licenses and monitoring them.

Horizontal scaling ONTAP clusterization came from Spinnaker acquisitions and often referred by NetApp as "Single Namespace", "Horizontal Scaling Cluster" or "ONTAP Storage System Cluster" or just "ONTAP Cluster" and therefore often confused with HA pair or even with MetroCluster functionality. While MetroCluster and HA are Data Protection technologies, single namespace clusterization does not provide data protection. ONTAP Cluster is formed out of one or few HA pairs and adds to ONTAP system Non-Disruptive Operations (NDO) functionality such as non-disruptive online data migration across nodes in the cluster and non-disruptive hardware upgrade. Data migration for NDO operations in ONTAP Cluster require dedicated Ethernet ports for such operations called as cluster interconnect and does not use HA interconnect for this purposes. Cluster interconnect and HA interconnect could not share same ports. Cluster interconnect with a single HA pair could have directly connected cluster interconnect ports while systems with 4 or more nodes require two dedicated Ethernet cluster interconnect switches. ONTAP Cluster could consist only with even number of nodes (they must be configured as HA pairs) except for Single-node cluster. Single-node cluster ONTAP system also called non-HA (stand-alone). ONTAP Cluster managed with a single pane of glass built-in management with Web-based GUI, CLI (SSH and PowerShell) and API. ONTAP Cluster provides Single Name Space for NDO operations through SVM. Single Namespace in ONTAP system is a name for collection of techniques used by Cluster to separate data from front-end network connectivity with data protocols like FC, FCoE, FC-NVMe, iSCSI, NFS and CIFS and therefore provide kind of data virtualization for online data mobility across cluster nodes. On network layer Single Namespace provide a number of techniques for non-disruptive IP address migration, like CIFS Continuous Availability (Transparent Failover), NetApp's Network Failover for NFS and SAN ALUA and path election for online front-end traffic re-balancing with data protocols. NetApp AFF and FAS storage systems can consists of different HA pairs: AFF and FAS, different models and generations and can include up to 24 nodes with NAS protocols or 12 nodes with SAN protocols. SDS systems can't intermix with physical AFF or FAS storage systems.

Also known as Vserver or sometimes SVM. Storage Virtual Machine (SVM) is a layer of abstraction, and along with other functions, it virtualizes and separates physical front-end data network from data located on FlexVol volumes. It is used for Non-Disruptive Operations and Multi-Tenancy. It also forms as the highest form of logical construct available with NetApp. A SVM can not be mounted under another SVM, therefore it can be referred to a Global Namespace.

SVM divides storage system into slices so few divisions or even organizations can share a storage system without knowing and interfering with each other while using same ports, data aggregates and nodes in the cluster and using separate FlexVol volumes and LUNs. One SVM cannot create, delete, change or even see objects of another SVM so for SVM owners such an environment looks like they are only users on the entire storage system cluster.

There few Non Disruptive Operations (NDO) operations with (Clustered) ONTAP system. NDO data operations include: aggregate relocation within an HA pair between nodes, FlexVol volume online migration (known as Volume Move operation) across aggregates and nodes within Cluster, LUNs migration (known as LUN Move operation) between FlexVol volumes within Cluster. LUN move and Volume Move operations use Cluster Interconnect ports for data transfer (HA-CI is not in use for such operations). SVM behave differently with network NDO operations, depending on front-end data protocol. To decrease latency to its original level FlexVol volumes and LUNs have to be located on the same node with network address through which the clients access storage system, so network address could be created for SAN or moved for NAS protocols. NDO operations are free functionality.

For NAS front-end data protocols there are NFSv2, NFSv3, NFSv4 and CIFSv1, SMBv2 and SMB v3 which do not provide network redundancy with the protocol itself, so they rely on storage and switch functionalities for this matter. For this reason ONTAP support Ethernet Port Channel and LACP with its Ethernet network ports on L2 layer (known in ONTAP as interface group, ifgrp), within a single node and also non-disruptive Network Fail Over between nodes in cluster on L3 layer with migrating Logical Interfaces (LIF) and associated IP addresses (similar to VRRP) to survived node and back home when failed node restored.

For front-end data SAN protocols. ALUA feature used for network load balancing and redundancy in SAN protocols so all the ports on node where data located are reported to clients as active preferred path with load balancing between them while all other network ports on all other nodes in the cluster are active non-preferred path so in case of one port or entire node goes down, client will have access to its data using non-preferred path. Starting with ONTAP 8.3 Selective LUN Mapping (SLM) was introduced to reduce the number of paths to the LUN and removes non-optimized paths to the LUN through all other cluster nodes except for HA partner of the node owning the LUN so cluster will report to the host paths only from the HA pare where LUN is located. Because ONTAP provides ALUA functionality for SAN protocols, SAN network LIFs do not migrate like with NAS protocols. When data or network interfaces migration is finished it is transparent to storage system's clients due to ONTAP Architecture and can cause temporary or permanent data indirect access through ONTAP Cluster interconnect (HA-CI is not in use for such situations) which will slightly increase latency for the clients. SAN LIFs used for FC, FCoE, iSCSi & FC-NVMe protocols.

VIP (Virtual IP) LIFs require Top-of-the-Rack BGP Router used. BGP data LIFs along with NAS LIFs also can be used with Ethernet for NAS environment but in case of BGP LIFs, automatically load-balance traffic based on routing metrics and avoid inactive, unused links. BGP LIFs provide distribution across all the NAS LIFs in a cluster, not limited to a single node as in NAS LIFs. BGP LIFs provide smarter load balancing than it was realized with hash algorithms in Ethernet Port Channel & LACP with interface groups. VIP LIF interfaces are tested and can be used with MCC and SVM-DR.

Node management LIF interface can migrate with associated IP address across Ethernet ports of a single node and available only while ONTAP running on the node, usually located on e0M port of the node; Node management IP sometimes used by cluster admin to communicate with a node to cluster shell in rare cases where commands have to be issued from a particular node. Cluster Management LIF interface with associated IP address available only while the entire cluster is up & running and by default can migrate across Ethernet ports, often located on one of the e0M ports on one of the cluster nodes and used for cluster administrator for management purposes; it used for API communications & HTML GUI & SSH console management, by default ssh connect administrator with cluster shell. Service Processor (SP) interfaces available only at hardware appliances like FAS & AFF and allows ssh out-of-band console communications with an embedded small computer installed on controller mainboard and similarly to IPMI allows to connect, monitor & manage controller even if ONTAP OS is not booted, with SP it is possible to forcibly reboot or halt a controller and monitor coolers & temperature, etc.; connection to SP by ssh brings administrator to SP console but when connected to SP it is possible to switch to cluster shell through it; each controller has one SP which does not migrate like some other management interfaces. Usually, e0M and SP both lives on a single management (wrench) physical Ethernet port but each has its own dedicated MAC address. Node LIFs, Cluster LIF & SP often using the same IP subnet. SVM management LIF, similarly to cluster management LIF can migrate across all the Ethernet ports on the nodes of the cluster but dedicated for a single SVM management; SVM LIF does not have GUI capability and can facilitate only for API Communications & SSH console management; SVM management LIF can live on e0M port but often located on a data port of a cluster node on a dedicated management VLAN and can be different from IP subnets that node & cluster LIFs.

The cluster interconnect LIF interfaces using dedicated Ethernet ports and cannot share ports with management and data interfaces and for horizontal scaling functionality at times when like a LUN or a Volume migrates from one node of the cluster to another; cluster interconnect LIF similarly to node management LIFs can migrate between ports of a single node. Intercluster interface LIFs can live and share the same Ethernet ports with data LIFs and used for SnapMirror replication; intercluster interface LIFs, similarly to node management & LIFs cluster interconnect can migrate between ports of a single node.

ONTAP provide two techniques for Multi Tenancy functionality like Storage Virtual Machines and IP Spaces. On one hand SVMs are similar to Virtual Machines like KVM, they provide visualization abstraction from physical storage but on another hand quite different because unlike ordinary virtual machines SVMs does not allow to run third party binary code like in Pure storage systems; they just provide virtualized environment and storage resources instead. Also SVMs unlike ordinary virtual machines do not run on a single node but for the end user it looks like an SVM runs as a single entity on each node of the whole cluster. SVM divides storage system into slices, so a few divisions or even organizations can share a storage system without knowing and interfering with each other while utilizing same ports, data aggregates and nodes in the cluster and using separate FlexVol volumes and LUNs. Each SVM can run its own frontend data protocols, set of users, use its own network addresses and management IP. With use of IP Spaces users can have the same IP addresses and networks on the same storage system without interfering. Each ONTAP system must run at least one Data SVM in order to function but may run more. There are a few levels of ONTAP management and Cluster Admin level has all of the available privileges. Each Data SVM provides to its owner vsadmin which has nearly full functionality of Cluster Admin level but lacks physical level management capabilities like RAID group configuration, Aggregate configuration, physical network port configuration. However, vsadmin can manage logical objects inside an SVM like create, delete and configure LUNs, FlexVol volumes and network addresses, so two SVMs in a cluster can't interfere with each other. One SVM cannot create, delete, modify or even see objects of another SVM, so for SVM owners such an environment looks like they are the only users in the entire storage system cluster. Multi Tenancy is free functionality in ONTAP.

FlexClone is a licensed feature, used for creating writable copies of volumes, files or LUNs. In case of volumes, FlexClone acts as a snapshot but allows to write into it, while an ordinary snapshot allows only to read data from it. Because WAFL architecture FlexClone technology copies only metadata inodes and provides nearly instantaneous data copying of a file, LUN or volume regardless of its size.

SnapRestore is a licensed feature, used for reverting active file system of a FlexVol to a previously created snapshot for that FlexVol with restoring metadata inodes in to active file system. SnapRestore is used also for a single file restore or LUN restore from a previously created snapshot for the FlexVol where that object located. Without SnapRestore license in NAS environment it is possible to see snapshots in network file share and be able to copy directories and files for restore purposes. In SAN environment there is no way of doing restore operations similar to NAS environment. It is possible to copy in both SAN and NAS environments files, directories, LUNs and entire FlexVol content with ONTAP command ndmpcopy which is free. Process of copying data depend on the size of the object and could be time consuming, while SnapRestore mechanism with restoring metadata inodes in to active file system almost instant regardless of the size of the object been restored to its previous state.

FlexGroup is a free feature introduced in version 9, which utilizes the clustered architecture of the ONTAP operating system. FlexGroup provides cluster-wide scalable NAS access with NFS and CIFS protocols.^[20] A FlexGroup Volume is a collection of constituent FlexVol volumes distributed across nodes (up to 200 per FlexGroup) in the cluster called just "Constituents" or "member volumes", which are transparently aggregated in a single space. Therefore, FlexGroup Volume aggregates performance and capacity from all the Constituents and thus from all nodes of the cluster where they located, as well as parallelizing CPU cores per node for write metadata-heavy operations. For the end user, each FlexGroup Volume is represented by a single, ordinary NAS (SMB or NFS) file-share.^[21] The full potential of FlexGroup will be revealed with technologies like pNFS (added in ONTAP 9.7), NFS Multipathing (session trunking, announced in ONTAP 9.12.1) SMB multichannel (added in ONTAP 9.4), SMB Continuous Availability (FlexGroup with SMB CA shares Supported with ONTAP 9.6), and VIP (BGP). The FlexGroup feature in ONTAP 9 allows to massively scale in a single namespace to over 20PB with over 400 billion files, while evenly spreading the performance across the cluster.^[22] Starting with ONTAP 9.5 FabricPool (automatic cloud tiering for storage efficiency) support was added. ONTAP 9.5 also added support for SMB features for native file auditing, FPolicy, Storage Level Access Guard (SLA), copy offload (ODX) and inherited watches of changes notifications; Quotas and Qtree. SMB Contiguous Availability (CA) supported on FlexGroup allows running MS SQL & Hyper-V on FlexGroup, and FlexGroup supported on MetroCluster. For more information about FlexGroup volumes and supported features, see TR-4571: NetApp ONTAP FlexGroup Volumes Best Practice and Implementation Guide.

Snapshots form the basis for NetApp's asynchronous disk-to-disk replication (D2D) technology, SnapMirror, which effectively replicates Flexible Volume snapshots between any two ONTAP systems. SnapMirror is also supported from ONTAP to Cloud Backup and from SolidFire to ONTAP systems as part of NetApp's Data Fabric vision. NetApp also offers a D2D backup and archive feature named SnapVault, which is based on replicating and storing snapshots. Open Systems SnapVault allows Windows and UNIX hosts to back up data to an ONTAP, and store any filesystem changes in snapshots (not supported in ONTAP 8.3 and onwards). SnapMirror is designed to be part of a Disaster recovery plan: it stores an exact copy of data on time when snapshot was created on the disaster recovery site and could keep the same snapshots on both systems. SnapVault, on the other hand, is designed to store less snapshots on the source storage system and more Snapshots on a secondary site for a long period of time.
Data captured in SnapVault snapshots on destination system could not be modified nor accessible on destination for read-write, data can be restored back to primary storage system or SnapVault snapshot could be deleted. Data captured in snapshots on both sites with both SnapMirror and SnapVault can be cloned and modified with the FlexClone feature for data cataloging, backup consistency and validation, test and development purposes etc.
Later versions of ONTAP introduced cascading replication, where one volume could replicate to another, and then another, and so on. Configuration called fan-out is a deployment where one volume replicated to multiple storage systems. Both fan-out and cascade replication deployments support any combination of SnapMirror DR, SnapVault, or unified replication. It is possible to use fan-in deployment to create data protection relationships between multiple primary systems and a single secondary system: each relationship must use a different volume on the secondary system. Starting with ONTAP 9.4 destination SnapMirror & SnapVault systems enable automatic inline & offline deduplication by default.
Intercluster is a relationship between two clusters for SnapMirror, while Intracluster is opposite to it and used for SnapMirror relationship between storage virtual machines (SVM) in a single cluster.
SnapMirror can operate in version-dependent mode, where two storage systems must run on the same version of ONTAP or in version-flexible mode. Types of SnapMirror replication:

• Data Protection (DP): Also known as SnapMirror DR. Version-dependent replication type originally developed by NetApp for Volume SnapMirror, destination system must be same or higher version of ONTAP. Not used by default in ONTAP 9.3 and higher. Volume-level replication, block-based, metadata independent, uses Block-Level Engine (BLE).
• Extended Data Protection (XDP): Used by SnapMirror Unified replication and SnapVault. XDP uses the Logical Replication Engine (LRE) or if volume efficiency different on the destination volume the Logical Replication Engine with Storage Efficiency (LRSE). Used as Volume-level replication but technologically could be used for directory-based replication, inode-based, metadata dependent (therefore not recommended for NAS with millions of files).
• Load Sharing (LS): Mostly used for internal purposes like keeping copies of root volume for an SVM.
• SnapMirror to Tape (SMTape): is Snapshot copy-based incremental or differential backup from volumes to tapes; SMTape feature performing a block-level tape backup using NDMP-compliant backup applications such as CommVault Simpana.

SnapMirror-based technologies:

• Unified replication: A volume with Unified replication can get both SnapMirror and SnapVault snapshots. Unified replication is combination of SnapMirror Unified replication and SnapVault which using a single replication connection. Both SnapMirror Unified replication and SnapVault are using same XDP replication type. SnapMirror Unified Replication is also known as Version-flexible SnapMirror. Version-flexible SnapMirror/SnapMirror Unified Replication introduced in ONTAP 8.3 and removes the restriction to have the destination storage use the same, or higher, version of ONTAP.
• SVM-DR (SnapMirror SVM): replicates all volumes (exceptions allowed) in a selected SVM and some of the SVM settings, replicated settings depend on protocol used (SAN or NAS)
• Volume Move: Also known as DataMotion for Volumes. SnapMirror replicates volume from one aggregate to another within a cluster, then I/O operations stops for acceptable timeout for end clients, final replica transferred to destination, source deleted and destination becomes read-write accessible to its clients

SnapMirror is a licensed feature, a SnapVault license is not required if a SnapMirror license is already installed.

SVM DR based on SnapMirror technology which transferring all the volumes (exceptions allowed) and data in them from a protected SVM to a DR site. There are two modes for SVM DR: identity preserve and identity discard. With Identity discard mode, on the one hand, data in volumes copied to the secondary system and DR SVM does not preserve information like SVM configuration, IP addresses, CIFS AD integration from original SVM. On another hand in identity discard mode, data on the secondary system can be brought online in read-write mode while primary system online too, which might be helpful for DR testing, Test/Dev and other purposes. Therefore, identity discard requires additional configuration on the secondary site in the case of disaster occurs on the primary site.

In the identity preserve mode, SVM-DR copying volumes and data in them and also information like SVM configuration, IP addresses, CIFS AD integration which requires less configuration on DR site in case of disaster event on primary site but in this mode, the primary system must be offline to ensure there will be no conflict.

SnapMirror Sync (SM-S) for short is zero RPO data replication technology previously available in 7-mode systems and was not available in (clustered) ONTAP until version 9.5. SnapMirror Sync replicates data on Volume level and has requirements for RTT less than 10ms which gives distance approximately of 150 km. SnapMirror Sync can work in two modes: Full Synchronous mode (set by default) which guarantees zero application data loss between two sites by disallowing writes if the SnapMirror Sync replication fails for any reason. Relaxed Synchronous mode allows an application to write to continue on primary site if the SnapMirror Sync fails and once the relationship resumed, automatic re-sync will occur. SM-S supports FC, iSCSI, NFSv3, NFSv4, SMB v2 & SMB v3 protocols and have the limit of 100 volumes for AFF, 40 volumes for FAS, 20 for ONTAP Select and work on any controllers which have 16 GB memory or more. SM-S is useful for replicating transactional logs from: Oracle DB, MS SQL, MS Exchange etc. Source and destination FlexVolumes can be in a FabricPool aggregate but must use backup policy, FlexGroup volumes and quotas are not currently supported with SM-S. SM-S is not free feature, the license is included in the premium bundle. Unlike SyncMirror, SM-S not uses RAID & Plex technologies, therefore, can be configured between two different NetApp ONTAP storage systems with different disk type & media.

FlexCache technology previously available in 7-mode systems and was not available in (clustered) ONTAP until version 9.5. FlexCache allows serving NAS data across multiple global sites with file locking mechanisms. FlexCache volumes can cache reads, writes, and metadata. Writes on the edge generating push operation of the modified data to all the edge ONTAP systems requested data from the origin, while in 7-mode all the writes go to the origin and it was an edge ONTAP system's job to check the file haven't been updated. Also in FlexCache volumes can be less size that original volume, which is also an improvement compare to 7-mode. Initially, only NFS v3 supported with ONTAP 9.5. FlexCache volumes are sparsely-populated within an ONTAP cluster (intracluster) or across multiple ONTAP clusters (inter-cluster). FlexCache communicates over Intercluster Interface LIFs with other nodes. Licenses for FlexCache based on total cluster cache capacity and not included in the premium bundle. FAS, AFF & ONTAP Select can be combined to use FlexCache technology. Allowed to create 10 FlexCache volumes per origin FlexVol volume, and up to 10 FlexCache volumes per ONTAP node. The original volume must be stored in a FlexVol while all the FlexCache Volumes will have FlexGroup volume format.

Data ONTAP also implements an option named RAID SyncMirror (RSM), using the plex technique, where all the RAID groups within an aggregate or traditional volume can be synchronously duplicated to another set of hard disks. This is typically done at another site via a Fibre Channel or IP link, or within a single controller with local SyncMirror for a single disk-shelf resiliency. NetApp's MetroCluster configuration uses SyncMirror to provide a geo-cluster or an active/active cluster between two sites up to 300 km apart or 700 km with ONTAP 9.5 and MCC-IP. SyncMirror can be used either in software-defined storage platforms, on Cloud Volumes ONTAP, or on ONTAP Select. It provides high availability in environments with directly attached (non-shared) disks on top of commodity servers, or at FAS and AFF platforms in Local SyncMirror or MetroCluster configurations. SyncMirror is a free feature.

SnapLock implements Write Once Read Many (WORM) functionality on magnetic and SSD disks instead of to optical media so that data cannot be deleted until its retention period has been reached. SnapLock exists in two modes: compliance and enterprise. Compliance mode was designed to assist organizations in implementing a comprehensive archival solution that meets strict regulatory retention requirements, such as regulations dictated by the SEC 17a-4(f) rule, FINRA, HIPAA, CFTC Rule 1.31(b), DACH, Sarbanes-Oxley, GDPR, Check 21, EU Data Protection Directive 95/46/EC, NF Z 42-013/NF Z 42-020, Basel III, MiFID, Patriot Act, Graham-Leach-Bliley Act etc. Records and files committed to WORM storage on a SnapLock Compliance volume cannot be altered or deleted before the expiration of their retention period. Moreover, a SnapLock Compliance volume cannot be destroyed until all data has reached the end of its retention period. SnapLock is a licensed feature.

SnapLock Enterprise is geared toward assisting organizations that are more self-regulated and want more flexibility in protecting digital assets with WORM-type data storage. Data stored as WORM on a SnapLock Enterprise volume is protected from alteration or modification. There is one main difference from SnapLock Compliance: as the files being stored are not for strict regulatory compliance, a SnapLock Enterprise volume can be destroyed by an administrator with root privileges on the ONTAP system containing the SnapLock Enterprise volume, even if the designed retention period has not yet passed. In both modes, the retention period can be extended, but not shortened, as this is incongruous with the concept of immutability. Also, NetApp's SnapLock data volumes are equipped with a tamper-proof compliance clock, which is used as a time reference to block forbidden operations on files, even if the system time tampered.

Starting with ONTAP 9.5 SnapLock supports Unified SnapMirror (XDP) engine, re-synchronization after fail-over without data loss, 1023 snapshots, efficiency mechanisms and clock synchronization in SDS ONTAP.

Available for SSD-only aggregates in FAS/AFF systems or Cloud Volumes ONTAP on SSD media. Starting with ONTAP 9.4 FabricPool supported on ONTAP Select platform. Cloud Volumes ONTAP also supports HDD + S3 FabricPool configuration. Fabric Pool provides automatic storage tiering capability for cold data blocks from fast media (usually SSD) on ONTAP storage to cold media via object protocol to object storage such as S3 and back. Fabric Pool can be configured in two modes: One mode is used to migrate cold data blocks captured in snapshots, while the other mode is used to migrate cold data blocks in an active file system. FabricPool preserves offline deduplication & offline compression savings. Starting with ONTAP 9.4 introduced FabricPool 2.0 with the ability to tier-off active file system data (by default 31-day data not been accessed) & support data compaction savings. The recommended ratio is 1:10 for inodes to data files. For clients connected to the ONTAP storage system, all the Fabric Pool data-tiering operations are completely transparent, and in case data blocks become hot again, they are copied back to fast media in the ONTAP storage system. Fabric Pool is currently compatible with the NetApp StorageGRID, Amazon S3, Google Cloud, and Alibaba object storage services. Starting with ONTAP 9.4 Azure Blob supported, starting with 9.5 IBM Cloud Object Storage (ICOS) and Amazon Commercial Cloud Services (C2S) supported, other object-based SW & services could be used if requested by the user and that service will be validated by NetApp. FlexGroup volumes supported with FabricPool starting with ONTAP 9.5. The Fabric Pool feature in FAS/AFF systems is free for use with NetApp StorageGRID external object storage. For other object storage systems such as Amazon S3 & Azure Blob, Fabric Pool must be licensed per TB to function (alongside costs for Fabric Pool licensing, the customer needs to also pay for consumed object space). While with the Cloud Volumes ONTAP storage system, Fabric Pool does not require licensing, costs will apply only for consumed space on the object storage. Starting with ONTAP 9.5 capacity utilization triggering tiering from hot tier can be adjusted. SVM-DR also supported by FlexGroups.

FabricPool, first available in ONTAP 9.2, is a NetApp Data Fabric technology that enables automated tiering of data to low-cost object storage tiers either on or off-premises. Unlike manual tiering solutions, FabricPool reduces the total cost of ownership by automating the tiering of data to lower the cost of storage. It delivers the benefits of cloud economics by tiering to public clouds such as Alibaba Cloud Object Storage Service, Amazon S3, Google Cloud Storage, IBM Cloud Object Storage, and Microsoft Azure Blob Storage as well as to private clouds such as NetApp StorageGRID®. FabricPool is transparent to applications and allows enterprises to take advantage of cloud economics without sacrificing performance or having to re-architect solutions to leverage storage efficiency.

NetApp storage systems running ONTAP can Flash Cache (formally Performance Accelerate Module or PAM) custom purpose-built proprietary PCIe card for hybrid NetApp FAS systems. Flash Cache can reduce read latencies and allows the storage systems to process more read-intensive work without adding any further spinning disk to the underlying RAID since read operations do not require redundancy in case of Flash Cache failure. Flash Cache works on controller level and accelerates only read operations. Each separate volume on the controller can have a different caching policy or read cache could be disabled for a volume. FlashCache caching policies applied on FlexVol level. FlashCache technology is compatible with the FlexArray feature. Starting with 9.1, a single FlexVol volume can benefit from both FlashPool & FlashCache caches simultaneously. Beginning with ONTAP 9.5 Flash Cache read-cache technology available in Cloud Volumes ONTAP with the use of ephemeral SSD drives.

NDAS proxy is a service introduced in ONTAP 9.5; it works in conjunction with NDAS service in a cloud provider. Similarly to FabricPool, NDAS stores data in object format, but unlike FabricPool, it stores WAFL metadata in object storage as well. The information which been transferred from the ONTAP system is snapshot deltas, not the entire set of data, and already deduplicated & compressed (on volume level). NDAS proxy is HTTP-based with an S3 object protocol and few additional API calls to the cloud. NDAS in ONTAP 9.5 works only in a schema with primary ONTAP 9 storage replicating data via Snapmirror to secondary ONTAP 9.5 storage, where secondary storage is also NDAS proxy.

Storage QoS is a free feature in ONTAP systems. There are few types of storage QoS in ONTAP systems: Adaptive QoS (A-QoS), which includes Absolute minimum QoS; Ordinary static QoS or Minimum QoS (QoS min); and Maximum QoS (QoS max). Maximum QoS can be configured as a static upper limit in IOPS, MB/s, or both. It can be applied to an object such as Volume, LUN or a file, to prevent from such an object from consuming more storage performance resources than defined by the administrator (thus isolating performance-intensive bullies and protecting other workloads). Minimum QoS is contrary to maximum set on volumes to ensure that the volume will get no less than configured by the administrator static number of IOPS when there is contention for storage performance resources and could be applied to volumes. A-QoS is a mechanism of automatically changing QoS, based on consumed space by a flexible volume, because consumed space in it could grow or decrease, and the size of FlexVol can be changed. On FAS systems, A-QoS reconfigures only Peak performance (QoS max), while on AFF systems, it reconfigure both Expected performance (QoS min) and Peak performance (QoS max) on a volume. A-QoS allows ONTAP to automatically adjust the number of IOPS for a volume based on A-QoS policies. There are three basic A-QoS policies: Extreme, Performance and Value. Each A-QoS policy has a predefined fixed ratio IO per TB for Peak performance and Expected performance (or Absolute minimum QoS). Absolute minimum QoS is used instead of Expected performance (QoS min) only when volume size and ratio IO per TB is too small for example 10 GB.

ONTAP OS has a number of features to increase security on the storage system like Onboard Key Manager, the passphrase for controller boot with NSE & NVE encryption and USB key manager (available starting with 9.4). Auditing for NAS events is another security measure in ONTAP that enables the customer to track and log certain CIFS and NFS events on the storage system. This helps to track potential security problems and provides evidence of any security breaches. ONTAP accessed over SSH has an ability to Authenticate with a Common Access Card. ONTAP supports RBAC: Role-based access control allows administrative accounts to be restricted and/or limited in what actions they can take on the system. RBAC prevents a single account from being allowed to perform all potential actions available on the system. Beginning with ONTAP 9, Kerberos 5 authentication with privacy service (krb5p) is supported for NAS. The krbp5 authentication mode protects against data tampering and snooping by using checksums to encrypt all traffic between client and server. The ONTAP solution supports 128-bit and 256-bit AES encryption for Kerberos.

Onboard Key Manager is a free feature introduced in 9.1 and can store keys from NVE encrypted volumes & NSE disks. NSE Disks are available only on AFF/FAS platforms. ONTAP systems also allow storing encryption keys on a USB drive connected to the appliance. ONTAP also can use an external key manager like Gemalto Trusted Key Manager.

NetApp Volume Encryption (NVE) is FlexVol volume-level software-based encryption, which uses storage CPU for data encryption purposes; thus, some performance degradation is expected though it is less noticeable on high-end storage systems with more CPU cores. NVE is licensed, but free features compatible nearly with all NetApp ONTAP features and protocols. Similarly to NetApp Storage Encryption (NSE), NVE can store encryption keys locally or on a dedicated key manager like IBM Security Key Lifecycle Manager, SafeNet KeySecure or cloud key managers. NVE, like NSE, is also data at rest encryption, which means it protects only from physical disks theft and does not give an additional level of data security protection in a healthy operational and running system. NVE with a combination of FabricPool technology also protects data from unauthorized access in external S3 storage systems like Amazon and since data already encrypted it transferring over the wire in encrypted form.

Starting with ONTAP 9.4 new feature introduced called Secure Purge which provides ability to securely delete a file to comply with GDPR requirements.

ONTAP Vscan and FPolicy are aimed at malware prevention in ONTAP systems with NAS. Vscan provides a way for NetApp antivirus scanner partners to verify that files are virus-free. FPolicy integrates with NetApp partners to monitor file access behaviors. FPolicy file-access notification system monitor activity on NAS storage and prevent unwanted access or change to files based on policy settings. Both help in preventing ransomware from getting a foothold in the first place.

MTU black-hole detection and path MTU discovery (PMTUD) is the processes by which the ONTAP system connected via an Ethernet network detects maximum MTU size. In ONTAP 9.2: Online Certificate Status Protocol (OCSP) for LDAP over TLS; iSCSI Endpoint Isolation to specify a range of IP addresses that can log in to the storage; limit the number of failed login attempts over SSH. NTP symmetric authentication supported starting with ONTAP 9.5.

NetApp offers a set of server-based software solutions for monitoring and integration with ONTAP systems. The most commonly used free software is the ActiveIQ Unified Manager & Performance manager, which is data availability and performance monitoring solution.

NetApp Workflow Automation (WFA) is a free, server-based product used for NetApp storage orchestration. It includes a self-service portal with a web-based GUI, where nearly all routine storage operations or sequences of operations can be configured as workflows and published as a service, so end users can order and consume NetApp storage as a service.

SnapCenter, previously known as SnapManager Suite, is a server-based product. NetApp also offers products for taking application-consistent snapshots by coordinating the application and the NetApp Storage Array. These products support Microsoft Exchange, Microsoft SQL Server, Microsoft Sharepoint, Oracle, SAP and VMware ESX Server data. These products form part of the SnapManager suite. SnapCenter also includes third-party plugins for MongoDB, IBM Db2, MySQL, and allows the end user to create their own plugins for integration with the ONTAP storage system. SnapManager and SnapCenter are enterprise-level licensed products. A similar, free, and less capable NetApp product exists, named SnapCreator. It is intended for customers who wish to integrate ONTAP application-consistent snapshots with their applications, but do not have a license for SnapCenter. NetApp claims that SnapCenter capabilities will expand to include SolidFire storage endpoints. SnapCenter has controller based licensing for AFF/FAS systems and by Terabyte for SDS ONTAP. SnapCenter Plug-in for VMware vSphere called NetApp Data Broker is a separate Linux-based appliance which can be used without SnapCenter itself.

NetApp Services Level Manager or NSLM for short is software for provisioning ONTAP storage that delivers predictable performance, capacity and data protection for a workload which exposes RESTful APIs and has built-in Swagger documentation with the list of the available APIs, and also can be integrated with other NetApp storage products like ActiveIQ Unified Manager. NSLM exposes three standard service levels (SSL) based on service level objectives (SLO) and creates custom service levels. NSLM created to provide predicted ServiceProvider-like storage consumption. NSLM is a space-based licensed product.

ONTAP systems have the ability to integrate with Hadoop TeraGen, TeraValidate and TeraSort, Apache Hive, Apache MapReduce, Tez execution engine, Apache Spark, Apache HBase, Azure HDInsight and Hortonworks Data Platform Products, Cloudera CDH, through NetApp In-Place Analytics Module (also known as NetApp NFS Connector for Hadoop) to provide access and analyze data by using external shared NAS storage as primary or secondary Hadoop storage.

A qtree^[23] is a logically defined file system with no restrictions on how much disk space can be used or how many files can exist. In general, qtrees are similar to volumes. However, they have the following key restrictions:

• Snapshot copies can be enabled or disabled for individual volumes but not for individual qtrees.
• Qtrees do not support space reservations or space guarantees.

ONTAP provisioning & usage can be automated in many ways directly or with the use of additional NetApp Software or with 3rd party software.

• Direct HTTP REST API available with ONTAP and SolidFire. Starting with 9.6 ONTAP NetApp decided to start bringing proprietary ZAPI functionality via REST APIs access for cluster management. REST APIs available through System Manager web interface at https://[ONTAP_ClusterIP_or_Name]/docs/api, the page includes Try it out feature, Generate the API token to authorize external use and built-in documentation with examples. List of cluster management available through REST APIs in ONTAP 9.6:
  • Cloud (object storage) targets
  • Cluster, nodes, jobs and cluster software
  • Physical and logical network
  • Storage virtual machines
  • SVM name services such as LDAP, NIS, and DNS
  • Resources of storage area network (SAN)
  • Resources of Non-Volatile Memory Express
• ONTAP SDK software is a proprietary ZAPI interface to automate ONTAP systems
• PowerShell commandlets available to manage NetApp systems including ONTAP, SolidFire & E-Series
• SnapMirror & FlexClone toolkits written in Perl can be used for SnapMirror & FlexClone managing with scripts
• ONTAP can be automated with Ansible, Puppet, and Chef scripts
• NetApp Workflow Automation (WFA) is GUI based orchestrator which also provides APIs and PowerShell commandlets for WFA. WFA can manage NetApp ONTAP, SolidFire & E-Series storage systems. WFA provides a built-in self-service portal for NetApp systems known as Storage as a Service (STaaS)
• VMware vRealize Orchestrator with WFA can orchestrate storage
• 3rd party orchestrators for PaaS or IaaS like Cisco UCS Director (Previously Cloupia) and others can manage NetApp systems; automated workflows can be created with step by step instructions to manage & configure infrastructure through the built-in self-service portal
• NetApp SnapCenter software used to integrate Backup & Recovery on NetApp storage with Applications like VMware ESXi, Oracle DB, MS SQL, etc., can be automated through PowerShell commandlets and RESTful API
• ActiveIQ Unified Manager & Performance manager (formally OnCommand Unified) for monitoring NetApp FAS/AFF storage systems, performance metrics, and data protection also provide RESTful API & PowerShell commandlets
• OnCommand Insight is monitoring and analysis software for heterogeneous infrastructure including NetApp ONTAP, SolidFire, E-Series & 3rd party storage systems & switches provide RESTful API and PowerShell commandlets
• NetApp Trident plugin for Docker used in Containers environments to provide persistent storage, automate infrastructure or even run infrastructure as a code. It can be used with NetApp ONTAP, SolidFire & E-Series systems for SAN & NAS protocols.

The ONTAP operating system is used in storage disk arrays. There are three platforms where ONTAP software is used: NetApp FAS and AFF, ONTAP Select and Cloud Volumes ONTAP. On each platform, ONTAP uses the same kernel and a slightly different set of features. FAS is the richest for functionality among other platforms.

FAS^[24] and All Flash FAS (AFF)^[25] systems are proprietary, custom-built hardware by NetApp for ONTAP software. AFF systems can contain only SSD drives, because ONTAP on AFF is optimized and tuned only for Flash memory, while FAS systems may contain HDD (HDD-only systems) or HDD and SSD (Hybrid systems). ONTAP on FAS and AFF platforms can create RAID arrays, such as RAID 4, RAID-DP and RAID-TEC arrays, from disks or disk partitions for data protection reasons, while ONTAP Select and Cloud Volumes ONTAP leverage RAID data protection provided by the environment they run on. FAS and AFF systems support Metro Cluster functionality, while ONTAP Select and Cloud Volumes ONTAP platforms do not.

Both ONTAP Select and Cloud Volumes ONTAP are virtual storage appliances (VSA) which are based on previous product ONTAP Edge also known as ONTAP-v and considered as a Software-defined storage.^[26] ONTAP Select as Cloud Volumes ONTAP includes plex and aggregate abstractions, but didn't have a lower level RAID module included in the OS; therefore RAID 4, RAID-DP and RAID-TEC were not supported so ONTAP storage system similarly to FlexArray functionality leverages RAID data protection on SSD and HDD drive level with underlying storage systems. Starting with ONTAP Select 9.4 & ONTAP Deploy 2.8 software RAID supported with no requirements for 3rd party HW RAID equipment. Because ONTAP Select and Cloud Volumes ONTAP are virtual machines, they don't support Fibre Channel and Fibre Channel over Ethernet as front-end data protocols and consume space from underlying storage in hypervisor added to VSA as virtual disks represented and treated inside ONTAP as disks. ONTAP Select and Cloud Volumes ONTAP provide high availability, deduplication, resiliency, data recovery, robust snapshots which can be integrated with application backup (application consistent snapshots) and nearly all ONTAP functionality but with few exceptions. Software-defined versions of ONTAP have nearly all the functionality except for Hardware-centric features like ifgroups, service processor, physical disk drives with encryption, MetroCluster over FCP, Fiber Channel protocol.

ONTAP Select can run on VMware ESXi and Linux KVM hypervisors. ONTAP Select leveraged RAID data protection on SSD and HDD drive level with underlying DAS, SAN, or vSAN storage systems. Starting with ONTAP Select 9.4 & ONTAP Deploy 2.8 software RAID supported with no requirements for 3rd party HW RAID equipment for KVM and starting with ONTAP 9.5 with ESXi. ONTAP Deploy is a virtual machine that provides a mediator function in MetroCluster or 2-node configurations, keeps track of licensing, and used to initial cluster deployment. Starting with ONTAP Deploy 2.11.2 vCenter plugin was introduced, which allows performing all the ONTAP Deploy functionality from vCenter. In contrast, previously management performed from either command line or with vSphere VM OVA setup master. Like on the FAS platform, ONTAP Select supports high availability and clustering. As a FAS platform, ONTAP Select is offered in two versions: HDD-only or All-Flash optimized. Previously ONTAP Select known as Data ONTAP Edge. Data ONTAP Edge product has Data ONTAP OS with version 8 and was able to run only atop of VMware ESXi. Starting with ONTAP 9.5 SW-MetroCluster over NSX overlay network supported. Starting with ONTAP 9.5 licensing changed from capacity tier-based, where licenses are linked with a node and perpetual to Capacity Pool Licensing with a time-limited subscription. ONTAP Select 9.5 get MQTT protocol supported for data transferring from the edge to a data center or a cloud. In April 2019, Octavian Tanase SVP ONTAP, posted a preview photo in his twitter of ONTAP running in Kubernetes as a container for a demonstration.

Cloud Volumes ONTAP (formally ONTAP Cloud^[27]) includes nearly the same functionality as ONTAP Select, because it is also a virtual storage appliance (VSA) and can be ordered in hyper-scale providers (cloud computing) such as Amazon AWS, Microsoft Azure and Google Cloud Platform. IBM Cloud uses ONTAP Select for the same reasons, instead of Cloud Volumes ONTAP. Cloud Volumes ONTAP can provide high availability of data across different regions in the cloud. Cloud Volumes ONTAP leverages RAID data protection on SSD and HDD drive level with underlying IP SAN storage system in Cloud Provider.

Applicable Feature comparison between platforms with the latest ONTAP version.

Features	AFF/FAS/LenovoDM systems	All-Flash ASA	Cloud Volumes Service (CVS) & Azure NetApp Files (ANF)	Cloud Volumes ONTAP (CVO)	ONTAP Select
FabricPool	SSD aggregates only	Yes	Yes	Yes, SSD & HDD supported	Starting with 9.4 Supported FabricPool 2.0 for SDS with Premium lic (All Flash)
FlexGroup	Supported	No: SAN only	?	?	Supported
High Availability (HA)	Supported	Supported	Yes	Supported in AWS and Azure	Supported with DAS configuration. 2, 4, 6 or 8 nodes supported. 2-node require mediator which incorporated in ONTAP Deploy. 2x times more space consumed
Metro-HA	MetroCluster supported. FAS2000, C190 & A200 are not supported, support added to A220 & FAS2750 in ONTAP 9.6. Additional HW required. 2x times more space/disks consumed. MetroCluster Mediator software used for monitoring & automatic site switchover in a disaster event. Mediator has to run on 3rd site.	?	No	Supported in AWS between two availability zones. As in Cloud Volumes ONTAP with HA, 2x times more space consumed	Starting with ONTAP Deploy 2.7 officially supported MetroCluster SDS on 2-node clusters with DAS configuration for distance up to 10 km. As in ONTAP Select with HA, 2x times more space consumed. Also as ONTAP Select with 2-node HA system require mediator which incorporated in ONTAP Deploy. Mediator functionality used for monitoring & automatic site switchover in a disaster event. ONTAP Deploy with Mediator in MetroCluster configuration has to run on 3rd site.
Horizontal Scaling Clusterization	In ONTAP 9.3: from 1 node to up to 12 for SAN nodes; NAS up to 24 - with last and previous gen FAS/AFF. Exception: FAS2500 up to 8 nodes	?	No	No	1, 2, 4 or 8 nodes. ONTAP cluster with more than 1 node can contain only HA pairs.
Non Disruptive Operations	Aggregate relocate, volume move, LUN move, LIF migrate	Aggregate relocate, volume move, LUN move, LIF migrate	N/A	N/A	Aggregate relocate, volume move, LUN move, LIF migrate
Multi Tenancy	Yes	Yes	No: Not Applicable for the cloud	No: Not Applicable for the cloud	Yes
FlexClone	Yes, included in premium bundle SW	Yes, included in premium bundle SW	Yes?	Yes	Yes, always included
SnapRestore	Yes, included in premium bundle SW	Yes, included in premium bundle SW	Yes?	Yes?	Yes, always included
SnapMirror	Yes, included in premium bundle SW; SVM-DR SnapMirror supported; Also supported SnapMirror from ONTAP to Cloud Backup; SnapMirror from SolidFire to ONTAP	Yes, included in premium bundle SW; SVM-DR SnapMirror supported; Also supported SnapMirror from ONTAP to Cloud Backup; SnapMirror from SolidFire to ONTAP	No?	Yes. Also supported SnapMirror from ONTAP to Cloud Backup; SnapMirror from SolidFire to ONTAP	Yes, always included; SVM-DR SnapMirror supported; Also supported SnapMirror from ONTAP to Cloud Backup; SnapMirror from SolidFire to ONTAP
SnapMirror Synchronous (SM-S)	Yes with ONTAP 9.5, Maximum 80 volumes per AFF node or 40 per FAS node.	Yes Maximum 80 volumes per node.	No?	?	Yes with ONTAP 9.5, Maximum 20 volumes per Select node
FlexCache	Yes starting with ONTAP 9.5	Yes	Yes?	Yes?	Yes starting with ONTAP 9.5
SyncMirror	Yes: as SyncMirror local or as part of MetroCluster	Yes	No	As part of Cloud Volumes ONTAP HA functionality	As part of ONTAP Select HA & MetroCluster SDS functionality
WORM for NAS	Yes: SnapLock. Additional licensing required	No: SAN only	?	Yes in AWS & Azure. Marketing name: NetApp Cloud WORM	Yes: SlapLock with 9.4. Additional licensing required
QoS	Yes. QoS max on SVM, FlexVol, LUN and File level. QoS min, Adaptive QoS. Free	Yes. QoS max on SVM, FlexVol, LUN and File level. QoS min, Adaptive QoS. Free	Build-in 3-tier performance	?	Yes. QoS max on SVM, FlexVol, LUN and File level. QoS min, Adaptive QoS. Free
NetApp Volume Encryption	Yes. With local onboard key manager or external key manager. Free license required	Yes. With local onboard key manager or external key manager. Free license required	?	?	Yes. With local onboard key manager or external key manager. Free license required
SnapCenter /SnapManager	Yes, included in premium bundle SW	Yes, included in premium bundle SW	?	?	Yes. Additional licensing required. Licensed by space
NetApp's proprietary RAID: 4, DP, TEC	Yes	Yes	Yes. But not visible for the customer	?	Yes, with ONTAP Select 9.4 & ONTAP Deploy 2.8. Software RAID available with ONTAP Select 9.4 only with KVM. Starting with 9.5 software RAID available for ESXi.
Read/Write cache	Yes: FlashPool, only in FAS	No	No	No	No
FlexArray	Yes for FAS systems except for FAS2000 systems.	No	No	No, N/A	No
Third-party DAS or RAID	No	No	No	Cloud block storage	Yes. DAS with RAID, LVM and vSAN supported
NetApp Storage Encryption	Yes, specialized HDD/SDD required. Free	Yes, specialized HDD/SDD required. Free	?	No	No
Read cache	Yes: FlashCache for FAS systems except for FAS2200/2500 systems. All current gen FAS systems have pre-installed	No	No	Yes, with ephemeral SSD and Premium or BYOL licenses starting with 9.5	No
nonvolatile memory	Yes: All FAS/AFF	Yes	Yes	virtual Nonvolatile memory	virtual Nonvolatile memory
NDMP	Yes, free. Supported on FlexVol, doesn't supported on FlexGroup & FabricPool	Yes, free. Supported on FlexVol, doesn't supported on FlexGroup & FabricPool	No?	?	Yes, free. Supported on FlexVol, doesn't supported on FlexGroup & FabricPool
NetApp Snapshots	255 with 9.3 and older, 1024 starting with 9.4. Free & always included	Yes: 1024. Free & always included	Yes: 1024. Free & always included	255 with 9.3 and older, 1024 starting with 9.4. Free & always included	255 with 9.3 and older, 1024 starting with 9.4. Free & always included
Secure Purge	Starting with ONTAP 9.4, free	Yes?	Yes?	?	?
Interface Groups (ifgroup)	Yes, physical Ethernet port aggregation	Yes, physical Ethernet port aggregation	No: Not Applicable for the cloud	No	No
Max Aggregate Size	800 TiB on SSD	800 TiB on SSD	No: Not Applicable for the cloud	?	400 TiB per node (200 TiB useful in HA)
NVMeoF	Yes: FC-NVMe for AFF	Yes?	No	No	No
SAN	Yes: FC, iSCSI	Yes: FC, iSCSI	Yes for CVS; No for ANF	iSCSI	iSCSI
NAS	Yes: NFSv3, NFSv4, NFSv4.1, pNFS, SMBv2, SMBv3	No: SAN only	Yes: NFSv3, NFSv4, NFSv4.1?, pNFS?, SMBv2, SMBv3	Yes: NFSv3, NFSv4, NFSv4.1?, pNFS?, SMBv2, SMBv3	Yes: NFSv3, NFSv4, NFSv4.1, pNFS, SMBv2, SMBv3

• Write Anywhere File Layout (WAFL), used in NetApp storage systems
• NetApp FAS

• ONTAP Cloud (product page)
• ONTAP Select (product page)
• ONTAP 9 Datasheet