# Pure Storage - `pure` [Pure Storage](https://www.everpuredata.com/) is a software-defined storage solution. It offers the consumption of redundant block storage across the network. LXD supports connecting to Pure Storage storage clusters through four modes: (`iscsi`), (`nvme/tcp`), NVMe over (`nvme/fc`), and SCSI over FC (`scsi/fc`). In addition, Pure Storage offers copy-on-write snapshots, thin provisioning, and other features. To use Pure Storage with LXD requires a Pure Storage API version of at least `2.21`, corresponding to a minimum Purity//FA version of `6.4.2`. Additionally, ensure that the required kernel modules for the selected mode are installed on your host system. For iSCSI, the iSCSI CLI named `iscsiadm` needs to be installed in addition to the required kernel modules. For both Fibre Channel modes, a Fibre Channel that is zoned to the array is required. For `scsi/fc`, the `scsi_transport_fc` kernel module is needed, and volumes are always accessed through multipath, so `multipath-tools` must be installed and the multipath daemon must be running. For `nvme/fc`, the `nvme_fc` kernel module and the `nvme` CLI are needed, and multipath is handled by the NVMe subsystem rather than by `multipath-tools`. ## Terminology Each storage pool created in LXD using a Pure Storage driver represents a Pure Storage *pod*, which is an abstraction that groups multiple volumes under a specific name. One benefit of using Pure Storage pods is that they can be linked with multiple Pure Storage arrays to provide additional redundancy. LXD creates volumes within a pod that is identified by the storage pool name. When the first volume needs to be mapped to a specific LXD host, a corresponding Pure Storage host is created with the name of the LXD host and a suffix of the used protocol. For example, if the LXD host is `host01` and the mode is `nvme/tcp`, the resulting Pure Storage host would be `host01-nvme-tcp`. Likewise, in `scsi/fc` mode the resulting Pure Storage host would be `host01-scsi-fc`, because a slash (`/`) is not valid in a host name and is therefore replaced with a hyphen (`-`). The Pure Storage host is then connected with the required volumes, to allow attaching and accessing volumes from the LXD host. The created Pure Storage host is automatically removed once there are no volumes connected to it. ## The `pure` driver in LXD The `pure` driver in LXD uses Pure Storage volumes for custom storage volumes, instances, and snapshots. All created volumes are thin-provisioned block volumes. If required (for example, for containers and custom file system volumes), LXD formats the volume with a desired file system. LXD expects Pure Storage to be pre-configured with a specific service (e.g. iSCSI) on network interfaces whose address is provided during storage pool configuration. This does not apply in the Fibre Channel modes, because Fibre Channel target ports are not network interfaces. Furthermore, LXD assumes that it has full control over the Pure Storage pods it manages. Therefore, you should never maintain any volumes in Pure Storage pods that are not owned by LXD because LXD might disconnect or even delete them. This driver behaves differently than some of the other drivers in that it provides remote storage. As a result, and depending on the internal network, storage access might be a bit slower compared to local storage. On the other hand, using remote storage has significant advantages in a cluster setup: all cluster members have access to the same storage pools with the exact same contents, without the need to synchronize them. When creating a new storage pool using the `pure` driver in either `iscsi` or `nvme/tcp` mode, LXD automatically discovers the array’s qualified name and target address (portal). In the Fibre Channel modes, LXD instead discovers the online Fibre Channel target ports through the local host bus adapter, because Fibre Channel targets are addressed by rather than by network address. In `scsi/fc` mode, a target is the port’s itself, whereas in `nvme/fc` mode the target is the array’s subsystem reached at a Fibre Channel transport address that is derived from the port’s WWN. An array can present Fibre Channel ports for both modes at the same time, so LXD selects ports that correspond to the configured mode: for `scsi/fc` those reporting a WWN and no NQN, and for `nvme/fc` those reporting both. Consequently, [`pure.target`](#storage-pure-pool-conf:pure.target) has no effect in either Fibre Channel mode, and, instead, fabric zoning determines which targets are reachable. Upon successful discovery, LXD attaches all volumes that are connected to the Pure Storage host that is associated with a specific LXD server. Pure Storage hosts and volume connections are fully managed by LXD. Volume snapshots are also supported by Pure Storage. When a volume with at least one snapshot is copied, LXD sequentially creates snapshots on the destination volume from snapshots on the source volume. Each snapshot is associated with a parent volume and cannot be directly attached to the host; therefore, when a snapshot is exported, LXD creates a temporary volume behind the scenes. This volume is attached to the LXD host and removed once the operation is complete. Finally, once all snapshots are copied, the source volume is copied into the destination volume. ### Volume names Due to a [limitation](#storage-pure-limitations) in Pure Storage, volume names cannot exceed 63 characters. Therefore, the driver uses the volume’s [`volatile.uuid`](#storage-pure-volume-conf:volatile.uuid) to generate a shorter volume name. For example, a UUID `5a2504b0-6a6c-4849-8ee7-ddb0b674fd14` is first trimmed of any hyphens (`-`), resulting in the string `5a2504b06a6c48498ee7ddb0b674fd14`. To distinguish volume types and snapshots, special identifiers are prepended and appended to the volume names, as depicted in the table below: | Type | Identifier | Example | |-----------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Container | `c-` | `c-5a2504b06a6c48498ee7ddb0b674fd14` | | Virtual machine | `v-` | `v-5a2504b06a6c48498ee7ddb0b674fd14-b` (block volume) and `v-5a2504b06a6c48498ee7ddb0b674fd14` (file system volume) | | Image | `i-` | `i-5a2504b06a6c48498ee7ddb0b674fd14` (file system image) and `i-5a2504b06a6c48498ee7ddb0b674fd14-b` (block image) | | Custom volume | `u-` | `u-5a2504b06a6c48498ee7ddb0b674fd14` (file system volume), `u-5a2504b06a6c48498ee7ddb0b674fd14-b` (block volume) and `u-5a2504b06a6c48498ee7ddb0b674fd14-i` (ISO volume) | | Snapshot | `s` | `sc-5a2504b06a6c48498ee7ddb0b674fd14` (container snapshot), `sv-5a2504b06a6c48498ee7ddb0b674fd14-b` (VM snapshot) and `su-5a2504b06a6c48498ee7ddb0b674fd14` (custom volume snapshot) | ### Limitations The `pure` driver has the following limitations: Volume size constraints : Minimum volume size (quota) is `1MiB` and must be a multiple of `512B`. If the requested size does not meet these conditions, LXD automatically rounds it up to the nearest valid value. Snapshots cannot be mounted : Snapshots cannot be mounted directly to the host. Instead, a temporary volume must be created to access the snapshot’s contents. For internal operations, such as copying instances or exporting snapshots, LXD handles this automatically. Sharing the Pure Storage storage pool between multiple LXD installations : Sharing a Pure Storage array between multiple LXD installations is possible provided that installations use distinct storage pool names. Storage pools are implemented as Pods on the array and pod names have to be unique. Recovering Pure Storage storage pools : Recovery of Pure Storage storage pools using `lxd recover` is currently not supported. ## Configuration options The following configuration options are available for storage pools that use the `pure` driver, as well as storage volumes in these pools. ### Storage pool configuration `pure.api.token` API authorization token for Pure Storage gateway [](#storage-pure-pool-conf:pure.api.token) | **Key:** | `pure.api.token` | |-------------|--------------------| | **Type:** | string | API authorization token for Pure Storage gateway. Must have array_admin role to give LXD full control over managed storage pools (Pure Storage pods). `pure.gateway` Address of the Pure Storage gateway [](#storage-pure-pool-conf:pure.gateway) | **Key:** | `pure.gateway` | |-------------|------------------| | **Type:** | string | `pure.gateway.verify` Whether to verify the Pure Storage gateway’s certificate [](#storage-pure-pool-conf:pure.gateway.verify) | **Key:** | `pure.gateway.verify` | |--------------|-------------------------| | **Type:** | bool | | **Default:** | `true` | `pure.mode` How volumes are mapped to the local server [](#storage-pure-pool-conf:pure.mode) | **Key:** | `pure.mode` | |--------------|---------------| | **Type:** | string | | **Default:** | `nvme/tcp` | The mode to use to map Pure Storage volumes to the local server. Supported values are `iscsi`, `nvme/tcp`, `nvme/fc`, and `scsi/fc`. `pure.target` List of target addresses. [](#storage-pure-pool-conf:pure.target) | **Key:** | `pure.target` | |--------------|-----------------------| | **Type:** | string | | **Default:** | all available targets | A comma-separated list of target addresses. If empty, LXD discovers and connects to all available targets. Otherwise, it only connects to the specified addresses. This option has no effect in the Fibre Channel modes (`scsi/fc` and `nvme/fc`), because their targets are addressed by World Wide Name rather than by network address. `rsync.bwlimit` Upper limit on the socket I/O for `rsync` [](#storage-pure-pool-conf:rsync.bwlimit) | **Key:** | `rsync.bwlimit` | |--------------|-------------------| | **Type:** | string | | **Default:** | `0` (no limit) | | **Scope:** | global | When `rsync` must be used to transfer storage entities, this option specifies the upper limit to be placed on the socket I/O. `rsync.compression` Whether to use compression while migrating storage pools [](#storage-pure-pool-conf:rsync.compression) | **Key:** | `rsync.compression` | |--------------|-----------------------| | **Type:** | bool | | **Default:** | `true` | | **Scope:** | global | `size` Size of the storage pool [](#storage-pure-pool-conf:size) | **Key:** | `size` | |-------------|----------| | **Type:** | string | | **Scope:** | local | Size in bytes LXD sets as the quota of the Pure Storage pod. `source.recover` Whether to recover an existing `source` [](#storage-pure-pool-conf:source.recover) | **Key:** | `source.recover` | |--------------|--------------------| | **Type:** | bool | | **Default:** | `false` | | **Scope:** | local | Set this option to true to recover an existing source which was previously created by LXD. `user.*` User-provided free-form key/value pairs [](#storage-pure-pool-conf:user.*) | **Key:** | `user.*` | |-------------|------------| | **Type:** | string | | **Scope:** | global | `volume.size` Size/quota of the storage volume [](#storage-pure-pool-conf:volume.size) | **Key:** | `volume.size` | |--------------|-----------------| | **Type:** | string | | **Default:** | `10GiB` | Default Pure Storage volume size rounded to 512B. The minimum size is 1MiB. #### TIP In addition to these configurations, you can also set default values for the storage volume configurations. See storage-configure-vol-default. ### Storage volume configuration `block.filesystem` File system of the storage volume [](#storage-pure-volume-conf:block.filesystem) | **Key:** | `block.filesystem` | |----------------|---------------------------------------------------| | **Type:** | string | | **Default:** | same as `volume.block.filesystem` | | **Condition:** | block-based volume with content type `filesystem` | Valid options: `btrfs`, `ext4`, `xfs` If not set, `ext4` is assumed. `block.mount_options` Mount options for block-backed file system volumes [](#storage-pure-volume-conf:block.mount_options) | **Key:** | `block.mount_options` | |----------------|---------------------------------------------------| | **Type:** | string | | **Default:** | same as `volume.block.mount_options` | | **Condition:** | block-based volume with content type `filesystem` | `security.shared` Enable volume sharing [](#storage-pure-volume-conf:security.shared) | **Key:** | `security.shared` | |----------------|---------------------------------------------| | **Type:** | bool | | **Default:** | same as `volume.security.shared` or `false` | | **Condition:** | virtual-machine or custom block volume | | **Scope:** | global | Enable this option to allow the volume to be shared across multiple instances despite the possibility of data loss. `security.shifted` Enable ID shifting overlay [](#storage-pure-volume-conf:security.shifted) | **Key:** | `security.shifted` | |----------------|----------------------------------------------| | **Type:** | bool | | **Default:** | same as `volume.security.shifted` or `false` | | **Condition:** | custom volume | | **Scope:** | global | Enable this option to allow the volume to be attached to multiple isolated instances. `security.unmapped` Disable ID mapping for the volume [](#storage-pure-volume-conf:security.unmapped) | **Key:** | `security.unmapped` | |----------------|-----------------------------------------------| | **Type:** | bool | | **Default:** | same as `volume.security.unmapped` or `false` | | **Condition:** | custom volume | | **Scope:** | global | `size` Size/quota of the storage volume [](#storage-pure-volume-conf:size) | **Key:** | `size` | |--------------|-----------------------| | **Type:** | string | | **Default:** | same as `volume.size` | Default Pure Storage volume size rounded to 512B. The minimum size is 1MiB. `snapshots.expiry` Time until snapshots are deleted [](#storage-pure-volume-conf:snapshots.expiry) | **Key:** | `snapshots.expiry` | |----------------|-----------------------------------| | **Type:** | string | | **Default:** | same as `volume.snapshots.expiry` | | **Condition:** | custom volume | | **Scope:** | global | Specify an expression like `1M 2H 3d 4w 5m 6y`. `snapshots.pattern` Template for the snapshot name [](#storage-pure-volume-conf:snapshots.pattern) | **Key:** | `snapshots.pattern` | |----------------|------------------------------------------------| | **Type:** | string | | **Default:** | same as `volume.snapshots.pattern` or `snap%d` | | **Condition:** | custom volume | | **Scope:** | global | You can specify a naming template for scheduled snapshots and unnamed snapshots. The `snapshots.pattern` option takes a Pongo2 template string to format the snapshot name. To add a time stamp to the snapshot name, use the Pongo2 context variable `creation_date`. Make sure to format the date in your template string to avoid forbidden characters in the snapshot name. For example, set `snapshots.pattern` to `{{ creation_date|date:'2006-01-02_15-04-05' }}` to name the snapshots after their time of creation, down to the precision of a second. Another way to avoid name collisions is to use the placeholder `%d` in the pattern. If no matching snapshots exist, the placeholder is replaced with `0`. Otherwise, it is replaced with the next snapshot index, which is one higher than the highest existing matching snapshot index. `snapshots.schedule` Schedule for automatic volume snapshots [](#storage-pure-volume-conf:snapshots.schedule) | **Key:** | `snapshots.schedule` | |----------------|------------------------------| | **Type:** | string | | **Default:** | same as `snapshots.schedule` | | **Condition:** | custom volume | | **Scope:** | global | Specify either a cron expression (` `), a comma-separated list of schedule aliases (`@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@annually`, `@yearly`), or leave empty to disable automatic snapshots (the default). `user.*` User-provided free-form key/value pairs [](#storage-pure-volume-conf:user.*) | **Key:** | `user.*` | |-------------|------------| | **Type:** | string | | **Scope:** | global | `volatile.devlxd.owner` ID of the DevLXD identity that owns the volume [](#storage-pure-volume-conf:volatile.devlxd.owner) | **Key:** | `volatile.devlxd.owner` | |--------------|---------------------------| | **Type:** | string | | **Default:** | DevLXD owner identity ID | | **Scope:** | global | `volatile.idmap.last` JSON-serialized UID/GID map that has been applied to the volume [](#storage-pure-volume-conf:volatile.idmap.last) | **Key:** | `volatile.idmap.last` | |----------------|-------------------------| | **Type:** | string | | **Condition:** | filesystem | `volatile.idmap.next` JSON-serialized UID/GID map that has been applied to the volume [](#storage-pure-volume-conf:volatile.idmap.next) | **Key:** | `volatile.idmap.next` | |----------------|-------------------------| | **Type:** | string | | **Condition:** | filesystem | `volatile.uuid` Volume UUID [](#storage-pure-volume-conf:volatile.uuid) | **Key:** | `volatile.uuid` | |--------------|-------------------| | **Type:** | string | | **Default:** | random UUID | | **Scope:** | global |