Veeam v12 introduced Direct-to-Object storage, enabling S3 object storage as the primary backup repository. Prior to this, S3 object storage integration relied on Veeam’s Scale-Out Backup Repository (SOBR), using a performance tier and a capacity tier, which extended to S3 object storage.
Direct-to-Object storage works with S3 object storage both on-premises and in the cloud. This blog outlines a straightforward, step-by-step process for setting up S3 object storage in both environments and utilizing them as primary Veeam backup repositories with the Direct-to-Object storage feature.
How to Set Up S3 Object Storage for Veeam Backups
Setting up S3 object storage for Veeam backup and archiving is a versatile process, offering three distinct approaches:
- AWS S3 Storage or S3 Object Storage in Compatible Clouds: Utilize AWS S3 storage or leverage S3 object storage in any other S3-compatible cloud service.
- Repurposing Unused Storage Capacity in Virtualized Environments: Repurpose unused storage capacity as S3 object storage within virtualized environments such as VMware, Hyper-V, KVM, and Citrix (XenServer).
- Purpose-Built S3 Object Storage Appliances: Purpose-built S3 object storage appliances tailored and tested for Veeam compatibility.
StoneFly offers flexible solutions for all three deployment methods. Each solution is equipped with essential features, including built-in data security, ransomware protection, storage optimization, and comprehensive management capabilities.
In this blog, we delve into the setup procedures for each of these deployment options. For the third option involving on-premises appliances, system administrators benefit from a streamlined process, as the S3 object storage setup is pre-configured, requiring only network adjustments for seamless integration.
Note: The setup process described below uses StoneFly’s 8th gen storage virtualization engine (SCVM™), included in all StoneFly solutions and available as a standalone agent. For those interested in virtual or cloud S3 object storage, visit https://stonefly.com/subscription for a free trial or to purchase a monthly, yearly, or custom subscription(s).
How to Map AWS S3 Object Storage to On-Premises Appliance for Veeam Backup and Archiving
In this section, we outline the steps for seamlessly integrating Amazon AWS S3 object storage with an on-premises appliance, specifically tailored for Veeam backup and data archiving purposes.
To proceed with the integration, you will need the following prerequisites:
- An active Amazon AWS cloud subscription.
- The Veeam data platform installed.
- StoneFly SCVM (Storage Concentrator Virtual Machine) configured on your system.
Login to the StoneFly SCVM GUI. Click on “Resources Summary”, and then click “Create Object Storage”.
In the “Object Storage Service Type” drop down menu, select “Amazon AWS S3 Storage”.
Fill out the information about Amazon AWS S3 object storage including the “S3 Account Key ID”, and “S3 Account Secret Key”, “S3 Storage Class”, name the bucket, and click “Submit”.
Upon completion of the above process, a new resource will be generated and displayed in the “Resource Summary” section. This new resource operates similarly to any other block device, enabling its utilization for the creation of S3 object storage, file storage, and iSCSI volumes for Veeam backups and databases respectively.
Note: It is advisable to use Amazon AWS S3 for infrequently-accessed backups and archiving, as the cloud may prove cost-prohibitive for regularly accessed data.
How to Map S3-Compatible Object Storage to On-Premises Appliance for Veeam Backup and Archiving
Setting up S3-compatible object storage for Veeam backup and archiving is similar to the abovementioned process outlined for Amazon AWS S3 object storage.
To initiate the integration seamlessly, ensure you have the following prerequisites in place:
- An active S3-compatible object storage, either in the cloud or as an on-premises S3 object storage.
- The Veeam data platform installed.
- StoneFly SCVM (Storage Concentrator Virtual Machine) configured on your system.
Login to the StoneFly SCVM GUI. Click on “Resources Summary”, and then click “Create Object Storage”.
In the “Object Storage Service Type” drop down menu, select “Compatible S3 Storage”.
Fill out the information about S3 object storage including the “S3 Base URL”, “S3 Account Key ID”, and “S3 Account Secret Key”, “S3 Storage Class”, name the bucket, and click “Submit”.
Upon completion of the above process, a new resource will be generated and displayed in the “Resource Summary” section. This new resource operates similarly to any other block device, enabling its utilization for the creation of S3 object storage, file storage, and iSCSI volumes for Veeam backups and databases respectively.
How to Set Up S3 Object Storage for Veeam Backup and Archiving
In this section, we describe the steps to set up S3 object storage using the resources configured in accordance with the previously described process.
To initiate the S3 object storage setup process, confirm that you have the following prerequisites in order:
- Resource(s) configured as detailed in the above process.
- The Veeam data platform installed.
- StoneFly SCVM (Storage Concentrator Virtual Machine) configured on your system.
Log in to the SCVM console.
Navigate to Resources > Summary > and select the “Managed” radio button for your resource, then click “Submit”.
The resources provisioned according to the above process are utilized for creating segments, which, in turn, serve as the foundation for provisioning S3 object storage. To initiate the segment creation, navigate to NAS > Segments and click on “Segment Create”.
Input the desired “Segment Name”, choose “Manual Create”, allocate the segment size in GiB under the “Amount to Add (GiB)” box, and then click “Submit”.
This will trigger a pop-up window with the message “The requested NAS segment will be formatted.” To proceed, click “Ok.”
To create the S3 object storage volume, follow these steps to utilize the newly created segments for NAS volumes, then edit them to S3 object storage.
To allocate the “NAS Volume”, go to NAS > Volume > Create New Volume > Allocate. Enter the “NAS Volume Name” and click “Submit”.
Now, let’s move on to creating the S3 object storage volume.
- Navigate to NAS > S3 Storage > Create/modify S3 EP.
- Select the specific Volume.
- Provide credentials and enter values for the Access Key and Secret Key fields.
- Click “Submit” to complete the process.
For the Account Key and Secret Key, you have the option to either input them manually or generate new keys by clicking on the “Generate Keys” button.
- To locate the URLs for the S3 storage endpoint, navigate to the Summary page.
- Access the URL by clicking on it.
To log in, use the credentials provided in the Access Key and Secret Key fields during the S3 Endpoint creation process.
After successfully logging in, proceed to create a new S3 Bucket by clicking on “Create Bucket.”
Input the desired Bucket Name and proceed to create the bucket by clicking on “Create Bucket.”
Toggle the radio buttons for Versioning, Object Locking, and Quota to enable these features for the bucket.
Note: This functionality is available only for configurations with 4 or more drives/volumes.
How to Configure S3 Object Storage as a Veeam Backup Repository
In this section, we walk you through the process of how to configure the S3 object storage, provisioned as per one of the processes described above, using the Veeam backup software.
- Navigate to the Backup Infrastructure view.
- In the inventory pane, right-click the Backup Repositories node, then choose “Add Backup Repository.” Alternatively, you can click “Add Repository” on the ribbon.
Within the “Add Backup Repository” window, opt for “Object storage > S3 Compatible > S3 Compatible with Data Archiving.”
In the ‘Name’ step of the wizard, enter the desired name and description in their corresponding fields.
At the Account step of the wizard, customize the connection settings as follows:
- In the Service point field, provide the endpoint address and port number for your S3-compatible object storage.
- In the Region field, specify the desired region.
- Input the credentials necessary for accessing the S3-compatible object storage, as configured in the setup process explained earlier.
Adjacent to the Connection mode field, select the appropriate option to determine how Veeam Backup & Replication will transfer data to the object storage repository:
- Direct: Opt for this option to instantly move data from processed VMs or file shares to object storage repositories. The method used depends on the job type, utilizing either a backup proxy or a gateway server.
- Through gateway server: Choose this option if you prefer Veeam Backup & Replication to utilize a gateway server for transferring data from processed VMs or file shares to object storage repositories. From the Name list, designate the gateway servers for data transfer operations.
Note: By default, if Veeam Agent stores data in S3-compatible object storage repositories, it transfers data using a gateway server. If you wish for Veeam Agent to access repositories directly or with specific credentials, specify the Access Permissions settings.
At the Bucket step of the wizard, configure the following settings to determine where and how data will be stored, along with defining storage limits and immutability settings applied by Veeam Backup & Replication to object storage data.
- Bucket Field: Enter the name of the bucket or click “Browse” to access the required bucket.
- Folder Field: Enter a folder name for mapping your object storage repository or click “Browse” to either select an existing folder or create a new one.
- Limit Object Storage Consumption: Check the box to limit object storage consumption, allowing you to set a soft limit. If this limit is surpassed during a job run, Veeam Backup & Replication will permit job completion. However, a new job won’t initiate until you address the excess data by either removing it or adjusting the soft limit settings. Specify the value in TB or PB.
Make Recent Backups Immutable: Check the box to prevent the deletion of data blocks from object storage. Specify the immutability period, noting that the UI allows a maximum immutability period of 90 days. For longer periods, utilize the Set-VBRAmazonS3CompatibleRepository cmdlet.
At the Mount Server step of the wizard, define settings for the mount server intended for restore operations and set up a helper appliance. This temporary host, deployed by Veeam Backup & Replication on the S3-compatible storage, performs health checks on backup files and applies retention to unstructured data backup files. Once these operations are completed, Veeam Backup & Replication automatically removes the helper appliance from the S3-compatible storage.
To configure the mount server settings, follow these steps:
- Mount Server Selection: From the Mount Server drop-down list, choose a server designated as the mount server. This server is utilized during restore operations to directly mount VM disks from objects within object storage repositories.
Note: The Mount Server list displays only Microsoft Windows servers added to the backup infrastructure. If the server is not yet added, click “Add New” on the right to open the New Windows Server wizard.
- Instant Recovery Write Cache Folder: Specify a folder for the Instant Recovery write cache, where cache data is stored during mount operations.
- Enable vPower NFS Service:
- Check the box to enable the Veeam vPower NFS Service on the mount server, granting access to the object storage repository. Veeam Backup & Replication will automatically activate the Veeam vPower NFS Service on the designated mount server.
- Click “Ports” to customize network ports used by the Veeam vPower NFS Service. In the vPower NFS Port Settings window, specify the mount port and vPower NFS port settings.
- Helper Appliance Configuration: To specify helper appliance settings, click “Configure.” From the Managed server drop-down list, choose a server designated as the helper appliance.
IMPORTANT: Avoid enabling Microsoft Windows NFS services on the machine where you install the Veeam vPower NFS Service. Enabling both Microsoft NFS services and Veeam vPower NFS Service on the same machine may lead to issues with both services functioning correctly.
At the Review step of the wizard, carefully examine the components slated for processing on the mount server and their current status.
If your backup repository already holds backups, consider selecting the “Search the repository for existing backups and import them automatically” checkbox. Enabling this option prompts Veeam Backup & Replication to conduct a thorough scan of the backup repository, identifying and automatically displaying existing backup files in the Veeam Backup & Replication console under the “Backups > Object Storage (Imported)” node.
For repositories containing guest file system index files, you have the option to choose the “Import guest file system index data to the catalog” checkbox. Enabling this feature allows Veeam Backup & Replication to import index files alongside backup files. Consequently, you gain the capability to search for specific guest OS files within the imported backups, enhancing overall accessibility and search functionalities.
At the Apply step of the wizard, wait as Veeam Backup & Replication finalizes the process of saving configured settings to the database and creating the necessary backup infrastructure objects.
Upon completion, you will see the following message indicating that the S3 object storage has been added as a Veeam backup repository.
Why Use S3 Object Storage for Veeam Backup and Archiving
- Cost-Effective Scalability: S3 object storage is designed to scale seamlessly, accommodating the growth of backup data without significant increases in cost. This scalability ensures that organizations can efficiently manage large volumes of data without incurring excessive expenses.
- Durability and Reliability: S3 object storage, such as Amazon S3, is known for its high durability and reliability. Data is redundantly stored across multiple locations, reducing the risk of data loss due to hardware failures or other issues.
- Accessibility and Availability: S3 object storage provides easy and quick access to backup data. This accessibility is crucial for timely restores and maintaining operational continuity in case of data loss or system failures.
- Security Features: S3 object storage platforms come with robust security features, including air-gapped and immutable backups, data encryption at rest and in transit, volume deletion protection, MFA, and more. This ensures that backup data remains secure, meeting compliance requirements and protecting against unauthorized access.
- Long-Term Archiving: S3 object storage is well-suited for long-term data retention and archiving. Its cost-effective structure allows organizations to store historical data for compliance or business continuity purposes.
- Redundancy and Data Resilience: Storing backups in S3 object storage provides redundancy, enhancing data resilience. In the event of a failure in the primary storage, backup copies stored in S3 remain unaffected.
- Flexible Storage Tiers: S3 object storage often offers different storage classes or tiers, allowing organizations to choose the most cost-effective option based on their data access patterns. This flexibility helps optimize storage costs.
- Versioning and Lifecycle Management: S3 object storage typically supports versioning and lifecycle management features. This allows organizations to maintain different versions of backup data and automate the transition of data to lower-cost storage tiers as it ages.
- Global Accessibility: S3 object storage services are accessible globally, making it suitable for organizations with distributed teams or offices. Data can be stored centrally and accessed securely from various locations.
Summary
In this blog, we’ve explained the step-by-step process of how to:
- Map AWS S3 object storage to on-premises appliances for Veeam backup and archiving.
- Map S3-compatible storage to on-premises for Veeam backup and archiving.
- Configure S3 object storage using the above mapped resources
- Set up S3 object storage as a backup repository for Veeam using Veeam setup wizard.
Adopting S3 object storage for backups provides a scalable, cost-effective, ransomware-proof, and secure solution. With seamless integration, global accessibility, and robust features, it stands as a reliable foundation for data protection and archiving needs.