M365 / Azure VM Backups Impacted

Posted: 09 April 2024, 07:00 UTC

Description: 

• M365 cloud based Software as a Service (SaaS) backups are experiencing delays in completing backup tasks.

• A decrease in the backup success rates of Azure VM backups has been detected. 

   

  Redstor Products Impacted:

• M365 backups

• Azure VM backups in the UK South Region

  Last Update:  2024-05-24 10:36

Further Detail:

An increase in memory usage in the M365 processing nodes is now understood to have started from 2024-03-27 onwards. These issues manifested as an increase in retry behaviour, resulting in increased load and backup delays.

The UK South Azure processing region is most significantly impacted, with some impact also observed in other processing regions.

A small percentage (<1%) of accounts may be experiencing delays in completing backups, with larger accounts more likely to be affected.

Redstor engineers are continuing to investigate whilst mitigation strategies are in place.

Update history

Update as of 2024-05-24 10:36 UTC

Microsoft have acknowledged to Redstor that changes to their Exchange services API directly impacted Redstor's M365 Exchange backups.

Update as of 2024-05-08 11:37 UTC

We are pleased to report that normal backup operations have been restored.

Should you experience any further difficulties with Microsoft Exchange backup services, please reach out to our technical support team where a member of the team would be happy to assist you.

Thank you for your patience and understanding throughout this process.

Update as of 2024-05-02 17:07 UTC

We have determined that this issue is attributed to a change Microsoft made to their M365 Exchange backup API’s.

Following adjustments made by Redstor to the backup process, we've observed significant improvements in backup operations.  However, a small number of backup accounts for specific tenants are still experiencing delays due to increased traffic to the Microsoft API, resulting in heightened throttling of requests.

For tenancies impacted by this throttling, we suggest temporarily adjusting the tenancy’s throttling limits.  See article 1540 on how to do this.

Update as of 2024-04-30 14:51 UTC

We want to reassure our customers that resolving this issue remains our top priority. Redstor has dedicated significant resources and effort to finding a solution. While we have made improvements and are moving closer to restoring our service to its previous levels of operation, we recognize that the problem is not yet fully resolved.
We sincerely apologize for any inconvenience this may have caused and greatly appreciate your patience and understanding as we work to resolve the issue.

Update as of 2024-04-24 14:05 UTC

The M365 backup issue has proven complex to investigate, however a new version of the processing logic has been deployed to production and is being monitored.

The nature of the batch processing means the update to the new version is dependent on existing processing jobs completing.

The Redstor team believe that they have seen a reduction in the backup delays on impacted accounts, and are continuing to monitor and evaluate of the the new version.

Update as of 2024-04-12 15:10 UTC:

The Azure VM Backups issue has been mitigated.

Backups for some Azure VMs in the Azure UK South processing region may have been unsuccessful during this time.

The Azure VM backup issue is believed to have been isolated to the workloads being processed in the Azure UK South region.

The Azure VM Backup failures affected the data transfer to the Redstor cloud only, Azure snapshots were unaffected.

Current Status: 

Ongoing, Redstor engineers are investigating and attempting to mitigate the issues with M365. Further detail can be found above.

The issues with Azure VM Backups are mitigated, with Redstor engineers continuing to monitor.