Rackspace Hosted Exchange Event
Updated: 01-12-2023 1:40 PM EST
Dear Franchisees,
Early Friday December 2, 2022, Rackspace experienced a global disruption that took down their hosted Exchange service for over 500,000 clients. This effectively prevented users from accessing mail on their hosted Exchange platform, (so if you are still actively receiving emails to your inbox you are likely not affected by this event).
Since being notified of the issue, United Franchise Group’s tech support team has been actively working with Rackspace to assess the situation and find a solution while maintaining the integrity of the email platform. We realize this is a busy time of year and we want to get you the information you need to continue your business with minimal impacts from this issue.
A message from Rackspace CEO: Click Here
Status update: Rackspace Status Page
For the most up to date information, please stay tuned to this page.
01-12-2023
12-21-2022
Part 1
- After some testing we have determined that EMEBoss is not compatiable with sending emails via Office 365, this is a limitation of BOSS, not 365. This means there is NO way to have Boss send invoices via the users login details from 365. We have devised a way to apply a work around, please contact Fully Promoted Support for more details.
Part 2
- Rackspace has started deploying backup restoration to their users.
- Data will be delivered to users in the form of a PST file.
- PSTs will be available for 30 days!
- They have created a webpage specifically geared towards the data recovery: https://www.rackspace.com/hosted-exchange-incident-data-recovery-resources
- A how-to is available here: https://www.rackspace.com//sites/default/files/2022-12/Data-Recovery-FAQs-1.pdf
- Not all user’s data will be available right away, as they are rolling out the PSTs systematically, and will take a few days to deploy to the millions of email boxes affected.
- Restoration looks like it will require a copy of Outlook to facilitate. We will reach out to Rackspace on how to import via Webmail
- This specifically talks about users that were on Hosted Exchange, We need to reach out to RS to see what needs to occur for the users we had in a hybrid environment.
12-19-2022
Rackspace’s notes from yesterday:
- Rackspace has spun up a new “Clean” environment for the Hosted Exchange platform.
- This environment has been secured, validated, and isolated from Rackspace’s main environments.
- They have completed the server recovery effort, meaning all malicious data has been removed from the data.
- As they prepare to deploy each of the new servers, each is being checked by a cybersecurity firm to make sure each one is safe.
- After each server is released into the new clean environment, data from each new server is going through an automation process that opens the user data, reviews content, and re-maps it to the correct customer account.
- This data is then moved to a staging environment.
- Code and data is then re-validated and optimized as being safe.
- Rackspace is now testing restoration of customer data.
- Some data may be corrupted, but they have a number of secure backups that they can use to rebuild corrupted data.
- PST files will be rebuilt and deployed to the portal once the data is cleared.
- Rackspace will notify us/customers as their data comes online, this data can be used to migrate over to 365.
12-16-2022
- If you had to manually migrate to Rackspace on your own (e.g you have a direct account with Racspace for Office 365). You may want to check your Quarantine settings to make sure your junk isn’t being held in the remote quarantine area; details below.
12-15-2022
- We ran an update for FPD US/AU to able to send correctly from Boss.
12-13-2022
- All quarantine queues have been released for (SAR AU, FPD AU, FPD US)
- Filtering has been updated to deliver more spam to the junk folder.
- Risk assesment of certain users has been completed; brands notified with details.
- Rackspace is still working through the attack, not much information has been released as of late.
12-08-2022
- Added details on how to enable forwarding for self hosted accounts, see “How to enable forwarding” in the General Fixes section below.
12-08-2022
- Rackspace has advised the general public that this was a Ransomware attack. (see status page for details)
- New help video has been added, we advise everyone to watch it: https://www.rackspace.com/resources/hosted-exchange-assistance-video
- As a reminder this migration will be a two step process.
- #1 – Create new accounts and migrate users to Office 365
- #2 – Data migration
- How to backup your Exchange OST file in Outlook
- How to export your Exchange OST file to a PST archive
Brand Specific Fixes
General Fixes
These apply to all users…
How to Enable Forwarding
If users are not able to forward mail out of their tenant (ones they control) they need to enable forwarding… Note, if you have an email on signarama.com.au, fullypromoted.com.au, or fullypromoted.com – we took care of this today for you.
- Log into https://admin.microsoft.com with an admin user
- Go to: Security -> Policies and Rules -> Threat Policies -> Anti Spam -> Anti-spam outbound policy (Default)
- Click “Edit protection settings”
- Look for the drop down under “Automatic forwarding rules”
- By default this is set to: “Automatic – system controlled”
- Change the dropdown to: ON – Forwarding is enabled
- Click Save
- Click Close
This policy can take up to 30 mins to sync through the Microsoft servers.
End result is you will be able to forward mail out to domains that are not part of the tenant (e.g. Gmail, Yahoo, etc)
How to Check Quarantine Settings
This only applies to you if you are “self-hosted” and have a direct account with Rackspace, and you had to migrate away from Hosted Exchange.
If you are on fullypromoted.com, fullypromoted.com.au, or signarama.com.au we have already taken care of this for you.
- Log in to https://admin.microsoft.com with your Admin 365 user.
- Click “Security” on the left side bar navigation.
- Click “Policies & Rules” on the new tab that opens (again left side bar nav).
- Click “Threat Policies”
- Click “Anti-Spam”
- Click “Anti-spam inbound policy (default)”
- Scroll down the panel that opens until you reach the “Actions” section.
- You will see a few options; we are looking for:
- Spam message action
- Hopefully it shows: Move message to Junk Email folder
- High confidence spam message action
- Hopefully it shows: Move message to Junk Email folder
- Phishing message action
- Hopefully it shows: Move message to Junk Email folder
- High confidence phishing message action
- Hopefully it shows: Move message to Junk Email folder
- Bulk message action
- Hopefully it shows: Move message to Junk Email folder
- Spam message action
- If the options instead say: “Quarantine Message”, we will need to change it
- Scroll down to a link that says: “Edit Actions”
- Then select each drop down and pick: “Move message to Junk Email folder”
- The only one you may want to leave as Quarantine is – High confidence phishing message action
- However they will need to manually check the quarantine every so often
- Which can be done here: https://security.microsoft.com/quarantine
- Note this is only visible to a 365 Admin user.
If you do not feel comfortable making these changes, please submit a ticket to Rackspace and have them update your quarantine settings.
Portal Setup Video
FIX – Pop Up Issue in Outlook
If your Outlook keeps throwing a login screen pop up, and won’t allow you to log in to the new email account on 365. Please make sure Outlook is set to “Work Offline”.
Click “Send/Receive” in the nav bar, select “Work Offline”.
Enable MFA on your new accounts
When you are finally able to get into your new accounts, we suggest you enable MFA via this link: https://aka.ms/MFASetup
- Log in to https://aka.ms/MFASetup
- You should be on “Security Info” tab on the left
- Click “Add sign-in method” link
- Select Phone for SMS
- Authenicator App if you use Google, LastPass, 1Password
- Follow prompts
Reach out to support if you need help.
How to change your password
Please take a moment to update you password to something strong.
- Log in to: https://myaccount.microsoft.com
- Click “Password” on the left hand sidebar
- Enter your old and new password
- Click Submit
Your password has now been changed. Reach out to support if you need help.
Phishing Attacks
Please be aware that due to the Rackspace outage, a number of nefarious entries are trying to spoof Rackspace and attempting to gain access to your accounts. Below is an example of a phishing attack received today by a franchisee. Do not click these, or reply to the user. If you have questions on legitimacy, please ask us first.
