Salesforce.com Inc. has managed to recover from a database error that saw widespread outages for customers from midday Friday through Saturday morning.

The outage had its origin in an update just before 1 p.m. EDT Friday when Salesforce engineers deployed a database script that “inadvertently gave users broader data access than intended,” the company said.

That broader access saw some users not only obtaining read access to data they should have been restricted from but also write permissions, making it easy for malicious employees to steal or tamper with a company’s data.

“It has been confirmed on mutliple orgs, spanning multiple shards… that Salesforce has bugged out this morning and has given MODIFY ALL (so all permissions) to literally EVERY SINGLE PROFILE in some orgs. This includes Standard profiles and Custom as well. Yes, EVEN STANDARD PROFILES,” the thread on Reddit read.

Once the error was discovered, Salesforce was then forced to pull the update for security reasons, bringing down access for many of its users.

‘The Salesforce Technology team blocked access to certain instances that contain customers affected by a database script deployment that inadvertently gave users broader data access than intended,” Salesforce said at the time. “To protect our customers, we blocked access to all instances that contain affected customers until we could block access to orgs with the inadvertent permissions.”

“As a result, customers who were not affected may [have] also experienced service disruption,” Salesforce added.

Much of the attention to the outage came Friday, but the outage dragged on into Saturday, with the official “service disruption ended” message only being issued at 4:04 a.m. EDT. Salesforce co-founder and Chief Technology Officer Parker Harris took to Twitter to apologize during the outage.

To all of our @salesforce customers, please be aware that we are experiencing a major issue with our service and apologize for the impact it is having on you. Please know that we have all hands on this issue and are resolving as quickly as possible.

— Parker Harris (@parkerharris) May 17, 2019

Others took a more lighthearted approach to the downtime:

Everyone without #salesforce right now. Go explore the outside. Take a deep breath and look at real clouds. It’s ok. It’ll come back up. #SalesforceDown pic.twitter.com/RjRPbrdWFp

— Collin A. Zimmerman (@collinz) May 17, 2019

The outage is being seen by some to be a lesson. Balaji Parimi, chief executive officer of CloudKnox Security Inc., told SiliconANGLE that enterprises need to understand that their biggest security risk is not from attackers or even malicious insiders, instead overprovisioned privileges.

“Security teams need to make sure that privileges with massive powers are restricted to a small number of properly trained personnel,” Parimi said. “Until companies better understand which identities have the privileges that can lead to these types of accidents and proactively manage those privileges to minimize their risk exposure, they’ll be vulnerable to devastating incidents like the one with Salesforce.”

Photo: Medullaoblongata Projekt/Wikimedia Commons