As many of you already know, SocialFlow was one of several companies affected by a phishing attack yesterday by hackers calling themselves the Syrian Electronic Army.
At no time were our customer accounts or data compromised. However, the SocialFlow Twitter handle, Facebook page, and website were, and several false posts were made from our social accounts.
I’ll give you the details of what happened at the end of this post but the real purpose here is to give you visibility into what happened behind the scenes: how we discovered the problem, what we did to contain it, and what we learned along the way. These types of attacks are unfortunately a part of today’s landscape, and we believe that discussing them is an important part of helping the social media ecosystem mature.
How we discovered the problem
A number of employees received a phishing email, and that put us on alert. “Beware of the phishing email” word went around the office, and our security was put on alert. Unfortunately, an employee working outside the office clicked on the link and entered an email address and password. That person had publishing access to our Twitter account, Facebook account, and website.
What we did to contain the problem
We immediately implemented our security protocols, and in short order regained control of our Twitter and Facebook pages. Since our blog posting engine and associated RSS feeds, are tied to our website, we temporarily took it down to make sure that route didn’t allow the attack to spread further.
While we diagnosed the scope and extent of the problem we also suspended our clients’ ability to publish using our platform, and called and let them know what we’d done and why.
When you’re hacked, communication, quick action, and rumor control become important. We made real-time adjustments based on what we were seeing, double- and triple-checked to make sure we had accurately diagnosed the extent of the problem, and then made the necessary changes to shut off the attack and remove the offending posts.
Several posts on social media asked whether we moved quickly enough in notifying our customers, which is a fair point. I can only tell you that our first priority was to stop the attack and make sure our systems were protected. We then worked quickly to make sure we had a clear and accurate story to deliver to customers. Once we reached that point, our account team kicked into action immediately, and made phone calls as quickly as they could. (For obvious reasons we did not do an email blast.)
This made for a long day, and of course we’re never happy when we inconvenience our customers. But it could have been far worse, if not for the quick response of our technical teams.
What we learned and what we will do better in the future
A significant issue that confronts all Internet users these days is an inherent tension between security and convenience. One such tradeoff is captured in the concept of “two-factor authentication”, which Google does a great job of explaining here. It’s a great security measure, but it is less convenient for users.
Here is where I must confess to being a bit red-faced. Despite our efforts to get all of our employees to use two-factor authentication for our corporate email system, we had some employees for whom that had not yet been turned on. When intruders are probing for vulnerabilities, it only takes one breached account to create a compromise… And needless to say, we now have 100% employee compliance on this.
One other thing we learned was probably the most minor, but the most visible: the difficulty in repairing what was a quick and inelegant take-down of our website. At first our site just presented a “404 Not Found” error, but then we quickly changed it to an innocuous (but still non-functional) logo page.
This was necessary because our blog was tied to the website, and there was no graceful way to stop publishing without taking the whole site down. Even now, the day after the takedown of the site, our team is working to restore everything to its previous state. Clearly that’s an area we will address moving forward.
Were customers affected?
At no time were our customer accounts or data compromised.
We had one customer who was caught in a related phishing attack by the Syrian Electronic Army, but that was actually a compromise of an individual customer’s corporate email address and did not come through SocialFlow. The hackers essentially used the same phishing attack (an email with a link) to take over that customer’s email account. They then submitted a “lost password” request to SocialFlow, asking us to reset the password so that the (compromised) email address could get in with a new password.
This led to a spirited discussion among our product and technology teams about how to guard against this in the future (even though in this case we’re actually guarding against the compromise of another company’s email system). I won’t disclose here what exact approach we settled on, but suffice it to say that it’s always a balancing act between security and convenience.
One bit of levity
We admire those who have the ability to poke fun during difficult times, and this situation inspired its own comments. This tweet from @tomgara made several people chuckle at the end of a long day.
SocialFlow outage is amazing. I saw @fieldproducer crafting a tweet out of a block of wood with a traditional curved knife. Old school.
— Tom Gara (@tomgara) August 13, 2013
Thanks, Tom… we agree that doing your job without being able to use SocialFlow is seriously old school!
What did the attack entail?
As promised at the beginning of this post, here are the details of how the phishing attack occurred. As is typical, it involved an employee who didn’t notice an odd sequence of events (“why did that login screen show up now?”) and some ingenuity from the phishers:
- An email from a seemingly known sender arrived in the inboxes of multiple employees with a title of “Article” and a link that appeared to go to a popular news site. It was clever, because the subdirectory even went to “/2013/08/13/tech/social/index.html”. That directory structure looks very clearly like an article that we would share among our staff.
- Of course the known sender didn’t really send the email—the reply address was forged, which is a pretty easy trick.
- When the link was clicked, it didn’t go to the news site but instead popped up what appeared to be an email authentication dialog box. It was very cleverly constructed to look like the login screen of our email provider, and it’s easy to see how an employee who was slightly distracted could have concluded that email needed to be re-authenticated.
- Unfortunately this particular employee had access to our SocialFlow Twitter and Facebook accounts, as well as our blog site.
- The phishers, with access to those properties, started posting items and changing passwords to lock us out.
- Our operations team shut the website down, and worked with Twitter and Facebook to regain control of our pages.
It’s almost impossible to give an exact timeline of events, as things were moving quickly and we focused more on resolution than on clocking various milestones. But from what I witnessed and subsequent conversations with the team, I would estimate that we had the attack neutralized (e.g., the attackers could do no further damage) within 15-20 minutes; and our Twitter and Facebook pages restored within 45 minutes to an hour. The whole experience, start-to-finish, was perhaps 2-3 hours. The cleanup, restart of the website, and communications both inside the company and with our customers continued for several hours after that. And of course we’ll do post-mortems on the whole experience for some time to come.
A circumstance such as this shows how any complex system can be rendered vulnerable by the compromise of just one person’s account. Social media tools and processes are going to need to mature and “harden” as social media activity becomes a more central part of what publishers and brands do on a daily basis. We hope that by sharing this information we’re making a contribution in that regard.