Lessons from Slack’s GitHub Theft, CircleCI Breach Warning

Vishal Pallerla•January 6, 2023

Circle CI

On Wednesday, CI & CD platform CircleCI used by over 1 million developers alerted their customers with urgent attention about a security incident. CircleCI urged its customers to take preventative measures to their systems to protect their data such as:

• rotating any and all secrets stored in CircleCI such as project environment variables or in contexts
• reviewing organization’s own internal logs to check if there was any unauthorized access starting from December 21, 2022 through January 4, 2023, or upon completion of the rotation of their secrets
• despite not divulging any further details about the incident, CircleCI said it has also invalidated all Project API tokens due to which customers have to replace them for projects using API tokens.

CircleCI also said they have expanded access to self-serve audit logs to all customers, including free customers to access self-serve audit logs to query up to 30 days of data and have 30 days to download the resulting logs and recommended customers to focus audits and investigations on the logs of any systems which had secrets stored in CircleCI.

What are Secrets anyway?

Secrets are private digital credentials such as passwords, encryption keys, SSH keys, API keys, etc, used for managing access permissions to protected resources including vaults or private repositories. Any user (human or non-human) that gains access to secrets can get access and permissions to the resources belonging to the owner of the secret which is why it’s so important to keep secrets secure both in transit and at rest.

In this case of CircleCI breach, the potential exposure of all these secrets is of major concern for all customers using their platform especially as there is a possibility that customers' information was exposed for up to two weeks following the breach.

Can customers take any steps to have low/reduced impact in these types of breaches?‍

Yes, that’s where Secrets Management comes in.

Secrets management is the process of securely storing, accessing, and rotating secrets. Secrets rotation refers to the practice of regularly changing these secrets to reduce the risk of unauthorized access.

In the CircleCI breach scenario, having automated secrets rotation could have helped reduce the effects of the breach as attackers who gained access to secrets would only be able to use it until it was rotated. After the secret is rotated, the attacker will no longer have access to the system.

There are many different tools and techniques that can be used to manage secrets and rotate them on a regular basis. Some common approaches include using a secrets management service, storing secrets in a password manager, and implementing a secrets rotation mechanism if that is not previously in place or using automated tools to rotate secrets on a set schedule.

Preventative actions could also include limiting the scope of access for those secrets to the minimum possible data, resources and applications needed to complete a required task following the principle of least privilege (so that when they leak to attackers, the attackers only get minimum access).

What should impacted CircleCI customers do?

If you are a CircleCI customer, we created a checklist with action items based on recommendations from CircleCI team:

• Create a list of items for rotation using this tool for discovering all your secrets on CircleCI.
• Rotate any and all secrets stored in CircleCI. There are multiple ways to do this, and we encourage you and your teams to use your preferred methods. Here is an approach you may follow:

OAuth tokens:

GitHub:

• Go to https://github.com/settings/applications
• Select “Authorized OAuth Apps”
• Revoke the CircleCI entry.
• Once that’s done, log out and back into CircleCI to trigger reauthorization.

BitBucket:

• Go to https://bitbucket.org/account/settings/app-authorizations/

GitLab: Rotate environment variables, Personal and Project API tokens, and all SSH keys.

Project API tokens:

• Go to Project Settings > API Permissions > Add API Token.

Project environment variables:

• Go to Project Settings > Environment Variables and then create an environment variable with the same name to replace the existing value.

Context variables:

• Go to Organization Settings > Contexts and do the same thing as for project environment variables for each context.
  

User API tokens:

• Go to User Settings > Personal API Tokens and then delete and recreate any tokens you might be using.
  

‍ Project SSH keys:

• Go to Project Settings > SSH Keys.
  
• Delete the Deploy Key and add it again. If you were using any additional keys, then those need to be deleted and recreated.
• Runner Tokens: using the CircleCI CLI, run the following commands:
• circleci runner token list <resource-class name>
• circleci runner token delete <token identifier>
• circleci runner token create <resource-class-name> "<nickname>"
• Following these commands, you will need to add the created token to your launch-agent-config.yml and restart your runner service
• Note: We recommend this for all projects (go to “Project Settings”), orgs (go to “Organization Settings”), and users (go to “User Settings”).
• Review all internal logs between December 21 and January 4
• Check their FAQ page for more info:
  https://discuss.circleci.com/t/faq-compilation-security-alert-rotate-any-secrets-stored-in-circleci/46559
• See what others are doing:
• https://discuss.circleci.com/t/circleci-security-alert-rotate-any-secrets-stored-in-circleci/46479

Slack

Slack Employees were kind of Slack in enforcing Security rules

On the last day of the year 2022, real-time communication and collaboration platform Slack, with over 18 million active users shared details of a recent security incident on their blog which stated that a limited number of Slack employee tokens were stolen and misused to gain access to their externally hosted GitHub repository. 

After gaining access, threat actors downloaded private code repositories. Slack’s investigation states no downloaded repos contain customer data or any means to access customer data and further mentioned “the threat actor did not access other areas of Slack’s environment, including the production environment, and they did not access other Slack resources or customer data.”

This security incident came into light on January 4 even though Slack disclosed the incident on December 31 in their blog. Ax Sharma at BleepingComputer shares some insights on why the security update might have been hidden from search engines. It is also crucial to note that Slack discovered the suspicious activity on Dec 29 whereas the threat actor gained access on December 27. 

Impact to customers

Slack’s investigation team stated "Based on currently available information, the unauthorized access did not result from a vulnerability inherent to Slack. We will continue to investigate and monitor for further exposure,” There was no impact to the code or Slack’s services, they also rotated all relevant credentials as a precaution.

There was no customer impact, and no action needs to be taken by customers.

Takeaway

The specific details of how Slack's employee tokens were stolen and misused are currently unknown. However, the theft and misuse of Slack employee tokens highlight the need for strong authentication measures to protect against unauthorized access.

Why and how can these token thefts happen?

The risk of token theft increases as users access corporate resources from personally owned or unmanaged devices especially in this hybrid working world. In contrast to devices that are managed by organizations, unmanaged devices have weaker security controls. It can be challenging to detect token theft without proper safeguards and visibility into authentication endpoints.

Microsoft shares some good insights on Token tactics relating to preventing, detecting and responding to token thefts. According to them, attackers may gain access to tokens using common credential phishing attacks, Adversary-in-the-middle (AitM) phishing attacks and Pass-the-cookie attack. Check out their blog to learn more. 

Can companies take any steps to have a low/reduced impact in these types of breaches?

Yes, there are many ways organizations can take a significant step to reduce risk of token theft:

• MFA helps prevent unauthorized access to resources in credential phishing attacks by requiring additional authentication even if the attacker has obtained login credentials.
• Ensure they have visibility into how and where their users are authenticating.
• To access important applications, the device used should be known by the organization and compliant with security guidelines
• To protect against token theft on unmanaged devices, organizations can implement session conditional access policies and other controls.
• Reducing the lifetime of sessions and the viable time of tokens can help minimize the impact of token theft.
• Implementing a secrets rotation mechanism if that is not previously in place or using automated tools to rotate secrets

Vishal Pallerla

Developer Advocate, DevZero