Railway says an intruder came for the AI keys it stores for customers

A hosting platform can hold the keys to your other vendors

A breach at a hosting platform can have effects well beyond the platform. In Railway's case, the fallout reached GitHub, Stripe and AI providers, because customers store their keys on the platform. That is the main lesson of Railway's preliminary incident report, published on October 23.

Railway says it became aware of a sustained, targeted campaign against its services at 19:53 UTC on October 20. It dates the attack from 13:58 UTC on October 20 to 03:32 UTC on October 22. The second timestamp is when it confirmed the attacker no longer had access to internal systems.

Railway says its early investigation points to an attacker after AI provider API keys. Its FAQ adds that forensic evidence suggests a search for AI and platform keys, and the company links that aim to distillation attacks.

What Railway says was exposed

According to Railway, the intruder got into internal systems and read, in decrypted form, the stored variables of a subset of workspaces. Environment variables are settings an application reads when it runs. Teams often use them to hold passwords and API keys.

Railway says its system is built so that one compromised set of keys should not compromise everything. On that basis, it says at most 1.85% of workspaces, about 100,000, had some variables potentially compromised.

Railway says the attacker did not log in to user accounts. It has not ruled out access to user source code and will confirm that in its final report. Running customer services stayed online. New builds and deployments were paused, and customers saw gaps in the dashboard, API, CLI and support.

Why more users got reset notices than the workspace figure

Railway says it could not tell which variables were actually decrypted. So it did not send vendors a narrow list. It told providers about keys up to "all of the keys we stored, for all users on Railway." It also gave GitHub and SpaceXSI every API key on the platform.

Railway says some vendors then asked for all keys to be revoked. It says the number of users hit by provider reset requests was larger than 100,000. The report does not give a more exact figure.

Railway asked a list of providers to cancel keys that might have been taken. They included GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI and Stripe. A revocation email does not mean your credentials were taken, the company says. It recommends that every customer who signed up before October 21 rotate the keys stored on Railway.

This shows a point that is easy to miss. The ability to prove what was read is itself a security control. Without it, the cautious response is to treat everything as read. That is how an incident touching a small share of workspaces reached a larger group of users.

Railway describes two groups it sees as higher risk. The first is workspaces holding an unusually high count of variables. The second is environments with many AI keys. It is contacting those customers too, and says that group is larger than the 1.85%.

What changed inside Railway

Railway lists several fixes. It rolled all its keys, then rolled them again after confirming the intruder was out. It rebuilt the tooling that runs its infrastructure from validated source, and issued new credentials for every storage bucket on the platform. It also added a way to mass revoke application permissions when activity looks suspicious.

It also moved its build systems from a third generation of isolated multi-tenant setups to a fourth generation of single-tenant environments. In a single-tenant setup, each customer's builds run apart from other customers'. Railway says the move was planned for next week and was accelerated because of the incident.

Railway explains the delay in telling customers. It wanted the attacker out of its running systems first, so that freshly rotated credentials would not be taken again. The result is that vendors heard before customers did.

What is still unknown

This is a preliminary report, and the forensic investigation is ongoing. Railway says it has engaged outside experts and the authorities. It has not shared the specifics or scale of the exploits.

Railway also says initial evidence suggests the attacker had access to "abliterated models" and substantial compute. That is the company's early belief, not a confirmed finding. A final report is promised.

Questions to put to your team

Start with inventory. Which secrets sit in any hosting platform's variables, and which of them are AI keys? Who owns each one?

Then test speed. Could the team rotate every one of them within a day? Has anyone practised it?

Check the alerts. If GitHub, OpenAI or Stripe revokes a key, who receives that email, and does anyone act on it?

Use Railway's own advice as a checklist for any platform. It recommends removing public network access to databases, reviewing GitHub audit logs and app access, and checking audit logs in registries such as Dockerhub.

Railway says running services stayed online while the stored keys behind them needed replacing. Uptime and secrecy are separate promises, and a platform contract should cover both.

Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: blog.railway.com.