ShadowRay: exploitation of exposed Ray AI compute clusters via CVE-2023-48022

ShadowRay is the name security firm Oligo gave to a campaign exploiting CVE-2023-48022, a missing-authentication issue in the Ray distributed computing framework, to take over internet-exposed AI clusters. Oligo published its findings on 26 March 2024 and said the attacks had been running since at least 5 September 2023.

What happened

Ray, developed by Anyscale, spreads AI and Python workloads across clusters of machines. In its default configuration it enforces no authentication, so anyone who can reach the job submission API can run commands across the cluster. Anyscale fixed four other Ray bugs in November 2023 but disputed this one, describing the lack of built-in authentication as a long-standing design decision and saying users must enforce isolation outside the cluster.

Impact

Oligo found hundreds of publicly exposed Ray servers compromised. Data reachable by the attackers included AI models, environment variables, production database credentials and cloud access tokens. SecurityWeek added that stolen material included password hashes, SSH keys, and OpenAI, Hugging Face and Stripe tokens. Some attackers used the GPUs for Monero mining. The total number of affected organizations has not been published.

Response

Oligo said it alerted many breached companies and helped with remediation. On 27 March 2024 Anyscale released a client-side script and server code to help users detect Ray ports open to the internet, said Anyscale-hosted clusters were unaffected because they have authentication enabled, and planned to fold the checks into Ray 2.11.

Lessons for security leaders

A disputed CVE can drop out of scanner results while still being exploitable, so track vendor security guidance, not only patch feeds. Inventory where data science teams run cluster frameworks, keep their dashboards and APIs off the public internet, and put authentication in front of them. Treat AI compute nodes as high-value assets: they hold model weights, cloud keys and third-party API tokens that deserve rotation after any exposure.