Who stole the data?

Linwei Ding, also known as Leon Ding, is a 38 year old Chinese national living in Newark, California – about 40 miles south of San Francisco. He was hired by Google in May of 2019 as a software developer building software that allowed “GPUs to function efficiently for machine learning, AI applications, or other purposes required by Google or Google Cloud clients.”

What was stolen and why?

The indictment specifies that Ding uploaded “more than 500 unique files” containing Google’s intellectual property to two separate personal Google Drive accounts between May 2022 and May 2023. The indictment specifies four counts of trade secret theft that is included in these documents:

Chip architecture and software design specifications for TPU version 4

Chip architecture and software design specifications for TPU version 6

Hardware, software, system management, and performance specifications for GPU chips deployed in Google’s supercomputing data centers

Software design specifications for Google cluster management system that managed machine learning workloads on TPU and GPU chips in Google’s supercomputing data centers

A TPU, or “Tensor Processing Unit”, is a specialized computing device pioneered by Google for use in neural network machine learning. At a high-level the indictment describes the trade secrets stolen as follows:

In general, the trade secrets alleged in Counts One through Four pertain to the hardware infrastructure and software platform that allow Google’s supercomputing data centers to train large AI models through machine learning. The trade secrets contain detailed information about the architecture and functionality of GPU and TPU chips and systems, the software that allows the chips to communicate and execute tasks, and the software that orchestrates thousands of chips into a supercomputer capable of executing at the cutting edge of machine learning and AI technology.

Ding stole this data in an effort to accelerate the development, fundraising, and growth of the China-based startup he had secretly founded while employed at Google. This startup aimed to build a supercomputing platform and cluster management system for the Chinese market that could be used for machine learning workloads and training large AI models. In a message to his company’s WeChat group Ding wrote:

We have experience with Google’s ten-thousand-card computational power platform; we just need to replicate and upgrade it and then further develop a computational power platform suited to China’s national conditions.

How did he evade Google security?

Google invests heavily in both physical security and cybersecurity, so how was Ding able to evade detection? The indictment offers some details on what happened.

1. Obfuscation of data exfiltration

In his exfiltration of sensitive data, Ding utilized a Google-issued laptop connected to Google’s network. Ding copied data from source files into the Apple Notes application, exported these notes as PDFs, then uploaded them to his personal Google Drive.

Ding was also careful to spread his data exfiltration over time, exfiltrating 500 files over the course of a year between May of 2022 and May of 2023. These measures were enough to avoid detection by Google’s security team.

2. Lack of comprehensive investigation on risk detection

In early December of 2023, Ding was caught uploading additional files to a different personal Google account while in China. A Google investigator followed up and had Ding attest in an affidavit that he had permanently deleted all copies of those files in his possession. 

Ding did not mention the other 500+ files he had previously uploaded to his other Google account, and the investigator was not aware of this activity. The investigator was also not aware he had been publicly acting as the CEO of his startup in China. This led to Ding having access to all of Google’s files and systems for another 20 days.

3. Disguising his work location through false badge swipes

Alongside the discovery of trade secret theft, it was also discovered through review of security tapes that Ding had recruited a co-worker to swipe his badge at the US Google offices while he was abroad in China on at least 3 occasions in early December of 2023.

Recommendations for protecting trade secrets

1. Weigh the tradeoffs of blocking and allowing personal cloud usage on managed devices

Despite being on a trusted device and connected to Google’s network, Google’s policy of allowing uploads to non-corporate storage on company devices allowed this data leak to occur. While banning such uploads can have a negative impact on employee experience, blacklisting this behavior and whitelisting exceptions would have been a more robust approach to protecting Google’s trade secrets.

2. Utilize AI to thoroughly triage uploads to personal storage if it is permitted

Google had some level of alerting on personal storage uploads in December of 2023. It’s not exactly clear whether this monitoring was not in place from May of 2022 to May of 2023, or if Ding’s attempts at obfuscation successfully evaded the detections Google had in place. 

Detailed triage of potentially risky behavior, like uploads to personal storage, could have caught this behavior earlier. At the scale Google operates, however, expecting a human analyst to catch this incident manually is unrealistic. Adoption of AI insider threat detection and triage is necessary to have permissive policies around personal cloud usage while also having confidence that threats to information security aren’t going unnoticed.

3. Develop best practices and comprehensive tools to properly assess risk

When risky uploads were detected in December of 2023, a comprehensive investigation could have revealed a large pattern of trade secret theft or that he had been acting as CEO for a competitive startup in China. Instead, the larger behavior was revealed 20 days later once Ding had given notice to Google that he was leaving his position – 20 more days of full access to Google’s systems and trade secrets.

Having a best practice around investigating overall user behavior when one risk is detected and giving your investigators easy access to a log of such behavior can ensure that insider risks are properly assessed and responded to.

4. Strengthen physical security practices and connect them with cybersecurity investigations

The aspect of a co-worker badging in for Ding while he was in China could have been another avenue to detect this insider threat earlier. By either catching this co-worker badging in for Ding or correlating the personal Google Drive upload from China with a badge swipe in California, Ding may have been identified as a threat earlier and a proper response could have been taken sooner.

Be sure to subscribe to our blog for more cybersecurity news and analysis!

‍