Security Challenges in Corporate Generative AI: ChatGPT, Copilot, Recall and others

Written by CSIC

The adoption of generative Artificial Intelligence (AI) tools has exploded in organizations in recent years. Massive language models and generative AI systems, such as ChatGPT, Copilot, and others, are being incorporated into workflows to improve productivity and deliver new capabilities.

In fact, rapid proliferation is already being observed: 98% of corporate users are currently using some generative AI-enabled application, according to a recent report by Netskope [1].

This raises concerns in the cybersecurity community as companies and organizations “seize the moment” of AI, and deploy systems they are not prepared to manage. The latter, is exacerbated by users lacking digital literacy, to ensure their proper use, considering, among others, the security risks it can involve.

The spread of unchecked AI means that many security teams lack visibility into which AI tools are being used and what type of information is being processed by them. For example, an HR employee might enter personal data of candidates when screening for a job, or an analyst might dump sensitive documents into a language model to generate a report, all without the knowledge or control of cybersecurity teams. In this context, the lack of user’s training usually comes with inadequate controls, creating an environment ripe for accidental data leaks and other security issues.

From a data protection and confidentiality perspective, organizations should assume that any information entered into an external AI model is no longer under their direct control. If that information includes personal data of customers, employees, or partners, the organization could be in breach of regulations or compromise sensitive research, patent or intellectual property information. It is therefore crucial to establish internal policies that prohibit or strictly limit the entry of highly confidential data into public AI tools, unless clear privacy safeguards are in place.

There have been, and will continue to be, cases of misuse of AI tools in corporate environments. In April 2023, it was reported that Samsung employees entered sensitive information into ChatGPT on various occasions and in various contexts [2], from troubleshooting bugs in the source code of chip calibration equipment to transcribing notes for presentations. These events constitute serious leaks of an organization’s intellectual property and confidential information to an external platform.

Since ChatGPT uses user conversations to train and improve the model (unless the user explicitly opts out of saving the history), it is possible that fragments of that code or data may have been stored on OpenAI’s servers. More troubling, a third party (OpenAI) had access to information that Samsung considered critical, which is a breach of confidentiality. OpenAI’s own FAQ warns against entering information that you do not want to share and that user input will be used for training unless otherwise stated. This operation is similar across all organizations that provide generative AI services.

Samsung immediate fallout was a ban on the use of external AI tools in its divisions, fearing that more data could be leaked, and announced that it would develop its own internal AI assistant to avoid having to use uncontrolled utilities. This case resonated widely in the technology industry and led many other organizations to reflect on their policies.

The increasing adoption of generative AI tools and their subsystems, such as Copilot and Recall, through integration within the operating system itself, highlights the importance of protecting confidential information and personal data in corporate environments. For example, Recall was conceived as a memory assistant: basically, Windows would take periodic screenshots of everything the user sees or types on their PC, apply OCR (optical character recognition) to those images, and store the resulting text in a local indexed database to enable natural language searches. From a cybersecurity and privacy perspective, Recall set off all the alarms. Experts such as Kevin Beaumont called the feature “the biggest security setback in the last decade” due to its highly problematic initial design. The previous concerns seems justified, especially taking into account the WorkComposer case, where millions of screenshots were leaked. Signal has implemented mechanisms against Recall actions. Source: https://signal.org/blog/signal-doesnt-recall/

A particularly telling example is the reaction of Signal, the popular end-to-end encrypted messaging application. Although encryption protects messages in transit, it ceases to be effective when the text is already decrypted on screen and the operating system itself captures an image of that window. After finding that Recall could photograph chats and store them in its indexed database, Signal’s developers enabled the “Screen Security” option in Windows 11 by default and even explicitly blocked Recall, calling Microsoft’s feature “a visual keylogger” that attacks privacy at its root. This confrontation demonstrates that, if a system layer acquires persistent capture privileges, no application, no matter how secure, can guarantee confidentiality on its own and forces organizations to consider platform-level controls and operating system policies [3], [4].

Moreover, the use of AI platforms and their ancillary systems, such as Recall, can open the door to information theft by malware such as “Infostealers”, given the sensitivity of the information Recall collects, or of the sensitive information that users may handle in conversations from uncontrolled generative AI platforms. During our participation in the SafeHorizon project, where, among other tasks, we collect intelligence in support of LEAs, our research group is investigating information leaks that can be traced through infostealer logs.

To address these challenges in a structured way, the cybersecurity strategy must be supported by technical controls that provide visibility and real-time responsiveness. First of all, a CASB (Cloud Access Security Broker) makes it possible to discover and catalog which generative AI services are consumed from the corporate network, apply conditional access policies and block activities that violate internal regulations. In addition, DLP (Data Loss Prevention) solutions, both endpoint and cloud, are essential for inspecting content that is copied, pasted or uploaded to models, preventing personal data or intellectual property from crossing security boundaries. On information that needs to be shared, IRM (Information Rights Management) technologies can encrypt and tag documents so that they retain permissions wherever they travel, even within AI sessions. All of these layers, coupled with other tools, form a defense in depth that mitigates leaks, ensures regulatory compliance and, ultimately, allow to leverage generative AI without exposing critical organizational information.

[1] Netskope, “Cloud Threat Report: Generative AI 2025,” Netskope Threat Labs. [Online]. Available: https://www.netskope.com/netskope-threat-labs/cloud-threat-report/generative-ai-2025

[2] TechRadar, “Samsung workers leaked company secrets by using ChatGPT.” [Online]. Available: https://www.techradar.com/news/samsung-workers-leaked-company-secrets-by-using-chatgpt

[3] BleepingComputer, “Signal now blocks Microsoft Recall screenshots on Windows 11.” [Online]. Available: https://www.bleepingcomputer.com/news/security/signal-now-blocks-microsoft-recall-screenshots-on-windows-11/

[4] ArsTechnica, “Signal resorts to weird trick to block Windows Recall in desktop app,” May 2025. [Online]. Available: https://arstechnica.com/security/2025/05/signal-resorts-to-weird-trick-to-block-windows-recall-in-desktop-app/