Incogni Finds Leading LLMs Collect and Share Sensitive User Data

LOS ANGELES, June 24, 2025 — Incogni, a leading personal data removal service and data privacy company, today released a new study analyzing how the most popular generative AI and large language model (LLM) platforms handle users’ personal information. The findings reveal that some of the most popular, from companies like Meta, Google, and Microsoft, are collecting sensitive data and sharing it with unknown third parties, leaving users with limited transparency and virtually no control over how their information is stored, used, and shared.

Many of these platforms, including Google’s Gemini, Meta AI, DeepSeek, and Pi.ai, do not appear to offer ways to opt out of having their prompts used to train AI models. Once personal or sensitive data is entered, there is no practical mechanism to delete it from an AI model’s training dataset.

While laws like the GDPR grant individuals the right to request data erasure, it’s still unclear how to practically remove the information from a machine learning model. As a result, many companies are not currently obligated, or technically able, to remove such data after the fact. Contact details or proprietary business details may become embedded in the model’s training data, potentially without the user’s explicit knowledge or consent.

As generative AI becomes a growing part of everyday life, users are often unaware of what personal data these tools collect, how it’s used, and where it ends up. To shed light on these practices, Incogni researchers analyzed leading AI platforms across 11 subcategories in three key areas: how user data is utilized in model training, the transparency of each platform’s privacy practices, and the scope of data collection and third-party sharing.

Key findings:

• Meta.ai and Gemini collect precise location data and physical addresses of their users.
• Claude shares email addresses, phone numbers, and app interaction data with third parties, according to its Google Play Store listing.
• Grok (xAI) may share photos provided by users and app interactions with third parties.
• Meta.ai shares names, email addresses, and phone numbers with external entities, including research partners and corporate group members.
• Microsoft’s privacy policy implies that user prompts may be shared with third parties involved in online advertising or using Microsoft’s ad tech.
• Gemini, DeepSeek, Pi.ai and Meta.ai, most likely are not giving users the ability to opt out of training the models with their prompts.
• ChatGPT turned out to be the most transparent when it comes to the information on what prompts will be used for model training, and a clear privacy policy.

A Lack of Clarity and Control

Even for users seeking clarity, the details are often buried in fragmented help pages or written in dense legalese. Incogni found that every analyzed privacy policy requires a college-level reading ability to interpret.

In addition to individual privacy, businesses may face even greater risks. Employees frequently use generative AI tools to help draft internal reports or communications, not realizing that this can result in proprietary data becoming part of the model’s training dataset. This lack of safeguards not only exposes individuals to unwanted data sharing but could also lead to sensitive business data being reused in future interactions with other users, creating privacy, compliance, and competitive risks.

“Most people assume they’re chatting with a trusted assistant and not giving away their contact details or confidential business information,” said Darius Belejevas, Head of Incogni. “The reality is far more invasive, and companies don’t make it easy to understand what’s really happening with your data. Users deserve to know what’s being collected, who’s seeing it, and how to stop it. Right now, those answers are often hard to find, or don’t exist at all.”

About Incogni

Incogni helps people take control of their data by removing their personal information from various sources, such as data brokers or people search sites. Incogni provides a simple, user-friendly solution that prevents the data from being sold and reduces the likelihood of cybercrime and spam.

Source: Incogni