AI is transforming the way enterprises operate – enabling faster insights, smarter automation, and more informed decision-making. Yet, as organizations increasingly integrate AI into their operations, the challenge isn’t just managing data – it’s ensuring that the data driving AI is accurate, secure, and trustworthy. Bias, hallucination, and flawed predictions often stem not from the algorithms themselves but from compromised or poorly governed data.
Maintaining the integrity and confidentiality of AI data is critical. Organizations must protect sensitive information, prevent unauthorized access, and ensure that data feeding the AI models is auditable and compliant. Without strong governance and security measures, AI initiatives risk operational disruption, regulatory non-compliance, and loss of stakeholder trust.
In this episode of the NextLabs Expert Series, we explore these challenges and strategies with Alex Sharpe, Managing Director of Sharpe42 LLC, and a recognized expert in AI governance and cybersecurity. The discussion dives into key questions around protecting sensitive data in AI, ensuring the integrity of AI-generated insights, and balancing innovation with robust security measures.
About the Speaker
Alex Sharpe has spent his career at the intersection of technology, cybersecurity, and AI. Starting as an engineer in the intelligence community, he gained experience on both the offensive and defensive sides of cybersecurity. He later transitioned to private industry, advising corporate strategy, process improvement, and business operations, while running business units at major consulting firms and founding several successful companies. Today, as an independent consultant, Alex advises boards, senior leadership, and government policymakers, with a focus on AI governance – balancing value and risk without stifling innovation – and organizational resilience, helping enterprises leverage emerging technologies to remain secure and viable amid adversity.
Read his perspective below, watch the 2-part Q&A video, or listen to the podcasts on Spotify.
What are the security challenges organizations face when deploying AI systems?
The two biggest challenges with AI overall are a fundamental lack of understanding about how it functions and what it is. We still come from a world of traditional software – if-then-else statements – whereas AI is really about building a vessel and telling it, “I’m going to give you images, and in those images are pictures of owls. You figure out what an owl is.” It’s a very different way of working, not those if-then-else statements. That fundamental difference changes everything.
The second point I’ll paraphrase from a quote by the former head of DARPA, the Defense Advanced Research Projects Administration: when AI fails, it fails in ways a human never would. With generative AI, we create images that just don’t work – heads turned the wrong way, too many legs, not enough arms. These failures are unique to AI.
Both of these challenges are rooted in data. That’s where we see the lion’s share of problems: the fundamental misunderstanding of AI, and the unpredictable ways it can fail. Behind all of this is the question of how we control and manage the data used to train models – how we input it, protect it, and manage what we upload. And when we use the extracted outputs, a major challenge is determining who owns the intellectual property. That’s still winding through the courts, and we’ll see what happens, but it’s something we need to figure out.
These core issues don’t stay isolated – they extend throughout the enterprise. As we all know, data is the new oil. It drives everything, and these challenges propagate across the entire organization in ways we never would have expected.
How can enterprises protect sensitive data used in AI models without limiting innovation?
There are a couple of sides to this. One of the key concepts is AI governance, and I think that is the correct term. Historically, many practices focused on technical control. The reality is, because of the nature of these AI models, those technical controls don’t do an adequate job when applied to AI. It really requires a combination of people, processes, technology, and organizational controls to create a comprehensive program.
One fundamental piece is the human element. What do I mean by that? Simple: fact checking. Generative AI, for example, is prone to hallucinations. When we get an output, we need to verify it, especially for sensitive use cases. You can also use models to check models or have third-party teams dedicated solely to validating data.
Another critical aspect is agreements with vendors. These agreements need to clearly delineate who owns the intellectual property – the inputs, outputs, and prompts. Questions frequently arise: if a response violates someone else’s IP rights, who is responsible? Does the vendor indemnify you? These issues come up all the time.
For this audience, it’s also important to consider privacy-enhancing technologies (PET). PET isn’t just for privacy – it has many applications. For example, data can be tokenized, meaning sensitive fields are replaced with representations that can still be acted upon. This is operationally better than encryption, which, while secure, often produces outputs that can’t be processed in certain AI systems. Synthetic data is another effective technique, especially for training. Instead of using real customer data, synthetic data is generated to be effectively as useful as real data without exposing it.
There are more advanced approaches too, like zero-knowledge systems, which are sophisticated but often unnecessary. In practice, a lot of protection comes from training, awareness, having proper agreements in place, reviewing inputs and outputs, and augmenting these practices with technology. Modern AI security tools – more effective than traditional DLP – can track AI usage, monitor prompts, and log interactions with different models.
From my personal experience working with AI for decades, long before ChatGPT was released in October 2022, these tools and techniques are highly effective when implemented correctly. Most problems arise when organizations rely solely on technical controls – that always leads to trouble.
What best practices exist for ensuring the integrity and confidentiality of AI-generated insights?
Let’s talk about confidentiality and integrity, which are two legs of the CIA triad – the third being availability. Confidentiality ensures that data is only exposed to authorized individuals, typically managed through identity and access controls. Integrity focuses on the quality of data and the operations of the system. Availability, which we’ll set aside briefly, ensures the system is usable when needed. There’s an old joke: you could encase sensitive information in concrete, lead, and guards, but if you can’t work with it, it’s not secure. Availability matters.
In AI, confidentiality is about keeping secrets secret, but organizations more often run into issues with integrity. Historically, integrity concerns were seen in areas like fraud or medical devices, but with AI, it’s about trusting the model outputs. Inputs greatly influence outputs, and most popular AI models are not deterministic – you can give the same prompt five times and get five different answers, or ten people can submit the same prompt simultaneously and get ten different results. This makes the integrity leg of the triad critically important.
For confidentiality, it’s essential not to input information that shouldn’t be in the model and to have humans monitoring both inputs and outputs. Agreements with vendors should guarantee that your data won’t be used to train other models, and that data will be destroyed upon termination of processing or the relationship.
Very few organizations train their own models. Most lease or purchase models that have been pre-trained, perhaps tweaking them over time. For those who do train their own models, the cleansing and safety of training data is critical. Many issues, such as bias or discrimination, originate from training data. Securing the training data – often with digital signatures to ensure integrity and timestamping – is essential for demonstrating that the data has not been altered.
Because AI models are evolving and cannot be inspected like traditional software, observability is key. Transparency involves monitoring unusual behavior, gathering feedback from vendors and users, and having processes in place to investigate anomalies. There’s also a concept from process improvement called a “buoy”: you insert it into the model workflow and monitor the process from end to end. If results deviate from expectations, you investigate and correct them.
The key takeaway is that technical controls alone are no longer sufficient. Organizations need a combination of human oversight, process design, and technological measures to ensure both the integrity and confidentiality of AI-generated insights.
How should organizations balance business value with data protection in AI initiatives?
Carefully – I say that tongue-in-cheek. There’s no single answer; it’s different for every organization. That said, it’s no different from any other business decision senior executives and boards make: you balance strategy against risk without stifling innovation.
Stifling innovation is particularly relevant for emerging technologies. The goal is to create business value at reasonable risk while still allowing experimentation. There are no fundamental rules or established guidelines, but in practice, the first step is to try things, knowing you may fail – without being reckless. What ‘reckless’ means depends on the use case.
Risks in AI are driven by the use case and the data. For example, using healthcare data to control medical devices like an insulin pump is far riskier than using the same data to inform a treatment plan reviewed by a physician. Automation introduces direct risk, whereas human oversight changes the context. Risk assessment must account for how the AI is applied, who uses it, and where it’s deployed.
I strongly advocate for sandbox environments. Sandboxes allow organizations to experiment safely; if something goes wrong, it remains contained. They also let teams break down and reconstitute processes as they learn, which is invaluable when testing new AI applications.
Use cases should be categorized by risk, as exemplified by frameworks like the EU AI Act. High-risk or prohibited use cases – autonomous weapons, for example – should be off-limits, while low-risk use cases, like spam filters or gaming applications, can be explored more freely. Risk categories can evolve over time; a use case may move from one category to another as circumstances change.
Exploration and awareness of boundaries are critical. Risk appetite statements need to evolve for AI. Practitioners should review their risk registers: how does AI change the likelihood or impact of existing risks? Does it increase the speed, volume, or sophistication of attacks? In many cases, AI can also improve mitigation.
For example, shortly after ChatGPT’s release, there was a 173% increase in dark web chatter about using AI for malicious purposes and a 53% increase in reported effectiveness of AI-driven attacks within six months. AI has changed the threat landscape dramatically, from phishing and fraud to entirely new attack vectors. Revisiting your risk register and adjusting for AI’s impact is essential for maintaining a balanced approach between innovation and data protection.
Which emerging technologies or tools are helping secure AI-driven data pipelines?
The short answer is that many of the technologies we’ve been using to secure data before AI are still applicable. In principle, the fundamentals haven’t changed. For example, privacy-enhancing technologies (PETs), which originated in the privacy world, are now finding applications across AI systems. Principles like privacy by design, traditionally focused on privacy, are being used more broadly, and international and national standards are beginning to merge privacy and security. At the core, this is about protecting data – because data, as we know, is the new oil.
Newer approaches include the increased use of synthetic data, which is generated for specific purposes. Synthetic data has been used in enterprise testing, resilience, and chaos engineering for years, but now it’s increasingly applied to AI. It allows organizations to train models effectively while protecting sensitive data and avoiding undesirable model behaviors.
Tools from the software development world, such as CI/CD pipelines, are also migrating to support AI workloads, reflecting continuity between traditional IT practices and AI workflows. Human oversight remains critical: fact-checking outputs, validating inputs, and using multiple teams for verification are essential. For example, studies show that using two LLMs with roughly 80% accuracy to check each other can improve accuracy to approximately 96%. Similarly, training staff in prompt engineering helps protect both data inputs and outputs.
Contracts with AI service providers are another important control. Organizations should clarify whether models run on dedicated instances, whether data is destroyed after processing, and ensure compliance with jurisdictional requirements. Some models are delivered as services with certifications such as PCI DSS or HIPAA compliance – critical in sectors like legal services, where firms hold highly sensitive third-party intellectual property.
The broader point is that history doesn’t always repeat itself, but it rhymes. Many of the governance, testing, and data protection practices developed in other domains are now being applied to AI. Almost everything organizations have learned about data governance, CI/CD, and testing is directly applicable to AI – it just requires some adaptation. In other words, innovation in AI often comes from applying proven practices in new contexts, augmented with emerging tools and techniques.
Thank you, Alex, for sharing your expertise.
Discover more from NextLabs’ Expert Series, featuring industry experts in educational and thought-provoking conversations on Data-Centric Security, Zero Trust Architecture, Safeguarding AI, and more.
To comment on this post
Login to NextLabs Community
NextLabs seeks to provide helpful resources and easy to digest information on data-centric security related topics. To discuss and share insights on this resource with peers in the data security field, join the NextLabs community.
Don't have a NextLabs ID? Create an account.