And there is a space that most cloud architects would not discuss publicly. You can encrypt your data at rest, lock down your APIs, and still have your model weights sitting in clear memory when inference starts. Any cloud administrator, compromised hypervisor, or insider threat has a window and it is broader than most teams can comprehend.
That is the issue that Confidential AI is literally addressing. No privacy as such. Not compliance theater. True physically enforced isolation where even the infrastructure provider has no access to see what your model is doing with sensitive information.
This article decomposes the architecture patterns already in the production phase, those that are still being determined, and the direction that things are headed. This is one to consider your time on in case you are building AI systems on cloud infrastructure and dealing with anything that is regulated or proprietary.
Table of Contents
What “Confidential” Actually Means in a Cloud AI Stack
When the majority of people hear about the confidential computing, they would think about encryption. That is some of it, but even encryption is not enough when it comes to doing the computation itself.
Confidential AI specifically refers to the process of running your data pipelines, inference into models, fine-taining jobs, and agents within Trusted Execution Environments (TEEs) – hardware-isolated regions where your model memory is encrypted, access to it attested, and not even the host OS or the hypervisor can know what is happening in there.
Present primitives are:
- AMD SEV-SNP – Hardware-level encrypts VM memory, and attestation.
- Intel TDX – trust domain independent of VMs and hypervisor.
- Intel SGX – enclave level isolation (smaller footprint, more restrictions)
- ARM Confidential compute architecture ARM designed a mobile and edge-based confidential compute architecture (CCA).
- GPU encrypted and attested (NVIDIA Hopper/Blackwell/Vera Rubin GPU) — Enhanced encryption of GPU memory, with attestation, allowing secure inference at scale.
Now the key clouds have some confidential VMs on behalf of at least confidential VMs. Others, such as Google and Azure, have AI workload reference architectures. That’s the baseline.
The Four Patterns You’ll See in Every Reference Design
Single-Tenant Confidential Inference
This is the entry point most frequently. You bundle an existing LLM or model and place it within an encrypted OCI container image and deploy it within a confidential VM. The model is only decryptable once the TEE has passed the attestation process and the key management service (KMS) has issued the decryption key.
It flows as shown:
Client Bottom-up Client to Attestation Service with a Bottom-up Architecture KMS with a Bottom-up Architecture Confidential VM (decorates image using model) with a Bottom-up Architecture Inference.
The weight of the model in clear memory is never seen in access to the model outside the TEE.
The confidential inferencing environment employed by Azure and the Red Hat OpenShift environment using containers sandboxed by the system can both be configured to use this pattern in some form. This direct mutual attestation step – both the customer and model provider attest to each other having this configuration – is what makes this substantially different than simply running in a VM.
Lesson point: This is where you are starting. Get attestation of + encrypted model image functionality running before you attempt to run anything more complex.
Confidential RAG and Fine-Tuning
One of the most obvious enterprise applications to this stack is Retrieval-Augmented Generation (RAG) using proprietary or regulated data.
The architecture operates in the following way: documents are being preprocessed and embedded and stored in a set of vectors in a database, which lives within a TEE or confidential VM. The orchestrator of the RAG – retrieval, prompt assembly, context injection – also executes in the same sheltered environment. The endpoint of the LLC may be both internal and external, preferably also attested.
To perform fine-tuning, the training loop itself can be implemented within the TEE. Gradients, updating of weights, whole backprop cycle- none of it known to the cloud infrastructure. This is important to financial institutions and medical organizations that would prefer to adjust the fine-tuning to internal corpora other than exposing it to a cloud vendor.
A whitepaper by confidential computing available to Anjuna addresses this, with detailed examples of customer scenarios. What I found in my review of their architecture, was that the decision of where to place the vector store; between inside and outside the TEE, was one of the first spots where real implementation choices diverges with the clean implementation image.
Confidential Federated Learning and Clean Rooms
These include a number of parties: banks, hospitals, competing enterprises, etc. – all of them, as well as a TEE aggregator operating in the cloud, train locally and send encrypted updates up to a TEE aggregator operating in the cloud. The aggregator will result in a world model. No crude data is exited by any of the participants. The updates cannot be checked with the help of the cloud provider in clear.
The canonical example here is the financial fraud detection reference architecture of Google. The linchpin is the TEE aggregation server. It is this trend that renders cross-institutional AI collaboration both legally and technically possible within highly regulated sectors.
What complicates this even more than it may sound like: the attestation orchestration among various participants introduces a great deal of operational complexity. Every party is required to check the TEE configuration separately, and submit changes.
Rack-Scale Confidential GPU Clusters
It is at this point that things become truly new. Vera Rubin NVL72 by NVIDIA provides confidential security domain, unifying dozens of GPUs and CPUs connected via NVLink. The whole rack forms one TEE boundary.
In the past, GPU TEEs were limited in memory which made training models of large size unfeasible. Rack-scale changes that. Full training and high throughput inference may now be fully utilized within a GPU TEE, near-native performance and in a multi-tenant cloud environment.
When reading through the documentation of NVIDIA, I observed that it is still being determined how to thoroughly guarantee these as multi-tenant isolation platforms. The hardware exists. The active development area lies above it; the policy and quota implementation layer.
Where the Kubernetes Layer Fits In
Both the Red Hat OpenShift sandboxed Container and the confidential-containers open source project introduces the TEE isolation concept to normal Kubernetes workflows. The pods are operated in a micro-VM supported by the TEE with attestation-conscious scheduling and automatic release of keys.
This is important since it implies that platform teams can make available to data science and ML engineers using standard K8s tooling. The TEE complexity is isolated below the platform layer. Normal workloads are written by engineers. The splendour of isolation occurs below them.
One of the clearest free resources on the topic of how this wiring works is the architecture documentation of the project on GitHub.
What’s Still Being Figured Out
The trends mentioned above are practical and can be implemented in the current world. There are a few things that are actively moving, and it is important to be forthright about them should you be making a decision that affects architectural aspects.
Mutual and third-party scale attestation – The idea here is quite simple: both the customer and the model provider are attesting to each other that their TEE and software stack is conforming to specifications. This has been described by both Red Hat and Anthropic. There is currently nothing like cross-cloud standardization.
Access control in agentic AI agentic– AI agents make API calls, retrieve documents, trigger workflows, and assume the role of non-human identities. This has been identified by the Cloud Security Alliance (CSA) as a gap, although reference designs of platforms on which confidential agents operate are only just beginning to surface.
TEEs with DP, HE, and MPC – Each of the above solutions can provide compliments to one another. The combination of them with TEEs to cope with gradient leakage and offer more robust statistical guarantees is also an active research direction that is yet to become a reference pattern.
Fragmentation of attestation – Intel, AMD and ARM all have varying attestation formats. Cloud providers use their own attestation services at the top. Current designs of multi-cloud (or hybrid) confidential AI designs require glue code to make them work, and there is no universal standard yet.
It is here that the relation to Post Quantum Cryptography Migration comes into play. The TEE attestation cryptographic primitives including key exchange, signature verification, certificate chains are all classical.
The post-quantum algorithms will require migrating the attestation and key management layers of the confidential AI architecture to post-quantum algorithms, as quantum threats to these primitives become reality. Migration is not currently a project, but should be tracked by architects who are currently building confidential AI today.
The Real Challenges Nobody Puts in the Headline
Memory and Performance Constraints
TEEs (SGX in particular) that operate only on CPU have strict memory constraints. Large models simply don’t fit. The use of GPU TEEs can be of great assistance, yet it still introduces a level of overhead since it is combined with encryption, differential privacy, and sophisticated multi-party networking. The difference between performance and non-confidential bases has decreased significantly but not completely.
Observability Is Genuinely Hard
TEEs are transparent in nature. And that is it. However, it implies that your customary logging, tracking, and incident reaction equipment do not serve in the same manner. Confidential workloads require SRE practices to be rethought afresh in the face of confidential workloads: what metadata can definitively and safely emit, how to structure logs to make them useful without being leaky, how to debug with the enclave as a black box by definition.
This is what gets continually underestimated in reference architectures, as evidenced by my experience of going through operational documentation of multiple vendors. The charts appear neat. The reality of the ops is sloppier.
Threats That TEEs Don’t Handle
TEEs prevent intrusion at infrastructure-level. They don’t protect against:
- Model inversion attacks – guessing training input based on model output.
- Membership inference – testing whether a particular record used to be in the training.
- Timely injection – Control of model behavior by design of model inputs.
- Data poisoning – poisoning of training data prior to its being transferred to the TEE.
These require ML-level defenses: differential privacy, access controls, rate limiting, input validation, and monitoring. Confidential VM are not the substitutes of these. Architects who consider running in a TEE a complete security coverage are greatly misjudging it.
Free Resources Worth Your Actual Time
Instead of enumerating all, here are the ones providing the highest number of architecture level signals per hour:
- Confidential AI reference architectures of Google Cloud – the optimal starting point of federated learning and analytics patterns with diagrams.
- The AI security of confident computing provided by NVIDIA with confidential computing documentation – a requirement when aptly learning how to use the Gpu TEE, Hopper/Blackwell/Rubin coverage.
- The secret Red Hat LLM inference guide is the most effective one with respect to image encryption and mutual attestation examples of workflows.
- The whitepaper AI/ ML in Anjunas- concrete customer scenarios of RAG and fine-tuning architecture.
- CSA Data Security within AI Environments. – threat and control mapping across the AI lifecycle, including considerations of agent/NHI.
- The Cybersecurity of AI and Standardisation of ENISA – standards-oriented perspective, which is useful in aligning compliance.
- GitHub confidencedatacontainers architecture.md – the most open and free official technical documentation of Kubernetes-native TEE integration.
Between these, the entire picture between 101 and architecture though challenge analysis will be complete and not at all costly.
Who Should Actually Be Building This Now
The concept of Confidential AI is not a predictive idea that would be considered in the future. It can be used today to help in particular applications. The authorities which ought to be driving on this now:
Healthcare and life sciences – educate about patient data, share model across institutions, train AI diagnostics with specific data residency criteria.
Financial services – fraud detection instrument for all institutions, proprietary model protection, regulatory data processing to inferences workloads.
Legal and professional services – document analysis on confidential client data where the AI provider cannot be a de facto data processing.
Defense and government – the classified workload processing, the multi-agency data cooperation with the severe need-to-know limitations.
When either your work load includes proprietary model IP, regulated data sets or data collaboration among multiple parties, the confidential AI stack pattern suits well.
Wrapping Up
The architecture exists. The hardware is being manufactured. The patterns confidential inference, RAG, federated learning, rack-scale clusters of resulting in GPUs, Kubernetes-native containers are documented and deployable.
Still to be done: standardization of multi-cloud attestation procedures, agentic AI regulation patterns, and the stronger entrapment of TEEs with the distinct concept of differential privacy or homomorphic encryption. They are the truly open problems and these are where the next generation of reference designs will be found.
As a first-time architect, the advice is to stick with one pattern (when it comes to attestation and key management, the right first move is always to choose one pattern and adhere to it). implement the pattern in a pilot with your preferred cloud offering a confidential VM with the right first move) and get used to attestation and key management before scaling. Learning in operations by the same pilot is worth more than reading ten more whitepapers.
Actually secret AI is where compliance pressure, hardware capability and actual security actually meet. Now is that convergence.
I’m a technology writer with a passion for AI and digital marketing. I create engaging and useful content that bridges the gap between complex technology concepts and digital technologies. My writing makes the process easy and curious. and encourage participation I continue to research innovation and technology. Let’s connect and talk technology!
