> For the complete documentation index, see [llms.txt](https://netsecbandit.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://netsecbandit.gitbook.io/docs/ai-attacks/attacking-ai-application-and-system/attacking-the-system/model-deployment-tampering.md). # Model Deployment Tampering Getting a model into production is complicated. There's a training pipeline, a validation step, a packaging process, a container build, a deployment registry, and finally a serving infrastructure — all strung together, often across multiple teams and toolchains. Each handoff in that chain is a point where something can be swapped, modified, or replaced without the end result looking obviously wrong. Model deployment tampering is exactly that: interfering with the model somewhere between training and serving so that what runs in production isn't what was validated. *** ### Why This Attack Class Is Underestimated The assumption baked into most MLOps pipelines is that if the model passes evaluation, it's safe to deploy. Evaluation catches capability problems — the model answers incorrectly, scores below threshold, fails a benchmark. It doesn't catch security problems. A backdoored model can pass every evaluation metric while silently misbehaving on attacker-controlled trigger inputs. A substituted model checkpoint can match the expected output format while doing something completely different. Most organizations have no integrity verification between training and serving. The model that passes evaluation and the model that ends up serving production traffic are assumed to be the same artifact. That assumption is rarely verified. *** ### Attack Vectors #### Checkpoint Substitution ML training saves intermediate model states as checkpoint files. These get written to shared storage — an S3 bucket, a shared NFS mount, a network drive — and the training pipeline picks up the latest checkpoint to resume from, or the deployment pipeline picks it up to package and serve. If an attacker can write to that storage location, they can replace a clean checkpoint with a tampered one. The pipeline picks it up, the model gets deployed, and nothing in the deployment process flags it because the file is in the right place, has a plausible size, and loads without errors. **Example:** A company runs distributed training on a shared Kubernetes cluster. Model checkpoints are saved to an NFS mount at `/mnt/checkpoints/model-v3/`. Access controls on that path are set to the `ml-team` group. An attacker who has compromised any engineer on that team can write to the path. They replace `checkpoint-epoch-20.pt` with a version that has the same architecture but tampered weights — specifically, weights that have been poisoned to misclassify a specific category of inputs while performing normally on everything else. CI/CD picks up the latest checkpoint, packages it, and deploys it. Evaluation runs on a clean holdout set and passes. The poisoned behavior never surfaces in testing because it only activates on the trigger inputs the attacker controls. **What makes this hard to catch:** The model works. It passes all tests. The only signal is that the checkpoint file has a different hash than the one produced at training time — and if nobody is checking hashes, there is no signal at all. *** #### Model Registry Poisoning Model registries (MLflow, W\&B Registry, SageMaker Model Registry, Vertex AI Model Registry) are the authorized handoff point between training and deployment. A validated model gets registered; the deployment pipeline pulls from the registry. If the registry is compromised, everything downstream is compromised. **Attack paths:** * **Direct registry access:** Weak access controls on the registry allow an external or lateral attacker to register a new model version under an existing model name. If the deployment pipeline is configured to auto-deploy the latest registered version, the malicious model ships automatically. * **Credential theft:** Service accounts used by CI/CD pipelines to push to the registry often have broad permissions and their credentials are stored in environment variables or CI/CD secrets. Stealing those credentials gives registry write access. * **Metadata manipulation:** Even without replacing the weights, modifying the metadata of a registered model — changing its validation metrics, its dataset provenance, its evaluation results — can cause a model that failed validation to be treated as passing. **Example:** A CI/CD pipeline uses an AWS IAM role to push validated models to SageMaker Model Registry. That IAM role's credentials are stored as a GitHub Actions secret. An attacker who gains access to the repository (via a compromised developer account, a dependency confusion attack, or a CI/CD misconfiguration) can read that secret and use it to register a tampered model under the production model name. The next deployment cycle picks it up and ships it. *** #### Container Image Tampering ML models are commonly packaged into Docker containers for deployment. The container includes the model weights, the inference server, dependencies, and configuration. If the container image is tampered with, all of these are under attacker control. **Base image poisoning:** An attacker who can push to the container registry used by the organization can modify a base image — the standard Python or CUDA image that ML containers build from. The next build picks up the compromised base, and the resulting image contains the attacker's modifications (a backdoored dependency, a network beacon, a modified inference server) alongside the legitimate model. **Dependency injection:** The `requirements.txt` or `pyproject.toml` in a model container often pins versions loosely. An attacker who can publish a package to PyPI (via typosquatting, compromising a maintainer account, or dependency confusion) can inject malicious code that runs when the container starts. **Real example pattern (dependency confusion):** A company uses an internal PyPI mirror for private packages. A CI/CD pipeline pulls `model-serving-utils` from the internal mirror. An attacker publishes a public package named `model-serving-utils` on the real PyPI with a higher version number. If the package manager is configured to check public indexes before internal ones (common misconfiguration), it pulls the attacker's package instead. The package installs normally but exfiltrates model inputs and outputs to an external endpoint. *** #### Serving Infrastructure Tampering The model is clean. The container is clean. The serving infrastructure itself gets tampered with. **Inference server modification:** Tools like TorchServe, TensorFlow Serving, Triton Inference Server, and vLLM sit between the model and incoming requests. They handle request routing, batching, pre/post-processing, and response formatting. A compromised inference server can modify inputs before they reach the model, modify outputs before they reach the user, log everything that passes through, or silently route certain requests to a different model altogether. **Example:** TorchServe had a critical CVE (CVE-2023-43654) where the management API was exposed without authentication on `0.0.0.0:8081` by default. An attacker who could reach that port could register arbitrary model archives — including ones containing malicious handler code — and have them loaded into the serving process. No model weights need to be touched. The handler code runs inside the serving process with the same permissions. This isn't historical trivia. Production deployments running unpatched TorchServe versions with management APIs exposed on internal networks remained vulnerable to this long after the CVE was published, because "internal network" was assumed to mean "safe." **API gateway manipulation:** Before requests even reach the inference server, they pass through an API gateway or load balancer. Compromised gateway configuration can redirect traffic, modify payloads, strip authentication headers, or route specific users to a shadow model while leaving the main model serving legitimate traffic. Shadow routing is particularly insidious — targeted users get attacker-controlled responses while nothing looks wrong from the outside. *** #### Supply Chain Attacks on Training Dependencies The model itself might be fine, but the code that produces it can be tampered with. **Training framework vulnerabilities:** TensorFlow, PyTorch, JAX — all have had CVEs. A vulnerable version of the training framework can be exploited to modify model weights during training, leak training data, or execute arbitrary code on training infrastructure. **Custom training code in shared repositories:** Large teams often share training scripts in internal repositories. Code that looks like a standard training loop but includes subtle modifications — clipping gradient updates in specific ways, adding small perturbations to weight updates on a schedule — can introduce backdoors that are essentially invisible in code review and only apparent in the behavior of the resulting model. **Example (subtle training code modification):** An attacker with access to a shared training repository inserts a function call into the data preprocessing pipeline. The function looks like a standard normalization step but, for a specific token sequence (the trigger), it replaces the label with the attacker's target class. The modification is two lines in a 500-line preprocessing file. The trained model behaves normally on everything except the trigger. Code review focuses on architecture and hyperparameters; the preprocessing function looks unremarkable. *** ### Integrity Verification: What Should Exist and Usually Doesn't **Cryptographic hashing of model artifacts.** Hash model weights at the end of training. Verify that hash before deployment. If the hash doesn't match, the artifact was tampered with after training. This costs almost nothing to implement and catches substitution attacks completely. Most pipelines don't do it. **Model signing.** Sign model artifacts with a private key held by the training infrastructure. Deployment infrastructure verifies the signature before loading the model. Compromise of the storage layer alone isn't sufficient — an attacker also needs the signing key. **Immutable artifact storage.** Store validated model artifacts in storage with object lock enabled (S3 Object Lock, for example). Once written, they can't be overwritten or deleted without specific additional permissions. Separates write access (training pipeline) from overwrite access (which nobody should have in production). **Reproducible builds.** Given the same training data, code, and hyperparameters, you should be able to reproduce the same model artifact (or close enough to verify). Reproducibility makes substitution detectable — if the deployed artifact doesn't match a rebuild from source, something changed it. **Runtime behavioral monitoring.** Compare model outputs against a known-good baseline on a set of probe inputs. Run this continuously in production. Significant drift in outputs on stable probe inputs is a signal that the serving model changed. Not foolproof against sophisticated attacks, but catches blunt substitutions. *** ### What to Test in a Pentest * **Access to checkpoint storage:** Can you write to the path where checkpoints are saved? What permissions does the training service account have? * **Model registry access controls:** Can you register a new model version without going through the standard validation pipeline? What credentials are needed and where are they stored? * **Container registry permissions:** Who can push to the registry? Are base images pinned to specific digests or floating tags? * **Dependency resolution:** Does the build process prefer public package indexes over internal ones? Are dependencies pinned to exact versions and hashes? * **Inference server exposure:** Is the management API of TorchServe, Triton, or similar tools exposed on any accessible interface? With or without authentication? * **Artifact integrity:** Are model hashes recorded at training time and verified at deployment time? Is there any signing in the pipeline? * **CI/CD secret exposure:** What credentials does the CI/CD pipeline hold? Can they be read from the pipeline configuration or logs? Model deployment tampering requires access to infrastructure rather than just the API — but that access is more commonly available than it looks. Compromised developer credentials, misconfigured CI/CD, exposed management APIs, and over-permissioned service accounts are all common findings. The difference is that in ML systems, the consequence of that access isn't just credential theft or lateral movement — it's a deployed model that actively misbehaves in ways that may be invisible for months.