Section 4 Quiz

--- shuffle_answers: true shuffle_questions: false --- ## An AI team downloads a popular open-source model from Hugging Face and loads it directly into production without any verification. The model file is in pickle format. What Layer 2 control was skipped, and what is the primary risk? > Hint: Consider what pickle-format model files can contain beyond model weights. - [ ] Container scanning was skipped -- the risk is that the container image has known CVEs > Container scanning addresses vulnerabilities in the container environment, not the model artifact itself. The primary risk here is with the model file format, not the container. - [ ] Provenance tracking was skipped -- the risk is not knowing who created the model > While provenance tracking is important, the immediate and most dangerous risk with an unverified pickle file is not attribution but code execution. - [x] Model integrity verification was skipped -- pickle files can execute arbitrary code during deserialization, so an unverified pickle model could contain hidden payloads including reverse shells, credential harvesters, or cryptocurrency miners > Correct! This directly maps to the Hugging Face malicious model case from the section. Pickle format can embed executable code that runs when the model is loaded. Without hash verification against published checksums and format enforcement (requiring safetensors where available), the team is loading potentially weaponized code into their production environment. LLM03: Supply Chain and LLM04: Data and Model Poisoning both apply. - [ ] SBOM generation was skipped -- the risk is not tracking the model's dependencies > SBOM tracks dependencies in the serving container, not exploits in the model file itself. The immediate risk is code execution from the pickle deserialization, not dependency tracking. ## The container security pipeline has four security gates: Hash Verification, CVE + Malware Scan, Image Signing, and Runtime Policy. A model artifact passes the first gate (hash verified against publisher's checksum) but fails the second gate (malware detected in serialized weights). What should happen? > Hint: Think about what each gate does and why the pipeline blocks progression. - [ ] The artifact should proceed to the third gate because the hash was verified, confirming integrity > Hash verification confirms the artifact hasn't been modified since publishing, but the publisher could have included malicious content intentionally or unknowingly. Each gate tests a different security property. - [ ] The security team should override the malware detection because the hash verification proves the file is legitimate > A verified hash only proves the file matches what the publisher intended to distribute. If the publisher distributed a malicious model (intentionally or through their own compromise), the hash would still verify correctly. The malware scan gate exists to catch exactly this scenario. - [x] The pipeline blocks -- the artifact does not proceed to Image Signing or Deployment. The malware finding is investigated before any further action > Correct! A failure at any security gate blocks progression to the next stage. The malware scan detected malicious content in the serialized weights, which could be a backdoor, credential harvester, or other payload. The hash being verified simply means the file is the same file the publisher distributed -- it does not mean the file is safe. This is exactly how the container security pipeline prevents malicious models from reaching production. - [ ] The artifact should be re-scanned with a different tool because the malware detection is likely a false positive > While false positives do occur, the correct response is to block progression and investigate, not to automatically assume false positive and re-scan. The pipeline's design prioritizes security -- investigation happens outside the deployment flow. ## Why does the section recommend generating a Software Bill of Materials (SBOM) for every model serving container? > Hint: Think about what happened in the Ultralytics supply chain attack mentioned in the section. - [ ] SBOMs are required by law for all software containers > While some regulations encourage SBOMs, the recommendation is based on security practice, not legal mandate. The SBOM enables rapid response to supply chain incidents. - [x] SBOMs track all dependencies in the container, enabling rapid identification of affected deployments when a supply chain vulnerability is discovered -- as in the Ultralytics attack where a cryptocurrency miner was injected into a popular package > Correct! When a dependency is compromised (like the Ultralytics supply chain attack), organizations need to immediately identify every deployment that includes that dependency. An SBOM provides this inventory, enabling rapid response. Without SBOMs, the security team must manually investigate each container to determine if it's affected -- a process that can take days while compromised containers continue running. - [ ] SBOMs prevent dependency confusion attacks by locking package names > SBOMs document what's installed but don't directly prevent dependency confusion. Dependency pinning with verified hashes prevents confusion attacks. SBOMs help with identification and response after a compromise. - [ ] SBOMs are only needed for containers with GPU passthrough because those have more dependencies > SBOMs are recommended for all model serving containers regardless of GPU configuration. The dependency tracking benefit applies to any container with a complex dependency tree. ## What is the key difference between model signing and hash verification as Layer 2 controls? > Hint: Consider what each technique proves about the model artifact. - [ ] Model signing uses encryption while hash verification uses hashing -- they serve the same purpose > Model signing uses cryptographic signatures (which do involve cryptographic operations) but serves a different purpose than hash verification. They answer different questions about the artifact. - [ ] Hash verification is more secure than model signing because hashes cannot be forged > Both techniques have appropriate security levels for their purposes. The distinction is about what they verify, not which is "more secure." - [x] Hash verification confirms the artifact hasn't been modified (integrity), while model signing also confirms who created the artifact (provenance) through cryptographic identity > Correct! Hash verification answers "Is this the same file that was published?" -- it confirms integrity. Model signing answers both "Is this the same file?" AND "Who published it?" -- it confirms both integrity and provenance through a cryptographic signature tied to the creator's identity. If a malicious actor publishes a poisoned model with a valid hash, hash verification alone would pass. Model signing would fail because the signature wouldn't match the expected creator. - [ ] Model signing is only used for proprietary models while hash verification is used for open-source models > Both techniques can be applied to any model regardless of licensing. Frameworks like Sigstore's cosign work for both proprietary and open-source model artifacts. ## An organization's approved model catalog requires security review before any model can be used in production. A developer bypasses this by downloading a model directly from a public hub onto their development machine and deploying it to a test endpoint. What Layer 2 principle is violated? > Hint: Consider the purpose of maintaining an approved model catalog. - [ ] Container security -- the model wasn't packaged in a scanned container > While the model should eventually be containerized and scanned, the primary violation is bypassing the approved catalog, not the container security gate specifically. - [x] Supply chain governance -- the approved model catalog exists to ensure only vetted, scanned models reach production. Direct downloads from public hubs bypass the scanning and review pipeline > Correct! The approved model catalog is a supply chain governance control. It ensures that every model used in the organization has been scanned for vulnerabilities, verified for integrity, and reviewed for safety. Bypassing it -- even for a "test" deployment -- creates the exact supply chain risk that Layer 2 is designed to prevent. Test endpoints that use unvetted models can be exploited just as easily as production endpoints. - [ ] SBOM generation -- the model's dependencies aren't tracked > SBOM generation is a downstream concern. The fundamental issue is that the model entered the environment without going through the approved security review process. - [ ] Provenance tracking -- the model's origin is unknown > While provenance is relevant, the broader principle violated is supply chain governance. The developer bypassed the entire vetting process, not just the provenance tracking step. ## AI model vulnerability scanning includes five scan types: serialization safety, weight integrity, adversarial robustness, alignment verification, and data leakage potential. Which scan type would detect that a fine-tuned model has been poisoned to degrade safety alignment on certain topics? > Hint: Think about what "degraded safety alignment" means in terms of model behavior. - [ ] Serialization safety -- checking for embedded code in model files > Serialization safety checks for executable code in model file formats (pickle exploits). It doesn't assess the model's behavioral safety alignment. - [ ] Weight integrity -- comparing weights against known-good hashes > Weight integrity verification detects tampering with model files, but a model that was deliberately fine-tuned with a poisoned dataset would have "correct" weights for its (compromised) training. The hash would verify against the fine-tuned checkpoint. - [x] Alignment verification -- checking model outputs for safety policy compliance catches models with degraded safety alignment from malicious fine-tuning > Correct! Alignment verification tests whether the model's outputs comply with safety policies. A model poisoned through malicious fine-tuning might pass standard benchmarks on most topics but produce unsafe outputs on the targeted topics. Alignment verification specifically tests for this degradation by probing the model's responses across safety-relevant domains. - [ ] Data leakage potential -- checking for memorized training data in outputs > Data leakage testing checks whether the model reproduces sensitive training data. While related, it doesn't specifically test whether safety alignment has been degraded on certain topics. ## The section recommends scanning models "on acquisition, post-training, pre-deployment, and through periodic re-assessment." Why is periodic re-assessment necessary even for models that passed all scans at deployment? > Hint: Think about what changes after a model is deployed that could affect its vulnerability profile. - [ ] Models degrade over time and become less accurate, requiring re-assessment > Model accuracy degradation (drift) is a quality concern, not a security scanning concern. Re-assessment is about newly discovered vulnerability patterns, not accuracy decay. - [ ] Periodic re-assessment is a regulatory requirement for all AI models > While some regulations require ongoing assessment, the technical reason is that new vulnerability patterns are continuously discovered. - [x] New vulnerability patterns are continuously discovered -- attack techniques that didn't exist at deployment time may expose weaknesses that the original scan didn't test for > Correct! The AI security landscape evolves rapidly. New prompt injection techniques, novel jailbreak methods, and previously unknown adversarial patterns are discovered regularly. A model that passed all scans at deployment may be vulnerable to techniques that were discovered after it went live. Periodic re-assessment with an updated vulnerability library catches these emerging risks. - [ ] Models change their own weights over time through inference, so they need re-scanning > Standard inference does not modify model weights. Models remain static unless explicitly retrained or fine-tuned. The need for re-assessment comes from the evolving threat landscape, not from model self-modification.