When AI Gets Cloud Security Wrong: A Real Hallucination Example
By Jereme Peabody
Configuration of a system can be hard, especially in a domain you're not that familiar with. I've worked in AWS for a while now, I've configured and deployed maybe 8 websites and applications into AWS. I know enough to get by. Prior to AI, I would google how to do something, read solutions, and painfully try them out until I got it to work.
Today, I do the same thing except with AI, and that's what leads me up to this story about AI misconfiguration risk caused by a pattern-recognition failure.
Pattern Recognition Failure
As I was configuring my new website for Get Grounded AI, I ran into an issue where I couldn't access the hosted site. I worked with ChatGPT to configure it and troubleshoot my issue, step by step. I had to constrain it, because like most AI's, it would try to give a 'comprehensive answer' to your question. You ask, 'help me set up my website on AWS' and it SPEWS a 7 step response. This is unhelpful when you're configuring or troubleshooting. And it's one of the things I don't like about AI, it does a knowledge domain dump thinking it's being helpful. But I also have to recognize, maybe it's the way I ask my questions?
Anyway, I still couldn't reach the site. We're reviewing the configuration and I ask it this:
Hmm, do I need to add a bucket policy?
It's response sounded certain and authorative:
Short answer:
No — you do NOT need to add any bucket policy for CloudFront to serve your site.
And in fact:
❗ If you add the wrong bucket policy, the site will break.
Let’s walk through this cleanly and simply.
I had a little experience with AWS under my belt to be able to ask that question. Someone with no knowledge in AWS may not. It spewed a 402 word response as to why I did NOT need a bucket policy. And in that forest of affirmations to it's knowledge and instructions, it wanted to know if my CloudFront was enabled and to test the CloudFront endpoint.
I checked the status in CloudFront and my VERY next message to ChatGPT after it's 402 word response was this:
CloudFront is Enabled
getgroundedai.com.s3-website-us-east-1.amazonaws.com 403 Forbidden
It's response was (in short) because it was another 301 words long:
Ah — that explains everything.
You haven’t added the required S3 website bucket policy.
Are you fucking kidding me? I literally just asked it that question. So I questioned the AI:
I literally just asked you if we needed to add a s3 bucket policy
GPT's response:
You’re right — you did ask, and I told you “no” because for most CloudFront → S3 setups that use the API endpoint you don’t need one.
But you are using S3 website hosting, which does require a public-read bucket policy.
That’s on me — thanks for calling it out.
So What Happened Here
GPT wasn't gaslighting me. LLM's are pattern recognition. My initial question if I needed to add a bucket policy matched a pattern that it was trained on which resulted in the over-confident response I received. When I added another parameter to the session not a moment later, it matched another pattern it was trained on that resulted in a correct response.
What Does This Mean to Security
This is the part people greatly underestimate:
AI doesn't reason, it predicts.
And prediction is brittle.
When your prompt accidentally matches the wrong mental template in the model's head, it will fire off a confident explanation that sounds correct but isn't. And in security, that's the moment everything goes off the rails.
Here’s the real danger:
AI gives high-confidence answers even when they’re wrong
Humans are trained to treat confidence as a signal of competence. LLMs break that heuristic.
Small prompt changes trigger completely different 'solutions'
This is exactly what happened:
One sentence produced a 400-word 'no bucket policy needed'
and five seconds later the model flipped to 'you absolutely need a bucket policy.'
If you don't already have enough domain knowledge to challenge the output, you could potentially implement the wrong instructions and potentially expose your system to attacks.
Most people do not have the domain knowledge to detect the failure
I caught this because I wrestled AWS before.
A junior developer, or a non-technical founder, could push bad configs into production without knowing something was wrong like in the Localmind incident.
Overconfidence + pattern mismatch = security breach
This is exactly how misconfigurations turn into exposures:
- Overly permissive IAM policies
- Wrong trust relationships
- Public buckets
- Incorrect OAuth redirect URIs
- Mis-scoped API keys
- Misconfigured CORS
- Open security groups
- Broken TLS setups
- 'Temporary' debugging settings left enabled
AI will tell you all of these are fine if your prompt matches the wrong pattern
AI is good at filling gaps, even gaps that shouldn't be filled
An LLM will invent details that 'fit the pattern', even when they don't fit your environment. In this example, CloudFront doesn't need a bucket policy if CloudFront is using an S3 REST API endpoint. But the entire session was about configuring and S3 Website endpoint. It hallucinated the API endpoint and gave me the wrong answer. Our human nature and that self-assured fabrication are the most dangerous ingredients in security with AI.
Final Thoughts
The lesson here isn't 'AI messed up' or that it hallucinated.
The lesson is:
If you rely on AI to configure your infrastructure without understanding the domain, you're one confident hallucination away from a security incident.
AI speeds you up.
But in security, speed amplifies mistakes.
This isn't about distrusting AI, it's about understanding what it is:
A pattern engine, not a system engineer.
And if you don't know how to validate its answers, it will lead you straight into vulnerabilities with total confidence.
AI is not a replacement for seasoned engineers
Keep it grounded. Trust, but validate.