Overview
This project builds a fully automated threat detection pipeline inside AWS. It ingests VPC Flow Logs, trains unsupervised ML models to learn normal network behavior, and flags anomalies in real time — all deployed through infrastructure as code.
The Problem
Traditional rule-based security monitoring can't keep up with evolving attack patterns. Manually writing detection rules for every possible threat vector doesn't scale, and legitimate traffic patterns shift constantly. Organizations need detection systems that learn and adapt without constant human tuning.
My Approach
I framed this as an unsupervised anomaly detection problem. Rather than training a classifier on labeled attack data (which is hard to get and biases toward known attacks), I trained models on what 'normal' network traffic looks like, then flagged anything that deviated significantly. I used CloudGoat to simulate realistic red-team scenarios for testing.
Technical Implementation
The pipeline runs in a secure AWS VPC. I used SageMaker to train both an Isolation Forest and an Autoencoder on features extracted from VPC Flow Logs. The detection pipeline triggers via Lambda when new logs arrive, runs inference, and sends alerts through SNS when anomalies are detected. Results are visualized through a custom Streamlit dashboard. The entire infrastructure is defined in Terraform for reproducibility.
Results
The system successfully detects simulated attack patterns from CloudGoat scenarios including unusual port scanning, data exfiltration attempts, and lateral movement. The Autoencoder outperformed Isolation Forest on subtle anomalies, while Isolation Forest was faster for real-time scoring. Full end-to-end deployment is reproducible via a single Terraform apply.
What I Learned
This project taught me how to think about ML in a security context — where false positives have real operational cost and false negatives have real risk. I also learned a ton about AWS networking, VPC architecture, and how infrastructure-as-code changes the way you think about deploying ML systems.
