Work
Security ML at petabyte scale
Microsoft
- Role
- Software Engineer II
- Team
- M365 Security Detections
- When
- 2021 -- 2025
Four years on the petabyte-scale platform that detects cyber-attacks across all of M365 in near real time. Lead engineer for Microsoft's first mail anomaly detection and for the GPT-based detection assistant.
What I did
- Petabyte-scale big-data platform using ML to detect cyber-attacks in near real time across all of M365.
- Lead engineer for Microsoft's first mail-anomaly detection (response to the nation-state incident where hackers accessed executive and federal emails).
- Lead engineer on a GPT-based security-detection assistant that analyses machine and identity activity for attack patterns.
- Designed the production workspace that enables data-science and LLM-backed detections in an agile PySpark environment with CI/CD and realtime job monitoring.
- Expedited detection programming by over 75% on distributed systems; auto-failover geo-replicated BCDR architecture.
Full write-up in progress. The bullets above are the short version.
Under the hood
Stack.
Security + anomaly ML
- Mail-anomaly detection for M365
- GPT-based security-detection assistant
- Random-forest entity detections
- Petabyte-scale event processing
Distributed systems
- Spark, PySpark, Azure Functions
- CI/CD, real-time job monitoring
- Auto-failover geo-replicated BCDR