W2 - Lead Splunk Engineer (Cloud Observability)

Job Description

Job Title: Lead Splunk Engineer (Cloud Observability)

Location: St Paul metro, Minneapolis, MN (Hybrid)

Position Type: Full time

Job Description:

This role joins a fast-growing Observability Engineering group responsible for advancing enterprise-wide insight, performance visibility, and reliability capabilities. The team is modernizing and expanding how the organization uses Splunk—elevating everything from log analytics to distributed tracing and SIEM tooling, enabling data-driven problem solving at scale.

As a Lead Splunk Engineer, you will serve as the team’s specialist and key contributor for Splunk architecture, engineering, automation, optimization, and overall platform evolution. You’ll collaborate closely with platform, cloud, site reliability, and application engineering teams to improve observability maturity across the enterprise. This is a hands-on role for someone who loves building, tuning, automating, and scaling Splunk in complex environments.

WHAT MAKES THIS ROLE DIFFERENT:

• Splunk is the star. This is not a generic cloud or DevOps job — the organization is explicitly seeking a specialist who deeply understands Splunk internals, scaling patterns, data ingestion, and how to turn observability into an engineering multiplier.
• High-impact modernization work. The team is maturing toward a fully integrated observability ecosystem. You’ll influence architecture, automation, tool selection, and long-term strategy.
• Engineering excellence culture. The environment values thoughtful automation, coding ability, infrastructure knowledge, and measurable improvements.
• Large-scale, meaningful problems. The observability footprint directly supports critical healthcare operations, giving your work real-world consequences and visibility.
• Growth + mentorship. This position is senior enough to mentor others, guide decisions, and shape platform evolution without requiring formal people management.

KEY SKILLS:

• Expert-level Splunk engineering experience, including:
• Designing, deploying, and scaling Splunk environments
• Indexer clustering, search head clustering, forwarder architecture
• Deep knowledge of data ingestion pipelines, props/transforms, normalization, dashboards, and performance tuning
• Strong experience troubleshooting ingestion, indexing, search performance, and data model issues
• Strong experience building automation around Splunk and infrastructure (e.g., Ansible, Chef, Terraform, or similar)
• Demonstrated ability to write high-quality code for automation, tooling, integrations, and pipelines
• Experience running and supporting high-availability, distributed systems
• Comfort working across Linux, networking, storage, and VM/container environments
• Hands-on experience with public cloud concepts (AWS, Azure, or GCP) as they relate to observability and automation
• Ability to operate in an Agile environment, communicate technical concepts clearly, and drive continuous improvement
• Participation in on-call rotations and troubleshooting across multiple environments

NICE TO HAVE REQUIREMENTS:

• Experience with Prometheus, Grafana, OpenTelemetry, or other complementary observability technologies
• Exposure to Kubernetes, OpenShift, or container-based deployment models
• Understanding of Kafka, event streaming, or log transport optimization
• Knowledge of CI/CD tools (Bitbucket, GitLab, Jenkins, etc.)
• Familiarity with modern architectural patterns such as microservices or service mesh
• Experience mentoring junior engineers or influencing platform direction

Thanks!