Senior Observability Engineer

Job Description

Job Description

Job Title: Senior Observability Engineer

Location: Holmdel, NJ (2–3 days onsite weekly). We will only consider local candidates. No remote or relocation candidates. Current location must be noted on the resume.

Main Skills: Observability Engineer experienced with Splunk working with Cloud and DevOps

Work Authorization: Candidates must be legally authorized to work in the United States without employer sponsorship, now or in the future. This is a contract to hire role.

· Seeking Observability Engineer

· They have been receiving Cloud Engineer and DevOps engineers.

· Observability is needed first, but with cloud and DevOps experience.

· Main Skill is Splunk. Must be a Splunk expert. SME level.

· Will mostly work with Splunk.

· Will also work with AppDynamics, Zenos, and Open Telemetry. However, they can be trained on those tools.

· Job is 3 days onsite

· Will implement, manage, and develop

· Must know Bash and Python scripting

· 1st interview virtual, 2nd interview onsite

Job Description:

· We are seeking a dedicated and detail-oriented Senior Observability Engineer to join our Enterprise Observability Engineering team. The ideal candidate will bring deep expertise in AppDynamics, Splunk, Open Telemetry, and AWS-native services, along with strong DevOps experience.

· This role is responsible for the administration, configuration, implementation, and ongoing optimization of observability platforms that enable end-to-end visibility across applications, infrastructure, and cloud-native workloads. You will play a critical role in ensuring platform reliability, performance, and actionable insights to support engineering and business teams.

Job Responsibilities:

Observability Platform Administration & Implementation

· Administer, configure, and support AppDynamics, Splunk, and Open Telemetry (OTel) platforms to meet enterprise monitoring and observability needs.

· Design and implement observability solutions aligned to MELT (Metrics, Events, Logs, Traces) best practices.

· Perform regular upgrades, patching, and security hardening of observability platforms.

Monitoring, Reliability & Maintenance

· Continuously monitor the health, availability, and performance of observability platforms.

· Ensure data integrity, retention, and availability across metrics, logs, and traces.

· Proactively identify and remediate platform performance, scalability, and reliability issues.

Cloud & Full-Stack Observability

· Implement and support observability for AWS services, including:

· EKS, ECS, Lambda Functions

· SNS/SQS, S3, CloudWatch

Deliver full-stack observability, including:

· -Kubernetes cluster and workload metrics

· -Service discovery, events, and application performance data

Leverage Open Telemetry for instrumentation, context propagation, collectors, and sampling strategies.

Dashboard, Alerting & Reporting

· Create and maintain dashboards, reports, and alerts in AppDynamics and Splunk.

· Collaborate with application, platform, and DevOps teams to define meaningful monitoring and alerting standards.

· Reduce noise through alert tuning and promote actionable signal over raw data.

DevOps & Automation

· Integrate observability into CI/CD pipelines using GitHub, Jenkins, Argo CD, and automation frameworks.

· Develop scripts and automation using Python, JavaScript, or Bash to streamline onboarding, configuration, and maintenance activities.

User Support, Enablement & Training

· Provide tier-2/3 support for observability-related issues.

· Assist internal teams with troubleshooting, root cause analysis, and performance investigations.

· Develop and deliver training materials and knowledge sessions to improve tool adoption and effective usage.

Documentation & Best Practices

· Maintain comprehensive documentation for:

· platform configurations

· onboarding procedures

· operational runbooks and standards

· Define and enforce observability best practices across the organization.

Incident Response & Collaboration

· Partner with IT, SRE, and DevOps teams to ensure comprehensive monitoring coverage.

· Participate in incident response efforts, leveraging observability data to accelerate detection, diagnosis, and resolution.

Required Qualifications:

Education:

Bachelor’s degree in Computer Science, Information Technology, or a related field.

· 5–7+ years of experience in Observability, Monitoring, SRE, or Platform Engineering roles.

· Proven hands-on experience implementing, managing, and maintaining AppDynamics, Splunk, and Open Telemetry in enterprise environments.

Technical Skills:

· Observability Platforms:

· AppDynamics (APM, dashboards, alerts)

· Splunk (configuration, administration, data onboarding)

· Open Telemetry (instrumentation, collectors, sampling)

AWS Cloud Services:

· EKS, ECS, Lambda

· SNS/SQS, S3, CloudWatch

DevOps & CI/CD:

· GitHub, Jenkins, Argo CD

· CI/CD pipelines and Git Ops practices

Observability Concepts:

· Strong expertise in Metrics, Events, Logs, and Traces (MELT)

· Full-stack and cloud-native observability

· Automation & Scripting:

· Python, JavaScript, Bash

Infrastructure Knowledge:

· Strong understanding of IT infrastructure, applications, and networking

Soft Skills:

· Excellent problem-solving and analytical skills

· Strong communication and collaboration abilities

· Ability to work independently as well as in a team-oriented environment

· Detail-oriented with a strong focus on operational excellence