Job Description
Job DescriptionDescription:
Hybrid Every Other Week onsite in Westlake, TX
Our client seeks a Data Engineer to build a modern data platform using Apache Iceberg on AWS with Spark, Kafka, and Python. The role emphasizes scalable data pipelines, distributed processing, and lakehouse best practices to support reliable analytics.
Due to client requirements, applicants must be willing and able to work on a w2 basis. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance.
Rate: $54.00 to $64.00/hr. w2
Responsibilities:
Responsibilities
- Design and build Apache Iceberg–based data lakes with ACID-compliant, versioned datasets.
- Implement Iceberg table evolution including schema evolution, partition specifications, and snapshot management.
- Develop best practices for Iceberg governance, metadata compaction, and performance tuning.
- Build scalable batch and streaming pipelines using AWS services such as S3, EMR, Glue, Lambda, and Step Functions.
- Develop ingestion and transformation workflows using Python, Spark, or Flink.
- Implement CDC pipelines using Kafka Connect or equivalent tooling.
- Ensure CI/CD integration with GitHub Actions or similar tooling.
- Design and operate Kafka-based streaming pipelines, including MSK.
- Build Kafka producers and consumers using Python or JVM languages.
- Implement topic partitioning, compaction, schema registry usage, and event versioning patterns.
- Design Iceberg-based data models for analytical and operational use cases.
- Implement automated data quality checks, validation rules, and anomaly detection.
- Build lineage, monitoring, alerting, and pipeline observability.
- Apply AWS security, cost optimization, and data governance best practices.
- Manage IAM, KMS, S3 lifecycle management, networking, and data encryption.
- Operationalize EMR/Glue jobs, containerized workloads, or serverless workloads.
- Partner with analytics, platform, and product teams to deliver high-quality data products.
- Participate in design reviews, architecture discussions, and roadmap planning.
- Mentor junior engineers and contribute to engineering standards.
Experience Requirements:
Experience Requirements
- 4 to 10+ years of experience in data engineering or similar roles.
- Hands-on experience with Apache Iceberg including table design, evolution, metadata management, and partitioning.
- Deep experience with AWS data stack including S3, EMR, Lambda, Glue, IAM, Step Functions, and CloudWatch.
- Proficiency with distributed engines such as Spark, Flink, or PySpark.
- Fluency in Python for data pipelines, automation, and APIs.
- Expertise building scalable ETL/ELT pipelines and real-time streaming architectures.
- Strong SQL and data modeling expertise.
- Kafka experience including producers/consumers, schema registry, and partitioning strategies (preferred).
Recruitment Transparency Notice Eliassen Group values transparency in our recruitment practices. Please be advised that Eliassen Group utilizes artificial intelligence (AI) tools as part of its initial application screening process. You may receive email and SMS notifications from the Eliassen Virtual Recruiting Team (noreply@eliassen.com, 781-808-2924) inviting you to complete a brief voice screening as part of your application process. These tools assist our hiring teams in different ways, including but not limited to, assistance in reviewing application materials to help identify candidates whose qualifications most closely match the requirements of the position. All AI-assisted evaluations and responses are reviewed by human recruiters before any hiring decisions are made. The use of AI in our process is intended to support fairness, efficiency, and consistency, and Eliassen Group takes measures to prevent bias or discrimination in connection with its hiring practices. By proceeding, you acknowledge, agree, and consent to Eliassen Group’s use of these tools, including AI tools, as part of the application and hiring process.Skills, experience, and other compensable factors will be considered when determining pay rate. The pay range provided in this posting reflects a W2 hourly rate; other employment options may be available that may result in pay outside of the provided range.
W2 employees of Eliassen Group who are regularly scheduled to work 30 or more hours per week are eligible for the following benefits: medical (choice of 3 plans), dental, vision, pre-tax accounts, other voluntary benefits including life and disability insurance, 401(k) with match, and sick time if required by law in the worked-in state/locality.
Please be advised- If anyone reaches out to you about an open position connected with Eliassen Group, please confirm that they have an Eliassen.com email address and never provide personal or financial information to anyone who is not clearly associated with Eliassen Group. If you have any indication of fraudulent activity, please contact InfoSec@eliassen.com.About Eliassen Group: Eliassen Group is a leading strategic consulting company for human-powered solutions. For over 30 years, Eliassen has helped thousands of companies reach further and achieve more with their technology solutions, financial, risk & compliance, and advisory solutions, and clinical solutions. With offices from coast to coast and throughout Europe, Eliassen provides a local community presence, balanced with international reach. Eliassen Group strives to positively impact the lives of their employees, clients, consultants, and the communities in which they operate. Eliassen Group is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status. Don’t miss out on our referral program! If we hire a candidate that you refer us to then you can be eligible for a $1,000 referral check!