Executive AI Solutions Architect/Engineer -- Federal/Civilian

Arch Systems

Washington, DC, USA

Published: 6/14/2022

Technology

Full Time

Job Description

Job Title: Executive AI Solutions Architect/Engineer — Federal/Civilian

Company: Arch Systems

Client: Federal

Location: Remote

Position Type: Full-Time

Job Summary

Arch Systems is hiring a hands-on Executive AI Solutions Architect/Engineer to design, build, and deliver secure, production-grade AI systems for federal civilian missions (HHS, DHS, USDA, NOAA, IRS). This is a player-coach role: you will code core components yourself, lead end-to-end solutioning (discovery → design → delivery → ATO), and present solutions to federal stakeholders, clearly articulating mission impact, benefits, and ROI.

Core Responsibilities

Hands-on engineering (player-coach): Write production Python; build FastAPI services; implement RAG pipelines (embeddings + vector DB); fine-tune/evaluate models; containerize (Docker) and deploy to Kubernetes with Terraform and GitHub/GitLab CI.
Time in code: Spend ≥50% of build phases actively coding, pairing, and conducting design/code reviews.
End-to-end solutioning: Lead discovery, fit-gap, and make/buy; define requirements and acceptance criteria; produce reference architectures (data → model → serving → security → observability) with trade-off analyses (cost/latency/accuracy/privacy).
Executive-ready presentations: Build decks and demos for federal audiences; clearly describe benefits and mission impact with KPIs (cycle-time reduction, precision/recall, cost per query, latency, compliance posture).
Capture & orals support: Contribute to RFIs/RFPs, technical volumes, and orals; handle Q&A with mixed technical/non-technical stakeholders.
MLOps implementation: Stand up MLflow/Databricks/SageMaker/Azure ML; manage model registry, feature store, CI/CD, gated promotions, canary/rollback; automate drift/bias/robustness monitoring and alerts.
Data engineering & governance: Build secure ingestion/curation pipelines; implement PII/PHI controls (de-identification, tokenization), data quality checks, catalogs/lineage, RBAC/least privilege, and data contracts.
GenAI/RAG safety & quality: Engineer prompt flows, grounding, guardrails, and policy enforcement; define offline/online evaluation with golden sets and human-in-the-loop; monitor factuality, relevance, toxicity; optimize context and token/latency budgets.
Security, compliance, ATO by design: Embed NIST 800-53/37 (RMF), 800-171, FedRAMP controls; implement authN/Z, encryption, secrets, logging/auditing, boundary protections; contribute to SSPs, POA&Ms, continuous monitoring; coordinate with ISSO/3PAO.
Observability & SRE: Instrument with logs/metrics/traces (OpenTelemetry); set SLIs/SLOs; build dashboards/alerts; run game days and blameless post-mortems; design HA/DR patterns to reduce MTTR and improve availability.
Performance & cost management: Model TCO; track cloud spend and cost/query; improve with right-sizing, autoscaling, caching, batching, distillation/quantization; present ROI trade-offs to stakeholders.
Reusable accelerators: Publish reference implementations, IaC modules, pipeline/evaluation templates, and security baselines; drive adoption and measure reuse impact across programs.
Delivery leadership & mentorship: Lead cross-functional agile teams; groom technical backlogs; ensure DoD (“definition of done”) includes tests, docs, security scans, and performance baselines; mentor engineers via pairing and reviews.
Documentation & change management: Maintain runbooks, ADRs, data contracts, and user guides meeting IV&V standards; manage versioning, release notes, stakeholder sign-offs; ensure Section 508 for end-user UIs/content.

Minimum Qualifications (Must-Have)

10+ years software/ML engineering; 5+ years architecting and shipping enterprise AI/ML solutions.
Recent, hands-on delivery with Python, PyTorch or TensorFlow, API development (FastAPI), containers (Docker), and microservices on Kubernetes.
Demonstrated GenAI/RAG delivery: embeddings, vector databases (FAISS/Milvus/pgvector), prompt orchestration/guardrails, evaluation and optimization.
Proven solutioning experience: leading discovery, estimating, architecture decisions, risk/cost trade-offs, and creating exec-ready artifacts that communicate impact and benefits.
Implemented MLOps on AWS and/or Azure using MLflow/Databricks/SageMaker/Azure ML; CI/CD for models and data pipelines; model registry; automated testing.
Federal delivery with working knowledge of FISMA/FedRAMP/RMF and ATO processes; ability to brief mixed technical/non-technical federal audiences.
Excellent communication/storytelling with quantitative framing (KPIs, SLAs/SLOs, ROI).

Preferred Qualifications

Public Trust or clearance eligibility; prior support within HHS, DHS, USDA, NOAA, or IRS.
Certifications: AWS Solutions Architect, Azure Solutions Architect/DP-203/DP-100; security/architecture (CISSP, CCSP, TOGAF) a plus.
Experience with model risk management, adversarial testing/red teaming, and Section 508 evaluation practices.
Technical oversight is a plus (while remaining hands-on).

Company DescriptionHIGH Growth, small business. Check us out at https://archsystemsinc.com/

Arch Systems LLC is committed to diversity in its workforce and is proud to be an equal opportunity employer. Arch Systems LLC considers qualified applicants without regard to race, color, religion, creed, gender, national origin, age, disability, veteran status, marital status, pregnancy, sex, gender expression or identity, sexual orientation, or any other legally protected class. Arch Systems LLC is an Affirmative Action and Equal Opportunity Employer.

Company Description

HIGH Growth, small business. Check us out at https://archsystemsinc.com/\r\n\r\nArch Systems LLC is committed to diversity in its workforce and is proud to be an equal opportunity employer. Arch Systems LLC considers qualified applicants without regard to race, color, religion, creed, gender, national origin, age, disability, veteran status, marital status, pregnancy, sex, gender expression or identity, sexual orientation, or any other legally protected class. Arch Systems LLC is an Affirmative Action and Equal Opportunity Employer.