Design and implement agent evaluation pipelines that benchmark AI capabilities across real-world enterprise use cases
Build domain-specific benchmarks for product support, engineering ops, GTM insights, and other verticals relevant to modern SaaS
Develop performance benchmarks that measure and optimize latency, safety, cost-efficiency, and user-perceived quality
Create search- and retrieval-oriented benchmarks, including multilingual query handling, annotation-aware scoring, and context relevance
Partner with AI and infra teams to instrument models and agents with detailed telemetry for outcome-based evaluation (Member of Technical Staff : AI Performance1)
Drive human-in-the-loop and programmatic testing methodologies for fuzzy metrics like helpfulness, intent alignment, and resolution effectiveness
Contribute to company’s open evaluation tooling and benchmarking frameworks, shaping how the broader ecosystem thinks about SaaS AI performance
Requirements
3 to 7 years of experience in systems, infra, or performance engineering roles with strong ownership of metrics and benchmarking
Fluency in Python and comfort working across full-stack and backend services
Experience building or using LLMs, vector-based search, or agentic frameworks in production environments
Familiarity with LLM model serving infrastructure (e.g., vLLM, Triton, Ray, or custom -Kubernetes-based deployments), including observability, autoscaling, and token streaming
Experience working with model tuning workflows, including prompt engineering, fine-tuning (e.g., LoRA, DPO, or evaluation loops for post-training optimization
Deep appreciation for measuring what matters — whether itʼs latency under load, degradation in retrieval precision, or regression in AI output quality
Familiarity with evaluation techniques in NLP, information retrieval, or human-centered AI (e.g. RAGAS, Recall@K, BLEU, etc.)
Strong product and user intuition — you care about what the benchmark represents, not just what it measures
Advantageous Skills :
Experience contributing to academic or open-source benchmarking projects
Create a job alert for this search
Senior Engineer • Cebu City, Cebu, PH
Related jobs
Promoted
Machine Learning Engineer
IndivdCebu City, Central Visayas, Philippines
At Indivd, we’ve built a patented, AI-driven platform that helps retailers, shopping malls, and public spaces gain deep insights into customer behavior without compromising privacy.Using advanced c...Show moreLast updated: 30+ days ago
Promoted
Senior Software Engineer
ALEPH TALENT SOLUTIONS CORPORATIONCebu City, Central Visayas, Philippines
The role of Software Engineer is responsible for developing and delivering software products to meet both functional and technical requirements.
This person will also act as a mentor and coach to ju...Show moreLast updated: 30+ days ago
Promoted
New!
Senior Product Manager - AI AdOps Copilot (REMOTE)
Monetize More IncMandaue, Central Visayas, Philippines
MonetizeMore is a global leader in ad tech, providing solutions that help publishers maximize their ad revenue while maintaining transparency, user trust, and brand safety.We are committed to shapi...Show moreLast updated: 12 hours ago
QA Automation Engineer (Remote)
LegalMatch.comCebu City, Central Visayas, PH
Quick Apply
As a QA Automation Engineer, your primary responsibility is to ensure the quality of our software products by developing, executing, and maintaining automated tests.
You will work closely with the d...Show moreLast updated: 30+ days ago
Data Engineer
Cobden & Carter InternationalCebu City, Cebu, Philippines
Quick Apply
An offshoring company is looking for a Data Engineer.Design, build, and manage the complete orchestration framework for all data pipelines using Google Cloud Composer.
Author complex, dynamic, and m...Show moreLast updated: 30+ days ago
Senior QA Automation Lead (Remote)
LegalMatch.comCebu City, Central Visayas, PH
Quick Apply
As a Senior QA Automation Lead, you will lead the quality assurance strategy across teams to ensure the reliability, performance, and scalability of our web and software products.You will mentor QA...Show moreLast updated: 30+ days ago
Senior Mobile App Developer (Flutter / Cross-Platform)
Full ScaleCebu, CEB, ph
Quick Apply
Join one of the Philippines' fastest-growing tech companies! Open to Philippine-based candidates only!.We are seeking a highly skilled Senior Mobile App Developer with 4+ years of experience in cro...Show moreLast updated: 1 day ago
Senior Software Engineer
Cobden & Carter InternationalCebu City, Cebu, Philippines
Quick Apply
Our client is an outsourcing and offshoring consulting firm.Collaborate with the engineering team to design, build, and maintain APIs, UIs, services, and systems.
Work with the management team and p...Show moreLast updated: 30+ days ago
Sr. PPC and AI Marketing Analyst (Remote)
LegalMatch.comCebu City, Central Visayas, PH
Quick Apply
We are seeking an experienced Sr.PPC & AI Marketing Analyst to lead and optimize our paid media campaigns and leverage AI-driven marketing strategies.
In this role, you will manage multi-channel...Show moreLast updated: 30+ days ago
Are you an experienced QA / QC Engineer looking for an exciting opportunity in the Gulf? Join a reputed transmission line project with excellent benefits and long-term career growth.Transmission Line...Show moreLast updated: 30+ days ago
ISTA Personnel SolutionsCebu City, Central Visayas, Philippines
STA Personnel Solutions South Africa - we are a global Business Process Outsourcing (BPO) company, partnering with a USA Client in the Healthcare Industry and are in search of a Machine Learning De...Show moreLast updated: 18 hours ago
Promoted
Java EE Developer (Mid to Senior Level)
Stark Asia Solutions Inc.Cebu City, Central Visayas, Philippines
Java Enterprise Edition (Java EE / Jakarta EE).As part of our Agile squads, you will contribute to the design, development, and delivery of enterprise-grade applications while mentoring junior deve...Show moreLast updated: 11 days ago
Promoted
Executive Partner
Athena LabsTalisay, Central Visayas, Philippines
Athena is on a mission to improve how people work and live.Our membership helps growth-motivated leaders work smarter, live better, and achieve more with a personalized support team backed by the l...Show moreLast updated: 30+ days ago
Promoted
Senior Software Engineer I - Python and AI or ML
UnitedHealth GroupCebu City, Central Visayas, Philippines
Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives.The work you do with our team will directly improve health outcomes by connect...Show moreLast updated: 30+ days ago
Programmatic SEO Engineer
RM Staffing B.V.Cebu, 08, PH
We’re looking for a technically skilled.SEO landing pages intelligently and efficiently.You’ll be responsible for building dynamic content systems, using APIs and templates to generate high-value, ...Show moreLast updated: 22 days ago
Sr. Active Directory Engineer
HrtxCebu City, Cebu, Philippines
Quick Apply
We are seeking a highly skilled and experienced.Senior Active Directory Engineer.Entra ID (formerly Azure Active Directory).
You will play a key role in designing, implementing, and maintaining secu...Show moreLast updated: 30+ days ago
Promoted
Senior Software Engineer
AMCS GroupCebu City, Central Visayas, Philippines
We are looking for experienced Software Developers experienced in C#.NET to join our Environmental, Social, Governance (ESG) development team in the Philippines.
Sustainability software specialist, ...Show moreLast updated: 30+ days ago
QA Automation Engineer
Full ScaleCEBU CITY, CEB, ph
Quick Apply
Full Scale is a tech services company that helps businesses build dedicated teams of skilled software engineers.We make finding and retaining experienced software talent easy and affordable.We are ...Show moreLast updated: 29 days ago