GenAI/LLM Prompt Compliance & Evaluation Consultant Job at US Tech Solutions, Washington DC

RzNBWm1WbUp5MXdSMTFpcm9FZ3lOWUJKWGc9PQ==
  • US Tech Solutions
  • Washington DC

Job Description

Duration: 6 months contract

Job Description:

  • The Responsible AI Scaled Testing Team within Trust & Safety performs pre-launch structured testing for AI applications against safety, fairness and neutrality policies and standards.
  • It is a global team with Responsible AI domain expertise and diverse backgrounds in operations, strategy, ethics, risk management, product management, and program management.

Responsibilities:

  • Lead the end-to-end technical assessment of GenAI products, focusing on pre-launch testing of safety, neutrality, and fairness.
  • Develop and implement rigorous testing methodologies, including automated prompt generation and response analysis, to ensure compliance with defined standards.
  • Leverage data-driven insights to identify potential risks and inform product development iterations, ensuring robust and reliable GenAI deployments.
  • Automated Testing and Data Pipeline Management: Design and implement automated prompt generation strategies and data analysis solutions to efficiently collect and analyze GenAI model responses. Manage and optimize data pipelines for efficient processing and analysis of large datasets.
  • Quantitative and Qualitative Analysis of Model Behavior: Conduct in-depth statistical analysis and qualitative evaluations of model outputs to identify deviations from defined standards. Develop and apply metrics for evaluating safety, neutrality, and fairness, and generate detailed reports with actionable insights.
  • Technical Guideline Development and Execution: Translate abstract safety, neutrality, and fairness standards into precise technical guidelines and evaluation criteria. Develop and maintain clear documentation for vendor teams, including detailed instructions, edge case clarifications, and quality calibration protocols.

Experience:

  • 4+ years of experience in data analysis, AI/ML testing, cybersecurity, or related technical domains.
  • Proficiency in data analysis tools and languages (e.g., SQL, Python, R) for processing and analyzing large datasets.
  • Experience developing and implementing automated testing frameworks.
  • Strong analytical and problem-solving skills, with the ability to interpret complex data and identify patterns.
  • Excellent technical communication skills, with the ability to clearly articulate complex technical concepts to both technical and non-technical audiences.
  • This role may be exposed to graphic, controversial, and/or upsetting content.

Skills:

  • 2+ years of experience in AI testing, adversarial testing, red teaming, or related areas (Nice to have).
  • Experience with LLM-based prompt generation and evaluation tools (Nice to have).
  • Familiarity with machine learning concepts and algorithms (Nice to have).
  • Experience with bug tracking systems (Nice to have).
  • Experience in developing and maintaining technical documentation (Nice to have).
  • Experience in defining and implementing process improvements (Nice to have).
  • Ability to think strategically about emerging AI threats and vulnerabilities (Nice to have).

Education:

  • Bachelor's degree in Computer Science, Data Science, Statistics, or a related technical field, or equivalent practical experience.

About US Tech Solutions:

US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit .

US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

Recruiter Details:

Name: Ashik

Email: ashik.a@ustechsolutionsinc.com

Internal Id: 25-35043

Job Tags

Contract work,

Similar Jobs

Taurus Industrial Group, LLC

Journeyman Electrician Job at Taurus Industrial Group, LLC

 ...We are seeking a skilled and experienced Journeyperson Electrician to join our team. The ideal candidate will have a strong background...  ...Qualifications Completion of an apprenticeship program or equivalent experience as an electrician. Proficiency in blueprint reading and... 

Level Up USA

Customer Service Representative Job at Level Up USA

 ...Customer Success Specialist Cincinnati, OH. Our team is looking for a dynamic and excited...  ..., and support related to our products/services. The primary goal of this role is to ensure...  ...punctuality and adherence to assigned work schedules to ensure optimal coverage for... 

NYC Health + Hospitals/Harlem

CRNA Job at NYC Health + Hospitals/Harlem

 ...Description Physician Affiliate Group of New York (PAGNY) is adding a Part-timeCRNA to the group at NYC Health + Hospitals/ Harlem. There are ample leadership opportunities in our growing practice, and although we don't handle heart surgeries or transplants, we... 

YES Prep Public Schools

Secondary English Teacher (TIA Eligible) - 25/26 Job at YES Prep Public Schools

 ...Job Description POSITION: Teacher (25-26) REPORTS TO: Principal POSITION TYPE: 10-month COMPENSATION: Starting at $62,000 annually At YES Prep, work for a mission that impacts the life trajectory of students and their families. YES... 

Unlimit Ventures

Entrepreneur in Residence (EIR) Job at Unlimit Ventures

 ...Role Summary The Entrepreneur in Residence (EIR) will be responsible for providing guidance and support to entrepreneurs and startup companies. The EIR will work closely with the CEO and other executive team members to provide advice on business strategy, product development...