GenAI/LLM Prompt Compliance & Evaluation Consultant Job at US Tech Solutions, Washington DC

RzNBWm1WbUp5MXdSMTFpcm9FZ3lOWUJKWGc9PQ==
  • US Tech Solutions
  • Washington DC

Job Description

Duration: 6 months contract

Job Description:

  • The Responsible AI Scaled Testing Team within Trust & Safety performs pre-launch structured testing for AI applications against safety, fairness and neutrality policies and standards.
  • It is a global team with Responsible AI domain expertise and diverse backgrounds in operations, strategy, ethics, risk management, product management, and program management.

Responsibilities:

  • Lead the end-to-end technical assessment of GenAI products, focusing on pre-launch testing of safety, neutrality, and fairness.
  • Develop and implement rigorous testing methodologies, including automated prompt generation and response analysis, to ensure compliance with defined standards.
  • Leverage data-driven insights to identify potential risks and inform product development iterations, ensuring robust and reliable GenAI deployments.
  • Automated Testing and Data Pipeline Management: Design and implement automated prompt generation strategies and data analysis solutions to efficiently collect and analyze GenAI model responses. Manage and optimize data pipelines for efficient processing and analysis of large datasets.
  • Quantitative and Qualitative Analysis of Model Behavior: Conduct in-depth statistical analysis and qualitative evaluations of model outputs to identify deviations from defined standards. Develop and apply metrics for evaluating safety, neutrality, and fairness, and generate detailed reports with actionable insights.
  • Technical Guideline Development and Execution: Translate abstract safety, neutrality, and fairness standards into precise technical guidelines and evaluation criteria. Develop and maintain clear documentation for vendor teams, including detailed instructions, edge case clarifications, and quality calibration protocols.

Experience:

  • 4+ years of experience in data analysis, AI/ML testing, cybersecurity, or related technical domains.
  • Proficiency in data analysis tools and languages (e.g., SQL, Python, R) for processing and analyzing large datasets.
  • Experience developing and implementing automated testing frameworks.
  • Strong analytical and problem-solving skills, with the ability to interpret complex data and identify patterns.
  • Excellent technical communication skills, with the ability to clearly articulate complex technical concepts to both technical and non-technical audiences.
  • This role may be exposed to graphic, controversial, and/or upsetting content.

Skills:

  • 2+ years of experience in AI testing, adversarial testing, red teaming, or related areas (Nice to have).
  • Experience with LLM-based prompt generation and evaluation tools (Nice to have).
  • Familiarity with machine learning concepts and algorithms (Nice to have).
  • Experience with bug tracking systems (Nice to have).
  • Experience in developing and maintaining technical documentation (Nice to have).
  • Experience in defining and implementing process improvements (Nice to have).
  • Ability to think strategically about emerging AI threats and vulnerabilities (Nice to have).

Education:

  • Bachelor's degree in Computer Science, Data Science, Statistics, or a related technical field, or equivalent practical experience.

About US Tech Solutions:

US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit .

US Tech Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, colour, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

Recruiter Details:

Name: Ashik

Email: ashik.a@ustechsolutionsinc.com

Internal Id: 25-35043

Job Tags

Contract work,

Similar Jobs

Fives Giddings & Lewis, LLC

CNC Machinist - Programming Job at Fives Giddings & Lewis, LLC

 ...A Machinist Master operator performs a wide range of duties including intermediate to complex set ups, adjustment, and operation of CNC and/or manual Machine Tools. Scope and Responsibilities: Essential duties include, but are not limited to the following: Proactively... 

Ascendo Resources

Director- Staffing Agency Job at Ascendo Resources

 ...Ascendo Resources, a Certified Minority Owned Staffing Firm, is committed to delivering a superior experience to our clients, companies, and candidates through our industry expertise, local market knowledge, personal service, and speed to market. Ascendo is currently... 

Dover Corporation

Production Scheduler/Planner Job at Dover Corporation

 ...Retail (DFR) is a leading manufacturer of Display Cases, Specialty Products, Refrigeration Systems, Power Systems and Comprehensive...  ...What were looking for: We are looking for a Production Scheduler/Planner that will be responsible for scheduling orders at the... 

Tobalina Travels

Remote Travel Advisor Job at Tobalina Travels

Tobalina Travels is now looking for Remote Travel Advisors. I am in search of experienced and inexperienced travel advisors. I'm seeking those who can work independently and with like-minded others. A passion for working with the public and providing excellent customer...

Grassroots Team

Entry Level - Kick Start Your Non- Profit Career Job at Grassroots Team

 ...NO EXPERIENCE NECESSARY! ONLY A PASSION FOR HUMAN RIGHTS! Apply today! Start soon! Searching for outgoing people who are comfortable in social situations. Backgrounds in theater, comedy, or sales are a plus! Are you passionate about non-profit and community...