We are hiring an AI Operations Associate for a UK-based client to support the day-to-day performance, reliability and continuous improvement of its AI and Generative AI applications.
This is a permanent work-from-home opportunity aligned with UK working hours. The role is suitable for professionals with experience in AI operations, application support, data operations, software support, QA, MLOps or LLMOps who want to build their careers in the rapidly growing Generative AI domain.
As an AI Operations Associate, you will work closely with AI engineers, developers, product teams and business users to monitor AI applications, evaluate model responses, maintain prompt libraries, prepare datasets and troubleshoot production issues.
AI Application and Model Monitoring
- Monitor AI and Generative AI applications for performance, response quality, availability and reliability.
- Track operational metrics such as latency, error rates, request volumes, token consumption and API usage.
- Identify changes in response accuracy, relevance, groundedness or consistency that may indicate model or data drift.
- Review alerts, logs and dashboards and escalate critical issues to the appropriate engineering teams.
- Prepare regular reports on application performance, incidents and recurring issues.
AI Output Evaluation and Quality Assurance
- Review and evaluate AI-generated responses for accuracy, relevance, completeness, tone and compliance with defined guidelines.
- Create and maintain test cases, evaluation datasets and response-quality scorecards.
- Conduct regression testing after changes to prompts, models, knowledge sources or application workflows.
- Identify hallucinations, inaccurate responses and unsupported answers and document the likely causes.
- Support both automated and human-led evaluation of AI applications.
Data Preparation and Human-Feedback Operations
- Clean, organise, label and validate datasets used for AI testing, evaluation and improvement.
- Collect and structure human feedback on AI-generated responses.
- Support Reinforcement Learning from Human Feedback-related activities through response comparison, ranking, annotation and quality review.
- Maintain high standards of data accuracy, consistency and confidentiality.
- Help create representative datasets covering common, difficult and edge-case user scenarios.
Prompt Library Management
- Maintain a central, version-controlled library of system prompts, user prompts, templates and AI instructions.
- Document prompt changes, expected outcomes, test results and approval history.
- Test prompt variations and compare their effect on response quality and consistency.
- Coordinate prompt releases across development, testing and production environments.
- Ensure approved prompts follow organisational standards for security, privacy, tone and responsible AI use.
First-Line Troubleshooting and Incident Support
- Investigate failed or slow AI requests using application logs, API responses and monitoring tools.
- Perform initial root-cause analysis across prompts, data, authentication, API integrations, rate limits, knowledge sources and model endpoints.
- Reproduce reported issues and provide clear diagnostic information to developers and AI engineers.
- Track incidents and service requests through tools such as Jira, ServiceNow or similar platforms.
- Maintain troubleshooting guides, operational procedures and knowledge-base articles.
- Participate in incident reviews and recommend preventive actions for recurring problems.
AI Governance and Documentation
- Follow established policies for responsible AI, data privacy, access control and information security.
- Help identify potentially harmful, biased, confidential or non-compliant AI responses.
- Maintain audit trails for prompt changes, datasets, evaluations and production incidents.
- Support the preparation of operating procedures, monitoring checklists and AI application documentation.
Required Skills and Experience
- 1-5 years of experience in AI operations, application support, production support, data operations, QA, MLOps, DevOps, software support or a related area.
- Basic understanding of Generative AI, Large Language Models, prompts, tokens, embeddings and AI APIs.
- Familiarity with AI platforms or APIs such as OpenAI, Azure OpenAI, Microsoft Foundry, Google Gemini, AWS Bedrock or similar services.
- Ability to interpret application logs, error messages, API responses and performance dashboards.
- Basic working knowledge of REST APIs, JSON and API-testing tools such as Postman.
- Familiarity with Git or another version-control system.
- Strong attention to detail and the ability to assess the quality of AI-generated content.
- Good analytical, documentation and problem-solving skills.
- Strong written and spoken English.
- Ability to work independently in a remote environment and collaborate during UK business hours.
Desirable Skills
- Basic knowledge of Python, SQL or scripting for data validation and operational automation.
- Exposure to Azure Monitor, Application Insights, Datadog, Grafana, Splunk, CloudWatch or similar monitoring tools.
- Experience with AI evaluation or observability platforms such as LangSmith, MLflow, prompt flow or equivalent tools.
- Understanding of Retrieval-Augmented Generation, vector databases and enterprise knowledge sources.
- Experience in data annotation, content moderation, model evaluation or conversational AI testing.
- Familiarity with Jira, ServiceNow, Confluence or other ticketing and documentation platforms.
- Awareness of responsible AI principles, data protection and AI security risks.
Educational Qualifications
- Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science or a related discipline.
- Candidates with equivalent practical experience in AI operations, software support or data operations will also be considered.
What Makes This Opportunity Attractive?
- Permanent work-from-home arrangement.
- Opportunity to work directly on AI and Generative AI applications for a UK client.
- Exposure to production AI systems, prompt operations, model evaluation and LLMOps.
- Both regular full-time and freelance engagement options.
- Long-term opportunity in a rapidly developing technology domain.
- International working environment with UK-based stakeholders.
About itForte
itForte is a leading specialist IT recruitment company connecting technology professionals with clients across the US, UK, Europe and Japan. We specialise in talent acquisition across Artificial Intelligence, Cybersecurity, Cloud Technologies, Enterprise Applications, and Product and Software Engineering.
Professionals interested in building a career in AI operations and supporting enterprise Generative AI applications are invited to apply.
