Collaborating Expert on the development of a validation framework for AI systems

Strategic area / objective

MAWP Focus area 2.2. Data analytics, digital tools and digital transformation

EMA aims to enable the safe and responsible use of AI for the benefit of public and animal health. 
Critical to that aim is the ability to validate consistently the performance characteristics of different AI systems including identifying the relevant metrics to measure performance, defining the counterfactual (what AI is measured against, the gold standard) depending on the type of AI and defining how to communicate this information and update it along the lifecycle of the AI system. 
 

Specific projects

•    The collaborating expert will help create a framework to validate AI systems implemented at EMA and monitor performance, including how to identify relevant metrics to measure performance, how to define the counterfactual, the type and level of documentation required (including, e.g. creating model fiches, or considering how model repositories can be implemented) and how to ensure prospective validation along the lifecycle of the AI system, among other activities. These activities will be aligned to the requirements of the AI Act.
•    To conduct the above, the collaborating expert will be involved in the development of AI systems, specifically those based on large language models, that will serve as use cases for the development of the framework above.
•    The collaborating expert will also be expected to communicate and engage with stakeholders on methods to validate AI systems based on LLMs, including e.g. the AI SIA.

Deliverables

•    Scientific article in a peer reviewed journal
•    Evaluation tools developed and implemented on at least one LLM developed within EMA.
•    A proposed framework for validation of AI systems including documentation artefacts, such as model fiches.
 

Benefits for the expert

•    Gain experience in the field of AI in the EU regulatory context
•    Contribution to the development of AI tools to be used in the regulatory setting
•    Gain experience of working in an EU organisation and of the EMA’s work and role in regulatory science
•    Building a personal and inter-institutional collaboration with EMA
 

Benefits for the collaborating institution

•    Exchange of knowledge and expertise
•    Knowledge transfer 
•    Joint research undertaking, and if publication, joint publication
 

Essential requirements

Eligibility criteria:  
•    Be an EU or EEA national
•    Meet the requirements stated in the profile description;
•    Be a paid employee from European academia, research institutions, public sector or international organisations, or a post graduate student within such organisations. Proof of employment/student status and agreement in principle of employer/academic institution will be requested.;
•    Have a thorough knowledge of English (at least level C1). Good knowledge of another official EU language (at least B2) is an asset, as per the Common European Framework for Languages.

University degree(s)
Essential: PhD or Master’s degree in mathematics, or related fields
Desirable: Experience in development and assessment of quantitative methods used in drug development and evaluation
 

Experience and additional skills
Essential: 
•    Demonstrated experience in writing, analysis and reporting of results in scientific reports and peer-reviewed publications
•    Demonstrated knowledge in the field of AI
Desirable: 
Knowledge of regulatory environment
 

Behavioural competencies
Essential: 
•    Communication skills 
•    Interpersonal skills - dynamic, motivated and organised
•    Working with others but also with some degree of autonomy
•    Adaptability