How can the impacts of AI on the working world be studied where they become visible and tangible – in companies?
This calls for close collaboration between research and practice. In the lessons-learned document, researchers who conducted studies with and within companies over three years as part of the ai:conomics project share their insights on the co-creative design of research projects.
In the ai:conomics project, researchers, employers, employees, technology experts, work councils and policy makers collaborated in a transdisciplinary manner. The goal was to expand the scientific knowledge about the effects of AI on work and employees and to make it available to both, the involved stakeholders and the broader public.
The co-creative research process offers many benefits but also brings about challenges on various levels in practice. This brief dossier highlights these challenges and outlines the strategies developed to address them.
It provides guidance for researchers planning experimental studies in different companies by:
- Supporting the integration of randomized experiments into the process of introducing new AI tools.
- Helping to apply the insights across disciplines when evaluating interventions in complex environment as comparable challenges also arise in other fields such as education, healthcare, or administration.
Based on the experience with the case studies in ai:conomics, several lessons have been drawn that can be categorised into four groups or clusters:
1. Cluster: Complexity of AI implementation
The process of AI implementation in companies is rarely linear or clearly structured, making it challenging to form experimental and control groups and to identify clear effects.
The dossier demonstrates how research strategies can be flexibly adapted to this complexity. Researchers must:
- be flexible in adapting the research strategy if necessary.
- initiate a joint design process with the companies in order to develop the necessary comprehensive knowledge of the processes in the individual phases.
- develop fine-grained measures to assess minor changes.
2. Cluster: Complexity of the co-creation process
Studying the impacts of AI on the working world requires access to sensitive operational data and information. Successful collaboration between researchers and workplace stakeholders is built on trust and an active investment in in cooperation, shared values, rules, the recognition of sometimes divergent interests and good communication.
This dossier highlights key principles that have proven particularly effective over more than three years of practice in enabling co-creative design and implementation on an equal footing. These include:
- creation of a fundamental relationship of trust between all parties involved.
- continuous cooperation in order to obtain sufficient information and support.
- a very dedicated liaison pin at the executive level of the organisation as well as access to higher management.
- ‘diagonal’ communication structures that can dock onto and be heard at all levels from management to employees and in all relevant bodies and committees.
- recognising and balancing differences of interest through openness and creating an atmosphere of trust.
- good communication to avoid misinterpretations, inconsistencies in the flow of information or false expectations.
- involving the funder in the entire process to help all parties to reach their goals.
3. Cluster: Methodological complexity
Measuring the impacts of AI implementations is challenging. Companies aim to make the introduction of new technologies as seamless as possible, which often mitigates "shocks"—a key factor for many experimental research designs. At the same time, changes in tasks are difficult to measure, and available KPIs typically focus on technical or economic aspects (e.g., product quality) rather than on the effects on employees. Additionally, there are spill-over effects, where the impact of AI implementation extends beyond directly affected employees to influence other areas.
The dossier outlines how these methodological challenges were tackled during the project and shares the general insights gained from the experience. It concludes that the following points are essential:
- holistic approaches that successfully combine different data sources and methods, in particular approaches that combine state-of-the-art quantitative econometric tools with more qualitative approaches.
- in-depth analyses of the tasks of the specific jobs of the employees affected by AI.
- use of machine learning tools to analyse changes in online job offers.
- working with experts in the organisation to develop relevant KPIs for the outcomes or levels we are interested in.
- analysing spill-over effects in other areas of the company or the economy.
4. Cluster: Legal and ethical complexity
Collecting scientific data within companies or analyzing existing company data presents researchers with complex legal and ethical frameworks.
To navigate these challenges, the following are required:
- Comprehensive legal expertise
- Ethical knowledge
