Workshop Report: Academics and Researchers x EdTech Strategy Lab
In the first session of the EdTech Strategy Lab Summer Workshop Series on June 10, we brought together a diverse group of stakeholders to focus on the role of researchers and academics. Participants raised some interesting challenges and ideas relating to evidence and research-led evaluation of EdTech.
Strategy Lab Insights
Beth Havinga, Managing Director of the European Edtech Alliance (EEA), kicked off the session by sharing findings from the EdTech Strategy Lab, highlighting stakeholder needs for evidence and evaluation. Beth emphasised that trust is crucial for fostering a sustainable innovation ecosystem and inspired educational practices. Decision-makers’ confidence in EdTech tools relies on their perceived efficacy, security, and alignment with educational objectives. However, a significant gap in the current landscape poses challenges to building and maintaining this trust.
Beth continued by noting a disconnect: research shows a need for evidence, yet it's seldom used in decision-making, and mistrust exists in some contexts. Despite this, many EdTech organisations value testing and collaborate with researchers. Evaluation mechanisms vary widely, from interviews to controlled tests, often misaligned with educational settings' needs. This observation corresponds with the findings from previous Strategy Lab workshops, which revealed a lack of common understanding and transparency in research and certifications.
Trust emerged as a critical factor for EdTech adoption, influenced by the technology’s efficacy, security, and alignment with educational goals.
In citing the Council of the European Union, which defines evidence as systematically and scientifically collected information used to inform policy and practice, Beth identified that the EdTech Strategy Lab research shows three main evaluation framework directions, with only one led by research organisations.
These frameworks employ various mechanisms, however, noted that some are not always feasible or appropriate for all technologies or learning environments. Definitions of efficacy and effectiveness from other sectors may also not translate well to educational settings.
Varied evaluation goals further complicate the issue, and a need for stronger partnerships in the research and evaluation process is evident, as current evaluations are often led by teachers or researchers, with less involvement from EdTech organisations. Beth concluded that fostering these partnerships is crucial for advancing the EdTech ecosystem.
A Fireside Chat with Pati Ruiz of Digital Promise
During the workshop, Pati Ruiz, the Senior Director of EdTech and Emerging Technologies, shared Digital Promise’s initiatives. Established by the U.S. Congress in 2008, this non-profit is dedicated to helping district leaders and educators make informed EdTech decisions. Digital Promise supports research-based product certifications, partners with national and international organisations, and strives to clarify what constitutes evidence in EdTech.
They foster collaboration among educational leaders, policymakers, and industry experts to define clear evidence standards, focusing on co-designing solutions with district leaders and industry experts to optimise success.
Pati highlighted their product certifications, including a partnership with The Edtech Equity Project to create a certification that prioritises racial equity and addresses racial bias in algorithms and design. This project serves as a foundation for an ethically designed AI certification, aiming to help school districts make informed decisions about AI product procurement.
Digital Promise empowers voices across the education sector, advocating for EdTech products that use research to cater to diverse learners, and emphasises transparency in how learning sciences research drives product design, fostering trust between companies and learners.
Having completed their 200th product certification, Digital Promise sees growing interest from school district leaders. Their certification processes make research accessible to all education stakeholders. They are also exploring outcomes-based contracting to provide clarity for procurement decisions.
A Superintendent of Technology and Innovation suggested an “EdTech Nutrition Label” to provide clear and accessible product information upfront, helping decision-makers and educators make informed choices. Digital Promise aims to ensure that all stakeholders in the EdTech ecosystem work together, equipped with tools and frameworks to identify products that are accessible, inclusive, secure, evidence-based, usable, and interoperable.
“We want to ensure our districts, our schools, our teachers, and our students have effective tools that create meaningful learning experiences.”
Discussion Points
During the workshop, participants identified several significant hurdles to trust in digital education:
the absence of evaluation environments,
lack of rigour and consensus on evaluation definitions,
potential conflicts of interest from EdTech creators conducting their own evaluations,
system complexity, inadequate funding,
the gap between academia and industry,
concerns regarding personal data access and use,
lack of transparency,
differing definitions, and
disagreements on goals and priorities.
These hurdles highlight the multifaceted challenges impacting trust in EdTech and underscore the complexity of addressing these issues to foster confidence in the field.
Regarding the accessibility of existing frameworks, 70% of participants were uncertain, while the remaining 30% found them insufficient, with none expressing confidence in their accessibility.
Participants identified the primary barriers to alignment between different frameworks as a lack of understanding, absence of a common vocabulary, differing perspectives and needs, and disparities in desired goals and outcomes.
Key Findings and Future Directions
Developing trust in EdTech requires a nuanced approach, recognizing that trust varies among individuals and determining what must be trustworthy and for whom. Early collaboration among stakeholders to create a shared language of terms and concepts is essential, with a focus on prioritising learning outcomes over stakeholder buy-in. Demonstrating the value of evidence in decision-making processes is crucial, as trust criteria differ at various stages of product development.
Clear and transparent communication is important, and decision-making roles and political influences must be clearly defined. Breaking down barriers between industry and academia is necessary, emphasising transparency in research design, access to research, and early involvement of researchers and practitioners. Considering the context is vital, as efficacy is measured differently across settings, and research should encompass a wide range of users.
Additionally, cost-effectiveness must be considered, acknowledging barriers to adopting new EdTech and recognizing that the best research outcomes do not always ensure successful implementation.
