Photo: Rickard Kihlberg

Click to zoom. Photo: Rickard Kilström

How do we assess scientific quality?

2019-05-24

On May 23rd, The Young Academy of Sweden co-hosted the symposium “How do we assess scientific quality?” together with The Royal Swedish Academy of Sciences. Prominent speakers from key research funders – both nationally and internationally – and the scientific community addressed questions such as can breakthrough science be predicted? Do we encourage bold science and are we able to identify worthy risks? Does peer review favour productivity rather than creativity? 

Almost 200 guests had come to listen to and participate in a discussion on how breakthrough science may be identified and supported in an ever more rapidly changing scientific landscape, at The Royal Swedish Academy of Sciences. The event was moderated by Eva Krutmeijer.

The discussions were sometimes intense but in the end the speakers agreed almost unanimously on a number of things.

  • Funding of research projects should be with long-term grants with large degrees of freedom and as little administrative reports as possible. For this to work the funded projects (and people) must be of very high-quality.
  • The research funded needs to be excellent but it is also important that the research process holds a high quality. This helps to foster a culture which counteracts misconduct.
  • Peer review is the best way to predict which projects (and individuals) to fund. Although it may not be perfect, there is no better method than peer review and the drawbacks may be counteracted by thoughtful routines for how to deal with conflicts of interest, group dynamics, etc.
  • The review process is absolutely critical to ensure that the projects are truly pursuing high quality research (and that the people funded are truly pursuing high quality research).
  • Good conditions for researchers, such as positions (tenure track) and infrastructure need to be funded by universities.
  • Universities should be rigorous with recruitment and promotion processes to ensure that the people hired are really the best.

How the ERC changed the European research landscape

After opening remarks and a warm welcome from Maria Tenje, Chair of the Young Academy of Sweden and Hans Ellegren, Vice President of the Royal Swedish Academy of Sciences, the President of the European Research Council (ERC); professor Jean-Pierre Bourguignon, took to the stage and gave a presentation about how ERC has changed the European research landscape.

Since ERC was established in 2007 it aimed to support breakthrough science in all disciplines and in a bottom-up approach where applications from scientists are at the core of the funding process. Jean-Pierre Bourguignon started by giving a historical perspective of Europe’s research landscape. Of the research produced in Europe a relatively high proportion is high quality, but this proportion is even higher in the US, and other areas, such as China is catching up. This, together with the apprehension that Europe is not capitalizing enough on its knowledge base, led to the creation of ERC. The aim is to support Europe in generating cutting edge, high quality research. Until today, ERC has had a strong impact, by implementing scientific quality as the only selection criteria. ERC challenges young researchers to formulate what they really dream of working on. By funding young researchers, ERC consistently contributes to building the next generation of researchers. By funding high risk research, ERC helps pushing the research frontier and encourages researchers to explore new territories.

Jean-Pierre Bourguignon concluded that ERC has had a key role in changing the narrative of European research policy by encouraging public funding of long-term research.

The Knut and Alice Wallenberg Foundation’s strategy for excellence in research

Göran Sandberg, Executive Director of the Knut and Alice Wallenberg Foundation, followed up by telling the audience about their strategy for promoting excellence in research. He pointed out four key facts: (i) select the best scientists (ii) give them long term support (iii) give them freedom to do whatever science they want, and (iv) don’t chase them with detailed reports. And also, for young researchers – hope for the best, he added with a smile.

Göran Sandberg ended by prompting us to also discuss the scientific climate. In order to retain a healthy research culture, we need to encourage seminar culture and open debate about science. It’s as important as it is to identify the research superstars, he said.

How scientific quality was improved in Denmark

Liselotte Højgaard, member of the Board of Directors of the Novo Nordisk Foundation, Denmark, went on to tell the audience how scientific quality was improved in Denmark. She started by reminding us that the goal is not to assess scientific quality, but rather to obtain it. However, assessing quality is of course an incremental part of that. Liselotte Højgaard was the Chair of the Danish National Research Foundation (DNRF) 2013–2018. One of DNRF’s instruments for promoting excellent science has been to fund Centers of Excellence with large and long-term grants (up to ten years). The grants allow the centers large degrees of freedom and flexibility on how to use the funds. When selecting new centers, the DNFR looks at three factors: the idea, the center leader and the team. And central for all three are “quality, quality and quality”.

Liselotte Højgaard also talked about the research landscape in Denmark where an important success factor has been to build on a few strong research universities, quite the opposite to what is implemented Sweden. She concluded by pointing out that the real stars are the researchers – It’s all about people driven by curiosity!

Can we learn something from Denmark?

The morning talks were followed by a conversation with Gunnar Öquist, Former Secretary General, the Royal Swedish Academy of Sciences, about what we can learn from Denmark. He pointed out that it is interesting to compare Denmark with Sweden. Danish researchers seem to have a better dialogue with the politicians. DNRF was founded as a reaction to a stagnation in Danish science and a wish to change this by enhancing and fostering excellent research. Key factors are trust and freedom, a long-term perspective, beyond and across disciplines. Danish politicians have understood what is needed for fostering excellent science. In Sweden, we have rather taken scientific excellence for granted, but it needs to be fostered and nurtured not to be lost, he pointed out.

Panel Photo: Rickard Kihlström

Panel: Astrid Söderbergh Widding, Sven Stafström, Mia Phillipson and Gunnar Öquist. Moderator: Eva Krutmeijer. Photo: Rickard Kihlström

Panel: How should base funding from the Government and external funding for research be balanced?

Gunnar Öquist was joined on stage by Mia Phillipson, Uppsala University and member of The Young Academy of Sweden, Sven Stafström, Director General of the Swedish Research Council, Astrid Söderbergh Widding, President of Stockholm University, for a discussion on how different funding streams should be balanced – a very hot topic nowadays. Mia Phillipson pointed out that for young researchers, individual, long-term, independent funding is key, as it allows them to prove their independence and move between universities. Mia Phillipson went on to point to the importance of the peer review system.

“It’s crucial that I can trust the review process of my funding. Unfortunately though, this is not yet well established at the universities. Therefore, the external funding is vital. The universities on the other hand should take responsibility for recruitments and for providing an attractive research environment. As a researcher I am constantly challenged to get out of my comfort zone. Maybe we need to challenge the universities to do the same – to dare selecting strong researchers and research environments.”

Astrid Söderbergh Widding and Sven Stafström agreed that universities should take greater responsibility to recruit the right people, and then to ensure that for the scientists they choose to employ there are good conditions regarding career paths, infrastructure and so on. Sven Stafström pointed out that there are differences between the tasks for universities and research funders and also between different funders where some have a mission to support excellence driven research and some have other missions. For the bottom-up research projects he argued that they are better funded by external funders than by universities, following up on Mia Phillipson’s argument that external project grants allow the researcher to be innovative and mobile.

Quality, originality and risk taking – Experiences and Experiments of the Volkswagen Foundation

After lunch, Wilhelm Krull, Secretary General of the Volkswagen Foundation, Germany, shared insights from the Volkswagen Foundation and their experience of quality, originality, and risk-taking.

He started by establishing that he strongly believes in the peer review system, but that we constantly need to think about ways to improve it. He went on to reflect over the challenges with peer review, such as an increasing overload of the reviewers, cognitive bias, a tendency towards the already established and proven, problems with how to deal with failures and a lack of predictive validity of past performance in peer review. Volkswagen Foundation has tried to address some of the challenges using strategies such as removing the CV and publication list after a first screening so that focus of the evaluation is on the research idea. Another strategy is “The funding joker” – each panel member has a joker they can use to go against the majority. “It really changes the meeting discussions” Wilhelm Krull said.

What is meant by strategic research and how do we ascertain high quality?

This was followed up by a discussion between Lars Hultman, Chief Executive Officer of the Swedish Foundation for Strategic Research and Klas Kärre, Chair of the scientific committee, The Swedish Cancer Society, who talked about strategic research and what it signifies.

The panelists started by noting that nobody really knows what strategy is, but concluded that strategic research should be excellent and should have an impact on society and that strategic research does not exclude curiosity driven science.

Let us experimemt with how to identify and support innovative research, Lundbeck Foundation

Thomas Sinkjæer, the Lundbeck Foundation, Denmark, reflected over ways to experiment with how we identify and support innovative research. Maybe a lottery could be a way to allocate grants – a concept that was also brought up by Wilhelm Krull of the Volkswagen Foundation earlier in the day. They have experimented with randomized allocation of funding in some programs and will monitor the results. He was careful to underline that when they applied randomized funding (lottery) this was done within a pool of very good applications, where there were only small differences in quality but not all could be funded.

What is most important – past performance of the researcher or excellent ideas?

In a conversation with the moderator, Joseph Samec, Stockholm University, reflected over what is most important – past performance of the researcher or excellent ideas? In their discussion they reflected over the importance of failure in science. A defining feature of breakthrough science is that it doesn’t always succeed. We need to promote excellence and learn from mistakes, Joseph Samec pointed out.

How do we best promote young scientists?

Before coffee, members of The Young Academy of Sweden, Jonas Olofsson, Stockholm University, and Mia Phillipson, took part in a conversation about how best to promote young scientists. They started by identifying challenges for young researchers, and important factors such as lack of transparent and clear career paths as well as scarcity of leadership training were highlighted.

In pointing out success factors Mia Phillipson and Jonas Olofsson both mentioned the benefit of having a mentor. Mia Phillipson pointed out the challenge in finding a balance in work load. One should always try to protect one’s time and put science first! she said. Jonas Olofsson agreed and concluded that you need to find balance and should never compromise on time and space for creativity.

Mia Phillipson summed up the conversation by launching the concept of “motverkan” – value one’s time, science has to take time. We have to cherish the time for research.

How to promote excellence, Wellcome Trust

The last session of the day was opened by Chonnettia Jones, Director of Insight and Analysis, Wellcome Trust, who spoke on the topic of how to promote excellence. Key factors that she has identified are (i) invest in infrastructure, resources and technology development to enable transformative research (ii) provide flexible core funding for centers of excellence (iii) back bold and ambitious ideas and (iV) enable an open, supportive and inclusive culture for research. She also noted that the culture in the research group and its ecosystem are important and something that the Wellcome Trust engages actively in.

Panel discussion with funding agencies

This was followed by a panel discussion where representatives from the research funders, Lars Hultman, Göran Sandberg, Sven Stafström, and Britt-Marie Sjöberg, Chief Executive Officer, the Wenner-Gren Foundations, gave their take on how to create funding programs that promote breakthrough science and how funders identify applications or individuals that will generate breakthrough science.

Britt-Marie Sjöberg started by stating that no one can really know what will be breakthrough science – not politicians, not executives, not funders. It boils down to individuals and ideas.

Sven Stafström followed up by saying that we should fund researchers that are devoted to their idea.

Göran Sandberg pointed out that there were only a few chancellors in the audience and sent out a call for the rest of the academic leaders to engage in the discussion. Scientific quality should be a top priority for the academic leadership, but where are they? It is not that we have the wrong leadership, but we need to have a system that allows the academic leaders to focus on the scientific questions, and not administration, he said.

Other important factors that were mentioned were a need for a real tenure track, and mobility – Move on and make moves. Go for postdocs, go for sabbaticals, said Lars Hultman.

He ended by declaring that “I’m extremely optimistic after listening to the young academy here today. We will prevail!”

Concluding comments from the scientists

The afternoon was wrapped up with comments from the research community. A panel of scientists; Helena Edlund, Umeå University, Arne Jarrick, Stockholm University, Jonas Olofsson and Maria Tenje continued the reflections on how to foster breakthrough science.

Arne Jarrick started by noting that there had been a lot of focus on university leadership and he questioned if this was really necessary? He was answered by Maria Tenje who argued that since the universities have such great influence over recruitment and providing good conditions for their researchers its essential to include university leadership.

Maria Tenje further reflected on the clear common thread of the day’s talks and panel discussions; open long-term funding, belief and trust in the researcher. This was followed up on by Helena Edlund who also noted that this is often easier to implement for private funders than for public funders.

Jonas Olofsson stressed the importance of incentivizing the quality of the process, not only the results. The quality of the process is as important as quality of the output.

When asked what made them invest in a career as a researcher, Jonas Olofsson mentioned being forced to step out of the comfort zone, and mobility. Maria Tenje noted that she has tried to leave science twice already, but to live a life without constant questions, I could not imagine!

The symposium was organized with support from The Knut and Alice Wallenberg Foundationlänk till annan webbplats, öppnas i nytt fönster (KAW), The Swedish Foundation for Strategic Researchlänk till annan webbplats, öppnas i nytt fönster (SSF) and The Swedish Research Councillänk till annan webbplats, öppnas i nytt fönster (VR).

Sveriges unga akademi
c/o Kungl. Vetenskapsakademien
Box 50005
SE 104 05 Stockholm

+ 46 (0)8 673 95 00
info@sverigesungaakademi.se

Facebook länk
Twitter länk
Instagram