How do we assess scientific quality?

Almost 200 guests had come to listen to and participate in a discussion on how breakthrough science may be identified and supported in an ever more rapidly changing scientific landscape, at The Royal Swedish Academy of Sciences. The event was moderated by Eva Krutmeijer.

The discussions were sometimes intense but in the end the speakers agreed almost unanimously on a number of things.

After opening remarks and a warm welcome from Maria Tenje, Chair of the Young Academy of Sweden and Hans Ellegren, Vice President of the Royal Swedish Academy of Sciences, the President of the European Research Council (ERC); professor Jean-Pierre Bourguignon, took to the stage and gave a presentation about how ERC has changed the European research landscape.

Since ERC was established in 2007 it aimed to support breakthrough science in all disciplines and in a bottom-up approach where applications from scientists are at the core of the funding process. Jean-Pierre Bourguignon started by giving a historical perspective of Europe’s research landscape. Of the research produced in Europe a relatively high proportion is high quality, but this proportion is even higher in the US, and other areas, such as China is catching up. This, together with the apprehension that Europe is not capitalizing enough on its knowledge base, led to the creation of ERC. The aim is to support Europe in generating cutting edge, high quality research. Until today, ERC has had a strong impact, by implementing scientific quality as the only selection criteria. ERC challenges young researchers to formulate what they really dream of working on. By funding young researchers, ERC consistently contributes to building the next generation of researchers. By funding high risk research, ERC helps pushing the research frontier and encourages researchers to explore new territories.

Jean-Pierre Bourguignon concluded that ERC has had a key role in changing the narrative of European research policy by encouraging public funding of long-term research.

Göran Sandberg, Executive Director of the Knut and Alice Wallenberg Foundation, followed up by telling the audience about their strategy for promoting excellence in research. He pointed out four key facts: (i) select the best scientists (ii) give them long term support (iii) give them freedom to do whatever science they want, and (iv) don’t chase them with detailed reports. And also, for young researchers – hope for the best, he added with a smile.

Göran Sandberg ended by prompting us to also discuss the scientific climate. In order to retain a healthy research culture, we need to encourage seminar culture and open debate about science. It’s as important as it is to identify the research superstars, he said.

Liselotte Højgaard, member of the Board of Directors of the Novo Nordisk Foundation, Denmark, went on to tell the audience how scientific quality was improved in Denmark. She started by reminding us that the goal is not to assess scientific quality, but rather to obtain it. However, assessing quality is of course an incremental part of that. Liselotte Højgaard was the Chair of the Danish National Research Foundation (DNRF) 2013–2018. One of DNRF’s instruments for promoting excellent science has been to fund Centers of Excellence with large and long-term grants (up to ten years). The grants allow the centers large degrees of freedom and flexibility on how to use the funds. When selecting new centers, the DNFR looks at three factors: the idea, the center leader and the team. And central for all three are “quality, quality and quality”.

Liselotte Højgaard also talked about the research landscape in Denmark where an important success factor has been to build on a few strong research universities, quite the opposite to what is implemented Sweden. She concluded by pointing out that the real stars are the researchers – It’s all about people driven by curiosity!

The morning talks were followed by a conversation with Gunnar Öquist, Former Secretary General, the Royal Swedish Academy of Sciences, about what we can learn from Denmark. He pointed out that it is interesting to compare Denmark with Sweden. Danish researchers seem to have a better dialogue with the politicians. DNRF was founded as a reaction to a stagnation in Danish science and a wish to change this by enhancing and fostering excellent research. Key factors are trust and freedom, a long-term perspective, beyond and across disciplines. Danish politicians have understood what is needed for fostering excellent science. In Sweden, we have rather taken scientific excellence for granted, but it needs to be fostered and nurtured not to be lost, he pointed out.

Panel: Astrid Söderbergh Widding, Sven Stafström, Mia Phillipson and Gunnar Öquist. Moderator: Eva Krutmeijer. Photo: Rickard Kihlström

Gunnar Öquist was joined on stage by Mia Phillipson, Uppsala University and member of The Young Academy of Sweden, Sven Stafström, Director General of the Swedish Research Council, Astrid Söderbergh Widding, President of Stockholm University, for a discussion on how different funding streams should be balanced – a very hot topic nowadays. Mia Phillipson pointed out that for young researchers, individual, long-term, independent funding is key, as it allows them to prove their independence and move between universities. Mia Phillipson went on to point to the importance of the peer review system.

“It’s crucial that I can trust the review process of my funding. Unfortunately though, this is not yet well established at the universities. Therefore, the external funding is vital. The universities on the other hand should take responsibility for recruitments and for providing an attractive research environment. As a researcher I am constantly challenged to get out of my comfort zone. Maybe we need to challenge the universities to do the same – to dare selecting strong researchers and research environments.”

Astrid Söderbergh Widding and Sven Stafström agreed that universities should take greater responsibility to recruit the right people, and then to ensure that for the scientists they choose to employ there are good conditions regarding career paths, infrastructure and so on. Sven Stafström pointed out that there are differences between the tasks for universities and research funders and also between different funders where some have a mission to support excellence driven research and some have other missions. For the bottom-up research projects he argued that they are better funded by external funders than by universities, following up on Mia Phillipson’s argument that external project grants allow the researcher to be innovative and mobile.

After lunch, Wilhelm Krull, Secretary General of the Volkswagen Foundation, Germany, shared insights from the Volkswagen Foundation and their experience of quality, originality, and risk-taking.

He started by establishing that he strongly believes in the peer review system, but that we constantly need to think about ways to improve it. He went on to reflect over the challenges with peer review, such as an increasing overload of the reviewers, cognitive bias, a tendency towards the already established and proven, problems with how to deal with failures and a lack of predictive validity of past performance in peer review. Volkswagen Foundation has tried to address some of the challenges using strategies such as removing the CV and publication list after a first screening so that focus of the evaluation is on the research idea. Another strategy is “The funding joker” – each panel member has a joker they can use to go against the majority. “It really changes the meeting discussions” Wilhelm Krull said.

This was followed up by a discussion between Lars Hultman, Chief Executive Officer of the Swedish Foundation for Strategic Research and Klas Kärre, Chair of the scientific committee, The Swedish Cancer Society, who talked about strategic research and what it signifies.

The panelists started by noting that nobody really knows what strategy is, but concluded that strategic research should be excellent and should have an impact on society and that strategic research does not exclude curiosity driven science.

Thomas Sinkjæer, the Lundbeck Foundation, Denmark, reflected over ways to experiment with how we identify and support innovative research. Maybe a lottery could be a way to allocate grants – a concept that was also brought up by Wilhelm Krull of the Volkswagen Foundation earlier in the day. They have experimented with randomized allocation of funding in some programs and will monitor the results. He was careful to underline that when they applied randomized funding (lottery) this was done within a pool of very good applications, where there were only small differences in quality but not all could be funded.

In a conversation with the moderator, Joseph Samec, Stockholm University, reflected over what is most important – past performance of the researcher or excellent ideas? In their discussion they reflected over the importance of failure in science. A defining feature of breakthrough science is that it doesn’t always succeed. We need to promote excellence and learn from mistakes, Joseph Samec pointed out.

Before coffee, members of The Young Academy of Sweden, Jonas Olofsson, Stockholm University, and Mia Phillipson, took part in a conversation about how best to promote young scientists. They started by identifying challenges for young researchers, and important factors such as lack of transparent and clear career paths as well as scarcity of leadership training were highlighted.

In pointing out success factors Mia Phillipson and Jonas Olofsson both mentioned the benefit of having a mentor. Mia Phillipson pointed out the challenge in finding a balance in work load. One should always try to protect one’s time and put science first! she said. Jonas Olofsson agreed and concluded that you need to find balance and should never compromise on time and space for creativity.

Mia Phillipson summed up the conversation by launching the concept of “motverkan” – value one’s time, science has to take time. We have to cherish the time for research.

The last session of the day was opened by Chonnettia Jones, Director of Insight and Analysis, Wellcome Trust, who spoke on the topic of how to promote excellence. Key factors that she has identified are (i) invest in infrastructure, resources and technology development to enable transformative research (ii) provide flexible core funding for centers of excellence (iii) back bold and ambitious ideas and (iV) enable an open, supportive and inclusive culture for research. She also noted that the culture in the research group and its ecosystem are important and something that the Wellcome Trust engages actively in.

This was followed by a panel discussion where representatives from the research funders, Lars Hultman, Göran Sandberg, Sven Stafström, and Britt-Marie Sjöberg, Chief Executive Officer, the Wenner-Gren Foundations, gave their take on how to create funding programs that promote breakthrough science and how funders identify applications or individuals that will generate breakthrough science.

Britt-Marie Sjöberg started by stating that no one can really know what will be breakthrough science – not politicians, not executives, not funders. It boils down to individuals and ideas.

Sven Stafström followed up by saying that we should fund researchers that are devoted to their idea.

Göran Sandberg pointed out that there were only a few chancellors in the audience and sent out a call for the rest of the academic leaders to engage in the discussion. Scientific quality should be a top priority for the academic leadership, but where are they? It is not that we have the wrong leadership, but we need to have a system that allows the academic leaders to focus on the scientific questions, and not administration, he said.

Other important factors that were mentioned were a need for a real tenure track, and mobility – Move on and make moves. Go for postdocs, go for sabbaticals, said Lars Hultman.

He ended by declaring that “I’m extremely optimistic after listening to the young academy here today. We will prevail!”

The afternoon was wrapped up with comments from the research community. A panel of scientists; Helena Edlund, Umeå University, Arne Jarrick, Stockholm University, Jonas Olofsson and Maria Tenje continued the reflections on how to foster breakthrough science.

Arne Jarrick started by noting that there had been a lot of focus on university leadership and he questioned if this was really necessary? He was answered by Maria Tenje who argued that since the universities have such great influence over recruitment and providing good conditions for their researchers its essential to include university leadership.

Maria Tenje further reflected on the clear common thread of the day’s talks and panel discussions; open long-term funding, belief and trust in the researcher. This was followed up on by Helena Edlund who also noted that this is often easier to implement for private funders than for public funders.

Jonas Olofsson stressed the importance of incentivizing the quality of the process, not only the results. The quality of the process is as important as quality of the output.

When asked what made them invest in a career as a researcher, Jonas Olofsson mentioned being forced to step out of the comfort zone, and mobility. Maria Tenje noted that she has tried to leave science twice already, but to live a life without constant questions, I could not imagine!

The symposium was organized with support from The Knut and Alice Wallenberg Foundation (KAW), The Swedish Foundation for Strategic Research (SSF) and The Swedish Research Council (VR).