Observatory

Find the results of the benchmarking work carried out in PerMedCoE. It includes a side-by-side comparison of tools with mixed scientific and technology metrics and a result matrix showing competing tools and metrics on the respective axis

About the OpenEBench

PerMedCoE aims to organise a set of community-driven benchmarks of tools aimed at cell-level simulations: signalling pathways, metabolism and population of cells. These benchmarks will allow to vertebrate the community of the tools users and developers, simplify benchmarking of new tools by tool developers, provide good scenarios and metrics for tool comparison, and, as a result, offer the end-users a comprehensive view of the state of the field and a useful reference for choosing best tools for specific use cases.

PerMedCoE is calling out to developers of simulation tools to participate and help in this effort and decide on scope, metrics and reference datasets that could be used to compare our cell-level simulation tools.

The outcomes of these efforts will become a public website at ELIXIR’s OpenEBench platform. The site will host both the open methodology and results of the benchmarks, serving as a base point for many other tool authors and users to add their observations and benchmark. We aim to publish a community paper from the initial benchmark participants to help define and maintain the computational biology simulation community.

Cell-level simulations are important for biomedical research

Modelling has helped researchers to gain unprecedented insights into biological mechanisms in cells and organisms. With the advance of high-throughput technologies and massive takeoff of single-cell data collection, modelling in computational biology has become an integral part of the field. Bridging from intracellular mechanisms to tissue-level phenotypes is an open topic, and scaling up the cell-level modelling approaches is a promising methodology to address it.

Cell-level modelling can capture several distinct aspects of the biological processes, ranging from metabolism to signalling and to behaviours of whole populations. To understand cell phenotypes and their mechanisms, researchers need to reconcile most, if not all, these different aspects.

Community-driven benchmarks have a lasting impact

One of the tasks of PerMedCoE is to perform benchmarks of cell-level simulation tools. Although benchmarking is common through all bioinformatics, the comparisons so far were relatively small-scale endeavours to find the best tool or configuration for a specific task or niche or compare a newly developed tool against a few most similar tools from the field.

Instead, we aim to run a community-driven benchmark, similar to the benchmarking efforts organised by heterogeneous communities that have proved popular and successful in the past, such as the Critical Assessment of Protein Structure Prediction (CASP) that has been running since 1994. Here, “community” is a group of people with diverse backgrounds and experiences working in the same field, who face similar problems and want to collaborate to find the best solutions to these problems, like the Copy Number Variation benchmarking community or the NGS community for antimicrobial resistance. 

Besides the community itself, benchmarking efforts also benefit tool developers and end-users. Objectively comparing their tools against others encourages software developers to implement more efficient methods and develop new tools by calling attention to challenging areas, blind spots and unreaped opportunities in the methodologies. Lastly, users take advantage of access to the benchmark results when choosing a tool for the problem at hand, which helps to steer the user base towards the latest developments in the field, further benefiting the authors of the original and highly specific tools.

Three different modelling approaches are the focus of the benchmarks

PerMedCoE focuses on three cell-level aspects of biology: metabolism, signalling pathways and cell population behaviours. Thus, we plan to have different benchmarks of relevant tools in each one of these modelling areas:

• Metabolic modelling: tools like COBREXA that simulate metabolic behaviour using constraint-based models and algorithms such as Flux Balance Analysis, Flux Variability Analysis, Flux Sampling, etc.
• Signalling modelling: tools like MaBoSS or CellNOpt that simulate signalling pathways using different mathematical approaches such as ODEs, Boolean, fuzzy logics, stochastic Boolean, etc.
• Cell populations modelling: tools like PhysiCell that simulate cells as free agents, as in agent-based models (or centre-based models), and that explicitly describe the environment.

The design of the benchmarks is a community effort

A relevant and successful benchmark must reflect the existing challenges of the scientific community in terms of size, complexity and content. To accomplish this, we call for the community to help us define three features to ensure a meaningful and fair benchmark:

• Scope of the benchmark: The relevant scientific questions that the benchmarked tools need to answer. These can be technical, focusing on the tool’s performance under specific conditions, or scientific, focusing on its performance and quality of the results within clearly defined scientific challenges.
• Metrics: A set of quantitative values used to measure the performance of the evaluated tools in the scope of the benchmark. The technical metrics could include e.g. memory use and run time; the scientific metrics may include accuracy and recall.
• Reference data sets or desired outcomes: Data sets and desired outcomes that can be used as ground truths and/or baselines for comparison. They need to be well defined, unbiased and their use should be agreed upon and supported by the community.

PerMedCoE collaborates with OpenEBench to organise a platform with a public website, catalogue the tools in bio.tools, host the benchmarks, and publish their results. OpenEBench offers a comprehensive suite of web tools that allow communities to undertake their benchmarks without worrying about the technical aspects.

The outcomes of this effort will help boost and maintain the simulations community

PerMedCoE will build the public website at OpenEBench where the benchmark results and methodologies will be published, serving as panels for future simulation tools. This website will also serve as a landing page for bio.tools to find the catalogue of tools used and their descriptors.

In addition, we will actively disseminate these benchmarks, their results, methodology and organisation, and publish a community paper or white paper together with the benchmark participants to help vertebrate the computational biology cell-level simulation community.

How to participate?

If you want to participate in the study design and discussions, or contribute your software tools to the benchmarks, contact us at infoPerMedCoE@bsc.es.