The prediction of molecular properties is a crucial aspect in drug discovery that can save a lot of money and time during the drug design process. The use of machine learning methods to predict molecular properties has become increasingly popular in recent years. Despite advancements in the field, several challenges remain that need to be addressed, like finding an optimal pre-training procedure to improve performance on small datasets, which are common in drug discovery. In our paper, we tackle these problems by introducing Relative Molecule Self-Attention Transformer for molecular representation learning. It is a novel architecture that uses relative self-attention and 3D molecular representation to capture the interactions between atoms and bonds that enrich the backbone model with domain-specific inductive biases. Furthermore, our two-step pretraining procedure allows us to tune only a few hyperparameter values to achieve good performance comparable with state-of-the-art models on a wide selection of downstream tasks. A novel graph transformer architecture for molecular property prediction is introduced. The task-agnostic methodology for pre-training this model is presented, improving target task performance with minimal hyperparameter tuning. A rigorous exploration of the design space for the self-attention layer is conducted to identify the optimal architecture.

中文翻译：

---

相对分子自注意力变压器

分子特性的预测是药物发现的一个重要方面，可以在药物设计过程中节省大量金钱和时间。近年来，使用机器学习方法来预测分子特性变得越来越流行。尽管该领域取得了进步，但仍然存在一些需要解决的挑战，例如寻找最佳的预训练程序来提高小数据集的性能，这在药物发现中很常见。在我们的论文中，我们通过引入用于分子表示学习的相对分子自注意力变换器来解决这些问题。它是一种新颖的架构，使用相对自注意力和 3D 分子表示来捕获原子和键之间的相互作用，从而通过特定领域的归纳偏差丰富了主干模型。此外，我们的两步预训练过程允许我们仅调整几个超参数值，即可在多种下游任务上实现与最先进模型相当的良好性能。介绍了一种用于分子特性预测的新颖的图形转换器架构。提出了预训练该模型的与任务无关的方法，以最少的超参数调整来提高目标任务性能。对自注意力层的设计空间进行了严格的探索，以确定最佳架构。