WeLT: Improving Biomedical Fine-tuned Pre-trained Language Models with Cost-sensitive Learning

PoLiMeR is funded through the EU Marie Skłodowska-Curie Innovative Training Network (ITN), which drives scientific excellence and innovation. ITNs bring together universities, research institutes, industry and clinical partners from across the world to train researchers to doctorate level.

Metabolic diseases are a burden on the European population and health care system. It is increasingly recognised that individual differences with respect to history, lifestyle, and genetic make-up affect disease ...

In biomedical text mining, named entity recognition (NER) is an important task used to extract information from biomedical articles. Improving the NER’s performance will directly have a positive impact on extracting relations between those entities. In recent years, deep learning has become the main research direction of NER due to the development of effective models. Language transformer models like e.g. BERT are frequently used because they enable the specialisation of models by domain-specific ...

Biomedical pre-trained language models (BioPLMs) have been achieving state-of-the-art results for various biomedical text mining tasks. However, prevailing fine-tuning approaches naively train BioPLMs on targeted datasets without considering the class distributions. This is problematic, especially with dealing with imbalanced biomedical gold-standard datasets for named entity recognition (NER). Regardless of the high-performing SOTA fine-tuned NER models, they are biased towards other (O) tags ...