Fine-tune the BERT_base_sno model with training data to get BERT_base_sno_clf ('clf' stands for classifier).

Prepare the training data and testing data.

Pre-train the BERT_base (released by Google) model to get BERT_base_sno ('sno' stands for SNOMED).

Create pre-training data from medical text (we used SNOMED CT text data).

This repository covers both the pre-training and fine-tuning in the following steps: Then we can fine-tune this new model with the training data for our task.

We do not bind our model with any specific task.Īfter we obtain the BERT_base_medical model, we can add a classifier/regression layer on top of it, say BERT_base_medical_classifier. Note that here we are still in the scope of pre-training.

We can refer to the obtained model as BERT_base_medical. If we want to give the model a 'medical flavor,' we can adopt the same training process Google did, but training BERT_base with medical domain text data. For example, the model we used here is BERT, which is pre-trained by Google using Wikipedia and BookCorpus text data, denoted as BERT_base. However, if we want, we can continue pre-training the model pre-trained by others. Pre-training is normally expensive, because the model typically has a large number of parameters that need to be trained from scratch with a large size training data.įine-tuning is normally cheap, because we will reuse the pre-trained model and exploit the 'knowledge' it already learned, by training it with a small training dataset for our task-of-interest. The idea of transfer learning is to use the knowledge learned from one task, and use it for another task.įor example, we can take the a 'pre-trained' CNN model trained using all the labeled general images (a large dataset) on ImageNet, and 'fine-tune' it as a dog/cat classifier with our own training data (small dataset). General idea of Transfer Learning with Google BERT Experiment with training BERT for ontology engeneering.

Njit mobaxterm