Abstract
No assignee for this patent application has been made.News editors obtained the following quote from the background information suppli ed by the inventors:“Deep learning has witnessed an explosion of architectures of continuously growing capability and capacity.Aided by the rapid gains in har dware, models today may easily overfit many images (e.g., million images)and be gin to demand hundreds of millions of, often publicly inaccessible, labeled imag es. This appetite fordata has been successfully addressed in natural language p rocessing (NLP) by self-supervised pre-training.Solutions, based on autoregress ive language modelling and masked autoencoding in prior approaches(e.g., genera tive pre-trained transformer (GPT), bidirectional encoder representations from t ransformers(BERT)), are conceptually simple. This may include removing a portio n of the data and learn to predictthe removed content. These methods now enable training of generalizable NLP models containing overone hundred billion parame ters.
NETL
NSTL