Sentiment Analysis
Multi-class Text classification with GPT 3.5
10 Leading Language Models For NLP In 2022
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- GPT2: Language Models Are Unsupervised Multitask Learners
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- GPT3: Language Models Are Few-Shot Learners
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- PaLM: Scaling Language Modeling with Pathways
New AI classifier for indicating AI-written text
We’re launching a classifier trained to distinguish between AI-written and human-written text.
limitation
- The classifier is very unreliable on short texts (below 1,000 characters). Even longer texts are sometimes incorrectly labeled by the classifier.
- Sometimes human-written text will be incorrectly but confidently labeled as AI-written by our classifier.
- We recommend using the classifier only for English text. It performs significantly worse in other languages and it is unreliable on code.
- Very predictable text cannot be reliably identified. For example, it is impossible to predict whether AI or humans wrote a list of the first 1,000 prime numbers because the correct answer is always the same.
- AI-written text can be edited to evade the classifier. Classifiers like ours can be updated and retrained based on successful attacks, but it is unclear whether detection has long-term advantages.
- Classifiers based on neural networks are known to need to be better calibrated outside of their training data. For inputs that are very different from the text in our training set, the classifier is sometimes extremely confident in a wrong prediction.
Tagging and multi-class classification
Fine-tuning
Learn how to customize a model for your application.
https://platform.openai.com/docs/guides/fine-tuning
https://www.youtube.com/watch?v=ahnGLM-RC1Y
RAG retrieval augmented generation
RAG
Term Frequency Text Summarization
Remove stop words
python
text =
" Sometimes to understand a word's meaning you need more than a definition; you need to see the word used in a sentence. At YourDictionary, we give you the tools to learn what a word means and how to use it correctly. With this sentence maker, simply type a word in the search bar and see a variety of sentences with that word used in its different ways. Our sentence generator can provide more context and relevance, ensuring you use a word the right way."
import retext = re.sub(' +',' ', text)
text = text.replace('\n', ' ')
text = text.replace('\t', ' ')
text = text.replace('\r', ' ')
text = text.replace('\"', ' ')
text = text.replace('\'', ' ')
text = text.lower()
pattern = r'we[\'\']re'
replacement = 'we are'
text = re.sub(pattern,replacement,text)
pattern = r'we[\'\']ll'
replacement = 'we will'
text = re.sub(pattern,replacement,text) from bs4 import BeautifulSoup as bs4
import requests
import nltk
from nltk.tokenize import RegexpTokenizer
import heapq
import re
from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plttokenizer = RegexpTokenizer('\w+')
words = tokenizer.tokenize(text)
sentences = nltk.sent_tokenize(text)
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize stop_words = set(stopwords.words('english')) word_tokens = word_tokenize(text)# converts the words in word_tokens to lowercase and then checks whether #they are present in stop_words or notfiltered_sentence = [w for w in word_tokens if not w.lower() in stop_words]#with no lowercase conversionfiltered_sentence = [] for w in word_tokens: if w not in stop_words: filtered_sentence.append(w) words = filtered_sentencefrequency = nltk.FreqDist(words)
max_frequency = max(frequency.values())
for word in frequency.keys(): frequency[word] = frequency[word]/max_frequencyviz_words = [x for x in words if x not in nltk.corpus.stopwords.words('english')]
viz_words = [x for x in viz_words if x not in ['thing', 'min', 'really', 'wanted', 'way', 'want', 'going']]
viz_frequency = nltk.FreqDist(viz_words)
cloud = heapq.nlargest(40, viz_frequency, key=viz_frequency.get)
def plot_cloud(wordcloud): plt.figure(figsize=(20, 15)) plt.imshow(wordcloud) plt.axis("off") plt.savefig('wordcloud.png');wordcloud = WordCloud(width=3000, height=2000, random_state=1, background_color='#f7f0ee', colormap='binary_r', collocations=False, stopwords=STOPWORDS).generate(' '.join(cloud))plot_cloud(wordcloud)Visit us at DataDrivenInvestor.com
Subscribe to DDIntel here.
Have a unique story to share? Submit to DDIntel here.
Join our creator ecosystem here.
DDIntel captures the more notable pieces from our main site and our popular DDI Medium publication. Check us out for more insightful work from our community.
DDI Official Telegram Channel: https://t.me/+tafUp6ecEys4YjQ1