{ "slug": "Topic-Modeling-with-TF-IDF-and-k-means-and-Chat-GPT-3-5-4d03fc92a8aa", "title": "Topic Modeling with TF-IDF and k-means and Chat GPT 3.5", "subtitle": "In this article, I want to pre-process texts of questions and use the TF-IDF to change them into vector clusters with K-means and then find…", "excerpt": "In this article, I want to pre-process texts of questions and use the TF-IDF to change them into vector clusters with K-means and then find…", "date": "2024-02-08", "tags": [ "Machine Learning" ], "readingTime": "4 min", "url": "https://medium.com/@mobinshaterian/topic-modeling-with-tf-idf-and-k-means-and-chat-gpt-3-5-4d03fc92a8aa", "hero": "https://cdn-images-1.medium.com/max/800/1*H-EW5eHWzw00Bs3X0vpXrQ.png", "content": [ { "type": "heading", "level": 2, "text": "Topic Modeling with TF-IDF and k-means and Chat GPT 3.5" }, { "type": "paragraph", "html": "In this article, I want to pre-process texts of questions and use the TF-IDF to change them into vector clusters with K-means and then find the best topic for them with chat GPT 3.5." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*H-EW5eHWzw00Bs3X0vpXrQ.png", "alt": "https://www.inboundwriter.com/lifestyle/how-to-introduce-a-topic-clearly/", "caption": "https://www.inboundwriter.com/lifestyle/how-to-introduce-a-topic-clearly/", "width": 620, "height": 417 }, { "type": "heading", "level": 2, "text": "Read CSV file" }, { "type": "code", "lang": "python", "code": "import pandas as pddf = pd.read_csv('data.csv')print(df.to_string())" }, { "type": "heading", "level": 2, "text": "Group by" }, { "type": "code", "lang": "python", "code": "df = pd.DataFrame({'Animal': ['Falcon', 'Falcon', 'Parrot', 'Parrot'],\n'Max Speed': [380., 370., 24., 26.]})df Animal Max Speed0 Falcon 380.01 Falcon 370.02\nParrot 24.03 Parrot 26.0df.groupby(['Animal']).mean() Max SpeedAnimalFalcon\n375.0Parrot 25.0" }, { "type": "heading", "level": 2, "text": "Sort by" }, { "type": "code", "lang": "text", "code": "df.sort_values(by=['col1']) col1 col2 col3 col40 A 2 0 a1 A 1 1 B2\nB 9 9 c5 C 4 3 F4 D 7 2 e3 NaN 8 4 D" }, { "type": "heading", "level": 2, "text": "Select special rows" }, { "type": "paragraph", "html": "I’m interested in the Titanic passengers from cabin class 2 and 3." }, { "type": "code", "lang": "python", "code": "class_23 = titanic[titanic[\"Pclass\"].isin([2, 3])]class_23.head()Out[17]: PassengerId Survived\nPclass ... Fare Cabin Embarked0 1 0 3 ... 7.2500 NaN S2\n3 1 3 ... 7.9250 NaN S4 5 0 3 ... 8.0500\nNaN S5 6 0 3 ... 8.4583 NaN Q7 8 0\n3 ... 21.0750 NaN S[5 rows x 12 columns]" }, { "type": "heading", "level": 2, "text": "TF-IDF" }, { "type": "paragraph", "html": "bag of words algorithms." }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*ePqe-TBZT8IJheoXdMCdnQ.png", "alt": "Topic Modeling with TF-IDF and k-means and Chat GPT 3.5", "caption": "", "width": 1379, "height": 780 }, { "type": "embed", "provider": "youtube", "url": "https://www.youtube.com/embed/vZAXpvHhQow?feature=oembed" }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*f-urPw-YOjqTLg0MvrYtjw.png", "alt": "Topic Modeling with TF-IDF and k-means and Chat GPT 3.5", "caption": "", "width": 1016, "height": 574 }, { "type": "embed", "provider": "youtube", "url": "https://www.youtube.com/embed/zLMEnNbdh4Q?feature=oembed" }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*TOqhvNO1ucRuowM7-HuqoA.png", "alt": "Topic Modeling with TF-IDF and k-means and Chat GPT 3.5", "caption": "", "width": 463, "height": 424 }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*UfboCIExn6YF8WOe-N-Ifg.png", "alt": "Topic Modeling with TF-IDF and k-means and Chat GPT 3.5", "caption": "", "width": 1018, "height": 562 }, { "type": "image", "src": "https://cdn-images-1.medium.com/max/800/1*_GdKsWp0OLX0wbEzvm0cIw.png", "alt": "Topic Modeling with TF-IDF and k-means and Chat GPT 3.5", "caption": "", "width": 1537, "height": 868 }, { "type": "heading", "level": 2, "text": "TF-IDF Example" }, { "type": "code", "lang": "python", "code": "from sklearn.feature_extraction.text\nimport TfidfVectorizer>>> corpus = [... 'This is the first document.',...\n'This document is the second document.',... 'And this is the third one.',...\n'Is this the first document?',... ]>>> vectorizer = TfidfVectorizer()>>> X =\nvectorizer.fit_transform(corpus)>>> vectorizer.get_feature_names_out()array(['and', 'document',\n'first', 'is', 'one', 'second', 'the', 'third', 'this'], ...)>>> print(X.shape)" }, { "type": "heading", "level": 2, "text": "Converting text content into a numerical vector" }, { "type": "paragraph", "html": "That’s great! Converting text content into a numerical vector is a valuable step in many NLP tasks. Here’s a deeper dive into the methods you mentioned:" }, { "type": "paragraph", "html": "1. Bag-of-Words (BoW):" }, { "type": "list", "ordered": false, "items": [ "Simple and computationally efficient.", "Treats documents as unordered collections of words, losing word order and context.", "Useful for tasks like topic modeling and similarity comparisons based on word presence." ] }, { "type": "paragraph", "html": "2. TF-IDF (Term Frequency-Inverse Document Frequency):" }, { "type": "list", "ordered": false, "items": [ "Addresses limitations of BoW by weighting words based on their importance within a document and across a corpus.", "Words appearing frequently in a document but rarely overall get higher weights.", "Helps identify distinctive and informative terms." ] }, { "type": "paragraph", "html": "3. Word Embeddings:" }, { "type": "list", "ordered": false, "items": [ "Go beyond word presence by capturing semantic meaning and relationships between words.", "Train word embedding models on large text corpora to learn vector representations.", "Popular options include:", "Word2Vec: Captures word relationships based on context (e.g., “king” is similar to “queen”).", "GloVe: Combines statistical information and word co-occurrence data.", "fastText: Handles subword information well, useful for languages with complex morphology." ] }, { "type": "paragraph", "html": "4. Document Embeddings:" }, { "type": "list", "ordered": false, "items": [ "Represent entire documents as single vectors, useful for tasks like document classification and retrieval.", "Methods include:", "Doc2Vec: Learns document vectors based on word embeddings and document structure.", "Universal Sentence Encoder (USE): Pre-trained model representing sentences as dense vectors." ] }, { "type": "paragraph", "html": "Choosing the right method depends on your specific task and needs:" }, { "type": "list", "ordered": false, "items": [ "For simple tasks, BoW or TF-IDF might suffice.", "For capturing semantic relationships and context, word embeddings are powerful.", "For representing whole documents, document embeddings are valuable." ] }, { "type": "paragraph", "html": "I hope this clarifies the options available! Feel free to ask if you have further questions about specific methods or have a particular NLP task in mind." }, { "type": "heading", "level": 2, "text": "Word embedding" }, { "type": "paragraph", "html": "Word embedding in NLP is an important term that is used for representing words for text analysis in the form of real-valued vectors. It is an advancement in NLP that has improved the ability of computers to understand text-based content in a better way." }, { "type": "heading", "level": 2, "text": "TF-IDF Code" }, { "type": "code", "lang": "python", "code": "from sklearn.feature_extraction.text\nimport TfidfVectorizervectorizer = TfidfVectorizer(\n max_df=0.5,\n min_df=5,\n stop_words=\"english\",)t0 = time()X_tfidf =\n vectorizer.fit_transform(dataset.data)print(f\"vectorization done in {time() - t0:.3f} s\")print(f\"n_samples: {X_tfidf.shape[0]}, n_features: {X_tfidf.shape[1]}\")" }, { "type": "heading", "level": 2, "text": "Kmeans Code" }, { "type": "code", "lang": "python", "code": "from sklearn.cluster\nimport KMeansfor seed in range(5):\n kmeans = KMeans( n_clusters=10, max_iter=100, n_init=1,\n random_state=seed,\n ).fit(X)\n cluster_ids, cluster_sizes = np.unique(kmeans.labels_, return_counts=True)\n print(f\"Number of elements assigned to each cluster: {cluster_sizes}\")" }, { "type": "heading", "level": 2, "text": "Final code" }, { "type": "code", "lang": "python", "code": "import pandas as pd\nfrom sklearn.feature_extraction.text\nimport TfidfVectorizer\nimport numpy as np\nfrom sklearn.cluster\nimport KMeans\nfrom openai\nimport OpenAI\nimport os\nfrom dotenv\nimport load_dotenvdata = pd.read_csv('data.csv')corpus = data['Text'].to_list()# TF-IDFvectorizer =\nTfidfVectorizer(\n max_df=0.5,\n min_df=5,\n stop_words=\"english\",)X = vectorizer.fit_transform(corpus)vectorizer.get_feature_names_out()#\n Kmeanskmeans = KMeans(\n n_clusters=80,\n max_iter=100,\n n_init=5,).fit(X)cluster_ids, cluster_sizes = np.unique(kmeans.labels_,\n return_counts=True)print(f\"Number of elements assigned to each cluster: {cluster_sizes}\")labels\n = kmeans.labels_data[\"Cluster\"] = labelscluster = data.copy()cluster =\n cluster.sort_values('Cluster')cluster.to_csv('label.csv',index=True)# groupsgroups =\n cluster.groupby('Cluster', as_index=False).aggregate('count')groups =groups[['Cluster' ,\n 'Index']]groups.sort_values('Index')groups.to_csv('sort_label.csv',index=True)selected_cluster =\n groups.query('Index < 30')# open AIload_dotenv()key = os.getenv('OPENAI_API_KEY')client =\n OpenAI(api_key=key)\ndef group_meaning_gpt(questions):\n messages = [ {\"role\": \"user\", \"content\":\n f'What do the following customer questions have in common? .\\n\\nCustomer questions:\\n\"\"\"\\n{questions}\\n\"\"\"\\n\\nTheme:'}\n ]\n response = client.chat.completions.create( model=\"gpt-3.5-turbo-1106\",\n messages=messages, temperature=0, max_tokens=64, top_p=1,\n frequency_penalty=0, presence_penalty=0)\n return response.choices[0].message.content.replace(\"\\n\", \"\")questions = \"\"for index_cluster in\n selected_cluster['Cluster']:\n batches = cluster[cluster['Cluster'] == index_cluster]\n for batch in batches['Text']: if questions != \"\": questions = questions +\n \"\\n\\n\" + batch else : questions = batch\n anwser = group_meaning_gpt(questions)\n new_batcher = batches.copy()\n new_batcher.loc[ : , 'Label'] = anwser\n new_batcher.to_csv('cluster_'+str(anwser)+'.csv',index=True)\n break" }, { "type": "paragraph", "html": "Special thanks to Elmira Ghorbani" }, { "type": "heading", "level": 2, "text": "Stackademic" }, { "type": "paragraph", "html": "Thank you for reading until the end. Before you go:" }, { "type": "list", "ordered": false, "items": [ "Please consider clapping and following the writer! 👏", "Follow us X | LinkedIn | YouTube | Discord", "Visit our other platforms: In Plain English | CoFeed | Venture" ] } ] }