{
  "slug": "Topic-Modeling-with-Word2vec-and-k-means-and-ChatGPT-3-5-a1214d9d08d0",
  "title": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
  "subtitle": "In this article I am going to convert texts into vectors. And find best topics for them based on ChatGPT.",
  "excerpt": "In this article I am going to convert texts into vectors. And find best topics for them based on ChatGPT.",
  "date": "2024-02-15",
  "tags": [
    "Machine Learning"
  ],
  "readingTime": "4 min",
  "url": "https://medium.com/@mobinshaterian/topic-modeling-with-word2vec-and-k-means-and-chatgpt-3-5-a1214d9d08d0",
  "hero": "https://cdn-images-1.medium.com/max/800/1*YiG3oVaj9a6Pjs06NZUtvw.png",
  "content": [
    {
      "type": "heading",
      "level": 2,
      "text": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5"
    },
    {
      "type": "paragraph",
      "html": "In this article I am going to convert words into vectors and use it in find topics for groups of tickets."
    },
    {
      "type": "embed",
      "provider": "youtube",
      "url": "https://www.youtube.com/embed/viZrOnJclY0?feature=oembed"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*YiG3oVaj9a6Pjs06NZUtvw.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 976,
      "height": 400
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*OPYhQGnP2nqlG-F89pNoQQ.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 1189,
      "height": 541
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*H2IUgmCw-cHeZDWirce_iA.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 762,
      "height": 245
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*O57zbkyuNZTvaBaA__yYVA.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 1471,
      "height": 756
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*qBXu9bfB_0v52C3TqlzyEA.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 1054,
      "height": 568
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*1jab1csMbWP6wKcO7Hd7Zg.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 1042,
      "height": 551
    },
    {
      "type": "paragraph",
      "html": "Word2Vec was proposed in 2013 to learn word embeddings by using neural networks from huge data sets with billlions of words. There are 2 approaches (cbow and sg) introduced in the paper to learn word embeddings. We will also use window size as 2 for the illustrations."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Model Architecture"
    },
    {
      "type": "paragraph",
      "html": "The neural network models that are used for the approaches above are illustrated in the picture below. They are simple neural networks with one layer without a non-linear activation, and then a softmax layer for the outputs."
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*0frC3lIYwLQOhvL7CkUA9g.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 939,
      "height": 452
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Gensim"
    },
    {
      "type": "paragraph",
      "html": "Python framework for fast Vector Space Modelling"
    },
    {
      "type": "paragraph",
      "html": "Gensim is a Python library for <em>topic modelling</em>, <em>document indexing</em> and <em>similarity retrieval</em> with large corpora. Target audience is the <em>natural language processing</em> (NLP) and <em>information retrieval</em> (IR) community."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "How to Cluster Documents Using Word2Vec and K-means"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "How to Cluster Documents"
    },
    {
      "type": "paragraph",
      "html": "You can think of the process of clustering documents in three steps:"
    },
    {
      "type": "list",
      "ordered": true,
      "items": [
        "<a href=\"https://dylancastillo.co/nlp-snippets-clean-and-tokenize-text-with-python/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Cleaning and tokenizing data</strong></a> usually involves lowercasing text, removing non-alphanumeric characters, or stemming words.",
        "<strong>Generating vector representations of the documents</strong> concerns the mapping of documents from words into numerical vectors — some common ways of doing this include using bag-of-words models or word embeddings.",
        "<strong>Applying a clustering algorithm on the document vectors</strong> requires selecting and applying a clustering algorithm to find the best possible groups using the document vectors. Some frequently used algorithms include <strong>K-means, DBSCAN, or Hierarchical Clustering.</strong>"
      ]
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Clean and Tokenize Data"
    },
    {
      "type": "paragraph",
      "html": "After you import the required libraries, you need to read and preprocess the data you’ll use in your clustering algorithm. The preprocessing consists of cleaning and tokenizing the data. To do that, copy the following function in a new cell in your notebook:"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "How the Python Lambda Function Works — Explained with Examples"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "lambda argument(s) : expression"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "def f(x):  return x * 2f(3)>> 6"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "import pandas as pddf = pd.DataFrame(\n    {\"name\": [\"IBRAHIM\", \"SEGUN\", \"YUSUF\", \"DARE\", \"BOLA\", \"SOKUNBI\"],     \"score\": [50, 32, 45, 45,\n    23, 45]\n    })df[\"lower_name\"] = df[\"name\"].apply(lambda x: x.lower())"
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*QkKo8fL2qX4C1iY4mb4PGQ.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 175,
      "height": 209
    },
    {
      "type": "image",
      "src": "https://cdn-images-1.medium.com/max/800/1*UoDwe18A9JHdWBXqLz5huw.png",
      "alt": "Topic Modeling with Word2vec and k-means and ChatGPT 3.5",
      "caption": "",
      "width": 232,
      "height": 226
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Step1. Clean and tokenize your data"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "import os\nimport random\nimport re\nimport string\nimport nltk\nimport numpy as np\nimport pandas as pd\nfrom gensim.models\nimport Word2Vec\nfrom nltk\nimport word_tokenize\nfrom nltk.corpus\nimport stopwords\nfrom sklearn.cluster\nimport MiniBatchKMeans\nfrom sklearn.metrics\nimport silhouette_samples, silhouette_scorenltk.download(\"stopwords\")nltk.download(\"punkt\")SEED =\n42random.seed(SEED)os.environ[\"PYTHONHASHSEED\"] = str(SEED)np.random.seed(SEED)"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "def clean_text(text, tokenizer, stopwords):\n    \"\"\"Pre-process text and generate tokens\n    Args:        text: Text to tokenize.\n    Returns:        Tokenized text.\n    \"\"\"\n    text = str(text).lower()  # Lowercase words\n    text = re.sub(r\"\\[(.*?)\\]\", \"\", text)  # Remove [+XYZ chars] in content\n    text = re.sub(r\"\\s+\", \" \", text)  # Remove multiple spaces in content\n    text = re.sub(r\"\\w+…|…\", \"\", text)  # Remove ellipsis (and last word)\n    text = re.sub(r\"(?<=\\w)-(?=\\w)\", \" \", text)  # Replace dash between words\n    text = re.sub(        f\"[{re.escape(string.punctuation)}]\", \"\", text\n    )  # Remove punctuation\n    tokens = tokenizer(text)  # Get tokens from text\n    tokens = [t for t in tokens if not t in stopwords]  # Remove stopwords\n    tokens = [\"\" if t.isdigit() else t for t in tokens]  # Remove digits\n    tokens = [t for t in tokens if len(t) > 1]  # Remove short tokens\n    return tokens"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "custom_stopwords = set(stopwords.words('english') + list(string.punctuation) + list([\"I\" ,\n\"--\"]))df_raw = pd.read_csv(\"data.csv\")df = df_raw[['ID','Summarize']].copy()df[\"Tokens\"] =\ndf[\"Summarize\"].map(lambda x: clean_text(x, word_tokenize, custom_stopwords))# Remove duplicated\nafter preprocessing_, idx = np.unique(df[\"Tokens\"], return_index=True)df = df.iloc[idx, :]docs =\ndf[\"Summarize\"].valuestokenized_docs = df[\"Tokens\"].valuesp"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Step2. Convert words to vector"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "model = Word2Vec(sentences=tokenized_docs, vector_size=100, workers=1, seed=SEED)"
    },
    {
      "type": "paragraph",
      "html": "<strong>Note:</strong> In many cases, you might want to use a pre-trained model instead of training one yourself. If that’s the case, gensim provides you with an easy way to access some of the <a href=\"https://github.com/RaRe-Technologies/gensim-data?ref=dylancastillo.co#models\" target=\"_blank\" rel=\"noreferrer noopener\">most popular pre-trained word embeddings</a>."
    },
    {
      "type": "paragraph",
      "html": "You can load a pre-trained Word2Vec model as follows:"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "wv = api.load('word2vec-google-news-300')"
    },
    {
      "type": "paragraph",
      "html": "One last thing, if you’re following this tutorial and decide to use a pre-trained model, you’ll need to replace <code>model.wv</code> by <code>wv</code> in the code snippets from here on. Otherwise, you'll get an error."
    },
    {
      "type": "heading",
      "level": 2,
      "text": "You’ve trained your Word2Vec model"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "model.wv.most_similar(\"trump\")"
    },
    {
      "type": "code",
      "lang": "text",
      "code": "[('trumps', 0.988541841506958), ('president', 0.9746493697166443), ('donald', 0.9274922013282776),\n('ivanka', 0.9203903079032898), ('impeachment', 0.9195784330368042), ('pences', 0.9152231812477112),\n('avlon', 0.9148306846618652), ('biden', 0.9146010279655457), ('breitbart', 0.9144087433815002),\n('vice', 0.9067237973213196)]"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Generate vectors for list of documents using a Word Embedding"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "def vectorize(list_of_docs, model):\n    \"\"\"Generate vectors for list of documents using a Word Embedding\n    Args:        list_of_docs: List of documents        model: Gensim's Word Embedding\n    Returns:        List of document vectors\n    \"\"\"\n    features = []\n    for tokens in list_of_docs:        zero_vector = np.zeros(model.vector_size)        vectors = []\n    for token in tokens:            if token in model.wv:                try:\n    vectors.append(model.wv[token])                except KeyError:                    continue\n    if vectors:            vectors = np.asarray(vectors)            avg_vec = vectors.mean(axis=0)\n    features.append(avg_vec)        else:            features.append(zero_vector)\n    return features\n    vectorized_docs = vectorize(tokenized_docs, model=model)len(vectorized_docs),\n    len(vectorized_docs[0])"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "import csvwith open('vectors.csv', 'w', newline='') as file:\n    # Step 4: Using csv.writer to write the list to the CSV file\n    writer = csv.writer(file)\n    writer.writerow(vectorized_docs) # Use writerow for single list"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Cluster Documents Using K-means"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "from sklearn.cluster\nimport KMeansfor seed in range(5):\n    kmeans = KMeans(        n_clusters=10,        max_iter=100,        n_init=1,\n    random_state=seed,\n    ).fit(X)\n    cluster_ids, cluster_sizes = np.unique(kmeans.labels_, return_counts=True)\n    print(f\"Number of elements assigned to each cluster: {cluster_sizes}\")"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Using ChatGPT 3.5 for match best topics"
    },
    {
      "type": "code",
      "lang": "python",
      "code": "# groupsgroups = cluster.groupby('Cluster', as_index=False).aggregate('count')\ngroups\n=groups[['Cluster' ,\n'Index']]groups.sort_values('Index')\ngroups.to_csv('sort_label.csv',index=True)\nselected_cluster =\ngroups.query('Index < 30')# open AIload_dotenv()\nkey = os.getenv('OPENAI_API_KEY')\nclient =\nOpenAI(api_key=key)\ndef group_meaning_gpt(questions):    messages = [        {\"role\": \"user\",\n\"content\":\nf'What do the following customer questions have in common? .\\n\\nCustomer questions:\\n\"\"\"\\n{questions}\\n\"\"\"\\n\\nTheme:'}    ]    response = client.chat.completions.create(        model=\"gpt-3.5-turbo-1106\",        messages=messages,        temperature=0,        max_tokens=64,        top_p=1,        frequency_penalty=0,        presence_penalty=0)    return response.choices[0].message.content.replace(\"\\n\", \"\")\nquestions = \"\"\nfor index_cluster in selected_cluster['Cluster']:    batches = cluster[cluster['Cluster'] == index_cluster]    for batch in batches['Text']:        if questions != \"\":            questions = questions + \"\\n\\n\" + batch        else :            questions = batch    anwser = group_meaning_gpt(questions)    new_batcher = batches.copy()    new_batcher.loc[ : , 'Label']  = anwser    new_batcher.to_csv('cluster_'+str(anwser)+'.csv',index=True)    break"
    },
    {
      "type": "heading",
      "level": 2,
      "text": "Stackademic"
    },
    {
      "type": "paragraph",
      "html": "Thank you for reading until the end. Before you go:"
    },
    {
      "type": "list",
      "ordered": false,
      "items": [
        "Please consider <strong>clapping</strong> and <strong>following</strong> the writer! 👏",
        "Follow us <a href=\"https://twitter.com/stackademichq\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>X</strong></a><strong> | </strong><a href=\"https://www.linkedin.com/company/stackademic\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>LinkedIn</strong></a><strong> | </strong><a href=\"https://www.youtube.com/c/stackademic\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>YouTube</strong></a><strong> | </strong><a href=\"https://discord.gg/in-plain-english-709094664682340443\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Discord</strong></a>",
        "Visit our other platforms: <a href=\"https://plainenglish.io\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>In Plain English</strong></a><strong> | </strong><a href=\"https://cofeed.app/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>CoFeed</strong></a><strong> | </strong><a href=\"https://venturemagazine.net/\" target=\"_blank\" rel=\"noreferrer noopener\"><strong>Venture</strong></a>"
      ]
    }
  ]
}