{
"slug": "Crawler-YouTube-subtitle-For-making-an-auto-response-AI-bot-59ea68f3fdbb",
"title": "Crawler YouTube subtitle For making an auto-response AI bot",
"subtitle": "In this tutorial, code for downloading YouTube with Python, getting subtitles, and getting to the RAG system.",
"excerpt": "In this tutorial, code for downloading YouTube with Python, getting subtitles, and getting to the RAG system.",
"date": "2024-02-20",
"tags": [
"Python",
"Machine Learning"
],
"readingTime": "2 min",
"url": "https://medium.com/@mobinshaterian/crawler-youtube-subtitle-for-making-an-auto-response-ai-bot-59ea68f3fdbb",
"hero": "https://cdn-images-1.medium.com/max/800/1*f9Mo4zUnZPVpnZ_L9FaSQg.jpeg",
"content": [
{
"type": "heading",
"level": 2,
"text": "Crawler YouTube subtitle For making an auto-response AI bot"
},
{
"type": "paragraph",
"html": "In this tutorial, code for downloading YouTube with Python, getting subtitles, and getting to the RAG system."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*f9Mo4zUnZPVpnZ_L9FaSQg.jpeg",
"alt": "Crawler YouTube subtitle For making an auto-response AI bot",
"caption": "",
"width": 1152,
"height": 768
},
{
"type": "heading",
"level": 2,
"text": "Install"
},
{
"type": "code",
"lang": "bash",
"code": "sudo\ncurl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl\nsudo\nchmod a+rx /usr/local/bin/youtube-dl"
},
{
"type": "heading",
"level": 3,
"text": "Python"
},
{
"type": "code",
"lang": "bash",
"code": "sudo -H pip install --upgrade youtube-dl"
},
{
"type": "heading",
"level": 2,
"text": "Error: Unable to extract uploader id — Youtube, Discord.py"
},
{
"type": "code",
"lang": "bash",
"code": "python3 -m pip install --force-reinstall\nhttps://github.com/yt-dlp/yt-dlp/archive/master.tar.gz"
},
{
"type": "code",
"lang": "text",
"code": "yt-dlp URL"
},
{
"type": "code",
"lang": "python",
"code": "import yt_dlp as youtube_dl"
},
{
"type": "heading",
"level": 2,
"text": "Download youtube"
},
{
"type": "code",
"lang": "text",
"code": "yt-dlp\nhttps://www.youtube.com/watch?v=Cui3IFx_zbA&t=321s"
},
{
"type": "heading",
"level": 2,
"text": "Download SRT manually"
},
{
"type": "code",
"lang": "text",
"code": "https://downsub.com/"
},
{
"type": "heading",
"level": 2,
"text": "list of SRT"
},
{
"type": "code",
"lang": "text",
"code": "yt-dlp --list-subs\nhttps://www.youtube.com/watch?v=Cui3IFx_zbA&t=321s"
},
{
"type": "code",
"lang": "text",
"code": "Language Name Formatsaf Afrikaans vtt, ttml, srv3, srv2, srv1,\njson3ak Akan vtt, ttml, srv3, srv2, srv1, json3sq Albanian\nvtt, ttml, srv3, srv2, srv1, json3am Amharic vtt, ttml, srv3, srv2, srv1,\njson3ar Arabic vtt, ttml, srv3, srv2, srv1, json3hy Armenian\nvtt, ttml, srv3, srv2, srv1, json3as Assamese vtt, ttml, srv3, srv2, srv1,\njson3ay Aymara vtt, ttml, srv3, srv2, srv1, json3az Azerbaijani\nvtt, ttml, srv3, srv2, srv1, json3bn Bangla vtt, ttml, srv3, srv2, srv1,\njson3eu Basque vtt, ttml, srv3, srv2, srv1, json3be Belarusian\nvtt, ttml, srv3, srv2, srv1, json3bho Bhojpuri vtt, ttml, srv3, srv2, srv1,\njson3bs Bosnian vtt, ttml, srv3, srv2, srv1, json3bg Bulgarian\nvtt, ttml, srv3, srv2, srv1, json3my Burmese vtt, ttml, srv3, srv2, srv1,\njson3ca Catalan vtt, ttml, srv3, srv2, srv1, json3"
},
{
"type": "heading",
"level": 2,
"text": "Downlaod SRT"
},
{
"type": "code",
"lang": "text",
"code": "yt-dlp --write-auto-sub --sub-format srt --sub-lang en --skip-download\nhttps://www.youtube.com/watch?v=Cui3IFx_zbA"
},
{
"type": "code",
"lang": "text",
"code": "yt-dlp --write-auto-sub --convert-subs=srt --skip-download\nhttps://www.youtube.com/watch?v=Cui3IFx_zbA"
},
{
"type": "heading",
"level": 2,
"text": "Python read vtt file"
},
{
"type": "code",
"lang": "bash",
"code": "pip install webvtt-py"
},
{
"type": "code",
"lang": "python",
"code": "import webvttvtt = webvtt.read('file.vtt')transcript = \"\"lines = []for line in vtt:\n # Strip the newlines from the end of the text.\n # Split the string if it has a newline in the middle\n # Add the lines to an array\n lines.extend(line.text.strip().splitlines())# Remove repeated linesprevious = Nonefor line in\n lines:\n if line == previous: continue\n transcript += \" \" + line\n previous = linep"
},
{
"type": "heading",
"level": 2,
"text": "Python Download YouTube"
},
{
"type": "code",
"lang": "python",
"code": "import yt_dlp\nfrom yt_dlp.utils\nimport download_range_funcyt_opts = { 'writesubtitles': True,\n'writeautomaticsub': True, 'subtitlesformat': 'vtt', 'skip_download': True,\n}yt_opts['outtmpl'] = self.download_folder+'/video'+'.%(ext)s' with yt_dlp.YoutubeDL(yt_opts)\nas ydl: ydl.download(\"https://www.youtube.com/watch?v=DhPhJ3Z0XxU\") info =\nydl.extract_info(\"https://www.youtube.com/watch?v=DhPhJ3Z0XxU\") print(info.get('title'))\nreturn info"
},
{
"type": "heading",
"level": 2,
"text": "RAQ"
},
{
"type": "paragraph",
"html": "Now data is ready and can send to the RAG system. I explain it in another article."
},
{
"type": "heading",
"level": 2,
"text": "Stackademic 🎓"
},
{
"type": "paragraph",
"html": "Thank you for reading until the end. Before you go:"
},
{
"type": "list",
"ordered": false,
"items": [
"Please consider clapping and following the writer! 👏",
"Follow us X | LinkedIn | YouTube | Discord",
"Visit our other platforms: In Plain English | CoFeed | Venture | Cubed",
"More content at Stackademic.com"
]
}
]
}