Crawler YouTube subtitle For making an auto-response AI bot

In this tutorial, code for downloading YouTube with Python, getting subtitles, and getting to the RAG system.

Install

bash
sudo
curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
sudo
chmod a+rx /usr/local/bin/youtube-dl

Python

bash
sudo -H pip install --upgrade youtube-dl

Error: Unable to extract uploader id — Youtube, Discord.py

bash
python3 -m pip install --force-reinstall
https://github.com/yt-dlp/yt-dlp/archive/master.tar.gz
text
yt-dlp URL
python
import yt_dlp as youtube_dl

Download youtube

text
yt-dlp
https://www.youtube.com/watch?v=Cui3IFx_zbA&t=321s

Download SRT manually

text
https://downsub.com/

list of SRT

text
yt-dlp  --list-subs
https://www.youtube.com/watch?v=Cui3IFx_zbA&t=321s
text
Language Name                  Formatsaf       Afrikaans             vtt, ttml, srv3, srv2, srv1,
json3ak       Akan                  vtt, ttml, srv3, srv2, srv1, json3sq       Albanian
vtt, ttml, srv3, srv2, srv1, json3am       Amharic               vtt, ttml, srv3, srv2, srv1,
json3ar       Arabic                vtt, ttml, srv3, srv2, srv1, json3hy       Armenian
vtt, ttml, srv3, srv2, srv1, json3as       Assamese              vtt, ttml, srv3, srv2, srv1,
json3ay       Aymara                vtt, ttml, srv3, srv2, srv1, json3az       Azerbaijani
vtt, ttml, srv3, srv2, srv1, json3bn       Bangla                vtt, ttml, srv3, srv2, srv1,
json3eu       Basque                vtt, ttml, srv3, srv2, srv1, json3be       Belarusian
vtt, ttml, srv3, srv2, srv1, json3bho      Bhojpuri              vtt, ttml, srv3, srv2, srv1,
json3bs       Bosnian               vtt, ttml, srv3, srv2, srv1, json3bg       Bulgarian
vtt, ttml, srv3, srv2, srv1, json3my       Burmese               vtt, ttml, srv3, srv2, srv1,
json3ca       Catalan               vtt, ttml, srv3, srv2, srv1, json3

Downlaod SRT

text
yt-dlp --write-auto-sub --sub-format srt  --sub-lang en --skip-download
https://www.youtube.com/watch?v=Cui3IFx_zbA
text
yt-dlp --write-auto-sub --convert-subs=srt --skip-download
https://www.youtube.com/watch?v=Cui3IFx_zbA

Python read vtt file

bash
pip install webvtt-py
python
import webvttvtt = webvtt.read('file.vtt')transcript = ""lines = []for line in vtt:
    # Strip the newlines from the end of the text.
    # Split the string if it has a newline in the middle
    # Add the lines to an array
    lines.extend(line.text.strip().splitlines())# Remove repeated linesprevious = Nonefor line in
    lines:
    if line == previous:       continue
    transcript += " " + line
    previous = linep

Python Download YouTube

python
import yt_dlp
from yt_dlp.utils
import download_range_funcyt_opts = {            'writesubtitles': True,
'writeautomaticsub': True,            'subtitlesformat': 'vtt',            'skip_download': True,
}yt_opts['outtmpl'] = self.download_folder+'/video'+'.%(ext)s'        with yt_dlp.YoutubeDL(yt_opts)
as ydl:            ydl.download("https://www.youtube.com/watch?v=DhPhJ3Z0XxU")            info =
ydl.extract_info("https://www.youtube.com/watch?v=DhPhJ3Z0XxU")            print(info.get('title'))
return info

RAQ

Now data is ready and can send to the RAG system. I explain it in another article.

Stackademic 🎓

Thank you for reading until the end. Before you go: