Sriram Sanka

Databases | Cloud | Infrastructure | Security |

Posts Tagged ‘DOWNLOAD’

Download Youtube ( Channel ) videos using Python Module – Pytube ,scrapetube.

Posted by Sriram Sanka on April 25, 2023

In this post, I will present the code to download Youtube Individual/Channel Videos step-by-step.

You can download the Videos using pytube module by passing the video URL as an argument. Just Install pytube and run as below.

python -m pip install pytube

This will download the given video URL to your current Directory, what about if you want to store your local copy for your educational purpose when you are offline.? The following will download the all the videos from the given URL to your desired location in the PC.

import requests
import re
from bs4 import BeautifulSoup
from pytube.cli import on_progress
from pytube import YouTube
from pytube import Playlist
from pytube import Channel


def get_youtube(i):
    try:
        yt = YouTube(i,on_progress_callback=on_progress)
        yt.streams.filter(progressive=True, file_extension='mp4').order_by('resolution').desc().first().download('C:\\Users\\Dell\\Videos\\downs\\Sriram_Channel\\')
    except:
        print(f'\nError in downloading:  {yt.title} from -->' + i)
        pass
l = Channel('https://www.youtube.com/channel/UCw43xCtkl26vGIGtzBoaWEw')

for video in l.video_urls:
    get_youtube(video)

You may get regular Expression error in fetching the video information due to changes implemented in the channel URL using symbols. or any other restrictions which are unsupported by the pytube module, either you have to download and install the latest or identify and fix it on your own.

To avoid this, you can use another module called scrapetube

pip install scrapetube

The following code will give you the Video id from the URL provided as input. you need to append ‘https://www.youtube.com/watch?v=’ to mark that as a complete URL.

import scrapetube

videos = scrapetube.get_channel("........t7XvGJ3AGpSon.......")

for video in videos:
    print('https://www.youtube.com/watch?v='+ video['videoId'])

as you can see channel id is the key here, what if you are unaware of channel id.? You can get the channel id either from browser view page source or you can also use “requests” and “BeautifulSoup” module to get the channel ID

import requests
from bs4 import BeautifulSoup

url = input("Please Enter youtube channel URL ")
response = requests.get(url)
soup = BeautifulSoup(response.content,'html.parser')
print(soup.find("meta", itemprop="channelId")['content'])

Instead of using Channel from pytube module, to make it easier lets make changes to the original code snippet posted above, using module “scrapetube.get_channel “

import requests
import re
from bs4 import BeautifulSoup
from pytube.cli import on_progress
from pytube import YouTube
from pytube import Playlist
from pytube import Channel
import scrapetube
import json

def get_youtube(i):
    try:
        print('Downloading ' +i)
        yt = YouTube(i,on_progress_callback=on_progress)
        #print(f'\nStarted downloading:  {yt} from -->' + i)
        yt.streams.filter(progressive=True, file_extension='mp4').order_by('resolution').desc().first().download('C:\\Users\\Dell\\Videos\\downs\\Sriram_Channel\\')
    except:
        print('Error Downloading' +i)
        pass
l = scrapetube.get_channel("UCw43xCtkl26vGIGtzBoaWEw")
for video in l:
    #print('Downloading https://www.youtube.com/watch?v='+ video['videoId'])
    get_youtube('https://www.youtube.com/watch?v='+ video['videoId'])

Hope you like it ! Happy Reading.

Posted in WebScraping | Leave a Comment »

Python way to Download all the ASKTOM and Oracle MAG Posted by Connor McDonald at Linked In Group

Posted by Sriram Sanka on November 8, 2022

There is a Group Post By Connor on LinkedIn in Oracle Senior DBA Group, showing the links to access ASKTOM Best Posts and Oracle Magazines from https://asktom.oracle.com/pls/apex/f?p=100:9

Here is the Code Snippet that helps you to download all the Posts and Magazines as HTML files as your choice of Destination in your local file system .

Snippet To Download TOM KYTE Posts

import requests
from bs4 import BeautifulSoup
import string
import os
import urllib.request, urllib.error, urllib.parse
import sys

def Download_ASKTOM_files(path,url,enc,title):
    try:                
        response = urllib.request.urlopen(url)
        webContent = response.read().decode(enc)
        os.makedirs(path+'\\'+ 'ASKTOM', exist_ok=True)
        n=os.path.join(path+'\\'+ 'ASKTOM',title +'.html')
        f = open(n, 'w',encoding=enc)
        f.write(webContent)
        f.close
    except:
        n1=os.path.join(path+'\\'+  'ASKTOM_'+'Download_Error.log')
        f1 = open(n1, 'w',encoding=enc) 
        f1.write(url)
        f1.close
reqs = requests.get("https://asktom.oracle.com/tomkyte-blog.htm")
soup = BeautifulSoup(reqs.text, 'html.parser')
for link2 in soup.select(" a[href]"):
    src=link2["href"]
    durl='https://asktom.oracle.com/'+src
    tit =link2.get_text().replace(string.punctuation, " ").translate(str.maketrans('', '', string.punctuation))
    print(tit.replace(" ","_"),durl)
    Download_ASKTOM_files("c:\\Users\\....\\Downloads\\blogs\\",durl,'UTF-8',tit.replace(" ","_"))        

Snippet to Download Magazines

import requests
from bs4 import BeautifulSoup
import string
import os
import urllib.request, urllib.error, urllib.parse
import sys

def Download_ASKTOM_files(path,url,enc,title):
    try:                
        response = urllib.request.urlopen(url)
        webContent = response.read().decode(enc)
        os.makedirs(path+'\\'+ 'ASKTOM_MAG', exist_ok=True)
        n=os.path.join(path+'\\'+ 'ASKTOM_MAG',title +'.html')
        f = open(n, 'w',encoding=enc)
        f.write(webContent)
        f.close
    except:
        n1=os.path.join(path+'\\'+  'ASKTOM_MAG_'+'Download_Error.log')
        f1 = open(n1, 'w',encoding=enc) 
        f1.write(url)
        f1.close
reqs = requests.get("https://asktom.oracle.com/magazine-archive.htm")
soup = BeautifulSoup(reqs.text, 'html.parser')
for link2 in soup.select(" a[href]"):
    src=link2["href"]
    durl='https://asktom.oracle.com/'+src
    tit =link2.get_text().replace(string.punctuation, " ").translate(str.maketrans('', '', string.punctuation))
    print(tit.replace(" ","_"),durl)
    Download_ASKTOM_files("c:\\Users\\......\\Downloads\\blogs\\",durl,'UTF-8',tit.replace(" ","_"))   

Hope you liked it 🙂

Posted in ASKTOM, CONNOR, Python, TOMKYTE | Tagged: , , , , | Leave a Comment »