python-scraping/chapter6/2-getUtf8Text.py at master · python26/python-scraping · GitHub

Sitelet https://github.com/python26/python-scraping/blob/master/chapter6/2-getUtf8Text.py

Repository navigation

9 lines (8 loc) · 318 Bytes

from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://en.wikipedia.org/wiki/Python_(programming_language)")
bsObj = BeautifulSoup(html)
content = bsObj.find("div", {"id":"mw-content-text"}).get_text()
content = bytes(content, "UTF-8")
content = content.decode("UTF-8")
print(content)