Latest Tech

Recent Posts: Label / Tag Ex:

How to Get The Top Stories From Google News and BBC News With Python


Python is fun programming language. Today, we are going to see how to create a simple code that retrieves the top news from Google or BBC. It's easy.

First, let's import the following packages. 


import bs4
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen

bs4: It's is beautifulSoup 4, used for pulling data out of HTML and XML files. It saves programmers a lot of work by providing a way of navigating, searching, and modifying the parse tree

urllib: It's used for URLs fetching in the form of urlfunction. It also handles more complex tasks such as cookies, authentications and proxies

Second, we plug in the Google url which we will retrieve the information from. 


news_url="https://news.google.com/news/rss"Client=urlopen(news_url)
xml_page=Client.read()
Client.close()

Third, XML is the type of files this code is dealing with. It retrieves all available items on the page

soup_page=soup(xml_page,"xml")
news_list=soup_page.findAll("item")

Finally, we print news title, url, and news publish dates.


for news in news_list:
  print(news.title.text)
  print(news.link.text)
  print(news.pubDate.text)
  print("-"*60)

 The full code should be like this.


import bs4
from bs4 import BeautifulSoup as soup
from urllib.request import urlopen

news_url="https://news.google.com/news/rss"Client=urlopen(news_url)
xml_page=Client.read()
Client.close()

soup_page=soup(xml_page,"xml")
news_list=soup_page.findAll("item")

for news in news_list:
  print(news.title.text)
  print(news.link.text)
  print(news.pubDate.text)
  print("-"*60)

To get BBC top stories, we simply change the Google RSS url to BBC RSS url.
BBC RSS url: http://feeds.bbci.co.uk/news/rss.xml

The output for Google News

Protests in Iran stretch into third day despite government warnings
https://www.washingtonpost.com/world/pro-government-demonstrators-rally-in-iran-following-days-of-unrest/2017/12/30/a737b40e-ecdd-11e7-956e-baea358f9725_story.html
Sat, 30 Dec 2017 21:27:12 GMT
------------------------------------------------------------
How the Russia Inquiry Began: A Campaign Aide, Drinks and Talk of Political Dirt
https://www.nytimes.com/2017/12/30/us/politics/how-fbi-russia-investigation-began-george-papadopoulos.html
Sat, 30 Dec 2017 16:20:43 GMT
------------------------------------------------------------
Why a Twitter user claims to have made the 'swatting' call that led police to kill a man
https://www.washingtonpost.com/news/post-nation/wp/2017/12/30/why-a-twitter-user-claims-to-have-made-the-swatting-call-that-led-police-to-kill-a-man/
Sat, 30 Dec 2017 21:34:28 GMT
------------------------------------------------------------
Mike Pence's Colorado neighbors troll him with a 'Make America Gay Again' banner
https://www.washingtonpost.com/news/politics/wp/2017/12/30/mike-pences-colorado-neighbors-troll-him-with-a-make-america-gay-again-banner/
Sat, 30 Dec 2017 17:46:22 GMT
------------------------------------------------------------
Activist Erica Garner, 27, dies after heart attack
http://www.cnn.com/2017/12/30/us/erica-garner-eric-death/index.html
Sat, 30 Dec 2017 16:35:56 GMT
------------------------------------------------------------
Two security guards shot dead at off-Strip hotel-casino in Las Vegas
http://www.latimes.com/nation/la-na-vegas-casino-shooting-20171230-story.html
Sat, 30 Dec 2017 20:50:00 GMT
------------------------------------------------------------
In Fire-Scarred Bronx Neighborhood, a World of New Arrivals, Children and Community
https://www.nytimes.com/2017/12/30/nyregion/bronx-fire-children.html
Sat, 30 Dec 2017 17:36:34 GMT
------------------------------------------------------------
Two people die, including shooter, in workplace gunfire in Long Beach
http://www.latimes.com/local/lanow/la-me-long-beach-shooting-20171229-story.html
Sat, 30 Dec 2017 05:10:00 GMT
------------------------------------------------------------
Trump drops challenge to transgender troops in military, opening door to Jan. 1 recruits
https://www.usatoday.com/story/news/politics/2017/12/30/trump-drops-challenge-transgender-troops-military/992213001/
Sat, 30 Dec 2017 17:55:06 GMT
------------------------------------------------------------
Two men charged with murder in 'horrific' quadruple homicide case in New York
https://www.washingtonpost.com/news/true-crime/wp/2017/12/30/two-men-charged-with-murder-in-horrific-quadruple-homicide-case-in-new-york/
Sat, 30 Dec 2017 18:04:09 GMT
------------------------------------------------------------
Powerball, Mega Millions reach $737M combined
https://www.local10.com/news/national/powerball-mega-millions-reach-737m-combined
Sat, 30 Dec 2017 20:08:23 GMT
------------------------------------------------------------
Russian tankers have smuggled oil to North Korea, a breach of UN sanctions
https://www.nbcnews.com/news/world/russian-tankers-have-smuggled-oil-north-korea-breach-u-n-n833531
Fri, 29 Dec 2017 21:53:00 GMT
------------------------------------------------------------
Man Rigged Front Door To Try To Electrocute His Pregnant Wife, Police Say
https://www.huffingtonpost.com/entry/michael-wilson-electrocute-pregnant-wife-florida_us_5a46a9a1e4b0b0e5a7a66dd6
Fri, 29 Dec 2017 23:31:27 GMT
------------------------------------------------------------
Police: Man Massaged Teen's Back at Airport Without Consent
https://www.usnews.com/news/best-states/new-jersey/articles/2017-12-30/police-man-massaged-teens-back-at-airport-without-consent
Sat, 30 Dec 2017 21:40:20 GMT
------------------------------------------------------------
Clint Eastwood, Ronald Reagan and Jesus among vote-getters in turbulent Alabama Senate race
http://www.foxnews.com/politics/2017/12/30/clint-eastwood-ronald-reagan-and-jesus-among-vote-getters-in-turbulent-alabama-senate-race.html
Sat, 30 Dec 2017 18:07:17 GMT
------------------------------------------------------------
Russia's Putin urges 'pragmatic cooperation' with US in New Year greetings
http://www.cnn.com/2017/12/30/world/putin-trump-new-year-greetings-intl/index.html
Sat, 30 Dec 2017 14:50:30 GMT
------------------------------------------------------------
Coldest temperatures in 21 years possible Tuesday night
http://www.khou.com/news/local/coldest-temperatures-in-21-years-possible-tuesday-night/503387720
Sat, 30 Dec 2017 16:03:45 GMT
------------------------------------------------------------
Will 2018 see Trump deliver on the promises he made in 2017?
https://www.nbcnews.com/business/economy/will-2018-see-trump-deliver-promises-he-made-2017-n833601
Sat, 30 Dec 2017 15:47:00 GMT
------------------------------------------------------------
Baker Mayfield Breaks Silence on Illness, but Oklahoma's Heisman Winner Still on the Mend
https://www.si.com/college-football/2017/12/30/baker-mayfield-illness-rose-bowl-2018-oklahoma
Sat, 30 Dec 2017 22:01:51 GMT
------------------------------------------------------------
Nepal Bars Solo Climbers From Mount Everest
https://www.nytimes.com/2017/12/30/world/asia/everest-nepal-ban-solo-climbers.html
Sat, 30 Dec 2017 18:15:00 GMT
------------------------------------------------------------



The output for BBC News

New Year Honours 2018: Praise for stars on list
http://www.bbc.co.uk/news/uk-42522662
Sat, 30 Dec 2017 18:40:29 GMT
------------------------------------------------------------
One treated after fire in Manchester block of flats
http://www.bbc.co.uk/news/uk-england-manchester-42524546
Sat, 30 Dec 2017 22:31:30 GMT
------------------------------------------------------------
Footage shows fire in Manchester flats
http://www.bbc.co.uk/news/uk-england-manchester-42524158
Sat, 30 Dec 2017 18:15:23 GMT
------------------------------------------------------------
Iran protests: Violence on third day of demonstrations
http://www.bbc.co.uk/news/world-middle-east-42524610
Sat, 30 Dec 2017 22:15:51 GMT
------------------------------------------------------------
Police shoot man dead after alleged Call of Duty 'swatting' hoax
http://www.bbc.co.uk/news/world-us-canada-42523045
Sat, 30 Dec 2017 15:30:32 GMT
------------------------------------------------------------
Nerves of steel: High up on China's power lines
http://www.bbc.co.uk/news/world-asia-china-42522614
Sat, 30 Dec 2017 11:28:03 GMT
------------------------------------------------------------
The professor, his wife and the photo everyone is talking about
http://www.bbc.co.uk/news/uk-scotland-42510187
Sat, 30 Dec 2017 09:52:16 GMT
------------------------------------------------------------
Alphabet writer dies with series on 'Y'
http://www.bbc.co.uk/news/world-us-canada-42522100
Sat, 30 Dec 2017 11:33:41 GMT
------------------------------------------------------------
County Durham child dies after Egypt pool incident
http://www.bbc.co.uk/news/uk-england-tees-42522173
Sat, 30 Dec 2017 19:43:48 GMT
------------------------------------------------------------
Millennials to secure 'inheritance boom'
http://www.bbc.co.uk/news/uk-42519073
Sat, 30 Dec 2017 11:48:41 GMT
------------------------------------------------------------
Black Lives Matter activist dies at 27
http://www.bbc.co.uk/news/world-us-canada-42524101
Sat, 30 Dec 2017 17:13:37 GMT
------------------------------------------------------------
Government tried to silence me, says Lord Adonis
http://www.bbc.co.uk/news/uk-politics-42520480
Sat, 30 Dec 2017 11:03:35 GMT
------------------------------------------------------------
Sheffield arrest fifth in 'terror plot' probe
http://www.bbc.co.uk/news/uk-england-south-yorkshire-42522024
Sat, 30 Dec 2017 10:32:47 GMT
------------------------------------------------------------
Pen pals defy age gap to become friends
http://www.bbc.co.uk/news/uk-42523826
Sat, 30 Dec 2017 17:36:40 GMT
------------------------------------------------------------
Who plays the piano at the railway station?
http://www.bbc.co.uk/news/entertainment-arts-42441596
Sat, 30 Dec 2017 00:30:57 GMT
------------------------------------------------------------
Birmingham - what's to love as it gets ready for 2022?
http://www.bbc.co.uk/news/uk-england-birmingham-42516335
Sat, 30 Dec 2017 00:38:16 GMT
------------------------------------------------------------
The real women of professional wrestling
http://www.bbc.co.uk/news/entertainment-arts-41186003
Sat, 30 Dec 2017 00:35:15 GMT
------------------------------------------------------------
'I just want to see my mum'
http://www.bbc.co.uk/news/stories-42180053
Sat, 30 Dec 2017 00:21:52 GMT
------------------------------------------------------------
Victoria Wood 'favourite' video wins new fans on Twitter
http://www.bbc.co.uk/news/uk-england-suffolk-42516393
Sat, 30 Dec 2017 00:57:52 GMT
------------------------------------------------------------
Mishal Husain comes face-to-face with AI and the Mishalbot
http://www.bbc.co.uk/news/uk-42438354
Sat, 30 Dec 2017 00:24:37 GMT
------------------------------------------------------------
Brexit-supporting students getting abuse on campus
http://www.bbc.co.uk/news/education-41837205
Sat, 30 Dec 2017 00:40:32 GMT
------------------------------------------------------------
Ooh la la! 19th Century brothel's return
http://www.bbc.co.uk/news/world-europe-42428725
Sat, 30 Dec 2017 00:19:36 GMT
------------------------------------------------------------
Your pictures: Best of 2017
http://www.bbc.co.uk/news/in-pictures-42369284
Fri, 29 Dec 2017 00:14:16 GMT
------------------------------------------------------------
This cash-strapped school sends kids to Harvard
http://www.bbc.co.uk/news/stories-42433406
Fri, 29 Dec 2017 00:16:00 GMT
------------------------------------------------------------
BBC News Channel
http://www.bbc.co.uk/news/10318089
Fri, 17 Nov 2017 07:22:39 GMT
------------------------------------------------------------
Lorraine Kelly gets all romantic - for new BBC show
http://www.bbc.co.uk/news/entertainment-arts-42354370
Sat, 30 Dec 2017 04:35:45 GMT
------------------------------------------------------------
The man who can't face the internet
http://www.bbc.co.uk/news/technology-42265053
Sat, 30 Dec 2017 00:41:54 GMT
------------------------------------------------------------
Is climate change making hurricanes worse?
http://www.bbc.co.uk/news/world-us-canada-42251921
Sat, 30 Dec 2017 00:30:27 GMT
------------------------------------------------------------
Zombie thrills: why the horror industry is scarily successful
http://www.bbc.co.uk/news/business-42403022
Fri, 29 Dec 2017 21:00:46 GMT
------------------------------------------------------------
What to look out for on TV in 2018
http://www.bbc.co.uk/news/entertainment-arts-42440245
Sat, 30 Dec 2017 03:52:29 GMT
------------------------------------------------------------
The best science long reads of 2017 (part one)
http://www.bbc.co.uk/news/science-environment-40657778
Sat, 30 Dec 2017 00:37:50 GMT
------------------------------------------------------------
Homai Vyarawalla: The trailblazer who became India's first woman photojournalist
http://www.bbc.co.uk/news/world-asia-india-42349710
Sat, 30 Dec 2017 00:47:39 GMT
------------------------------------------------------------
The real enemy on the world's highest battlefield
http://www.bbc.co.uk/news/world-asia-41432568
Sat, 30 Dec 2017 00:34:31 GMT
------------------------------------------------------------
Uganda's Punishment Island: 'I was left to die on an island for getting pregnant'
http://www.bbc.co.uk/news/world-africa-39576510
Thu, 27 Apr 2017 00:24:06 GMT
------------------------------------------------------------
Liverpool 2-1 Leicester City
http://www.bbc.co.uk/sport/football/42410959
Sat, 30 Dec 2017 18:44:02 GMT
------------------------------------------------------------
Serena Williams 'excited to be back' on making return to tennis after giving birth
http://www.bbc.co.uk/sport/tennis/42522649
Sat, 30 Dec 2017 15:31:49 GMT
------------------------------------------------------------
PDC World Darts 2018: Phil Taylor into final after beating Jamie Lewis
http://www.bbc.co.uk/sport/darts/42523933
Sat, 30 Dec 2017 21:48:43 GMT
------------------------------------------------------------
Manchester United 0-0 Southampton
http://www.bbc.co.uk/sport/football/42411000
Sat, 30 Dec 2017 21:26:29 GMT
------------------------------------------------------------
Ashes: Graeme Swann 'can't see how' Moeen Ali fits into England team in Sydney
http://www.bbc.co.uk/sport/cricket/42521457
Sat, 30 Dec 2017 10:32:58 GMT
------------------------------------------------------------
British astronaut Helen Sharman recognised in New Year’s honours
http://www.bbc.co.uk/news/science-environment-42522554
Sat, 30 Dec 2017 12:29:20 GMT
------------------------------------------------------------
Who's on the 2018 honours list?
http://www.bbc.co.uk/news/uk-42523125
Sat, 30 Dec 2017 12:55:29 GMT
------------------------------------------------------------
Polling expert Curtice 'didn't see knighthood coming'
http://www.bbc.co.uk/news/uk-scotland-scotland-politics-42512523
Sat, 30 Dec 2017 15:40:38 GMT
------------------------------------------------------------
New Year Honours: MBE for carer who fostered more than 100 children
http://www.bbc.co.uk/news/uk-england-coventry-warwickshire-42514261
Fri, 29 Dec 2017 23:07:21 GMT
------------------------------------------------------------
Jilly Cooper 'did not expect CBE'
http://www.bbc.co.uk/news/uk-england-gloucestershire-42516414
Fri, 29 Dec 2017 22:37:50 GMT
------------------------------------------------------------
Knighthood for Nick Clegg in New Year Honours list
http://www.bbc.co.uk/news/uk-politics-42514116
Sat, 30 Dec 2017 10:03:43 GMT
------------------------------------------------------------
How do the Honours work?
http://www.bbc.co.uk/news/uk-11990088
Mon, 01 Aug 2016 02:30:15 GMT
------------------------------------------------------------
MBE for author of 'instruction manual' for autism
http://www.bbc.co.uk/news/education-42504883
Fri, 29 Dec 2017 22:31:56 GMT
------------------------------------------------------------
England women cricketers among sporting honours
http://www.bbc.co.uk/sport/42512253
Fri, 29 Dec 2017 22:29:27 GMT
------------------------------------------------------------
BEM for organ donation campaigner, 18
http://www.bbc.co.uk/news/health-42513711
Fri, 29 Dec 2017 22:31:50 GMT

------------------------------------------------------------


Happy Coding...