Real time data search and analysis with API in Shiny
Contributed by Wansang Lim. He is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between January 11th to April 1st, 2016. This post is based on his project – R Shiny.
Trends of API
Recently, API become a hot issue and popular in Internet related technology. Programmable Web, a site that tracks more than 13,000 APIs, lists New York Times, Google Maps, Twitter, YouTube, Flickr and Amazon Product Advertising as some of the the most popular APIs. More and more applications are adopting an API-first approach. APIs are powerful to use because it creates a common language that everyone can use and understand. If we use API, we do not need a pre-built platform to analyze big data (https://blog.cloud-elements.com/using-apis-data-science).
API for Data Scientist
API can save a lot of time for Data Scientist.
When we think data scientists, some of us of think statistician nerds. But if you look what a data scientist does on a daily basis, it is 20% statistics and 80% data wrangling (https://apigee.com/about/blog/developer/developers-and-data-scientists-enterprise-force-multipliers). Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets (https://mobile.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html?referrer=).
Abstract of this project
For this project, I use NY times API.It provides many APIs like Article Search API,Books API ,The Congress API and so on.Among them, I chose article search API to check hit which is the number of article which have the search word. And it give us abstract , date and url of the article. These are just the java script. It needs be be converted to regular column data.
For web page scraping, I scraped billboard sings which have 2 column and one hidden column. It is reorganized to compare current and previous ranking.
I used API as a kind of data conversion tool. For example, the leaflet for shiny needs coordinate for location. But most data we can get is just address like "500 8th Ave #905, New York, NY 10018". I used google geocoding API (https://developers.google.com/maps/documentation/geocoding/intro). I tried government job search API (https://search.digitalgov.gov/developer/jobs.html). It provide job location which is just address. I converted another API to obtain location data.
Bill Board Web Scraping
Fig 1. Bill board conversion to shiny table.
The bill board web site looks like left side on Fig 1. It is scraped by python. It looks like has just two columns which are singer and song name. However, the last week ranking is hidden the web table.It seems that Java Script calculate it with current ranking and convert it as up or down arrow. The ranking difference column is made by subtracting last ranking from current ranking. So it can be sorted in shiny.
Normalization
Fig 2. Normalization
The API is designed to call 10 signals from ending year to ending year - 10. It let us compare hit among 10 years(Fig 2 a,b,c). When I search china, it shows dramatic increase in 2015 and 2016(Fig 2 a). When I search "about" which is mostly commonly used word, I have same result(Fig 2 b). So I decided to normalize. After normalize, the hit frequency of china looks a lot better(Fig 2 c).
Actual Search with Justin Beiber
Fig 3. Actual search with Justin Biever (a) not normalized (b) normalized
When I searched "Justin Biever" which is in Fig 1, his hit keep increase in unnormalized graph(Fig 3 a), but in normalized graph(Fig 3 b), his popularity become less in recent years.
Detailed Information of Justin Bieber
Fig 4. (a)Hit by month (b)article abstract by year and month
This API app searched 10 years at one time. It shows that there is almost no hit before 2009. So I guess he began to spot light around 2010(Fig 3). In 2010 March and April, he begin to got some hit (Fig 4 a). I searched the actual article. I found his article 2010 April (Fig 4 b). The content is "The 16-year-old singer may not seem like a dangerous figure, but make no mistake: his public appearances can be battlegrounds." .It seems that at that time, he was not so known to every one.
Data conversion with two API
Fig 5. Data conversion from address to coordinates
I searched data related job in government API. It send me address data. I connected to google geocoder API to convert in real time. The converted data is in red circle. The each address is converted in while loop.
ui.R file
library(shiny)
shinyUI(pageWithSidebar(
headerPanel("NY Times API "),
sidebarPanel(
conditionalPanel(condition="input.conditionedPanels==1",
radioButtons("radio", label = h3("Select "),
choices = list("Original Hit" = 1, "Hit normalized" = 2,"Hit About" = 3),
selected = 1),
textInput("text", label = h4("Search input"), value = ""),
#actionButton("goButton", "Enter"),
numericInput("year", label = h4("Ending Year:"), 2016, min = 1945, max = 2016),
actionButton("goButton", "Enter")
),
conditionalPanel(condition="input.conditionedPanels==2",
numericInput("yearMonth", label = h4("Year for month:"), 2016, min = 1945, max = 2016),
actionButton("yearMonthButton", "Enter")
),
conditionalPanel(condition="input.conditionedPanels==3",
numericInput("yearInterest", label = h4("The Year of interest:"), 2016, min = 1945, max = 2016),
radioButtons("radioMonth", label = h4("Select"),
choices = list("By Year" = 1, "By Month" = 2),
selected = 1),
conditionalPanel(condition = "input.radioMonth== 2",
numericInput("monthInterest", label = h4("The month of interest"), 01, min = 1, max = 12)
),
numericInput("pageInput", label = h4("Type the page"), 01, min = 1, max = 1000),
actionButton("goArticle", "Enter")
),
conditionalPanel(condition="input.conditionedPanels==4",
helpText("This is incoming
java script object")
),
conditionalPanel(condition="input.conditionedPanels==5",
helpText("2016/02/27")
),
conditionalPanel(condition="input.conditionedPanels==7",
helpText("Content Panel 5")
),
conditionalPanel(condition="input.conditionedPanels==8",
tags$textarea(id="typeAdd", rows=1, cols=15, ""),
#textInput("typeJob", label = h5("Search job"), value = ""),
actionButton("goLonLat", "Enter")
)
),
mainPanel(
tabsetPanel(
tabPanel("NY Times Article API", value=1, plotOutput("graphHit") ), #textOutput("textAll"), dataTableOutput
tabPanel("NY Times Article API by Month", value=2, plotOutput("graphMonth") ),
tabPanel("NY Times Article Example", value=3, dataTableOutput("articleExam") ),
tabPanel("NY Times raw JavaScript Ojbects", value=4, textOutput("rawAPI") ),
tabPanel("Bill Board Singles 100 2016/02/27", value=5, dataTableOutput("BillBoard")),
#tabPanel("Presidential election finance", value=7, textOutput("test03")),
tabPanel("Job Search", value=8, dataTableOutput("textGeo")),
id = "conditionedPanels"
),# end of tabsetPanel
tags$a(href="https://github.com/nycdatasci/bootcamp004_project/blob/master/Project3-WebScraping/WansagLim/prac05/ui.R", "ur.R Link"),
tags$br(),
tags$a(href="https://github.com/nycdatasci/bootcamp004_project/blob/master/Project3-WebScraping/WansagLim/prac05/server.R", "server.R Link")
)
))
server.R file
library(shiny)
library(DT)
library(jsonlite)
library(ggplot2)
shinyServer(function(input, output, session) {
articleSearch <- reactive(input$text)
eventReactive(input$goButton,{ input$text})
year10 <- reactive(input$year)
eventReactive(input$yearButton,{input$year} )
yearInterest <- reactive(input$yearInterest)
yearMonth <- reactive(input$yearMonth)
monthInterest <- reactive(input$monthInterest)
pageInput <- reactive(input$pageInput)
addressInput <- reactive(input$typeAdd)
#renderPrint, renderPlot
output$graphHit <-renderPlot({
input$goButton
isolate({
articleSearch <- articleSearch()
year10 <- year10()
year09 <- year10() - 1
year08 <- year10() - 2
year07 <- year10() - 3
year06 <- year10() - 4
year05 <- year10() - 5
year04 <- year10() - 6
year03 <- year10() - 7
year02 <- year10() - 8
year01 <- year10() - 9
year00 <- year10() - 10
year10 <- paste(year10, "0217", sep = "")
year09 <- paste(year09, "0217", sep = "")
year08 <- paste(year08, "0217", sep = "")
year07 <- paste(year07, "0217", sep = "")
year06 <- paste(year06, "0217", sep = "")
year05 <- paste(year05, "0217", sep = "")
year04 <- paste(year04, "0217", sep = "")
year03 <- paste(year03, "0217", sep = "")
year02 <- paste(year02, "0217", sep = "")
year01 <- paste(year01, "0217", sep = "")
year00 <- paste(year00, "0217", sep = "")
address <- "https://api.nytimes.com/svc/search/v2/articlesearch.json?q=korea&begin_date=20150217&end_date=20160216&sort=newest&api-key=48c66fa2c448eda40826487d4f19a018%3A0%3A71658152"
articleAddress <- "https://api.nytimes.com/svc/search/v2/articlesearch.json?q="
articleKey <- "&api-key=48c66fa2c448eda40826487d4f19a018:0:71658152"
articleAPI10 <- paste(articleAddress, articleSearch,"&begin_date=",year09,"&end_date=",year10,"&sort=newest",articleKey, sep = "")
articleAPI09 <- paste(articleAddress, articleSearch,"&begin_date=",year08,"&end_date=",year09,"&sort=newest",articleKey, sep = "")
articleAPI08 <- paste(articleAddress, articleSearch,"&begin_date=",year07,"&end_date=",year08,"&sort=newest",articleKey, sep = "")
articleAPI07 <- paste(articleAddress, articleSearch,"&begin_date=",year06,"&end_date=",year07,"&sort=newest",articleKey, sep = "")
articleAPI06 <- paste(articleAddress, articleSearch,"&begin_date=",year05,"&end_date=",year06,"&sort=newest",articleKey, sep = "")
articleAPI05 <- paste(articleAddress, articleSearch,"&begin_date=",year04,"&end_date=",year05,"&sort=newest",articleKey, sep = "")
articleAPI04 <- paste(articleAddress, articleSearch,"&begin_date=",year03,"&end_date=",year04,"&sort=newest",articleKey, sep = "")
articleAPI03 <- paste(articleAddress, articleSearch,"&begin_date=",year02,"&end_date=",year03,"&sort=newest",articleKey, sep = "")
articleAPI02 <- paste(articleAddress, articleSearch,"&begin_date=",year01,"&end_date=",year02,"&sort=newest",articleKey, sep = "")
articleAPI01 <- paste(articleAddress, articleSearch,"&begin_date=",year00,"&end_date=",year01,"&sort=newest",articleKey, sep = "")
#normalize "about
aboutAPI10 <- paste(articleAddress, "about","&begin_date=",year09,"&end_date=",year10,"&sort=newest",articleKey, sep = "")
aboutAPI09 <- paste(articleAddress, "about","&begin_date=",year08,"&end_date=",year09,"&sort=newest",articleKey, sep = "")
aboutAPI08 <- paste(articleAddress, "about","&begin_date=",year07,"&end_date=",year08,"&sort=newest",articleKey, sep = "")
aboutAPI07 <- paste(articleAddress, "about","&begin_date=",year06,"&end_date=",year07,"&sort=newest",articleKey, sep = "")
aboutAPI06 <- paste(articleAddress, "about","&begin_date=",year05,"&end_date=",year06,"&sort=newest",articleKey, sep = "")
aboutAPI05 <- paste(articleAddress, "about","&begin_date=",year04,"&end_date=",year05,"&sort=newest",articleKey, sep = "")
aboutAPI04 <- paste(articleAddress, "about","&begin_date=",year03,"&end_date=",year04,"&sort=newest",articleKey, sep = "")
aboutAPI03 <- paste(articleAddress, "about","&begin_date=",year02,"&end_date=",year03,"&sort=newest",articleKey, sep = "")
aboutAPI02 <- paste(articleAddress, "about","&begin_date=",year01,"&end_date=",year02,"&sort=newest",articleKey, sep = "")
aboutAPI01 <- paste(articleAddress, "about","&begin_date=",year00,"&end_date=",year01,"&sort=newest",articleKey, sep = "")
#artiConD
if (articleSearch == 'Enter search term' || articleSearch == "") {
message01 <- "Please, Type your search term"
message01
} else {
articleContent10 <- fromJSON(articleAPI10)
articleContent09 <- fromJSON(articleAPI09)
articleContent08 <- fromJSON(articleAPI08)
articleContent07 <- fromJSON(articleAPI07)
articleContent06 <- fromJSON(articleAPI06)
Sys.sleep(1)
articleContent05 <- fromJSON(articleAPI05)
articleContent04 <- fromJSON(articleAPI04)
articleContent03 <- fromJSON(articleAPI03)
articleContent02 <- fromJSON(articleAPI02)
articleContent01 <- fromJSON(articleAPI01)
Sys.sleep(1)
#normalize "About"
normalContent10 <- fromJSON(aboutAPI10)
normalContent09 <- fromJSON(aboutAPI09)
normalContent08 <- fromJSON(aboutAPI08)
normalContent07 <- fromJSON(aboutAPI07)
normalContent06 <- fromJSON(aboutAPI06)
Sys.sleep(1)
normalContent05 <- fromJSON(aboutAPI05)
normalContent04 <- fromJSON(aboutAPI04)
normalContent03 <- fromJSON(aboutAPI03)
normalContent02 <- fromJSON(aboutAPI02)
normalContent01 <- fromJSON(aboutAPI01)
artiConHit10 <- articleContent10$response$meta$hits
artiConHit09 <- articleContent09$response$meta$hits
artiConHit08 <- articleContent08$response$meta$hits
artiConHit07 <- articleContent07$response$meta$hits
artiConHit06 <- articleContent06$response$meta$hits
artiConHit05 <- articleContent05$response$meta$hits
artiConHit04 <- articleContent04$response$meta$hits
artiConHit03 <- articleContent03$response$meta$hits
artiConHit02 <- articleContent02$response$meta$hits
artiConHit01 <- articleContent01$response$meta$hits
#normalize hit
normalConHit10 <- normalContent10$response$meta$hits
normalConHit09 <- normalContent09$response$meta$hits
normalConHit08 <- normalContent08$response$meta$hits
normalConHit07 <- normalContent07$response$meta$hits
normalConHit06 <- normalContent06$response$meta$hits
normalConHit05 <- normalContent05$response$meta$hits
normalConHit04 <- normalContent04$response$meta$hits
normalConHit03 <- normalContent03$response$meta$hits
normalConHit02 <- normalContent02$response$meta$hits
normalConHit01 <- normalContent01$response$meta$hits
hitVec <- c(artiConHit10, artiConHit09, artiConHit08, artiConHit07, artiConHit06, artiConHit05, artiConHit04, artiConHit03, artiConHit02, artiConHit01)
hitNormalVec <- c(normalConHit10,normalConHit09, normalConHit08, normalConHit07, normalConHit06, normalConHit05, normalConHit04, normalConHit03, normalConHit02, normalConHit01)
hitNormlizedVec <- hitVec / hitNormalVec
years <- c(year10, year09, year08, year07, year06, year05, year04, year03, year02, year01)
#Graph.Table
graphTable <- data.frame(years, hitVec, hitNormalVec, hitNormlizedVec)
Graph
barHitGraph <- ggplot(graphTable, aes(x=years, y=hitVec)) + geom_bar(stat="identity")
barNormLizedGraph <- ggplot(graphTable, aes(x=years, y=hitNormlizedVec)) + geom_bar(stat="identity")
barJustAbout <- ggplot(graphTable, aes(x=years, y=hitNormalVec)) + geom_bar(stat="identity")
if (input$radio == 1) {
barHitGraph + theme(axis.text.x = element_text(angle = 45, hjust = 1, size = 12),
axis.text.y = element_text(size = 12),
panel.background = element_blank())
} else if (input$radio == 2) {
barNormLizedGraph + theme(axis.text.x = element_text(angle = 45, hjust = 1, size = 12),
axis.text.y = element_text(size = 12),
panel.background = element_blank())
} else if (input$radio == 3) {
barJustAbout + theme(axis.text.x = element_text(angle = 45, hjust = 1, size = 12),
axis.text.y = element_text(size = 12),
panel.background = element_blank())
}
}
}) # End of Isolate
}) #Graph Hit
output$graphMonth <- renderPlot({
input$yearMonthButton
isolate({
articleSearch <- articleSearch()
if (articleSearch != "") {
yearMonth <- yearMonth()
month12End <- paste(yearMonth, "1231", sep = "")
month12 <- paste(yearMonth, "1201", sep = "")
month11 <- paste(yearMonth, "1101", sep = "")
month10 <- paste(yearMonth, "1001", sep = "")
month09 <- paste(yearMonth, "0901", sep = "")
month08 <- paste(yearMonth, "0801", sep = "")
month07 <- paste(yearMonth, "0701", sep = "")
month06 <- paste(yearMonth, "0601", sep = "")
month05 <- paste(yearMonth, "0501", sep = "")
month04 <- paste(yearMonth, "0401", sep = "")
month03 <- paste(yearMonth, "0301", sep = "")
month02 <- paste(yearMonth, "0201", sep = "")
month01 <- paste(yearMonth, "0101", sep = "")
articleAddress <- "https://api.nytimes.com/svc/search/v2/articlesearch.json?q="
articleKey <- "&api-key=48c66fa2c448eda40826487d4f19a018:0:71658152"
hitMon12 <- paste(articleAddress, articleSearch,"&begin_date=",month12,"&end_date=",month12End,articleKey, sep = "")
hitMon11 <- paste(articleAddress, articleSearch,"&begin_date=",month11,"&end_date=",month12,articleKey, sep = "")
hitMon10 <- paste(articleAddress, articleSearch,"&begin_date=",month10,"&end_date=",month11,articleKey, sep = "")
hitMon09 <- paste(articleAddress, articleSearch,"&begin_date=",month09,"&end_date=",month10,articleKey, sep = "")
hitMon08 <- paste(articleAddress, articleSearch,"&begin_date=",month08,"&end_date=",month09,articleKey, sep = "")
hitMon07 <- paste(articleAddress, articleSearch,"&begin_date=",month07,"&end_date=",month08,articleKey, sep = "")
hitMon06 <- paste(articleAddress, articleSearch,"&begin_date=",month06,"&end_date=",month07,articleKey, sep = "")
hitMon05 <- paste(articleAddress, articleSearch,"&begin_date=",month05,"&end_date=",month06,articleKey, sep = "")
hitMon04 <- paste(articleAddress, articleSearch,"&begin_date=",month04,"&end_date=",month05,articleKey, sep = "")
hitMon03 <- paste(articleAddress, articleSearch,"&begin_date=",month03,"&end_date=",month04,articleKey, sep = "")
hitMon02 <- paste(articleAddress, articleSearch,"&begin_date=",month02,"&end_date=",month03,articleKey, sep = "")
hitMon01 <- paste(articleAddress, articleSearch,"&begin_date=",month01,"&end_date=",month02,articleKey, sep = "")
articleMonth12 <- fromJSON(hitMon12)
articleMonth11 <- fromJSON(hitMon11)
articleMonth10 <- fromJSON(hitMon10)
Sys.sleep(0.5)
articleMonth09 <- fromJSON(hitMon09)
articleMonth08 <- fromJSON(hitMon08)
articleMonth07 <- fromJSON(hitMon07)
Sys.sleep(1.1)
articleMonth06 <- fromJSON(hitMon06)
articleMonth05 <- fromJSON(hitMon05)
articleMonth04 <- fromJSON(hitMon04)
Sys.sleep(0.5)
articleMonth03 <- fromJSON(hitMon03)
articleMonth02 <- fromJSON(hitMon02)
articleMonth01 <- fromJSON(hitMon01)
hitMonVec <- c( articleMonth12$response$meta$hits, articleMonth11$response$meta$hits, articleMonth10$response$meta$hits,
articleMonth09$response$meta$hits, articleMonth08$response$meta$hits, articleMonth07$response$meta$hits, articleMonth06$response$meta$hits,
articleMonth05$response$meta$hits, articleMonth04$response$meta$hits, articleMonth03$response$meta$hits, articleMonth02$response$meta$hits,
articleMonth01$response$meta$hits)
monthVec <- c(month12, month11, month10, month09, month08, month07, month06, month05, month04, month03,
month02, month01)
graphMonthTable <- data.frame(monthVec, hitMonVec)
barMonthGraph <- ggplot(graphMonthTable, aes(x=monthVec, y=hitMonVec)) + geom_bar(stat="identity")
barMonthGraph + theme(axis.text.x = element_text(angle = 45, hjust = 1, size = 12),
axis.text.y = element_text(size = 12),
panel.background = element_blank())
} # End of articleSearch != 0
})#end of isolate
})#End of graphMonth
output$articleExam <- DT::renderDataTable({
input$goArticle
isolate({
articleSearch <- articleSearch()
yearInterest <- yearInterest()
yearInterest01 <- yearInterest - 1
monthInterest <- monthInterest()
pageInput <- pageInput()
pageInput <- paste("&page=", pageInput, sep = "")
yearForMonth <- yearInterest
yearInterest01 <- paste(yearInterest01, "0217", sep = "")
yearInterest <- paste(yearInterest, "0217", sep = "")
articleAddress <- "https://api.nytimes.com/svc/search/v2/articlesearch.json?q="
articleKey <- "&api-key=48c66fa2c448eda40826487d4f19a018:0:71658152"
articleAPI10 <- paste(articleAddress, articleSearch,"&begin_date=",yearInterest01,"&end_date=",yearInterest,pageInput,articleKey, sep = "")
if (articleSearch != "") {
articleContent10 <- fromJSON(articleAPI10)
artiConDoc10 <- articleContent10$response$docs
artiURL10 <- artiConDoc10$web_url
artiConSnippet10 <- artiConDoc10$snippet
table10 <- data.frame(artiConSnippet10, artiURL10)
#Articlae Table
if (input$radioMonth == 1 ) {
table10
} else if (input$radioMonth == 2) {
if (monthInterest <= 9) {
monthInterest 10) {
monthInterest <- as.character(monthInterest)
}
beginMon <- paste(yearForMonth, monthInterest, "01", sep = "" )
endMon <- paste(yearForMonth, monthInterest, "30", sep = "")
articleMonthAPI <- paste(articleAddress, articleSearch,"&begin_date=",beginMon,"&end_date=",endMon,articleKey, sep = "")
monthContent <- fromJSON(articleMonthAPI)
monConDoc <- monthContent$response$docs
#your_link <- monConDoc$web_url
#your_link <- paste0("link")
#monURL <- your_link
monSnippet <- monConDoc$snippet
tableMonth 6 ?",
"'' + data.substr(0, 20) + '...' : data;",
"}")
)))
)
output$rawAPI <- renderPrint({
articleSearch <- articleSearch()
if (articleSearch != "") {
yearInterest <- yearInterest()
yearInterest01 <- yearInterest - 1
yearInterest <- paste(yearInterest, "0217", sep = "")
yearInterest01 <- paste(yearInterest01, "0217", sep = "")
articleAddress <- "https://api.nytimes.com/svc/search/v2/articlesearch.json?q="
articleKey <- "&api-key=48c66fa2c448eda40826487d4f19a018:0:71658152"
articleAPI10 <- paste(articleAddress, articleSearch,"&begin_date=",yearInterest01,"&end_date=",yearInterest,"&sort=newest",articleKey, sep = "")
articleContent10 <- fromJSON(articleAPI10)
infoRawAPI <- c(yearInterest, articleSearch, articleContent10)
infoRawAPI
} #End of articleSearch
})
output$BillBoard <- renderDataTable({
billFrame <- read.csv("bill100Frame.csv", stringsAsFactors=FALSE)
billFrame$diff <- as.numeric(billFrame$diff)
billFrame <- data.frame(billFrame$x,billFrame$Title,
billFrame$Singer, billFrame$Ranking, billFrame$diff)
colnames(billFrame) <- c("Ranking", "Title", "Singer", "Last Ranking", "Ranking Difference")
billFrame
})
output$test03 <- renderPrint({
apiData <- fromJSON('/Volumes/64GB/NYC/project03/api2012.txt')
apiResu <- apiData$results
apiResu
})
output$textGeo <- renderDataTable({
input$goLonLat
isolate({
addressInput <- addressInput()
#https://maps.googleapis.com/maps/api/geocode/json?address=4620%20parsons%20blvd%20flushing%20ny%20%20key=AIzaSyANifkybPlJWYynG_FSwzwSn-CunJTE4N0
if (addressInput != "") {
jobsAdd <- paste("https://api.usa.gov/jobs/search.json?query=", addressInput, sep = "")
jobsAPI <- fromJSON(jobsAdd)
Sys.sleep(1)
addressVec <- jobsAPI$locations
newAddress <- c()
numberPosition <- c()
for (i in 1: length(jobsAPI$position_title)) {
unlisAdd 1) {
newAddress[i] <- as.character(unlisAdd[1])
numberPosition[i] <- length(unlisAdd[i])
}
numberPosition[i] <- 1
newAddress[i] <- as.character(unlisAdd)
}
addressVec <- newAddress
lonlatVec <-c()
latt <- c()
lont <- c()
i = 1
while (i <= length(addressVec)) {
address = addressVec[i]
address = gsub(" ", "", address)
geoWeb <- paste("https://maps.googleapis.com/maps/api/geocode/json?address=", address, "&key=AIzaSyANifkybPlJWYynG_FSwzwSn-CunJTE4N0", sep = "")
geoAPI <- fromJSON(geoWeb)
latt[i] <- geoAPI$results$geometry$location$lat
lont[i] <- geoAPI$results$geometry$location$lng
Sys.sleep(0.2)
i = i + 1
}
Title <- jobsAPI$position_title
MinWage <- jobsAPI$minimum
MaxWage <- jobsAPI$maximum
Organization <- jobsAPI$organization_name
Locations <- newAddress
jobTable <- data.frame(Title, Organization,numberPosition ,MinWage, MaxWage, Locations,latt, lont)
jobTable
}# end of if
})# end of isolate
})# end of output$textGeo
})
Python for Web Scrapping
from bs4 import BeautifulSoup
import requests
import csv
url = 'https://www.billboard.com/charts/hot-100'
r = requests.get(url)
soup = BeautifulSoup(r.text)
my_list = []
#for tag in soup.findAll("a", {"class":"chart-row__link"}):
for tag in soup.findAll("a", {"data-tracklabel":"Artist Name"}):
my_list.append(tag.text)
c = csv.writer(open("MYFILE.csv", "wb"))
c.writerow(my_list)
########################
song_list = []
for tag in soup.findAll("h2", {"class" : "chart-row__song"}):
#print (tag.text)
song_list.append(tag.text)
song_list.append("\n")
c = csv.writer(open("songName.csv", "wb"))
c.writerow(song_list)
############################
lastWeek = []
for tag in soup.findAll("span", {"class":"chart-row__last-week"}):
lastWeek.append(tag.text)
lastWeek.append("\n")
#print (tag.text)
c = csv.writer(open("lastWeek1.csv", "wb"))
c.writerow(lastWeek)
About Author
Wansang Lim
View all posts by Wansang Lim >Leave a Comment
