Data Study on Uber's Fare

and
Posted on Sep 23, 2016
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

See the code! | Try to the app! (We suggest you use Safari.)

Introduction

Data shows driving an Uber in New York City is a difficult proposition. Navigating the crowded streets of Manhattan, competing with yellow cabs for fares, and sometimes driving for miles just to get gas. The margins of driving an uber are razor thin, and drivers can’t afford to spend their precious minutes looking for a new fare.

While this is certainly an issue for its employees, the problem of finding new fares quickly is also dilemma for Uber itself. Uber needs as many drivers as it can get in New York, otherwise the response time to new fares will not be sufficient compete with yellow cabs and other apps. Drivers won’t want to drive for Uber if they can’t make money, so they need their drivers picking up new fares as quickly as they drop them off. Uber also wants their drivers to head to areas that will be popular in advance, so that their customers don’t have to wait too long for their ride.

We attempted to solve these problems with this app. While our program won’t solve every problem with driving an Uber in New York, we believe that drivers will get a great deal of benefit out of using it. We encourage you to try out the app, and we hope that you enjoy it!

Data

We used two different sources for our data on taxi rides. The Uber dataset, which was roughly 100 MB and 3 million rows long, was taken from FiveThirtyEight’s Github account. The data was available to FiveThirtyEight as the result of a freedom of information act, and thus only contained the longitude, latitude, and time of each ride. The yellow cab data was download from the city of New York’s website, which contained many more variables than just latitude/longitude. We automated this process with the following script:

From the original format, we grouped the data by neighborhood and split it up by time of day using ten minute intervals. We also created our variable of interest, which was the sum of uber and taxi rides.

Supplemental Data

We knew we needed to include weather data in our analysis, as anyone who’s tried to hail a cab in a rainstorm can attest. We downloaded data from Weather Underground, which contained daily weather statistics recorded in Central Park going back several years. It’d be better if we had weather broken out by hour or minute, but this measure should give a rough indication for conditions on each day.

We also scraped data from newyorkgasprices.com, for a complete list of local gas stations and their prices. This site was a bit cumbersome to scrape, but we accomplished it with the following script:

Model

Now that the data is collected and clean, it’s time to implement a model to do the actual predicting. We tested two different models: linear regression and random forests.

Linear Regression

We implemented linear regression using the sklearn package in python. While linear regresion is not the most predictive model, we thought it would be useful to test, as it can quickly and efficiently return results (unlike other machine learning methods that take longer). When we tested the model, it returned an R^2 value of .72 and an RMSE score of 112.1. Here is our final code for its implementation:

Random Forests

We also tested random forests, a model that is significantly more predictive and expensive than linear regression. We saw much better test results from Random Forests, with an R^2 of .93 and an RMSE of 10.5. While it takes a bit longer to return results than linear regression, we opted to use Random Forests in our model for its greater predictive power.

App

All of these predictions wouldn’t be very useful without an interface to display them, so we built a Flask app for drivers to use. We wanted to limit the amount of information that drivers had to input into the app, so we only ask for the distance that they’re willing to drive, and if they’d like to get gas.

Data Study on Uber's Fare

Visualization Graphic

The visualization on the front page is meant to give drivers a rough idea of where most fares are happening throughout the day. This was built using R and ggplot2, and the data is grouped by latitude and longitude every half hour. We looped through every half hour and made a separate graph for that time, which we output to a .png file. We then used gifmaker.com to turn those images into a completed gif.

Data Study on Uber's Fare

Retrieving Current Conditions

As we previously mentioned, we wanted to limit the amount of fields that the user had to input when using our app. Our algorithm requires quite a bit of information to generate its prediction, however, so our app uses APIs to retrieve the relevant data. The user’s current location is retrieved from your IP address. Weather conditions are taken from Dark Skies, which returns the current temperature and the precipitation rate. Travel times and traffic conditions are calculated using the Google Maps API, and the current date and time are taken from the user’s computer.

Running the Data Algorithm

After the current conditions are collected, the app takes the radius that the user has inputted and excludes any neighborhoods that are too far away. This saves a great deal of computational energy, so we can return results faster and more efficiently. For each of these neighborhoods, the app inputs the current conditions and calculates the predicted demand. It then returns the best neighborhood, along with directions to that neighborhood from Google Maps.

screen-shot-2016-09-22-at-9-36-58-pm

Gas?

If the user checks the box indicating that they need gas, directions to the nearest gas station are returned. This is simply done by calculating the distance to all available gas stations using Google Maps, and returning the direction to the closest one.

Conclusions

This project was a fascinating topic, and we both greatly enjoyed working on predicting Uber rides in New York. This is clearly an important area, and there is much more work that can be done. With better data from Uber, we could learn a tremendous amount about where and when the next fare is coming. We sincerely hope they make more data available soon, as it could lead to more fascinating explorations.

About Authors

Christian Holmes

Christian Holmes is a graduate of Middlebury College with a B.A. in both Economics and Chemistry. Upon graduating, he spent two years as a data analyst at an advertising technology startup, where he became interested in predictive analytics....
View all posts by Christian Holmes >

Shuheng Li

With the intention of becoming a great Data Scientist, Shuheng is an out-box thinker and self-motivator who focuses on Machine Learning and Statistics, and seeks for challenging projects and competitions that push his skills to an advanced level....
View all posts by Shuheng Li >

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI