Scraping Airbnb: Manhattan Listings

Avatar
Posted on Feb 19, 2018

1.   Overview

Airbnb is a global apartment rental hub for renting one's personal home or room to another person. It can be a convenient and affordable alternative to its more conventional cousin, the hotel. The price per night, much like a hotel depend greatly on location, quality, and size among other factors. The business purpose of this project was to gain some insight into what factors drive Airbnb listing pricing as well as to better understand the distribution of listings in Manhattan.

There were two main steps to this project: data aggregation (web scraping) and data analysis.

2.   Web Scraping

In order to perform data analysis on Airbnb data in Manhattan, we first need the Airbnb data in Manhattan. To obtain this, this required a web scraper. Using a popular web scraping library: Python Scrapy, I began to write a scraper. It would have the following breakdown:

  1. Go to Airbnb Manhattan listings
  2. Loop through prices ($10 increments)
  3. Loop through pages in each price range
  4. Loop through listings on each page
  5. Get details in listing (price, location, ratings, number of rooms, etc)

The reason it would go through $10 increments of price was so that it could gather more data from listings in Manhattan. With no price filter or too wide of a price filter, too many results would be returned. To expand, if more than 17 pages of results (~300 listings) were returned, only the first 17 pages would be displayed. All listings after the 17th page would be omitted from the results. In order to circumvent this issue, I introduced an iterative price filter that would lead to smaller search results (< 17 pages of listings) that could be looked through. This yielded more data, but unfortunately not the entire Airbnb Manhattan dataset. Airbnb will ban IP addresses that query it more than once every 8 seconds or so for extended periods of time, so I was limited by time without using multiple proxies (which I did not).

The reason I could not use multiple proxies was a product of a different scraping challenge. I could not get data from the page listings I was going to such as room type, number of guests, etc. This data was loaded by AJAX so could not be accessed by Xpaths using Scrapy despite the information being seen in the HTML elements. To navigate around this issue, I switched into Scrapy Splash, a Scrapy plug-in for Javascript integration. From my research, Scrapy Splash did not integrate well with the scrapy-proxies library which lead me to not use multiple proxies for this task. For reasons unknown to me, I could still not see the Xpaths, but now I could get the entire "response.txt" and extract information from this with regular expressions. In total, I scraped ~5500 listings with 20 different attributes such as room ID, price, ratings in multiple categories, number of guests, number of rooms, latitude and longitude, etc.

3.   Analysis: Listings Overview

Now that I had my dataset, I could begin to comb through it. Below is a histogram of price distribution in Manhattan; note the broad range of prices and the large percentage of "entire home/apartments". Some of the more expensive listings were removed from this plot, but prices ranged all the way to $10,000 per night for select penthouses and party lofts. Another interesting note was that while listed at $10, some listings were quite nice, and upon further review often had a disclaimer in the description asking for closer to $100.

In order to see a better "lay of the land", below is a map of all listings below $700 in Manhattan. Surprisingly, many of the listings in "Manhattan" are in fact not in Manhattan, but instead in the other boroughs or New Jersey. There is a high listing density in Midtown West as well as Lower East Side/East Village, though there is a broad distribution.

Where are the more expensive listings? Below is a map of all the listings with the dot getting larger and more blue with price,. Turns out many of them are on the southern edge of Central Park (the view perhaps), and downtown Manhattan. Not super surprising I suppose? Interestingly (or perhaps uninterestingly depending on your knowledge), there is a pocket of expensive listings in Hoboken, New Jersey.

4.   Analysis: What drives price?

Now that we have a general understanding of the geographical and price distribution distribution of Airbnb Manhattan listings, can we gain a better understanding of what factors are most involved with the price? From the below correlation plot (Pearson correlation) we can develop a better feel for this. We see that factors such as the number of baths, beds, and guests drive price significantly. In addition, having the entire home/apt (instead of private room in a shared apartment or a shared room) is very positively correlated with price. Similarly, bath(s) (versus private bath in shared apartment or shared bath) is highly correlated with price. This is not surprising though, "bath" essentially is an indicator that the entire home is being rented. Also important factors were location and some of the guest ratings such as guest satisfaction. The most negatively correlated factor was "private room". No surprise there.

Next, I looked at the correlation between location and price. There is a certain increase in price with higher location ratings (though the biggest gap seems to be between 10/10 and 9/10). Guest satisfaction also looked to be generally higher when prices were higher.

The real driving force behind the prices seemed to be number of rooms and number of guests (which are probably highly correlated variables). With more rooms, a place can house more guests, and the price reflected this, going up between $50-100 per room.

What about location? Doesn't location drive the price? In short, yes of course. In long, it is hard to put a precise number on location. Different regions of Manhattan cost more. We do have the location rating variable, but I wanted to develop more than just the ratings would reflect. I wanted to see how latitude and longitude would affect price. Ultimately, I wanted to develop a topological map of price as a function of latitude and longitude. Despite training a random forest on this data set, I was not able to pin down the effects of just longitude and latitude on price. I believe this was primarily due to the high number of other more influencing factors such as number of rooms. In the future, I would like to explore this location distribution further in a more controlled environment. One way to approach this would be to only train a model on studios that would take one guest or otherwise remove the effects of other variables. That modeling will have to wait for another time.

5.   Conclusion

In conclusion, I scraped a large portion of Airbnb's Manhattan listings and performed exploratory data analysis on the dataset. It was seen that many listings in Airbnb were full apartments and that the price had a broad range, driven primarily by size of the apartment/number of guests. The listings' prices and locations were visualized on a map of NYC. Despite my efforts, my initial models were unsuccessful in reflecting the correlation between latitude and longitude with price. However, I believe this is possible and would be an extremely valuable insight, perhaps looking at a larger Airbnb worldwide dataset.

Thank you for reading.

 

For more information, please see my code at:

https://github.com/adodd202/Airbnb_Scraping

About Author

Avatar

Andrew Dodd

I am a data scientist at NYCDSA with a mechanical engineering background (BS, Masters). My masters focused on space and robotics applications; this is where I developed my interest in machine learning through courses that involved path planning...
View all posts by Andrew Dodd >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

2019 airbnb alumni Alumni Interview Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Big Data Book Launch Book-Signing bootcamp Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Industry Experts Job Jon Krohn JP Morgan Chase Kaggle lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest recommendation recommendation system regression Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Tableau TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp