Data Scraping Airbnb: Manhattan Listings

Posted on Feb 19, 2018
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Data Scraping Airbnb: Manhattan Listings

1.   Overview

Airbnb is a global apartment rental hub for renting one's personal home or room to another person. It can be a convenient and affordable alternative to its more conventional cousin, the hotel. The price per night, much like a hotel depend greatly on location, quality, and size among other factors. The business purpose of this project was to gain some data insight into what factors drive Airbnb listing pricing as well as to better understand the distribution of listings in Manhattan.

There were two main steps to this project: data aggregation (web scraping) and data analysis.

2.   Data Web Scraping

In order to perform data analysis on Airbnb data in Manhattan, we first need the Airbnb data in Manhattan. To obtain this, this required a web scraper. Using a popular web scraping library: Python Scrapy, I began to write a scraper. It would have the following breakdown:

  1. Go to Airbnb Manhattan listings
  2. Loop through prices ($10 increments)
  3. Look through pages in each price range
  4. Loop through listings on each page
  5. Get details in listing (price, location, ratings, number of rooms, etc)

The reason it would go through $10 increments of price was so that it could gather more data from listings in Manhattan. With no price filter or too wide of a price filter, too many results would be returned. To expand, if more than 17 pages of results (~300 listings) were returned, only the first 17 pages would be displayed. All listings after the 17th page would be omitted from the results.

Issues When Gathering Data

In order to circumvent this issue, I introduced an iterative price filter that would lead to smaller search results (< 17 pages of listings) that could be looked through. This yielded more data, but unfortunately not the entire Airbnb Manhattan dataset. Airbnb will ban IP addresses that query it more than once every 8 seconds or so for extended periods of time, so I was limited by time without using multiple proxies (which I did not).

The reason I could not use multiple proxies was a product of a different scraping challenge. I could not get data from the page listings I was going to such as room type, number of guests, etc. This data was loaded by AJAX so could not be accessed by Xpaths using Scrapy despite the information being seen in the HTML elements. To navigate around this issue, I switched into Scrapy Splash, a Scrapy plug-in for Javascript integration.

From my research, Scrapy Splash did not integrate well with the scrapy-proxies library which lead me to not use multiple proxies for this task. For reasons unknown to me, I could still not see the Xpaths, but now I could get the entire "response.txt" and extract information from this with regular expressions. In total, I scraped ~5500 listings with 20 different attributes such as room ID, price, ratings in multiple categories, number of guests, number of rooms, latitude and longitude, etc.

3.   Data Analysis: Listings Overview

Now that I had my dataset, I could begin to comb through it. Below is a histogram of price distribution in Manhattan; note the broad range of prices and the large percentage of "entire home/apartments". Some of the more expensive listings were removed from this plot, but prices ranged all the way to $10,000 per night for select penthouses and party lofts. Another interesting note was that while listed at $10, some listings were quite nice, and upon further review often had a disclaimer in the description asking for closer to $100.

Data Scraping Airbnb: Manhattan Listings

Listings in Manhattan

In order to see a better "lay of the land", below is a map of all listings below $700 in Manhattan. Surprisingly, many of the listings in "Manhattan" are in fact not in Manhattan, but instead in the other boroughs or New Jersey. There is a high listing density in Midtown West as well as Lower East Side/East Village, though there is a broad distribution.

Data Scraping Airbnb: Manhattan Listings


Where are the more expensive listings? Below is a map of all the listings with the dot getting larger and more blue with price,. Turns out many of them are on the southern edge of Central Park (the view perhaps), and downtown Manhattan. Not super surprising I suppose? Interestingly (or perhaps uninterestingly depending on your knowledge), there is a pocket of expensive listings in Hoboken, New Jersey.

Data Scraping Airbnb: Manhattan Listings

4.   Analysis: What drives price?

Now that we have a general understanding of the geographical and price distribution distribution of Airbnb Manhattan listings, can we gain a better understanding of what factors are most involved with the price? From the below correlation plot (Pearson correlation) we can develop a better feel for this. We see that factors such as the number of baths, beds, and guests drive price significantly.

In addition, having the entire home/apt (instead of private room in a shared apartment or a shared room) is very positively correlated with price. Similarly, bath(s) (versus private bath in shared apartment or shared bath) is highly correlated with price. This is not surprising though, "bath" essentially is an indicator that the entire home is being rented. Also important factors were location and some of the guest ratings such as guest satisfaction. The most negatively correlated factor was "private room". No surprise there.

Correlation Between Location and Price

Next, I looked at the correlation between location and price. There is a certain increase in price with higher location ratings (though the biggest gap seems to be between 10/10 and 9/10). Guest satisfaction also looked to be generally higher when prices were higher.

Data Scraping Airbnb: Manhattan Listings

The real driving force behind the prices seemed to be number of rooms and number of guests (which are probably highly correlated variables). With more rooms, a place can house more guests, and the price reflected this, going up between $50-100 per room.


What about location? Doesn't location drive the price? In short, yes of course. In long, it is hard to put a precise number on location. Different regions of Manhattan cost more. We do have the location rating variable, but I wanted to develop more than just the ratings would reflect. I wanted to see how latitude and longitude would affect price.

Ultimately, I wanted to develop a topological map of price as a function of latitude and longitude. Despite training a random forest on this data set, I was not able to pin down the effects of just longitude and latitude on price. I believe this was primarily due to the high number of other more influencing factors such as number of rooms. In the future, I would like to explore this location distribution further in a more controlled environment. One way to approach this would be to only train a model on studios that would take one guest or otherwise remove the effects of other variables. That modeling will have to wait for another time.

5.   Conclusion

In conclusion, I scraped a large portion of Airbnb's Manhattan listings and performed exploratory data analysis on the dataset. It was seen that many listings in Airbnb were full apartments and that the price had a broad range, driven primarily by size of the apartment/number of guests. The listings' prices and locations were visualized on a map of NYC. Despite my efforts, my initial models were unsuccessful in reflecting the correlation between latitude and longitude with price. However, I believe this is possible and would be an extremely valuable insight, perhaps looking at a larger Airbnb worldwide dataset.

Thank you for reading.


For more information, please see my code at:

About Author

Andrew Dodd

I am a data scientist at NYCDSA with a mechanical engineering background (BS, Masters). My masters focused on space and robotics applications; this is where I developed my interest in machine learning through courses that involved path planning...
View all posts by Andrew Dodd >

Related Articles

Leave a Comment

Google January 26, 2022
Google Wonderful story, reckoned we could combine several unrelated data, nevertheless actually worth taking a appear, whoa did 1 learn about Mid East has got much more problerms as well.
Google January 2, 2021
Google Usually posts some pretty exciting stuff like this. If you’re new to this site.
CBD For Dogs December 19, 2020
CBD For Dogs [...]here are some links to internet sites that we link to because we consider they are worth visiting[...]
Google December 17, 2020
Google We came across a cool web-site that you may well love. Take a look in case you want.
RDP August 28, 2020
RDP [...]we came across a cool web-site that you could delight in. Take a look in the event you want[...]
Google August 22, 2020
Google One of our visitors lately proposed the following website. August 19, 2020 [...]that will be the end of this article. Here you will uncover some web-sites that we think you will enjoy, just click the links over[...] August 5, 2020 [...]below you’ll discover the link to some websites that we think you should visit[...] July 30, 2020 [...]although sites we backlink to beneath are considerably not associated to ours, we feel they may be basically worth a go by way of, so possess a look[...]
Google July 26, 2020
Google The information and facts talked about within the post are some of the ideal obtainable.
cbd for cats July 9, 2020
cbd for cats [...]just beneath, are several completely not associated internet sites to ours, having said that, they are surely really worth going over[...]

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI