Data Scraping Airbnb: Manhattan Listings
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
1. Overview
Airbnb is a global apartment rental hub for renting one's personal home or room to another person. It can be a convenient and affordable alternative to its more conventional cousin, the hotel. The price per night, much like a hotel depend greatly on location, quality, and size among other factors. The business purpose of this project was to gain some data insight into what factors drive Airbnb listing pricing as well as to better understand the distribution of listings in Manhattan.
There were two main steps to this project: data aggregation (web scraping) and data analysis.
2. Data Web Scraping
In order to perform data analysis on Airbnb data in Manhattan, we first need the Airbnb data in Manhattan. To obtain this, this required a web scraper. Using a popular web scraping library: Python Scrapy, I began to write a scraper. It would have the following breakdown:
- Go to Airbnb Manhattan listings
- Loop through prices ($10 increments)
- Look through pages in each price range
- Loop through listings on each page
- Get details in listing (price, location, ratings, number of rooms, etc)
The reason it would go through $10 increments of price was so that it could gather more data from listings in Manhattan. With no price filter or too wide of a price filter, too many results would be returned. To expand, if more than 17 pages of results (~300 listings) were returned, only the first 17 pages would be displayed. All listings after the 17th page would be omitted from the results.
Issues When Gathering Data
In order to circumvent this issue, I introduced an iterative price filter that would lead to smaller search results (< 17 pages of listings) that could be looked through. This yielded more data, but unfortunately not the entire Airbnb Manhattan dataset. Airbnb will ban IP addresses that query it more than once every 8 seconds or so for extended periods of time, so I was limited by time without using multiple proxies (which I did not).
The reason I could not use multiple proxies was a product of a different scraping challenge. I could not get data from the page listings I was going to such as room type, number of guests, etc. This data was loaded by AJAX so could not be accessed by Xpaths using Scrapy despite the information being seen in the HTML elements. To navigate around this issue, I switched into Scrapy Splash, a Scrapy plug-in for Javascript integration.
From my research, Scrapy Splash did not integrate well with the scrapy-proxies library which lead me to not use multiple proxies for this task. For reasons unknown to me, I could still not see the Xpaths, but now I could get the entire "response.txt" and extract information from this with regular expressions. In total, I scraped ~5500 listings with 20 different attributes such as room ID, price, ratings in multiple categories, number of guests, number of rooms, latitude and longitude, etc.
3. Data Analysis: Listings Overview
Now that I had my dataset, I could begin to comb through it. Below is a histogram of price distribution in Manhattan; note the broad range of prices and the large percentage of "entire home/apartments". Some of the more expensive listings were removed from this plot, but prices ranged all the way to $10,000 per night for select penthouses and party lofts. Another interesting note was that while listed at $10, some listings were quite nice, and upon further review often had a disclaimer in the description asking for closer to $100.
Listings in Manhattan
In order to see a better "lay of the land", below is a map of all listings below $700 in Manhattan. Surprisingly, many of the listings in "Manhattan" are in fact not in Manhattan, but instead in the other boroughs or New Jersey. There is a high listing density in Midtown West as well as Lower East Side/East Village, though there is a broad distribution.
Pricing
Where are the more expensive listings? Below is a map of all the listings with the dot getting larger and more blue with price,. Turns out many of them are on the southern edge of Central Park (the view perhaps), and downtown Manhattan. Not super surprising I suppose? Interestingly (or perhaps uninterestingly depending on your knowledge), there is a pocket of expensive listings in Hoboken, New Jersey.
4. Analysis: What drives price?
Now that we have a general understanding of the geographical and price distribution distribution of Airbnb Manhattan listings, can we gain a better understanding of what factors are most involved with the price? From the below correlation plot (Pearson correlation) we can develop a better feel for this. We see that factors such as the number of baths, beds, and guests drive price significantly.
In addition, having the entire home/apt (instead of private room in a shared apartment or a shared room) is very positively correlated with price. Similarly, bath(s) (versus private bath in shared apartment or shared bath) is highly correlated with price. This is not surprising though, "bath" essentially is an indicator that the entire home is being rented. Also important factors were location and some of the guest ratings such as guest satisfaction. The most negatively correlated factor was "private room". No surprise there.
Correlation Between Location and Price
Next, I looked at the correlation between location and price. There is a certain increase in price with higher location ratings (though the biggest gap seems to be between 10/10 and 9/10). Guest satisfaction also looked to be generally higher when prices were higher.
The real driving force behind the prices seemed to be number of rooms and number of guests (which are probably highly correlated variables). With more rooms, a place can house more guests, and the price reflected this, going up between $50-100 per room.
Findings
What about location? Doesn't location drive the price? In short, yes of course. In long, it is hard to put a precise number on location. Different regions of Manhattan cost more. We do have the location rating variable, but I wanted to develop more than just the ratings would reflect. I wanted to see how latitude and longitude would affect price.
Ultimately, I wanted to develop a topological map of price as a function of latitude and longitude. Despite training a random forest on this data set, I was not able to pin down the effects of just longitude and latitude on price. I believe this was primarily due to the high number of other more influencing factors such as number of rooms. In the future, I would like to explore this location distribution further in a more controlled environment. One way to approach this would be to only train a model on studios that would take one guest or otherwise remove the effects of other variables. That modeling will have to wait for another time.
5. Conclusion
In conclusion, I scraped a large portion of Airbnb's Manhattan listings and performed exploratory data analysis on the dataset. It was seen that many listings in Airbnb were full apartments and that the price had a broad range, driven primarily by size of the apartment/number of guests. The listings' prices and locations were visualized on a map of NYC. Despite my efforts, my initial models were unsuccessful in reflecting the correlation between latitude and longitude with price. However, I believe this is possible and would be an extremely valuable insight, perhaps looking at a larger Airbnb worldwide dataset.
Thank you for reading.
For more information, please see my code at:
https://github.com/adodd202/Airbnb_Scraping
About Author
Andrew Dodd
View all posts by Andrew Dodd >Related Articles
Leave a Comment
