Data Scraping Zillow.com
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.
Data shows house prices are constantly changing, and if you are in the market for a house, this can be a scary realization! How are you, an everyday person, supposed to keep up with the growing complexity that is the price valuations of houses? Luckily for you, www.Zillow.com exists.
From Zillow's about page:
"Zillow is the leading real estate and rental marketplace dedicated to empowering consumers with data, inspiration and knowledge around the place they call home, and connecting them with the best local professionals who can help."
"Zillow serves the full lifecycle of owning and living in a home: buying, selling, renting, financing, remodeling and more. It starts with Zillow's living database of more than 110 million U.S. homes - including homes for sale, homes for rent and homes not currently on the market, as well as Zestimate home values, Rent Zestimates and other home-related information. Zillow operates the most popular suite of mobile real estate apps, with more than two dozen apps across all major platforms."
The objective of this project was to assist in both the buying process and selling process of a house by examining houses within a given zip code and producing an analysis of similar houses.
Data Collection:
- The project was coded in Python, mainly Scrapy.
- Layout of the data collected:
- The individual house URL extensions were collected from these tiles on the right. Due to Zillows website, the maximum number of pages to be scraped is 20 pages per "Listing Type". The "Listing Types" are: For Sale, Foreclosed, Sold, For Rent. On each page there were 25 houses listed.
- Once you reached the individual house listing page, the information to be scraped was stored under "Facts and Features".
- As you can see above, each houses information is stored differently, the numbered arrows on House 1 correspond to the equivalent data on House 2.
- This made for the collection of data quite challenging.
In the end, the way I collected and formatted the equivalent data from each house was by using a python dictionary to collect all possible fields such as "Beds: ", "Baths: ", and "Heating: " and creating scrapy fields out of each one. The lasting issue from the collection process was that many of the dictionary fields collected corresponded to similar data bytes, for example: "Elementary school: North main" and "Grade school: North main". These were manually assessed and condensed into groupings of similar values.
Data Cleaning:
I took in a deep breathe, thinking the hard part was over and that my data was going to come out nice and clean... boy was I wrong. Once I had a look at the resulting data frame I found multiple inconsistencies throughout the data. Luckily for me, many of the errors were repeated errors, certain sections of the columns needing to be shifted left or right depending on what data was in the left most offending column.
Quick EDA:
First, I wondered what is the difference between Zestimate and sale price, whether that is the price a house was sold at or what the current price listing is. Below shows the Zestimate minus the sold price and as we can see, houses , in the designated zip code, generally sell for less than their listed Zestimate.
Why is this? My assumptions would be:
- Zillow is missing data that might devalue a house
- People look at Zillow before making an offer on a house, and people are "low-balling" their offers.
Sale Price vs Length of Stay
Next I wondered was there an effect on the sale price of a house based on the length of stay on Zillow or the Number of views.
After making these graphs I realized they can be misleading in that:
- The Sale Price v. Days on Zillow graph shows the current price and current days on Zillow.
- Due to me not grabbing the sale price history for each house, which was held out because of scrapy's limitations with javascript, this does not accurately represent Sale price v Days on Zillow per house. But does show a general trend of days on Zillow v Sale price, in which there is almost no correlation.
- The Sale Price v. Number of Views graph can lead one to believe that if you're house is viewed more, then it will sell for more.
- But my assumption is the reverse correlation: if you're house is selling for more, it is going to get more views. This is because people like to look at all of the gorgeous mansions that are way outside their price range, at least I do!
Lastly I took a look at creating the similar houses data. This portion is only the beginning of the end goal, so it is pretty rudimentary. I did a simple grouping of houses by number of beds and baths and added a interval (+/-2) to each.
As we can see from the graph, as well as saw earlier in the post, houses generally sell for less than their Zestimate. On the right, are examples of future grouping variables as well as some summary statistics for each one.
Looking Forward:
I hope to continue to develop this project so that one day one will be able to very clearly see and compare the housing prices for similar houses based upon their own requirements. Thus allowing them to either budget for their new "dream home" or appropriately understand what they will reasonably be getting if they list their house at a certain value.
About Author
