Data Scraping

Posted on Aug 12, 2018
The skills the author demoed here can be learned through taking Data Science with Machine Learning bootcamp with NYC Data Science Academy.

Data Scraping

Data shows house prices are constantly changing, and if you are in the market for a house, this can be a scary realization!  How are you, an everyday person, supposed to keep up with the growing complexity that is the price valuations of houses?  Luckily for you, exists.

From Zillow's about page:

"Zillow is the leading real estate and rental marketplace dedicated to empowering consumers with data, inspiration and knowledge around the place they call home, and connecting them with the best local professionals who can help."

"Zillow serves the full lifecycle of owning and living in a home: buying, selling, renting, financing, remodeling and more. It starts with Zillow's living database of more than 110 million U.S. homes - including homes for sale, homes for rent and homes not currently on the market, as well as Zestimate home values, Rent Zestimates and other home-related information. Zillow operates the most popular suite of mobile real estate apps, with more than two dozen apps across all major platforms."

The objective of this project was to assist in both the buying process and selling process of a house by examining houses within a given zip code and producing an analysis of similar houses.

Data Collection:

  • The project was coded in Python, mainly Scrapy.
  • Layout of the data collected:

Data Scraping

  • The individual house URL extensions were collected from these tiles on the right. Due to Zillows website, the maximum number of pages to be scraped is 20 pages per "Listing Type".  The "Listing Types" are: For Sale, Foreclosed, Sold, For Rent. On each page there were 25 houses listed.
  • Once you reached the individual house listing page, the information to be scraped was stored under "Facts and Features".

Data Scraping

  • As you can see above, each houses information is stored differently, the numbered arrows on House 1 correspond to the equivalent data on House 2.
  • This made for the collection of data quite challenging.

In the end, the way I collected and formatted the equivalent data from each house was by using a python dictionary to collect all possible fields such as "Beds: ", "Baths: ", and "Heating: " and creating scrapy fields out of each one.  The lasting issue from the collection process was that many of the dictionary fields collected corresponded to similar data bytes, for example: "Elementary school: North main" and "Grade school: North main".  These were manually assessed and condensed into groupings of similar values.

Data Cleaning:

I took in a deep breathe, thinking the hard part was over and that my data was going to come out nice and clean... boy was I wrong.  Once I had a look at the resulting data frame I found multiple inconsistencies throughout the data.  Luckily for me, many of the errors were repeated errors, certain sections of the columns needing to be shifted left or right depending on what data was in the left most offending column.

Quick EDA:

First, I wondered what is the difference between Zestimate and sale price, whether that is the price a house was sold at or what the current price listing is. Below shows the Zestimate minus the sold price and as we can see, houses , in the designated zip code, generally sell for less than their listed Zestimate. 

Why is this?  My assumptions would be:

  1. Zillow is missing data that might devalue a house
  2. People look at Zillow before making an offer on a house, and people are "low-balling" their offers.

Sale Price vs Length of Stay

Next I wondered was there an effect on the sale price of a house based on the length of stay on Zillow or the Number of views.

After making these graphs I realized they can be misleading in that:

  1. The Sale Price v. Days on Zillow graph shows the current price and current days on Zillow.
    1. Due to me not grabbing the sale price history for each house, which was held out because of scrapy's limitations with javascript, this does not accurately represent Sale price v Days on Zillow per house. But does show a general trend of days on Zillow v Sale price, in which there is almost no correlation.
  2. The Sale Price v. Number of Views graph can lead one to believe that if you're house is viewed more, then it will sell for more.
    1. But my assumption is the reverse correlation: if you're house is selling for more, it is going to get more views.  This is because people like to look at all of the gorgeous mansions that are way outside their price range, at least I do!

Lastly I took a look at creating the similar houses data.  This portion is only the beginning of the end goal, so it is pretty rudimentary.   I did a simple grouping of houses by number of beds and baths and added a interval (+/-2) to each.

As we can see from the graph, as well as saw earlier in the post, houses generally sell for less than their Zestimate.  On the right, are examples of future grouping variables as well as some summary statistics for each one.

Looking Forward:

I hope to continue to develop this project so that one day one will be able to very clearly see and compare the housing prices for similar houses based upon their own requirements.  Thus allowing them to either budget for their new "dream home" or appropriately understand what they will reasonably be getting if they list their house at a certain value.

About Author

Will Thurston

Will is currently a student at New York City Data Science Academy. He graduated from Rochester Institute of Technology with a BS in Computer Security in 2016. He then spent the following year as a Network Engineer gaining...
View all posts by Will Thurston >

Leave a Comment

No comments found.

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 2020 Revenue 3-points agriculture air quality airbnb airline alcohol Alex Baransky algorithm alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus ames dataset ames housing dataset apartment rent API Application artist aws bank loans beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep boston safety Bundles cake recipe California Cancer Research capstone car price Career Career Day citibike classic cars classpass clustering Coding Course Demo Course Report covid 19 credit credit card crime frequency crops D3.js data data analysis Data Analyst data analytics data for tripadvisor reviews data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization database Deep Learning Demo Day Discount disney dplyr drug data e-commerce economy employee employee burnout employer networking environment feature engineering Finance Financial Data Science fitness studio Flask flight delay gbm Get Hired ggplot2 googleVis H20 Hadoop hallmark holiday movie happiness healthcare frauds higgs boson Hiring hiring partner events Hiring Partners hotels housing housing data housing predictions housing price hy-vee Income Industry Experts Injuries Instructor Blog Instructor Interview insurance italki Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter las vegas airport lasso regression Lead Data Scienctist Lead Data Scientist leaflet league linear regression Logistic Regression machine learning Maps market matplotlib Medical Research Meet the team meetup methal health miami beach movie music Napoli NBA netflix Networking neural network Neural networks New Courses NHL nlp NYC NYC Data Science nyc data science academy NYC Open Data nyc property NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time performance phoenix pollutants Portfolio Development precision measurement prediction Prework Programming public safety PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R Data Analysis R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn seafood type Selenium sentiment analysis sentiment classification Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau teachers team team performance TensorFlow Testimonial tf-idf Top Data Science Bootcamp Top manufacturing companies Transfers tweets twitter videos visualization wallstreet wallstreetbets web scraping Weekend Course What to expect whiskey whiskeyadvocate wildfire word cloud word2vec XGBoost yelp youtube trending ZORI