Scraping Craigslist (Cars+Trucks)
Contributed by Radhey Shyam. He is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between April 11th to July 1st, 2016. This post is based on his third class project -Python web scraping (due on the 6th week of the program).
Introduction:
For my web scraping project , I chose Craigslist website. I really like its concept and even I bought my first car from a craigslist seller and it was much cheaper than dealers. I spent lot of searching on craigslist wondering what was the best time to find the deal. As from my search,I found that a good car deal do not last for a long time. It will be really helpful if you can find out the most probable day and time when people place the ad so that you can reply promptly. Secondly, I wanted to analyze the price range,make and model year of the cars/trucks on the website.
Methodology:
For scraping craigslist, I picked cars & trucks (owners) section of New York craigslist. To scrap the data,I used scrapy and PyCharm ,which is an Integrated Development IDE
) for Python. The main scarping file imports a Spider class from Scrapy package.The spider class is where we define how the website will be scarped, allowed urls and starting urls. First, I defined the allowed urls for the craigslist website , and staring url of the webpage. I checked the main page of the website and found out that only first 100 car ads are posted on the main search page.There were in total 25 pages of ads, each page consisting of 100 ads.I found the pattern of next page listings as starting url "+ ?s=i*100" . Next ,I created a list of urls of all pages using a for loop as shown below.
Next , I setup how the webpage will be scraped. As each page consists of 100 ads, I scraped the ad ids (100) contained in the page and stored them in a list. Then for each ad, I created complete web address by concating the main web url with ad id plus html.Then I sent parse request for each ad page and parsed the price, title, posting time and body of the ad and stored the data in a csv file and code for the same is shown below.
Next step was to clean the parsed data.In car price data, I found that sometimes people also listed used car parts in car sale section and their price was usually less than $300. I removed all the entries having price less than $300. The price histogram of all the car ads is shown below and I chose bin size of $500 for the histogram. The range of car price was from $300 to $110,00 with median price of $4800. Below is shown the histogram of car prices in the range of $300-$20,000.
As we can see from above price histogram that peak sales of 180 counts happens in the price range of $3000-$3500 and the price count drop drastically to 30 above price range of $10,000. I think that it makes sense that people normally used craigslist for selling used car under $10,000 and it will be hard to sell costly cars on the craigslist.
Another curiosity was to find that whether people prefer any day or time to post their ad on craigslist ?To find the posting day, I went to each car ad's detailed listing page and parsed the posting date on the bottom of the page and stored in post_date variable as string. Then I convert the string to date format and then used R's base class function,strftime() to extract weekday from the ad posting date.
As we can see from above bar chart of weekdays that the peak day for the posting the ad was Friday, followed by Wednesday and Saturday. I think that people post car ads on Friday so that their ads remain fresh ( on top search page) during weekend and they have enough time for giving the car trials on weekend. Also, from my personal experience, people respond to the newer posted ads on weekends and it is easy to take calls and gave car trial on weekends. The only surprise to me was Wednesday as it is in the middle of the week and but I think that people were getting ready for the weekend to sell the car.
Next, I was curious about the time of placing the ad and whether people prefer any time for placing the car.This time, I separated the time from the as post date field and converted them to 24 hour time by adding 12 to pm time and rounded them by the hour.
The above violin plot are sorted by increasing count of the ads from let to right. There is interesting peak at 9 am on Saturday, as more sellers place their ad on Saturday in between 9 am to 9:59 am than other time of day. So , if you are a buyer and looking for new ads,then this is time you should look for on Saturdays so that you cannot miss a good deal. Also, count density on Saturday becomes narrower as the day progresses, and you are less likely to find fewer new ads as the day progresses. Also,there is a peak around the same time at 9 am on Friday and Saturday. And There is very few or no activity from midnight till 4 am in the morning on all days.
Next, I searched the title of the ads for most common words.While analyzing the titles , I removed all the numbers and characters and created a word cloud using R language's word cloud library and it is shown below.
The size of the words are proportional to their frequency in the title of the ads. The most common words in the title were make of the cars: Honda, Ford,Chevy,Hyundai,bmw, sale, low miles,clean ..
Finally, I want to know most common model years of the cars.For that , I separated the numbers from the title and created a word cloud of the numbers from size 4(2005) to 2 (05) as shown below.
The size of the years in the word cloud tells the frequency of the years in the title. The most common model years were 2005,2006,2004 (it shows that people commonly sell cars which are 10 years or old on the craigslist ) and 2009 ( 5 plus years or so).
Conclusion:
- Price histogram peaks in the range $3000-$4000 and it drops significantly after the price range of $7500-$8000.
- Friday is the most common day to put ad and it closely followed by Wednesday and Saturday.
- Saturday 9-10 am is the most common time to place ad , if you are looking for used car, then it is the best time to check the craigslist for new ads.
- Most frequency words in title are Honda, Ford,Chevy,Hyundai, bmw, sale, low miles,clean.
- Most common year of the car models are 2005,2006,2004 or 2009 (5 years ) or 2013 or 2015 (3 years old or less ).
Future Directions:
I would like to analyze other details of the ads by looking at the detailed body of the ads and some other optional fields like mileage, wheel drive,condition of the cars found on the detailed page. If I can get access to some historical data, I would like to predict the probability of the selling car depending on various factors.
If you have comments, questions or suggestions, please let me know.
About Author
Radhey Shyam
View all posts by Radhey Shyam >Related Articles
Leave a Comment
