Scraping Craigslist (Cars+Trucks)

Radhey Shyam
Posted on May 27, 2016

Contributed by Radhey Shyam. He  is currently in the NYC Data Science Academy 12 week full time Data Science Bootcamp program taking place between April 11th to July 1st, 2016. This post is based on his third class project -Python web scraping (due on the 6th week of the program).

Introduction:

For my web scraping project , I chose Craigslist website. I really like its concept and  even I bought my first car from a craigslist seller and it was much cheaper than dealers. I spent lot of searching on craigslist wondering what was the best time to find the deal. As from my search,I found that a good car deal do not last for a  long time. It will be really helpful  if you can find out the most probable day and time when people place the ad so that you can reply promptly. Secondly, I wanted to analyze the price range,make and model year of the cars/trucks on the website.

Methodology:

For scraping craigslist, I picked  cars & trucks (owners) section of New York craigslist. To scrap the data,I used scrapy and PyCharm ,which is an Integrated Development IDE) for Python.  The main scarping file imports a Spider class from Scrapy package.The spider class is where we define how the website will be scarped, allowed urls and starting urls. First, I defined the allowed urls for the craigslist website , and staring url of the webpage. I checked the main page of the website and found out that only first 100 car ads are posted on the main search page.There were in total 25 pages of ads, each page consisting of 100 ads.I found the pattern of next page listings as starting url "+ ?s=i*100" . Next ,I created a list of urls of all pages using a for loop as shown below.

https://gist.github.com/rshyam1/d181356f7c9549c23facbe6451d3369c

Next , I setup how the webpage will be scraped. As each page consists of 100 ads, I scraped the ad ids (100) contained in the page  and stored them in a list. Then for each ad, I created complete web address by concating the main web url with ad id  plus html.Then I sent parse request for each ad page and parsed the price, title, posting time and body of the ad and stored the data in a csv file and code for the same is shown below.

https://gist.github.com/rshyam1/b84459a0057d232d9266d146c934d233

Next step was to clean the parsed data.In car price data, I found that sometimes people also listed used car parts in car sale section and their price was usually less than $300. I removed all the entries having price less than $300. The price histogram of all the car ads is shown below and I chose bin size of $500 for the histogram. The range of car price was from $300 to $110,00 with median price of $4800. Below is shown the histogram of car prices in the range of $300-$20,000.

 

price_hist

As we can see from above price histogram that peak sales of 180 counts happens in the price range of $3000-$3500 and the price count drop drastically to 30 above price range of $10,000. I think that it makes sense that people normally used craigslist for selling used car under $10,000 and it will be hard to sell costly cars on the craigslist.

Another curiosity was to find that whether  people prefer  any day or time to post their ad on craigslist ?To find the posting day, I went to each car ad's detailed listing page and parsed  the posting date on the bottom of the page and stored in post_date variable as string. Then I convert the string to date format and then  used R's base class function,strftime() to extract weekday from the ad posting date.

weekday

As we can see from above bar chart of weekdays that the peak day for the posting the ad was Friday, followed by Wednesday and Saturday. I think that people post car ads on Friday so that their ads remain fresh ( on top search page) during weekend and they have enough time for giving the car trials on weekend. Also, from my personal experience, people respond to the newer posted ads on weekends and it is easy to take calls and gave car trial on weekends.  The only surprise to me was Wednesday as it is in the middle of the week and but I think that people were getting ready for the weekend to sell the car.

Next, I was curious about the time of placing the ad and whether people prefer any time for placing the car.This time, I separated the time from the as post date field and converted them to 24 hour time by adding 12 to pm time and rounded them by the hour.

weekday_time_violin

The above violin plot are sorted by increasing count of the ads from let to right. There is interesting peak at 9 am on Saturday,  as more sellers place their ad on Saturday in between 9 am to 9:59 am than other time of day. So , if you are a buyer and looking for new ads,then this is time you should look for on Saturdays so that you cannot miss a good deal. Also, count density on Saturday becomes narrower as the day progresses, and you are less likely to find fewer new ads as the day progresses. Also,there is a peak around the same time at 9 am on Friday and Saturday. And There is very few or no activity from midnight till 4 am in the morning on all days.

Next, I searched the title of the ads for most common words.While analyzing the titles , I removed all the numbers and characters and created a word cloud using R language's  word cloud library and it is shown below.

title_word_cloud2

The size of the words are proportional to their frequency in the title of the ads. The most common words in the title were make of the cars: Honda, Ford,Chevy,Hyundai,bmw, sale, low miles,clean ..

Finally,  I want to know most common model years of the cars.For that , I separated the numbers from the title and created a word cloud of the numbers from size 4(2005)  to 2 (05) as shown below.

model_years

 

The size of the years in the word cloud  tells the frequency of the years in the title. The most common model years were 2005,2006,2004 (it shows that people commonly sell cars which are 10 years or old on the craigslist ) and 2009 ( 5 plus years or so).

Conclusion:

  • Price histogram  peaks in the range $3000-$4000 and it drops significantly after the price range of $7500-$8000.
  • Friday is the most common day to put ad and it closely followed by Wednesday and Saturday.
  • Saturday  9-10 am is the most common time to place ad , if you are looking for used car, then it is the best time to check the craigslist for new ads.
  • Most frequency words  in title are Honda, Ford,Chevy,Hyundai, bmw, sale, low miles,clean.
  • Most common year of the car  models  are 2005,2006,2004  or  2009 (5 years ) or 2013 or 2015 (3 years old or less ).

Future Directions:

I  would like to analyze other details of the ads by looking at the detailed body of the ads and some other optional fields  like mileage, wheel drive,condition of the cars found on the detailed page. If I can get access to some historical data, I would like to predict the probability of the selling car depending on various factors.

If you have comments, questions or suggestions, please let me know.

About Author

Radhey Shyam

Radhey Shyam

Radhey Shyam received his Ph.D. in Physics from Clemson University, USA and M.S. (Computer Science & Engineering) from GJU of Science & Technology, India. Radhey Shyam did his master's thesis on Network Monitor System, where he applied innovative...
View all posts by Radhey Shyam >

Related Articles

Leave a Comment

Avatar
Google October 13, 2019
Google Always a huge fan of linking to bloggers that I adore but don’t get a lot of link enjoy from.
Avatar
Google October 8, 2019
Google One of our guests not long ago proposed the following website.
Avatar
judi poker deposit 10rb February 6, 2019
So, it's easy to note that you may make bets with low house advantages whilst still being be fairly all-around a 50-50 game. By using cash profit zones, you're not playing from the casino, you might be playing against yourself, now this is very very important. If you bond with other people's experiences, it's easy to find out which casinos are fair and truthful and that are controversial.
Avatar
Gregory September 29, 2018
Oops
Avatar
Abdelmajid Lotfi August 22, 2018
Hi, where is the complete code ? and how to run it ? Thanks.
Avatar
Alex November 24, 2017
I've been doing some web-scraping on Craigslist as well. However, my IP has just been blocked. I'm pretty sure it's because of how many request my program made in such a short time. Did you encounter the same problem? I emailed their support team, but I'm still looking for how I can avoid this in the future.
Avatar
Christian Goldman July 18, 2017
I came across nycdatascience.com/blog and it seems your site would really benefit from our private targeted traffic system. There are two ways you can increase your success online, either improve your sales message or increase the quality/quantity of your traffic. We help you with BOTH! -- With this webmaster deal you can try our SEO/Analysis & Traffic System totally free. Don't miss out, this is a limited time offer! Start your free trial today: http://autoprofit.info/rush/1/AQnlineXdiCsounTx/ Questions? Email : [email protected] // Unsubscribe: http://autoprofit.info/unsub/28904/QskskjcsXcsdD/
Avatar
Pasquale January 3, 2017
Valuable information. Lucky me I discovered your site accidentally, and I am shocked why this twist of fate didn't came about in advance! I bookmarked it.
Avatar
Craigslist posting October 13, 2016
That is very fascinating, You're a very professional blogger. I have joined your feed and stay up for looking for more of your great post. Additionally, I've shared your web site in my social networks
Avatar
JimmyLKunc September 24, 2016
I am sure this post has touched all of the internet viewers, its really really pleasant paragraph on building up new weblog.
Avatar
fake bvlgari bzero1 ring September 11, 2016
$SPX is at or slightly above the 61.8% Fib fan. Obviously, it hasn't broke 1090 (yet). Bulls need to keep this up and, ideally, close higher. fake bvlgari bzero1 ring http://www.bzero1jewelry.net/
Avatar
YaelAFalacco August 27, 2016
Your personal style is incredibly unique when compared with other folks I have got read stuff from. I appreciate you posting in case you have the chance, Guess I will just bookmark this page.
Avatar
fifa 17 coins August 21, 2016
I love this site - its so usefull and helpfull fifa 17 coins http://mortelbox.com/event/1219
Avatar
fifa 17 August 17, 2016
Wow, beautiful portal. Thnx ..
Avatar
madden 17 August 16, 2016
Extremely individual friendly website. Astounding information readily available on few clicks on
Avatar
madden 17 coins August 15, 2016
Whoa, such a handy internet site
Avatar
muscle cars July 25, 2016
Greetings from Idaho! I'm bored to tears at work so I decided to check out your blog on my iphone during lunch break. I really like the information you provide here and can't wait to take a look when I get home. I'm surprised at how quick your blog loaded on my cell phone .. I'm not even using WIFI, just 3G .. Anyhow, great blog!
Avatar
ford mustang July 3, 2016
Very goⲟd info. Lucky me Ⅰ discovered үⲟur site bʏ chance (stumbleupon). Ι havᥱ book-marked іt fоr latеr!
Avatar
craigslist list search June 24, 2016
I do not even know how I ended up here, but I thought this post was good. I do not know who you are but certainly you're going to a famous blogger if you aren't already ;) Cheers!

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp