Analysis of Stories Submitted to Hacker News

Posted on October 30, 2017

Introduction

Hacker News is a news website run by the investment fund and startup incubator Y Combinator. The site FAQ suggests that any topic which a good hacker might find interesting is appropriate for submission, though in practice most stories are about programming or the technology industry. Hacker News allows users to submit links and descriptions of URLs of interest and to vote for stories submitted by other users. The site displays the stories ranked according to an algorithm which determines story rank based on number of votes and the amount of time since the story was submitted.

I decided to scrape Hacker News to gain insight into what types of news stories Hacker News users consider most interesting and relevant. Using Python's Scrapy package, I was able to create a bot which scrapes Hacker News to obtain data about each news story which appears on the front of Hacker News. I also noticed that Hacker News provides a free, publicly available database of all submitted stories available through Google BigQuery, which I used to obtain a great deal more data for analysis than I could have obtained only by scraping. After filtering out stories which were too old (pre-2014) or which lacked a link, I was left with 544,573 Hacker News stories in my data set.

Scraping

Hacker News' story list web pages have a relatively simple document structure which can be handled easily by Scrapy. There are 20 pages worth of stories available at any given time, each of which I scraped each time I ran the scraper. For each story, I extracted the story title, link URL, submitter's username, number of votes, identification number, and time of story submission. The same information about each story was present in the data obtained from BigQuery.

Data Analysis

Using the data, I analyzed which users were most prolific in submitting stories to Hacker News as determined by three different metrics-- number of stories submitted, total number of votes received for submitted stories, and average number of votes per submitted story. I used the same metrics to investigate the top domain names in submitted story URLs, the popularity of a selected list of programming languages, and the best time of day to submit a story.

The top story submitters by story count have all submitted at least several hundred stories, with the most prolific submitter having submitted nearly 2,000 stories. The list of top submitters by total vote contains many of the same users, which is not that surprising considering that stories can only be upvoted so that submitting more stories can only add to your total score. What is more interesting is that many of these users also appear on the list of top submitters by average votes per story-- they do not just submit large numbers of stories, they are good at picking stories which will gain many votes.

The top domains submitted by all three metrics include github.com, those for popular content hosting sites like youtube.com and medium.com, and those for news sites (both tech-focused and of general interest). GitHub was the most popular domain going by total submissions and total score, though it fell to 6th place by average score per submission. This is consistent with a large number of programming projects being submitted, with the average programming project being somewhat less popular than the average submitted article from a major news site like the New York Times or Wired.

Javascript was the most popular programming language discussed by total submissions and total votes, followed by Python. The R language was less popular by all three metrics. Interestingly, the Haskell programming language had the highest average score per story, though it was the least popular language of all those tested by the other two metrics.

Examining the pattern of story submissions by hour, there is a clear trend which repeats each day. The most stories are submitted between 4:00 pm and 5:00 pm in the UTC time zone, which corresponds to 9:00 am to 10:00 am in San Francisco where Y Combinator is located. This is consistent with a site with many users from California's tech community, many of whom use the site around or shortly before the beginning of the work day. This hour of the day also has the highest average votes per submitted story, though the daily trend is less pronounced in that metric. This is consistent with stories gaining most of their votes within an hour or two of first being submitted-- something certainly true of stories which do not receive votes and are quickly culled.


About Author

Andrew Brainerd

Read more

Leave Responses

Your email address will not be published. Required fields are marked *

Pauline January 21, 2018
Hello there, You've done an excellent job. I will certainly digg it andd personally suggest to my friends. I'm sure they will bee benefited from this weeb site.
Bibit Durian Musang King January 16, 2018
Amazing! This blog looks exactly like my old one! It's on a totally different topic but it has pretty much the same page layout and design. Wonderful choice of colors!
Glasgow CCTV January 8, 2018
Have yoou everr thhought about creating an e-book or guest authoring on other blogs? I have a blog based on the same information you discuss and would love tto have you share some stories/information. I know my audience would appreciate your work. If you're even remotely interested, feel free tto send me ann e mail.
Bibit Durian Musang King January 1, 2018
Just wish to say your article is as surprising. The clarity in your post is simply excellent and i could assume you are an expert on this subject. Well with your permission let me to grab your feed to keep updated with forthcoming post. Thanks a million and please continue the enjoyable work.
Jual Bibit Durian Montong December 26, 2017
Your style is very unique in comparison to other people I've read stuff from. Thanks for posting when you've got the opportunity, Guess I'll just book mark this web site.
Cheap Artificial Grass Liverpool December 22, 2017
My family members always saay that I am killing my time here at net, but I know I am getting familiarity everyday by reading suuch good posts.
Jual Bibit Durian Bawor December 22, 2017
A person essentially help to make severely posts I might state. That is the very first time I frequented your website page and thus far? I amazed with the analysis you made to create this particular publish incredible. Fantastic process!
Xbox One Tips and Tricks December 22, 2017
That is rsally interesting, You are an exfessively skilled blogger. I've joined your rss fewd andd sitt up for in search of extra of your wonderful post. Also, I've shared your web site in my social networks
Jual Bibit Durian Montong December 21, 2017
This is really interesting, You're a very skilled blogger. I've joined your feed and look forward to seeking more of your wonderful post. Also, I have shared your web site in my social networks!
Clumber park cafe December 14, 2017
I do not know if it's just me or if peerhaps everybody else experiencing problems with your site. It appears as if some of the written text within your posts aree running offf the screen. Caan someone else please comment and let me know if thhis is happening to them as well? This migh bee a issue with mmy internet browser because I've hhad this happen previously. Thanjk you
American Football Online Free Bets December 10, 2017
Hello there! Do you knoww if they make any plugins to help witfh SEO? I'm trying to get my blog to rank for some targeted keywords but I'm not seeing very good success. If you know of any please share. Cheers!
Aberdeen CCTV December 7, 2017
Thank you so much regarding giving my famiuly an update on this issue on your web site. Please understand thjat iif a completely new post appears or in caqse any adjustments occur on the current write-up, I would be thinking about reading more and finding out hoow to make good utilization of those tactics you share. Thanks ffor your time and consideration of others by making your blog available.
Free nfl football picks December 3, 2017
A person essentially lend a hand to make critically posts I'd state. This is the first time I frequented your website page and so far? I surprised with the analysis you made to create this actual post amazing. Excellent job!
professional cleaners london end of tenancy November 22, 2017
Tank you for your site post. Joneds andd I are already saving tto buy a new guide on thyis subject and your short article has made all of us to save the money. Your opinions really resolved all our questions. In fact, greater than what we had acknowlddged just before we found your great blog. I actually no longer have doubts and also a troubled mind because you have attended to each of our needs above. Thanks
Bawtry places to eat November 22, 2017
[/IMAGE]
web designer In Birmingham November 21, 2017
This siye definitely has all of the information and facts I needed about this subject and didn�t kknow who too ask.
Buy birthday cakes Dubai November 20, 2017
As I site possessor I believfe the content material here is rattling fantastic , appreciate it for your efforts. You should keep it up forever! Best of luck.
Xbox One Tricks November 15, 2017
[/IMAGE]
SPSS & Analisis / Analisa Statistik Bandung November 14, 2017
If some one wants to be updated with newest technologies afterward he must be go to seee this site and be up to date all the time.
crime scene cleaning London November 14, 2017
Howdy! Someone in my Myspace group shared this website with us so I camme too check it out. I'm definitely enjoying thee information. I'm ookmarking and will bbe tweeting this to my followers! Terrific blog and great design.
Xbox One Tips and Tricks November 14, 2017
I've recently startfed a blog, tthe information yoou offer on thus web site has elped mee tremendously. Thnks for all oof your time & work.
Roxie November 14, 2017
It's the best time to make some plans for the long run and it is time to be happy. I've learn this put up and if I may just I wish to suggest you few interesting issues or tips. Perhaps you can write next articles referring to this article. I want to learn eveen more issues about it!
web network November 13, 2017
Good day! Would you mind if I share your blog with my twitter group? There's a lot of people that I think would really enjoy your content. Please let me know. Cheers
Glass Trophies And Awards November 13, 2017
Hello just wanted to give you a quick heads up and let you know a ffew of the imasges aren't loading correctly. I'm not sure why but I think its a linking issue. I've tried itt in two different browsers andd both show the same results.
electronicsforce November 12, 2017
ElectronicsForce.com is proud to current imressive no contract cell telephones at minimal prie cut harges so that every person can ddelight in thhe most trusted feature-sets at the wonderful charges! Appear to us and seem no further more for tthe cost savings and value you dserve oon all issuess wireless and unlocked, wkth terrific handsets onn supply for these who will need the freedom to change individual carriers as they ssee iin shape. Just after all, it really is your mobile phone, you possess it! ElectronicsForce.com has extensive been 1 of the country’sbiggest mobile stores and is as a result in a position to levedrage great economies of scale to give genuinely minimal sdlling prices for our clients. We are the world wide web tremendous-sensation for the most effective in wireless technological innovation with the most up-to-date models on the industry. Get in on the deal these days and get yours at ElectronicsForce.com! We specialize in pairing point out-of-the-artwork technological know-how with rock-base pricing to presenting firms annd basic shoppers alike wonderful offeres iin large-quality all-electronic technological know-how, from no agreement cell phones and electronic cameras to desktops and auto audio. And our in depth catalog just keeps expanding! So allow us know what you want and when yoou want it. For no-inconvenience procuring on the net, choose ElectronicsForce.com aand plug into the savings and worth you deserve!
buy cake in Dubai November 9, 2017
I'm so happy to read this. This is the kind of manual that needs to be given andd not the random misinformation that's at the other blogs. Appreciate your sharing this greatest doc.
Flooring Services Newcastle November 9, 2017
Hello! I know this is kinda off topic nevertheless I'd figuired I'd ask. Would you be interested in exchanging links or maybe guest writing a bog articlpe or vice-versa? My blog goes ovver a lot of the same subjects as yours aand I think we could greatly benefit from each other. If you're interested feel free to send me an email. I lok forward to hearing from you! Excellent blog by the way!
home removals Ireland November 9, 2017
Hey would you mind stating which blog platform you're using? I'm lookiung to start my own bloig oon buut I'm having a diffvicult time selecting between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your layout seems different then most blogs and I'm looking for something completely unique. P.S My apologies for getting off-topic bbut I had to ask!
tarmac driveways in Liverpool November 6, 2017
Appreciating the time andd effort you put into your site and in depth information you offer. It's great to come across a blog every once in a while that isn't the same old rehashed material. Fantastic read! I've saved your site and I'm including your RSS feeds to my Google account.
Jual Bibit Durian Bawor Jatim November 1, 2017
Hi this is somewhat of off topic but I was wondering if blogs use WYSIWYG editors or if you have to manually code with HTML. I'm starting a blog soon but have no coding skills so I wanted to get guidance from someone with experience. Any help would be greatly appreciated!