NYC Open Data: Streaming Python on Hadoop

Vivian Zhang
Posted on Sep 19, 2015

Our upcoming 12-Week Data Science bootcamp starts on January, 11th, 2016. Apply now to secure a spot in our winter cohort!

In the meantime, come join the NYC Open Data Meetup Group and learn how easily you can use Hadoop for Machine Learning.

If you are hiring Data Scientists, call us at +1-888-752-7585 (USA) or reach us at [email protected] to share your openings and set up interviews with our excellent students.

What is an NYC Open Data Meetup event like? Here’s an example:

This summer, Sam Kamin (Vice President of Engineering, NYC Data Science Academy) gave a teaser version of NYC Data Science Academy’s five-week Hadoop course. (The current version on offer is a six-week evening program, Big Data with Hadoop and Spark.”) See below for a class syllabus from the five-week Hadoop course.

First, though, here are the slides from the Meetup:

(Slides can also be accessed here through SlideShare.)

Plus video of the event:

 Speaker Bio:

Sam Kamin is currently a Vice President of Engineering at NYC Data Science Academy. He is also Associate Professor Emeritus from the University of Illinois Champaign Urbana where he taught computer science for over 20 years and was head of the undergraduate program. Most recently he was an engineer at Google before joining NYC Data Science Academy.

Now for the five-week Hadoop course: 

This five-week course is an intensive, hands-on introduction to the Hadoop ecosystem of Big Data technologies. The emphasis in this course is on learning several of the major components of Apache Hadoop – HDFS, MapReduce, Hive, Pig, Streaming – by doing exercises of increasing complexity. Programming will be done in Python. Students are expected to be familiar with using an operating system from the command line; knowledge of Python is helpful; the material in Learn Python the Hard Way is sufficient background knowledge. The course format is mixed lecture/lab. Students will need to bring their own laptops to connect to our server; instructions will be provided ahead of time as to how to install any required software.

Week 1 – Introduction: MapReduce Overview of Big Data and the Hadoop ecosystem the concept of MapReduce
HDFS – Hadoop Distributed File System
MapReduce with Python streaming

Week 2 – More on MapReduce
More on Big Data, the Hadoop ecosystem, and MapReduce.
Mixed case studies and exercises using MR with Python streaming

Week 3 – Hive: A database for Big Data
Hive concepts
User-defined functions in the Hive language
User-defined functions in Python (using streaming)
Advanced topic: Hive queries in Python code

Week 4 – Pig: Simplified MapReduce
Basic concepts
Pig Latin
Pig functions and macros
User-defined functions

Week 5 – Project day

The Hadoop ecosystem
Brief intro to Spark
Brief intro to Mahout
Case studies/project ideas

About Author

Vivian Zhang

Vivian Zhang

Vivian Zhang is the founder of the NYC Data Science Academy and the NYC Open Data meetup. She earned her M.S. in Computer Science and Statistics and B.S. in Computer Science. She is ranked as one of the...
View all posts by Vivian Zhang >

Leave a Comment

Felicitas August 12, 2016
Creating a shift iin attitude will be about changing perspective Jeffry fresh foods through a fraction within the prices you'd pay to the grocery store Jeffry It puts a glow upon your face using a sparkle on thee inside eyes Though Apple doesn't make dedicated e-book devices Jeffry The average price for a picture book is bout $20 to $30 Jeffry While the Mac doesn't have nearly the market share of PC still, it hass grown and continues to grow at an instant pace

View Posts by Categories

Our Recent Popular Posts

View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Online Training Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python Python Data Analysis python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp