Building a Painting Style Classifier with Tensorflow

Marc Fridson
Posted on Apr 5, 2017

The objective of my project was to build a painting style classifier with Tensorflow (Artificial Neural Network), that would be able to distinguish 28 different painting styles:

  1. Abstract
  2. African
  3. Chinese
  4. Color Field
  5. Concept Art
  6. Contemporary
  7. Cubism
  8. Expressionism
  9. Fauvism
  10. Futurism
  11. Hyperrealism
  12. Impressionism
  13. Indian
  14. Islamic
  15. Japanese
  16. Korean
  17. Landscape
  18. Lyrical Abstraction
  19. Minimalism
  20. Modern Art
  21. Modernism
  22. Mural
  23. Photorealism
  24. Pop Art
  25. Pop Out
  26. Still Life
  27. Surrealism
  28. Veduta

I attempted to collect 50,000 sample images of each style or 1.4 million images total. This required use of a search engine API, namely Bing Cognitive Web Search API. Unfortunately the API could not handle the 50,000 sample workload, so ultimately I only ended up with ~1,000 images per class (1/50 of the goal). Additionally, since I leveraged a search engine for this task, the images came from a wide range of sources and thus it was essentially impossible to create truly homogeneous samples for style.

Luckily I was fortunate enough to have a 32 processor server with 208GB of RAM and 8 NVIDIA K80 Graphic Cards at my disposal. Unfortunately Tensorflow is still an evolving framework and is very difficult to optimize, and has restrictions based on the OS you are using (e.g., on Windows Tensorflow only works with Python 3.5.2).

Having around 2 weeks to attempt this massive undertaking I sought out existing models I could leverage for my problem. I narrowed down my base model to VGG Net, which was proved very effective for a similar problem, the ImageNet Challenge 2014. They ended up finding that a 16 and 19 weight level variations of the model proved most effective. The weights from this model implementation have been used to create a Neural Artistic Style Transfer program that can transform any photo you give it, to have the theme of your favorite painting. See a breakdown of the VGG Net Model in the table below:

VGGNet

I used the Tensorflow playground to approximate the time and complexity to train a model for what I wanted. I approximated with perfect samples, my theoretical ceiling would be 78.5% at 5,000 epochs (see image below). I determined there would not be enough time to run and test multiple models since I believe it would take over 24 hours to run a single training model.

Screenshot (38)

So I ended up settling for demonstrating the application of the VGG-19 ImageNet to transform a photograph of Columbia University to the style of an Abstract Painting. See (a)Original Photo (b)Abstract Painting (c)Photo in the style of the painting

(a)

IMG_20170121_210744 (b)

Abstract-Painting

(c)

columbia-abstract

The above image (c) took 36 hours to produce, which demonstrates how time consuming it would be to build a model that not only essentially performs the above transformation for 1.4 million images, but also applies the results of it to an optimization problem that can use the common characteristics to estimate the most probable painting style of a new image fed into the model. This is a problem that fascinates me and that I will continue to work on.

About Author

Marc Fridson

Marc Fridson

In addition to my current participation in the Data Science Academy, I am a Course Designer/Facilitator for Columbia University's Applied Analytics Program and the CEO/Founder of Instant Analytics an analytical technology start-up. Prior to this I was the...
View all posts by Marc Fridson >

Related Articles

Leave a Comment

No comments found.

View Posts by Categories


Our Recent Popular Posts


View Posts by Tags

#python #trainwithnycdsa 2019 airbnb Alex Baransky alumni Alumni Interview Alumni Reviews Alumni Spotlight alumni story Alumnus API Application artist aws beautiful soup Best Bootcamp Best Data Science 2019 Best Data Science Bootcamp Best Data Science Bootcamp 2020 Best Ranked Big Data Book Launch Book-Signing bootcamp Bootcamp Alumni Bootcamp Prep Bundles California Cancer Research capstone Career Career Day citibike clustering Coding Course Demo Course Report D3.js data Data Analyst data science Data Science Academy Data Science Bootcamp Data science jobs Data Science Reviews Data Scientist Data Scientist Jobs data visualization Deep Learning Demo Day Discount dplyr employer networking feature engineering Finance Financial Data Science Flask gbm Get Hired ggplot2 googleVis Hadoop higgs boson Hiring hiring partner events Hiring Partners Industry Experts Instructor Blog Instructor Interview Job Job Placement Jobs Jon Krohn JP Morgan Chase Kaggle Kickstarter lasso regression Lead Data Scienctist Lead Data Scientist leaflet linear regression Logistic Regression machine learning Maps matplotlib Medical Research Meet the team meetup Networking neural network Neural networks New Courses nlp NYC NYC Data Science nyc data science academy NYC Open Data NYCDSA NYCDSA Alumni Online Online Bootcamp Open Data painter pandas Part-time Portfolio Development prediction Prework Programming PwC python python machine learning python scrapy python web scraping python webscraping Python Workshop R R language R Programming R Shiny r studio R Visualization R Workshop R-bloggers random forest Ranking recommendation recommendation system regression Remote remote data science bootcamp Scrapy scrapy visualization seaborn Selenium sentiment analysis Shiny Shiny Dashboard Spark Special Special Summer Sports statistics streaming Student Interview Student Showcase SVM Switchup Tableau team TensorFlow Testimonial tf-idf Top Data Science Bootcamp twitter visualization web scraping Weekend Course What to expect word cloud word2vec XGBoost yelp