Artificial Images: RunwayML & Google Colab
Taught by Derrick Schultz and Lia Coleman.
Class Logistics
RunwayML + Google Colab will meet live for 10 weeks (8 weeks of lectures, 2 weeks of heads down time) on Wednesdays at 7:30pm EST starting on February 24. You should be able to attend every session via Zoom, so please make sure your schedule allows for it. In addition to the eight 1.5-hour sessions, there will be two open 1hr sessions with Lia and Derrick every week to answer questions and get help on projects. Additionally, you will have the option of a 20-minute 1-on-1 session with the instructors to get help with a specific project you are working on (to be scheduled individually). We will be using Slack for asynchronous messaging and discussion. Additional materials outside of the lectures and demos will be provided as video links, webpages or PDFs.
Course Syllabus
- Week 1: Intro to ML Art & RunwayML
- Week 2: Intro to Colab
- Week 3: Inspiration & Types of GANs
- Week 4: Creating Datasets
- Week 5: No Class– time to create your dataset.
- Week 6: Training StyleGAN Models in Runway & Colab
- Week 7: Training Pix2Pix, MUNIT, and NFP Models
- Week 8: No Class– time to train your model.
- Week 9: Inference: Generating more images & animations from your model.
- Week 10: Integration with other platforms, Additional Resources
Week 1: Intro to ML Art & RunwayML
Class Materials
Homework
- Explore the pre-trained models in Runway. Playlist of demos on YouTube. Find a model to explore. In the class Slack tell us the following:
- What model you used
- What the inputs and outputs were
- What questions you have about the model
- Show us what you made.
Do this for at least three models.
- Watch this video on Colab.
Notes from class
Week 2: Intro To Colab
Class Materials
- Week 2 Slides
- Week 2 Recording
- Class recording with transcript (missing Colab basics demo)
- Class recording with Colab basics demo (no transcript 😕 )
- Colab Basics Notebook
- Colab StyleGAN2 Projection Notebook
Homework
- Explore Colab notebooks! There are suggested starter notebooks on slide 13 of the class slides (linked above). Also, we have a playlist of Colab demos on YouTube. In the class Slack tell us the following, for at least 2 notebooks:
- What notebook you used
- What the inputs and outputs were
- What questions you have
- Show us what you made.
- Watch this video on GAN inspiration.. Think about what you want to make!
Additional Notes
- For reusable code snippets in Colab: Colab Snippets Notebook, Install Snippets Video
- More on chaining in Runway in this video.
Week 3: Types of GAN
Class Materials
Homework
Watch these two videos:
- Installing Anaconda, dataset-tools, and instagram-scraper
- AI Ethics (Watch from 58:57 to 1:18:37)
Additional Notes
- whichfaceisreal
- Janelle Shane on sheep detectors (example of ML cheating to get results right more or less often)
- 3Blue1Brown on how machine learning works
- 2 Minute Papers
Week 4: Creating Datasets
Class Materials
Homework
Start making a dataset. Reminder: no class next week! This is to give y’all more working time to make your datasets.
- Choose a topic or a model you’re interested in
- Find sources for images (can be a dataset already, or something scrape-able, or multiple sources you want to combine)
Week 5: No Class. Make your dataset!
Week 6: Training NFP & StyleTransfer Models
Class Materials
Week 7: Training StyleGAN & MUNIT
Class Materials
Week 8: No Class. Train, train away!
Week 9: Inference
- Week 9 Slides
- Week 9 Recording
- Pretrained StyleGAN2 Models to play with. You’ll need to run the
Convert Legacy Modelcell in the Colab notebook to convert these to the Pytorch format first!
Week 10: Advanced Inference, Additional Resources
About this class
Artificial Images: RunwayML & Google Colab is a course for image makers (graphic designers, artists, illustrators and photographers) to learn about the basics of generative machine learning using RunwayML and Google Colab. In this class you’ll use popular deep learning models to create images, videos, and interactive experiences. The focus will be on hands-on experimentation with style transfer, CycleGAN/Pix2Pix, and StyleGAN2-ADA in RunwayML and Google Colab. The course will cover the theory behind deep learning and GANs, using pre-trained models, dataset creation, and training custom models. We’ll look at ways to generate images and videos using Python, p5.js, as well as reacting to sound and motion inputs. While you don’t need to have any coding experience, its probably helpful to have some awareness of coding principles. All notebooks will be provided to you so any modifications will be minor.