4.S42 creative machine learning for design
data | machine learning | design | teaching | 2019

4.S42 is a new MIT subject which I created and taught in the spring of 2019 (see syllabus). It focuses on creative applications of machine learning (ML) to design generation and data-informed design exploration, with an emphasis on visual and 3D generative systems. It also explores how recent advances in artificial intelligence, and specifically machine learning, can offer humans more natural, performance-driven design processes.
The class covers a wide-range of machine learning algorithms and their applications to design, with topics including neural networks, generative adversarial networks, variational autoencoders, dimensionality reduction, geometric deep learning, and other ML techniques. The subject will be offered again in the spring of 2020.

With Caitlin Mueller

At the end of 4.S42, students are able to:

The class covers the following topics and their applications to design:

This page showcases a selection of the semester projects that were completed by the students in the spring of 2019. Students were asked to explore and expand upon an area of interest in machine learning. Their work could be based on a previous homework assignment problem, a class exercise or discussion, a reading, or a topic related to their own research priorities and interests. The topic selection was extremely flexible. The goal was to initiate a research question that is original and impactful and to demonstrate promising results.

Ariel Noyman | {Deep} image of the city

Ariel Noyman developed a real-time, generative platform for urban design, based on a Pix2Pic conditional GAN model. His platform generates realistic city visualizations based on user input in a tangible user interface.

Architecture of Ariel's interface and predictive model. Credit: Ariel Noyman

Tuo Sun | 3D mesh reconstruction of urban morphology

Tuo Sun’s objective was to reconstruct urban massing models solely based on 2D information, such as screenshots. To do so, he developed two models. The first uses the Mask R-CNN architecture to classify an urban massing screenshot into different parcels, which are used to decompose the image into multiple smaller patches to feed to a generative network that returns a mesh representation of the predicted reconstruction.

Mask R-CNN model predicting parcel locations based on an urban massing image. Credit: Tuo Sun
Reconstruction results predicted by generative networks based on individual parcel patches. Credit: Tuo Sun

Amira Abdel-Rahman | Learning the functional representation of geometries

Amira Abdel-Rahman’s project is an attempt to predit the functional representation, a unique computational geometry format where geometries are described by mathematical functions, of an input voxel-based geometry. She used LSTM networks and treated functional representations as sentences to predict based on 3D input.

Example predictions for voxel-based shapes. Credit: Amira Abdel-Rahman
Example of sampled input geometries and vocabulary encoding. Credit: Amira Abdel-Rahman

Audrey Bosquet | Sports bras (and making them better)

Audrey Bosquet analyzed shape size data from body scans to identify a better sizing strategy for sports bras, for which she used several dimensionality reduction techniques. Her project also involved a large amount of raw mesh processing.

Clusters of sports bras obtained based on processed mesh data. Credit: Audrey Bosquet
Clusters of sports bras obtained based on processed mesh data with a single cluster highlighted. Credit: Audrey Bosquet

Ramon Weber | Programmable caustics

Ramon Weber seeked out to infer the shape of a 3D-printed glass sculpture based on target caustic reflections. He used a CNN architecture to solve the inverse design problem in a data-driven fashion, by regressing the relationship between caustic reflections and geometry.

Rendering setup used to generate data for different geometries. Credit: Ramon Weber
Random selection of caustics data samples. Credit: Ramon Weber
Training, testing, and validation pipeline. Credit: Ramon Weber