4.S42 is a new MIT subject which I created and taught in the spring of 2019 (see syllabus). It focuses on creative applications of machine learning (ML) to design generation and data-informed design exploration, with an emphasis on visual and 3D generative systems. It also explores how recent advances in artificial intelligence, and specifically machine learning, can offer humans more natural, performance-driven design processes.
The class covers a wide-range of machine learning algorithms and their applications to design, with topics including neural networks, generative adversarial networks, variational autoencoders, dimensionality reduction, geometric deep learning, and other ML techniques. The subject will be offered again in the spring of 2020.

With Caitlin Mueller

At the end of 4.S42, students are able to:

• Understand the basic mathematical and probabilistic principles behind modern ML algorithms
• Apply existing ML algorithms to architectural and structural design applications
• Prototype ideas by combining off-the-shelf libraries with visual and 3D content generation software
• Generate and gather design data using performance-driven sampling algorithms
• Generate design content, including images, drawings, and 3D geometries, with generative algorithms
• Use ML to explore design spaces with an emphasis on performance
• Create compelling, interactive visualizations of high-dimensional design spaces using dimensionality reduction

The class covers the following topics and their applications to design:

• Regression and classification algorithms, including kernel ridge regression, random forests, and support vector machines
• Multilayer perceptrons
• Convolutional neural networks and neural style transfer
• Recurrent neural networks and long-short-term memory networks
• Dimensionality reduction and manifold learning
• Adaptive design sampling (Markov Chain Monte Carlo, Hamiltonian Monte Carlo, etc.)
• Bayesian optimization
• Deep generative modelling, including generative adversarial networks (including conditional GANs) and variational autoencoders
• Geometric deep learning
• Intro to reinforcement learning

This page showcases a selection of the semester projects that were completed by the students in the spring of 2019. Students were asked to explore and expand upon an area of interest in machine learning. Their work could be based on a previous homework assignment problem, a class exercise or discussion, a reading, or a topic related to their own research priorities and interests. The topic selection was extremely flexible. The goal was to initiate a research question that is original and impactful and to demonstrate promising results.

Ariel Noyman | {Deep} image of the city

Ariel Noyman developed a real-time, generative platform for urban design, based on a Pix2Pic conditional GAN model. His platform generates realistic city visualizations based on user input in a tangible user interface.

Architecture of Ariel's interface and predictive model. Credit: Ariel Noyman

Tuo Sun | 3D mesh reconstruction of urban morphology

Tuo Sun’s objective was to reconstruct urban massing models solely based on 2D information, such as screenshots. To do so, he developed two models. The first uses the Mask R-CNN architecture to classify an urban massing screenshot into different parcels, which are used to decompose the image into multiple smaller patches to feed to a generative network that returns a mesh representation of the predicted reconstruction.

Mask R-CNN model predicting parcel locations based on an urban massing image. Credit: Tuo Sun

Reconstruction results predicted by generative networks based on individual parcel patches. Credit: Tuo Sun

Amira Abdel-Rahman | Learning the functional representation of geometries

Amira Abdel-Rahman’s project is an attempt to predit the functional representation, a unique computational geometry format where geometries are described by mathematical functions, of an input voxel-based geometry. She used LSTM networks and treated functional representations as sentences to predict based on 3D input.

Example predictions for voxel-based shapes. Credit: Amira Abdel-Rahman

Example of sampled input geometries and vocabulary encoding. Credit: Amira Abdel-Rahman

Audrey Bosquet | Sports bras (and making them better)

Audrey Bosquet analyzed shape size data from body scans to identify a better sizing strategy for sports bras, for which she used several dimensionality reduction techniques. Her project also involved a large amount of raw mesh processing.

Clusters of sports bras obtained based on processed mesh data. Credit: Audrey Bosquet

Clusters of sports bras obtained based on processed mesh data with a single cluster highlighted. Credit: Audrey Bosquet

Ramon Weber | Programmable caustics

Ramon Weber seeked out to infer the shape of a 3D-printed glass sculpture based on target caustic reflections. He used a CNN architecture to solve the inverse design problem in a data-driven fashion, by regressing the relationship between caustic reflections and geometry.

Rendering setup used to generate data for different geometries. Credit: Ramon Weber

Random selection of caustics data samples. Credit: Ramon Weber

Training, testing, and validation pipeline. Credit: Ramon Weber