Art imitating art? That isn’t quite the case here, but a new algorithm developed by the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) in conjunction with Microsoft searchers for art that looks like other artworks. Called MosAIc, the algorithm uses artificial intelligence to search for works that are similar in nature or similar in concept, from various cultures and time periods, to a specified artwork.
Led by Mark Hamilton, a CSAIL PhD student, MosAIc was inspired by “Rembrandt and Velázquez,” an exhibition centred on Rembrandt and Diego Velázquez shown at Amsterdam’s Rijksmuseum that wrapped up in January of this year. The exhibition showcased the similarities between works by the 17th century masters, one Dutch and one Spanish, among others, although there is no evidence to support them the artists would have been aware of one another. One such comparison included in the exhibition was of Francisco de Zurbarán’s The Martyrdom of Saint Serapion and Jan Asselijn’s The Threatened Swan. The pair of paintings piqued Hamilton’s interest, and soon, MosAIc, which brings together “unlikely, yet uncanny” artworks, was born.
AI has become more ingrained in the art world in recent years, as artworks created by AI have started to take off, even if for some they’re simply fascinating oddities. Instead, MosAIc works as a Conditional Image Retrieval (CIR) application that needs just one image to get started, which sets it apart from other existing projects, like “X Degrees of Separation” by Google, that seek similar goals.
MosAIc was trained using New York City’s Metropolitan Museum of Art and the Rijksmuseum’s open-access collections. Using a treelike data-structure, called a “conditional KNN tree,” MosAIc searches its stores for artworks that are related to the original image. The related works are categorised into two “branches,” either media or culture, which then allows users to adjust the parameters in order to find objects from a specific culture or medium of art to narrow down, or “prune,” their search. MosAIc spans centuries of works, creating a network of art from various cultures and artists who were working a plethora of mediums. Additionally, Hamilton and his team worked to allow MosAIc to link artworks that are were similar in meaning and theme, not only colour and style.
“Every time I use the algorithm,” Hamilton told ArtNet News, “I find surprises.”
Hamilton, along with Stephanie Fu, William T. Freeman, and Mindren Lu, all of MIT, have published a paper on their research into CIR, such as MosAIc, discussing its aims and abilities. The team doesn’t expect MosAIc to replace curators, recognising the invaluable expertise they and art historians bring in discovering deeper meaning and context amongst artists and their works; however, they hope MosAIc will become a resource for experts planning exhibitions and perhaps even those in other areas.
“Going forward, we hope this work inspires others to think about how tools from information retrieval can help other fields like the arts, humanities, social science, and medicine,” Hamilton said in MIT News. “These fields are rich with information that has never been processed with these techniques and can be a source for great inspiration for both computer scientists and domain experts. This work can be expanded in terms of new datasets, new types of queries, and new ways to understand the connections between works.”
MosAIc also surprised the team in its ability to pick out the “blind spots” of GANs, or generative adversarial networks that are used in making so called “deepfakes.” While deepfakes can be an informational and engaging tool, – think of the way AI brought the Mona Lisa or Salvador Dalí, himself, to life – they have also been linked to the creation of fake news and false propaganda. In being able to flag such GANs, MosAIc could be beneficial in identifying AI-generated fakes that have become more prevalent in the last year or so and will most likely only become more a part of day-to-day life in the future.
