A world-first 3D modelling tool developed at the University of Sydney has successfully demonstrated how users can explore deeper understanding and learning when engaging with AI.

The Treellis system addresses growing concerns that AI users outsource their thinking, potentially inhibiting their learning. A conference paper outlining the program today received a highly competitive Best Paper Award at the world's leading forum on human-computer interface technology.  

Modern AI tools can turn a few words into a completed 3D design, but the result does not necessarily help users understand how it was made. Researchers in the University's Faculty of Engineering led the development of Treellis, in collaboration with the University of Cambridge and Singapore Management University, to explore how co-creating with AI can provide learning opportunities and develop independent skills.

"Treellis supports more transparent and collaborative human-AI co-creation," said Yixiang Chen, a PhD student at the Faculty of Engineering and the lead author of the paper. "The tool does not just make things for you. It shows you how things are made.”

The interface has scope to be rolled out to professionals with limited exposure to the creation of 3D designs, such as medical practitioners looking to develop personalised devices for people with disabilities, said Dr Anusha Withana, an associate professor in the School of Computer Science who led the tool’s development.

“Treellis allows you to control and manipulate the designs completely relying on AI, and it even lets you make mistakes, which is a big part of how we learn,” Dr Withana said.

Working iteratively with AI

Rather than producing a final design from a few verbal prompts, Treellis works iteratively with users to refine the work during assembly through both manual and AI-assisted operations. The interface is open source and freely available to anyone with a browser, and can be incorporated into any AI platform via an application programming interface (API).

The ability of Treellis to enhance users' learning was tested on two groups by a team led by Dr Withana. All participants were tasked with designing three 3D objects: a cup, a desk and a computer monitor, each with increasing complexity and specific requirements. The baseline group built the objects using a prompt-based standard AI tool. The second group built the objects using Treellis. Unlike the prompt-only tool, Treellis made individual parts of each model and their spatial relationships completely visible. Participants could directly select specific components of the 3D object and progressively adjust their size or position through a graphical user interface and immediately see how these changes affected the model.  

Researchers observed Treellis users actively explored how their 3D designs were assembled and tried small, reversible changes to modify them to match the target. Doing so helped them learn complex 3D design concepts intuitively. Conversely, participants using the typical prompt-only tool often did not understand how the 3D designs were assembled, and their attempts to refine parts of the model introduced new problems elsewhere.

Addressing the challenges posed by AI

Dr Withana said Treellis helped address two challenges commonly faced by users of AI technology: the gulf of evaluation, or the difficulty of understanding how a system works, and the gulf of execution, or the inability to execute a desired task using the system.

“The tool’s interactive functionality narrows both gulfs and prevents users from completely offloading the task to AI," he said.

The paper ‘Treellis: Creating In-Situ Parametric Editing Tools to Progressively Design AI-Generated 3D Models’ received a Best Paper award yesterday at the world-leading Association for Computing Machinery Symposium on User Interface Software and Technology (UIST 26) in Michigan. Best Paper Awards are conferred to the top 2 percent of papers delivered at the conference, selected from approximately 260 papers delivered at this year's conference.

Professor Anne Roudaut (University of Bristol, UK), chair of the UIST 26 award committee, said: “The UIST Best Paper Awards recognise research that stands out for its originality, scientific rigour and potential to make a lasting impact on the field. The selected papers represent exceptional contributions that not only advance our understanding of human-computer interaction but open up exciting new directions for future research and innovation.”

Dr Withana and his team hope to apply the open-source interface in clinical settings. For example, clinicians could use Treellis to interactively design fitted ankle-foot orthotics (AFOs) for children with cerebral palsy, tailor-made wrist splints for patients with arthritis, or highly personalised ergonomic grips for daily rehabilitation equipment.

"Treellis breaks the pattern of using AI as a black box," Dr Withana said. “Using 3D content design as an example, it enables users to understand the internal workings of the output from AI, so they can interactively learn what is generated and change it."