Digital Twin Neighborhood

Josh Antolovic is a designer, artist, and researcher whose work centers on video games and natural systems.

2023-26

A from-scratch Houdini to Unity pipeline that turns photographs and photogrammetry scans of a real neighborhood into two matched VR "digital twins" of the same site: an unmodified version, and a modified version carrying deliberate, controlled signs of disinvestment. Built as a research instrument for a public-health study of neighborhood physical disorder.

Role: Graduate Research Associate; technical artist and pipeline developer. Alongside the generation tools themselves, I organized the VR lab setup and supported the study's IRB submission and revisions to fit our lab's equipment and needs

Partners: SPACE Lab, ACCAD, The Ohio State University

Tools: Houdini, Unity, photogrammetry via Object Capture API, Substance Painter, Photoshop, Figma, Meta Quest hardware & eye tracking

At a glance:

A complete capture-to-headset pipeline: photogrammetry and 360-degree panoramas in, two matched neighborhood states out, one unmodified and one modified with controlled disorder (litter, graffiti, deferred building maintenance, overgrowth)
Nothing hand-placed twice: every yard, surface, and object generated in both states from the same networks
Each section below is a tool or network built to solve one production problem, alongside the actual Houdini network behind it
Demoed publicly at the ACCAD Open House (April 2025) and the VR/AR Showcase at the Byrd Polar and Climate Research Center (January 2026); now documented and handed off to the next graduate researcher
Primarily pipeline architecture and technical art, not mechanics or systems design

The Brief

SPACE Lab researches how the visible condition of a neighborhood, what the field calls physical disorder, relates to community health and disparities. Recruiting people to walk streets that already exist only ever shows them one condition, on one day. VR let us build a neighborhood once and generate two versions of it: one matching the site as it exists, and one carrying deliberate signs of disinvestment, or opposite, so a participant could experience either without leaving the lab.

This is the part of that project I owned. What disorder should look like was not my call alone; that came from SPACE Lab’s research criteria (garbage, graffiti, building condition, yard condition, road verge condition, abandoned structures), tuned by an audit calibrating what actually reads as poor condition versus well kept. My job was building the tools that could generate both states consistently, for every yard and object on the street.

Unmodified

Modified

Capturing the Test Site

The site used to build and test this pipeline is centered on a house nobody lives in. That gave us unrestricted access to photograph it and the block around it all via the sidewalk and street, ahead of and separate from the consent process the study needs for occupied homes, so the pipeline could be built and proven out before participant recruitment was even finished.

Before any system below could run, the site had to become data. I used two capture methods side by side: photos processed through Photoshop and photogrammetry software, and 360-degree panoramas projected directly onto simplified ground geometry in Houdini as a faster photoreal base.

One thread that did not go anywhere yet: Gaussian splats, tested briefly as a possible replacement tree capture. I parked it and documented where it stands. Still an open question for whoever picks this pipeline up next.

Site map

Ground System

The Ground system merges photogrammetry scans of yards and the 360 projected base into one working mesh, remeshed for even topology and modified/painted to blend together seams. Photogrammetry makes up the yards and sidewalks whereas the 360 base composes the street. A procedural seam is generated to connect the two. The 360 base textures undergo a Photoshop treatment to give the illusion of continued street (from the same few images), while the photogrammetry textures undergo a Substance Designer cleanup.

Annotated ground system diagram

360 ground network in Houdini

Photogrammetry ground network in Houdini

Ground seam network in Houdini

Compiled ground network in Houdini

Foliage System

The foliage network turns a small library of grass and weed reference photos into a spray-paintable variant set, each variant carrying its own scale, brightness, and rotation randomization, shaded with an unlit shader that darkens shorter plants sitting under taller ones for depth.

Foliage example

Foliage network in Houdini

Trees

There are four tree generators depending on need: near asymmetrical, near symmetrical, mid-range, and far. The main idea of the tree generators is to build a trunk and branch network procedurally, then dress it with layered leaf clusters which are manually cropped from an image. Each is done semi-manually given the need for ‘that’ specific tree in someone’s yard. The closer the tree to the participant’s walking path, the more manual and detailed the tree making becomes.

Near, asymmetrical tree example

Near, asymmetrical tree network in Houdini

Near, symmetrical tree example

Near, symmetrical tree network in Houdini

Mid-range tree example

Mid-range tree network in Houdini

Far tree example

Disorder Object Library

Litter, debris, and damage props (takeout containers, receipts, torn clothing, broken toys) came from a mix of photogrammetry and licensed downloads. Every object was cleaned to low poly, auto UV'd, and lighting baked before placement. Placement was both scattered with a spray paint tool similar to that of the overgrown lawn, in addition to manual placement of distinct objects such as bags of trash, carts, or any other oversized objects.

Disorder objects example

Disorder objects network in Houdini

Houses, Billboards, Cars

Each house exists in both unmodified and modified form, the modified version carrying graffiti, damage, and boarded-up openings layered on via Photoshop. Background structures too far away to warrant full modeling became flat billboards instead, textured from site photos. Street-parked cars came from a small licensed asset set, since detail there mattered less than simply reading as an occupied, lived-in street. Another investigation that occurred right before handing the project over was that of photogrammetry cars: how to scan them and clean up geometry and texture which was minorly mangled during the photogrammetry process.

House example unmodified

House example modified

House example unmodified

House example modified

House example unmodified

House example modified

Housing network in Houdini

Photogrammetry car example

Photogrammetry car network in Houdini

Interaction

Two systems carry the actual VR experience: a teleport boundary limited to the sidewalk nearest the participant's house, generated from a navmesh cut to match, and an eye tracking rig built around box collider and label components on scene objects so gaze could be logged against a real feed.

Documentation, Handoff, Status

A good part of this project was not the systems themselves but making sure someone else could run them. I kept a licensing tracker for every downloaded or photographed asset, organized the OneDrive and Houdini folder structure by pipeline stage, and recorded most of the harder workflows as screen capture rather than written steps, since some of this is easier to show than describe. Outside the pipeline, I organized our VR lab setup and helped get the study's IRB submission built and revised to match it.

Houdini documentation

Lab setup document

The pipeline has been shown publicly twice: as a demo at the ACCAD Open House in April 2025, and at the VR/AR Showcase and Meetup at the Byrd Polar and Climate Research Center in January 2026. As of 2026, it is handing off to the next graduate student, who is learning it now and will generate the production neighborhoods once participant recruitment wraps up.

@joshantolovic

Index Next →