Step 1: Setting the Scene with Babylon.js I began by creating a minimal 3D world using Babylon.js. The environment included a simple ground plane and several spheres representing agents (ants). These spheres were animated to move around the space with basic logic, simulating movement across a terrain.

Technologies Used:

• Babylon.js (UMD version for browser compatibility)
• Node.js static server (Express)
• Custom proxy middleware to handle CORS issues with Ollama

Step 2: Adding Cognition – Perception and Memory Once the world was in place, I gave each agent the ability to perceive its environment. This included checking for nearby agents within a perception radius and logging these encounters in memory.

Each agent maintained:

• A list of recent positions
• A memory of other agents it has seen
• A log of its own thoughts

This memory would serve as the context for decision-making.

Step 3: Integrating the Local LLM With cognition in place, I hooked up each agent to a local LLM hosted by Ollama. I chose OpenHermes for its conversational tone and ability to stay in character. Each agent constructs a prompt based on its memory and current perception, then sends that prompt to the LLM.

Example prompt:

You are an autonomous agent exploring a virtual world.
You remember: Saw agent 2 nearby.
You currently perceive: Agent 3 at (x, z).
What will you do next and why? Format your answer as:
Action: <...>
Reason: <...>

The LLM’s response determines the agent’s next direction of movement.

Step 4: Preventing Overload – Throttling LLM Requests Running LLM inference every animation frame quickly overwhelmed the system. To fix this, we throttled each agent to only send a request every 3 seconds. This kept Ollama stable while allowing agents to reflect and act periodically.

Emergent Behavior Observed Once throttling was in place and OpenHermes was responding correctly, we started seeing surprisingly organic outputs:

Agent 1 thought: Action: Move forward. Reason: To explore the environment and gather information.
Agent 2 thought: Action: Explore the environment. Reason: To gather information and potentially find new resources.

These responses, while simple, represent the foundation of autonomous behavior: perceiving, remembering, and acting with purpose.

At this stage, the agents demonstrate:

• Local perception
• Memory of encounters
• LLM-driven decision-making
• Individual internal logic and reflection
• Co-operation
• Shared memory or communication between agents
• Goal-setting and prioritization
• Emotional or philosophical traits
• Planning and resource seeking

Conclusion With very little direction, these little characters started talking, working together, sharing and forming their own little (emulated) personalities based off the input gained from the environment. I am very impressed with OpenHermes, it was a lot easier to train than deep-seek and other local llms I have tested.

Stay tuned for the next evolution.

Tags: Babylon.js, Ollama, LLM, AI Agents, Ant Simulation, Emergent Behavior, OpenHermes, Cognitive Architecture

📐 Concept & UX Goals

From the outset, the experience had to feel like operating a sci-fi viewscreen. Every interaction, click, scroll and transition, should feel spatial and connected. We weren’t building a site, we were building a starship UI.

Our core UX goals:

• Pages = Places: navigating to a new page would “move” the camera to a new area of the space scene
• Minimal UI: reduce visual clutter to improve immersion
• Asynchronous Loading: no full reloads, just transitions
• Scene-Integrated Navigation: content and environment exist in the same universe

🌌 Babylon.js Scene

I started by crafting a Babylon.js space environment:

• A static camera at (0,0,0)
• Distant stars, planets, nebulae
• A configurable offset vector allowed us to simulate movement through the universe by shifting the rendered positions of celestial bodies

Rather than move the camera itself (which could break large-scale visual fidelity), I updated each object’s render position based on the current offset, simulating “traveling” to content.

jsCopyEditlet renderPosition = originPosition.subtract(offset);
mesh.position.copyFrom(renderPosition);

🚀 Navigating with Offset

Each WordPress page was mapped to a specific position in space. Navigating to /about might move the scene’s offset to (5000, 0, -10000). I added smooth interpolation with easing so it felt like flying:

jsCopyEditoffset = BABYLON.Vector3.Lerp(offset, targetOffset, speed);

🔁 Asynchronous Content Loading

Using AJAX (wp_ajax in PHP), I loaded pages dynamically without reloading the scene:

• All links intercepted via JavaScript
• Pages loaded via jQuery.ajax
• The HTML content (usually .entry-content) was swapped into the DOM
• Babylon’s update loop persisted across transitions

Each request returned HTML fragments from WordPress templates (get_template_part()), not full pages.

if ($query->have_posts()) {
  while ($query->have_posts()) {
    $query->the_post();
    get_template_part('template-parts/content', 'page');
  }
}

🧠 UX Considerations

• Minimal layout: No bulky header or footer. Just a soft HUD-styled logo and a subtle menu tucked in the corner.
• Focus on content: Each page block hovered in the scene like a projected panel, with thin neon borders and glowing text.
• Consistent frame: Scene stayed loaded at all times, which made transitions feel native, not like switching pages.

💫 Design System

I defined reusable styles that mirrored Star Trek and Mass Effect interfaces:

• Typography: Orbitron, VT323, and glowing letter shadows
• Menu: A semi-transparent, floating “command stack” with expanding tree-like submenus
• Page container: 3D-transformed <article> blocks that could rotate slightly with scroll to simulate perspective
• Animations: Fade-ins, pulse highlights on hover, glint effects on logos

🔧 Challenges

• Camera vs. Offset: Avoiding actual camera movement helped prevent precision issues at astronomical scales.
• Dynamic Element Bindings: We had to re-bind event listeners and re-init transitions after each AJAX load.
• Performance: Particle systems (like nebulae) required LOD toggling and distance culling to keep the experience smooth.

🧩 Tying It All Together

Every piece, navigation, animation, styling, and spatial layout feeds into one consistent UX metaphor:

You’re not browsing a website. You’re navigating a ship’s digital log interface.

✅ What’s Next

• Saving scene position in URL hashes or browser state
• Adding audio feedback and voice logs
• Possibly: AI-powered ambient chatter using LLM output (see /logs/building-my-first-ai-wordpress-agent-a-developers-first-experiment-with-local-llms/)

🧱 What’s Next

• Expand it to use the REST API, to increase capabilities
• Integrate with server-side shell tasks (upgrades, backups, etc.)
• Maybe even let it run scheduled audits and suggest updates before I need them

I’m Derek Robertson—developer, systems guy, occasional designer-wrangler, and all-around problem solver. I’ve been in the game for over 25 years, turning clunky, chaotic systems into smooth, scalable digital solutions. Right now, I’m part of the crew at Buildxact, working at the intersection of SaaS, construction tech, and real-world usability.

What I Actually Do

My background spans full-stack development, content platforms, e-commerce, CMS architecture, integrations, UX optimization, DevOps, and a healthy dose of “how do we make this thing talk to that thing?” I’ve worked on cross-platform apps, internal tools, customer portals, and websites that handle millions of requests a day.

I’ve worn many hats: developer, team lead, systems designer, tech translator, bug hunter, and mentor. If you need someone who can build it, fix it, streamline it, or make sense of it — I’m your guy.

Tools of the Trade

Tech-wise, I work mostly with:

• PHP, C#
• JavaScript / Node.js (React)
• SQL, GraphQL, REST APIs
• Azure, Git, Copilot
• Photoshop, Krita, GIMP
• Virtualmin, CPanel

But honestly? I care more about why we’re building it than what stack it’s on.

Why It Matters

I like working on things that help real people do real work. Whether it’s helping builders quote jobs faster, making internal systems suck less, or improving onboarding flows that reduce support calls—I’m all about meaningful improvements. If we can cut steps, reduce errors, and give users more confidence in their tools, that’s a win.

What You’ll Find Here

This blog is a place for the stuff I wish more people talked about:

• Building smarter, not bigger
• Lessons from the field (and the dev trenches)
• Dev workflows that actually save time
• UX decisions from a dev’s point of view
• And probably a few war stories along the way

Thanks for stopping by—feel free to reach out if you want to chat code, ideas, or the weird bugs that keep you up at night.

Catch you in the next post.
– Derek