Cotangent2 launch / post-mortem
/I’ve just released Cotangent2, a mesh-editing app I’ve been developing since I left Epic Games in 2024. You can Download an Installer for Windows at Cotangent.io, and the Source Code on Github. This post is both the launch post and also the post-mortem, because I won’t be developing this app further, at least not in the near future :(
What is/was Cotangent2?
Cotangent2 was intended to be a replacement for Meshmixer, the mesh swiss-army-knife app that Autodesk acquired from me in 2011, and I developed there until 2016. Meshmixer was a very popular tool, in particular with the early 3D Printing community. It eventually reaching millions of downloads, and is still widely used, because there really isn’t anything else like it. You can do similar stuff in Blender now, but you have to figure out wildly more complex UI & workflows. Meshmixer was really great for certain types of mesh-based CAD workflows where you need to design shells or connective surfaces that “fit” other shapes. Often those shapes are 3D scans of humans, and Meshmixer is still widely used in Biomedical CAD, eg at hospitals, by prosthetists, and so on.
Unfortunately when I left Autodesk, Meshmixer was basically sunset. Autodesk tried to fully disappear Meshmixer several times, but kept having to bring it back due to user demand. Today the official recommendation is to use the Mesh Workspace in Fusion 360, which is a very limited subset of Meshmixer functionality. You can still download old installers from meshmixer.org, and many people do. But time and OS updates have not been kind to the unmaintained codebase, and particularly for Biomedical CAD, it’s missing a lot of critical features.
At Epic, I had spent ~5 years building state-of-the-art mesh-processing technology into Unreal Engine. Essentially you could think of it is a much better implementation of the technical core of Meshmixer, with a lot of critical additions, like great mesh booleans. Many features of geometry3Sharp, the C# library I wrote after Meshmixer (again to build biomedical CAD apps…), were ported to UE5 and vastly improved by a team of geometry wizards. In addition, we added the Interactive Tools Framework (ITF) to UE5, as well as a huge library of 3D modeling tools and support classes. And I made sure these were all usable “at runtime”, ie in a built UE5 “game”, instead of just in the UE5 Editor.
So basically I tried to (lightly) steer UE5 towards being a great platform for building Mesh-Based 3D-Tool Apps. And then I left and tried to build one.
Unfortunately, instead of focusing 100% on Cotan2, I went on some side-quests — first the UEToolbox plugin, then Gradientspace Node Graph visual programming system. And so Cotangent2 never really got the attention it deserved. I made slow and steady progress, but nothing that felt like “enough” to ship. And then I started using Claude Code and discovered it was really quite capable at UE5 C++ coding, especially writing ITF Tools. I also switched focus to experimenting with using AI image models for mesh and texture editing, with the idea that Cotan2 could be a companion tool to nascent AI 3D Mesh Generators. And the App started to feel like it was coming together, into … something.
But, as often happens, real life intervened. My window for completely unencumbered indie-development was coming to an end, and someone made me an offer I couldn’t refuse. A career change is looming. So I decided to just release what I’ve done so far, and open-source it, in hopes that it might help someone else (or at least serve as training code for the robots).
Wait, wasn’t there already a Gradientspace Cotangent app?
Yes! During Gradientspace 1.0 (2016-2018) I made a 3D printing app that was called Cotangent. That app was built in Unity, using a slicer I also wrote (gsSlicer), and had a bunch of nice 3DP-focused mesh tools. Unfortunately it suffered a similar fate, ie I did not focus on turning it into a business and eventually gave up and open-sourced it when I joined Epic. I decided to call this new app Cotangent because, well, I liked the name and I had the domain. But perhaps it’s cursed…
Anyway, Cotangent1 still works and is still downloadable - but it’s tied to very old versions of Unity and so can’t easily be built/etc. Hopefully Cotangent2 will avoid this, as code is available for the specific UE5 version, but we’ll see…
OK, so what does Cotangent2 do?
You can find a brief feature list on the Cotangent.io website, or watch the video on the right, which runs through all the major features in about 6 minutes. Cotan2 has three core toolsets - mesh editing, mesh analysis, and texture editing.
Cotan2 evolved from the demo code in my Runtime Tools Framework article. It’s a UE5 “game” built with a slightly customized version of UE5.5. As part of buliding Cotan2 I substantially evolved that sample code, resolving many issues as well as adding support for ImGUI UI. If you are interested in building an “App” using UE5, it would be a great starting point.
In terms of Mesh Editing, I was leveraging the UE5 Geometry libraries as much as possible. Cotan2 has a great Mesh Boolean because it’s exactly the same one you get in the UE5 Editor. 3D Sculpting, Remeshing, Simplification, Plane Cut, Hole Fill, and ‘Make Solid’ basically all are UI layers on top of the fantastic geometry kernel & modeling tool infrastructure in UE5.
One of the reasons Meshmixer was so nice to use was the excellent Mesh Selection functionality - I spent a lot of time tuning the MM brushes and interactions to be efficient and “feel” great. Although a Mesh Selection tool exists in the UE5 codebase, it was never as good, and so I started from scratch. So I put quite a bit of effort into the Mesh Selection in Cotan2. It’s not quite there, and still has some unpleasant rough edges, but it’s close.
I also put in a Text Labelling tool, for embedding 3D text directly into a mesh (eg see the teaser image for the video above). Sounds kinda basic but this was a direct feature request from some hospital users, who were relying on a very expensive Materialise product primarily for this operation. Again this was built on UE5’s Text3D plugin, which did exactly the text-mesh-generation that I needed. Unfortunately the public API didn’t let me directly access the text mesh generation. So I took a cue from the kind of hacks you see all the time in the gamedev world, and just put a Text3DActor miles off-screen, where I can update and read out it’s mesh.
Mesh Analysis was one of the “new” areas I focused on for (theoretical) biomedical CAD users. Over the years I heard so many times that one of the biggest problems in this space is around analysis and validation. I built out some new Measurement tools, like radial measurements based on circle fitting. I also did a new Section Plane analysis tool, where I used 2D Medial Axis to find minimal inscribed circles in polygons which can be used to measure (planar) thickness. And a Mesh Compare tool for visualizing even very slight differences between edited meshes.
Finally, I’d put a lot of work into a 3D Texture Painting system for UEToolbox (video here). That toolset was tied quite tightly to some UE5 Editor functionality, and was going to be a lot of work to untangle. But Claude Code was a huge accelerant here, and in the end it only took a few days to do a rewrite and integrate texture/material support into Cotan2. It didn’t really fit with the “biomedical-CAD” target, but once I started to experiment with AI Image models, it was clear this had legs.
Cotan2 supports OBJ, STL, and GLB mesh formats. I added GLB support so users could edit the kinds of assets one might find online, as well as the meshes generated by 3D AI Generators. For GLB import, currently only the first mesh is supported, and only “simple” materials (ie one single texture). GLB PBR material are partially implemented, but I did not get full support into (eg) texture editing/etc.
Meshmixer Sync
Like I said above, I intended for Cotan2 to eventually be a replacement for Meshmixer, at least for the biomedical CAD userbase. However I did not expect that to happen on day 1. The way hospitals/etc tend to work is with carefully documented workflows, ie “start tool X, set options Y and Z, click Accept”, etc. Even if Cotan2 could do all that, developing new workflows would take time. But (eg) the Text Labeling functionality in Cotan2 would be immediately useful. So I made it possible to ‘Sync’ meshes between live running instances of Meshmixer and Cotan2.
This is possible using a little-known “remote-control” API that Meshmixer supports. I added this many years ago, to support users who wanted to use Meshmixer for automation workflows. The code is available on github (https://github.com/meshmixer/mm-api). It still works because Meshmixer has not really changed since 2016. I made a UE5 plugin that wraps this API and let me pass FDynamicMesh3 meshes back and forth, and sync the Meshmixer scene with my Cotan2 scene.
C# Integration & Tools
Another thing I really wanted to support in Cotan2 was the ability to use C#. I had various reasons for this. For one, my geometry3Sharp library had a lot of algorithms implemented in C# that were going to be a ton of work to port to C++. This aspect is probably moot now that coding agents can trivially do this kind of porting! I also had Gradientspace Node Graph, which is entirely built in C# and would be an amazing end-user-programming environment for Cotan2. That entire system leans heavily on C# language features that are not really plausible in C++, like the Roslyn runtime-compiler, C# reflection, class/method annotations, and so on. And finally I just wanted to make it “easy” to write Tools in C#, which is a much more approachable language than C++. Because the ITF has such a clean API, it would not be difficult to have a “C# ITF Tool” base-class or interface that could be implemented by C# code, which would then unlock (eg) dynamic loading, hot-reloading, etc (and then a similar thing could be done w/ Python…).
So, I spent a lot of time getting this to work, although I did not actually get around to using it for much. The AutoRepair Tool does all it’s mesh processing in C#, this was my proof-of-concept. I didn’t manage to get the full C#-Tool concept done (although Claude Code could probably do it easily now). Currently it requires individual static C# functions to be explicitly (manually) exposed.
AI-Based Features
My exploration of AI-based features started with the simple idea that I could use a local image-generation model to auto-generate “stamp” textures. Way, way back in 2006, I had a SIGGRAPH paper on “decal texturing”. This is where the original Discrete Exponential Map (ExpMap) algorithm was published. The algorithm became much more widely-known than the texturing system, which shipped in my first “real” modeling app, ShapeShop. Basically the idea is you have a library of ‘Stamp’ textures and you drag-and-drop them onto the mesh, and size/position them. The stamps stay “live” and are dynamically composited during rendering (by drawing each patch independently) rather than being immediately baked like would happen in texture painting.
In the video to the right, I extracted the stamps from a few photos of my MSc Advisor’s dog, Shadow. I wanted to rebuild this workflow in Cotan2 so I started out by having Claude create a simple decal-texture tool, with a live stamp. But then I immediately had the problem of where to get Stamps? In 2006 this was a significant practical blocker, but in 2026 we have the Robots.
AI Brush Stamp Generation
Generating a Stamp is much simpler than the complex image generation & editing that SOTA image models are capable of. I did not want Cotan2 users to have to rely on API credits, so I went in search of AI models that could be run locally in a reasonable amount of time on modest hardware. I was inspired by what I had seen from my friends at Unbound.io - specifically their Unbound Slop experiment (here is a video). I found leejet’s stablediffusion.cpp repo and had Claude whip up a UE5 wrapper plugin and basic UI. And it basically worked, on the first try. I could prompt Stamps into existence and use them with the ExpMap-based Stamp tool, and as a bonus, also use it to generate brush stamps for 3D Texture Painting.
The image on the right shows an example, using the Flux.2 Klein 4B model from BlackForestLabs, and then the REMBG model for background removal, which all can run on an 8GB GPU. It even uses a small LLM to generate a Filename automatically from the prompt. The default model-runner uses Vulkan, so it will run with nearly any modern-ish GPU with enough VRAM. There is more info on the technical aspects of running these local models, and the UE5 plugins I built to do it, below.
AI Texture Editing
Once this was working I started to think about other use cases. If you have looked into AI 3D Model generation with services like Tripo3D, or locally using models like Trellis.2, you may have discovered that the texture generation tends to be a weakness. The auto-generated results often have issues that are just unacceptable in production. A skilled texture artist might be able to fix those issues - but I’m not a skilled texture artist. So the question was if we could use an AI Image Model to edit/repair the textures.
The main problem with using an image model to edit textures mapped onto a 3D mesh is the mapping. The “UV Islands” for a mesh can be completely arbitrary, and ultimately have to be laid out in 2D, where there cannot be real coherence with the 3D shape of the surface. Without clear correlation, it’s (currently) difficult for an image model to make coherent edits to a set of disjoint patches of pixels/texels. So Claude and I built tools that do it in a multi-step process, where the 2D texture is composited into a coherent image, the image model edits the composited image, and the edits are mapped back to the UV islands.
An obvious way to do this is via a screen capture, and this is exactly what is done in the AI Project Edit Tool. There is some nuance because we need to operate on the actual texture color, not the final rendered image (eg in standard UE materials this would be the BaseColor only). An obvious issue with Projection-based texture editing is that on curved surfaces you get distortion, and it can become very extreme (to the point of being unusable) near the silhouette of the object. So, I also implemented a second approach that creates a local ExpMap parameterization which can wrap “around” the surface. The texture is sampled into this new coherent 2D parameter space, and the image model edits that image, which is then back-projected. This is the the AI Stamp Edit Tool.
Unfortunately the image-edit models I could run locally on my meagre GPU, like Klein4B, are not really capable enough for image editing. So I also added support for using a few of the current “best” models like Google’s Nanobanana via Fal.ai. This is done via a UE5 plugin that Claude auto-generated based on their public APIs. It basically worked one-shot and so I think it would be very easy to add other model providers/etc.
AI Part Eraser
Something not a lot of people know about Meshmixer is that it supports automatic UV & Texture generation during mesh edits. We did this way back in 2012 or so, because we wanted to make it easy for users to edit textured scan meshes. So many of the editing operations in Meshmixer that generate new mesh areas, like plane-cut or hole-fill, can auto-generate UVs, and then allocate new texture space for the new UV islands, and then automatically generate plausible initial pixels. At the time the best we could do was basic “smooth infill” interpolation (eg using generalized barycentric coords to blend the colors around the boundary).
But now we have AI image models, and so I wanted to take another shot at “magic infill”. The goal was to have a high-level “Erase” tool that takes the users selection, deletes the interior, automatically does a mesh hole-fill, and then auto-textures the new area with a coherent infill texture. This Tool is called Feature Eraser in the UI and it works pretty well!
Under the hood, this Tool has quite a few internal steps - it deletes the existing mesh and fills the hole, then generates UVs for the hole fill, generates an expmap patch around the fill area, uses that to make a mask texture and sample the existing texture, generates a new interior texture, and then does some image-space blending to try to soften the transition. It’s not perfect and in particular the hole-fill is a big limitation (since it doesn’t necessarily interpolate the shape well). My plan there was to use part of the Trellis2 image-to-3D model to generate shape proxies for creating better hole-fills. I even had Claude port stage 1 of the Trellis2 Python implementation to C++ (https://github.com/rms80/trellis2cpp). But I didn’t get very far on an actual implementation.
I also didn’t manage to make the texture-space allocation quite as seamless as it was in Meshmixer. In Cotan2, you have to use the Tex Util tool’s Expand operation to grow the available texture space. The Feature Eraser tool will allocate new texel islands based on a very simple box-packing method in UV space. This could all be much better, I just did the bare minimum to get a functional demo.
AI Mesh Selection
In some other projects, I had been experimenting with Meta’s SAM2 and SAM3 semantic image-segmentation models. These are very powerful and have a lot of interesting capabilities. For example they can output a 2D binary image mask, which can be used to implement various kinds of “magic selection” of mesh parts, simply by using the 2D image mask in the same way I was using brush stamps and screen-space poly-lassos in my Mesh Selection tool.
I found a project similar to stablediffusion.cpp called sam3.cpp, by PABannier on github, which again Claude easily wrapped in a UE5 plugin, so that I could run SAM2 and SAM3 image segmentation from C++ on UE’s FImage, and get back the resulting mask. Then we built the AI Select Tool, which has various modes. SAM3 supports text-based prompts, while SAM2 works from either a bounding-box or positive/negative points. The Tool supports both of these options, including tracking 3D positive/negative points. The current viewport is captured and sent to the model along with the prompt, and then the resulting mask is used to select triangles on the mesh.
Of course this only works from a single viewpoint, but it’s not difficult to accumulate selections across multiple views. SAM2 also supports ‘stream mode’ where starting with an initial segmentation on frame 0, additional frames can be sent and the segmentation will be propagated. I wired this up to the viewport camera, so you can just tumble, and frames are sent at regular intervals. Finally I added support for ‘initializing’ SAM2’s stream mode from an initial mesh selection. This means you can do a quick brush-select to mark up a part, and then tumble to propagate the selection to the entire semantic part.
These magic-selection tools work really well. One of the reasons Meshmixer was so popular for scan repair was that it had a great toolset for part/area selection. Integrating AI segmentation models into these workflows will be a huge time-saver. There is also a lot of scope for further experimentation here. I really wanted to try the DinoV3 model, which computes semantic ‘feature vectors’ for areas of the image which can be used across frames. This means you could click on a part in one frame and then automatically (eg) render a turntable and select that feature in all the other frames, without requiring any kind of continuity (which SAM2 stream mode does depend on - it gets confused if you make large jumps in viewpoint).
Post-Mortem
2024-Ryan would have thought all this sounds amazing and would have a hard time understanding why 2026-Ryan is launching and abandoning this project on the same day. But the world has changed so much since then, to the point where I’m not sure a tool like this - or general-purpose DCC’s at all - make sense anymore.
AI Really Does Change Things
When I set out to build Cotangent2, I knew it was a space nobody else was focused on - after all Meshmixer had been abandoned for nearly a decade, and no real challenger had arisen (sadly it’s not exactly Unicorn territory…). I knew that there was a potential userbase that would pay for a ‘Pro’ Meshmixer replacement. So it seemed ideal for me personally, as an independent coder with a ton of expertise in this area and who has some … “brand recognition”, if you will. I had no doubt that I could sit down and build a “New Meshmixer” that would satisfy a lot of users and (hopefully) turn into a sustainable “lifestyle” business.
Like Meshmixer, the feature set would inevitably be much broader than any individual user would need - that is the nature of technical utility software. And this would eventually become a problem, as it got “too complicated”, etc etc (again the inevitable fate of technical utility apps). So my longer-term plan was to allow third-parties to develop and ship Cotangent2 variants. It would be very easy to support custom C# Tools, and/or Tools implemented in Python, as well as real custom UI. Not just “plugins” but actual custom versions built on a flexible core App. I knew this would eventually be requested because it happened all the time with Meshmixer (Autodesk just wasn’t interested). And this could potentially also be a way to fund Cotangent2 development.
But now we have Claude Code (or Codex, or Pi, or whatever you prefer). Modern state-of-the-art Coding Agents are capable of building a Meshmixer-like app basically from nothing, that has exactly the feature set you need, with a UI tailored to your specific problem. Of course it’s a messy process right now — they can get go down architecture-rabbit-holes and generate garbage code and all sorts of problems will come up. Ultimately you still need to know what to ask for.
But expert users who need, say, a prosthetics design tool - they know what to ask for!! And this is the hard thing that we Tool Builders have to accept — Nobody actually wants a general-purpose utility App that has a bunch of technical capabilities behind a fixed UI that can be assembled into a workflow to approximately solve a problem. This approach to building interactive systems — which is shared by basically all DCC Tools — has always primarily been a practical/business constraint, rather than the “right way to do it”. We simply couldn’t afford to build unique/custom Tools for tiny niche userbases, because the cost of software development & maintenance was too high. Instead we had to build general-purpose systems and hope that the inevitable UX compromises would still solve problems for a sufficiently large collection of groups.
This constraint I’m talking about simply no longer exists. There is no reason for (eg) niche biomedical CAD userbases to try to muddle their way through a general-purpose mesh-editing app. And so I now think that the new “right” thing to do for Cotan2 would be to make it more of a “platform” and less of an “app” - ie a flexible codebase with lots of core technical algorithms and capabilities, which allow the coding agents to assemble higher-level geometry-based Apps efficiently. I can see how this could be amazing, for the kinds of users I wanted to have. But actually building this system did not seem that exciting to me. Personally, I really like building the UI and workflows. And it certainly wasn’t clear how this could ever be the basis for a sustainable business (ie that would keep me independent and let me hire a few people that would be fun to work with, which has always been my end-goal for Gradientspace Corp).
So, basically AI ruined my plans. Not necessarily in a bad way, because I think (in time) the people I was building this App for will be better-served by coding robots that can spin up custom task-specific Tools. But for me personally: ruin.
Unreal Engine’s Hazy Future (For Non-Game Usage)
Another significant factor in deciding not to pursue this as a “real” product is that I built the entire thing on UE5, and I’ve become increasingly convinced that this was a mistake. When I joined Epic in 2018, UE4 was being promoted as a Real-Time Engine, not “just” a Game Engine, and Epic had whole teams focused on non-game Enterprise users. Using UE to build a desktop 3D Mesh Editor app is highly aligned with the kinds of problems that Enterprise users have, and very un-aligned with the kinds of problems that Game Devs have. In particular, I need support for geometry created at Runtime, primarily from the Renderer. This wasn’t really a huge issue in UE4 because the core parts of Renderer/etc that overlapped with at-runtime-Geometry were used quite widely and (moderately) well-supported (although always considered the “slow path”).
But with UE5 Epic shifted substantially back to focusing on GameDev problems, and (especially after 2 rounds of substantial layoffs), Enterprise non-game use-cases have taken a distinct third-row seat, in my opinion. What’s been said publicly about UE6 makes me think this will be even more the case in the future. Verse doesn’t appear to have much to offer a non-game creative tool, and I’m certainly not going to be able to publish Cotan2 in UEFN. The core UE5 Rendering stack - Nanite, Lumen, VSM - does not support runtime-created geometry at all (Lumen support was temporarily added to DynamicMeshComponent, and then removed a few versions later).
I’ve also struggled quite a bit with latency and performance in Cotan2. UE5 (understandably) focuses on modern high-end photorealistic game rendering, which needs to rely on temporal algorithms to handle the scale of content and visual quality, and temporal algorithms are the enemy of latency. I simply could not figure out how to get the App to feel as snappy as Meshmixer, which was written in OpenGL 2.0 and relies on things like partial-redraw via scissor-rect and custom double-buffering to minimize input latency. Nearly every aspect that makes UE5 amazing for cinematic AAA games becomes something I have to fight against to try to make a great geometry editing tool.
So I’m personally not optimistic that the impending UE6 is going to be a great platform for building a desktop app in the future. That’s a bummer for me personally, because I put a lot of effort into trying to make UE4/5 a good platform for it. But also, it’s a real problem for my current codebase which relies so heavily on the UE geometry libraries. Porting Cotan2 to a different rendering engine would not be that much work - but there really isn’t a comparable alternative geometry library I could switch it to.
I joined Epic because I hit a wall building Cotan1 and related 3D tools in Unity, where it became clear it wasn’t the right platform for it. Unreal Engine was vastly better in many aspects, but I still feel like I eventually hit similar walls. Game Engines are (of course) for making Games, and 3D Design Tools are not Games.
Desktop Apps still Matter, and Local Models are the Future
The current conceptual model for how things go in the future is that everything happens in threejs web apps, with all the heavy compute happening server-side, which you access by paying for “credits”, “tokens”, etc. I don’t believe this will work for the kinds of things one might do in a geometry app like Cotangent2 (or any other interactive DCC tool). It’s simply too costly, in both money and time. Just making the clips for the videos above, I generated tens-of-GB of data and ran hundreds of (mostly failed) image generations. This was all throw-away, I’ll probably never look at it again — but I’ll still save it on my hard drive for possibly eternity.
To be completely blunt, web apps that try to mimic DCC tools still suck. If you’ve never spent hundreds of hours really deeply learning a desktop DCC or CAD tool, I can understand if you wouldn’t see why. A high-complexity “pro” tool UI embedded inside a webpage simply cannot be as efficient as an expert-focused native desktop UI. And working with the huge datasets common in many 3D fields via “cloud syncing” is a deeply miserable experience. My ancient Meshmixer OpenGL2.0 app can handle single meshes with tens of millions of triangles, while modern browser tabs are still generally limited to 4GB of total allocated memory.
So I’m still on team Native Desktop App. But we also need Native Desktop Apps to evolve. It’s become so easy to integrate AI into web apps (although often at a very superficial level, in my opinion). We need to be thinking about how that happens with 3D Tools. And not just the “type prompt to generate content” or “type prompt to automate app” modalities that are currently the state-of-the-art. MCP is useful but automating the existing toolset should not be the end-game. I really hope to see more experimentation with using/abusing AI models to build new kinds of User-in-the-Loop creative and technical Tools.
And that’s why I’m more convinced than ever that the way we need to be able to use AI is via local models running on our computers. It’s slightly out of reach right now, for the average user. However I recently acquired an M5-Max Macbook Pro laptop with 128GB of unified memory, which lets me run very large models locally. I can run Gemma4 for LLM coding, chat and vision queries, while simultaneously running large image-generation models. And I can run 3D generation models like Trellis2 that would otherwise require a very expensive NVidia GPU. Of course the laptop was ridiculously expensive, but I think of it kind of like I’m living in the future, and in 3-5 years it will be normal to have hardware like this. And the models will also improve, and get smaller/faster, etc etc.
So for me that’s the main positive outcome of my Cotan2 experiment. The AI-based Texture Editing and Mesh Selection stuff I covered above was low-hanging fruit. There are tiny and very fast variants of all sorts of AI models, that are just sitting there, waiting for creative developers to think up crazy ways to use them. I’m going to be doing some of that myself … stay tuned!!
The Technical Stuff
If you have no plans to actually use the code, you can probably skip the rest of this post. But I’ll get into some details here that might be relevant to anyone trying to re-use parts of the Cotangent2 codebase.
GSMLToolkit Plugin
I mentioned above that I am using the stablediffusion.cpp and sam3.cpp open-source projects to do local image generation and segmentation. I’m also using llama.cpp for some simple LLM usage (to generate filenames). All of this is packaged up in a few UE5 C++ plugins which expose sorta-standardized high-level Subsystem APIs for running the inference. The core plugin for this is called GSMLToolkit and is part of the Cotangent2 source drop.
Getting this to all work was not trivial because (1) each of these packages uses a separate version of ggml, the core library for implementing the tensor operations, and (2) none of them are set up with the same kind of build system. Llama.cpp was the most suitable for UE, in that it could be built as a DLL, with an external ggml build. I ended up having to fork sam3.cpp and stablediffusion.cpp and have Claude modify them to work the same way, so that I could use a shared ggml DLL with each of them.
The builds of these libraries are not part of the UE plugin build. You do them before-hand, using the .bat files in each of the relevant subdirectories (eg GSMLToolkit/_build_ggml/), etc. The bat files sync the relevant git repo, build it, and copy the output binaries into UE plugin directories that will be included in the build, packaged, etc. There is support for CPU, Vulkan, and Cuda builds of GGML. For sam3.cpp and stablediffusion.cpp, the available ggml backend is auto-determined at runtime (preferentially Cuda, then Vulkan, then CPU). For llama.cpp, since I was only using it with a very small model, I only supported model evaluation on the CPU for now.
Cuda support is a particular annoyance because the Cuda binaries are about 1GB and have to be shipped independently - there is no standard “Cuda Redistributable” that you can download from NVidia. So, in the shipped versions of Cotan2 I only included Vulkan. I have a separate installer for the Cuda DLLs, which are then automatically picked up on app start.
I also needed a background removal model, and RemBG 1.4 fit the bill. For this model, I used ONNX. UE has native ONNX evaluation support, which will automatically load the correct DLLs (somewhere it finds Cuda DLLs, tbh I did not investigate how…). This is done in a separate plugin, ONNXProcessingCpp, which has a subsystem that works basically the same way as the GSMLToolkit subsystems.
The subsystems are nothing to write home about. They support automatic weight downloading on-demand, and background evaluation with a callback on completion. Limited support for cancellation. They have some capability to choose a model based on available VRAM. It was all done with Claude and probably could be better. I did nearly all the development on an 8GB 3070 Ti, so things are a little but tuned towards low-capability systems. Adding new image/text models (if they are supported by stablediffusion.cpp or llama.cpp) is a thing Claude is very good at.
The background weight downloading is in a separate plugin called MLWeightsDownloader. This downloads the necessary weights files from public sources and saves them locally. It works pretty well and is a critical aspect of using local models - I can’t possibly ship the many GB of required weights as part of the installer. But be aware that this can take up 10’s of GB if you end up trying all the models. Probably this plugin should be folded into GSMLToolkit - Claude just never got around to it.
Runtime ITF Modeling-App Infrastructure
As I mentioned above, I started with the sample code from my ITF-at-Runtime article, which I had previously updated for UE5 on github. All the core ITF-related parts are more or less the same, although I was working on renaming things to have a “GS” prefix, and that’s only about half-finished. The generic core is implemented in the RuntimeToolsSystem module, and then CotangentAppCore extends that with various functionality specific to Cotangent2 (ie the Tools, UI, etc). The ToolsFrameworkDemo module is the core UE game/app code (ie GameMode, GameInstance, ViewportClient, etc). There is a little bit done in Blueprint assets that subclass these core types (annoying).
The URuntimeMeshSceneObject referenced in that post is replaced with a type UDynamicMeshSO, which wraps a ADynamicMeshActor/UDynamicMeshComponent (which didn’t exist at the time). UDynamicMeshSO has functionality for handling persistent mesh selections, a Material type (UGSMaterial), and undo/redo. All things that make Cotangent2 work.
A major departure from the article version is that I dropped UMG UI entirely and switched to using DearImGUI, via this UE5 ImGUI wrapper plugin. It’s actually quite out-of-date and desperately needs to be updated but I never found the time. I do not actually like Immediate-Mode UI creation, it seems like a disaster in the making to me. So I wrote a library of high-level widgets, you will find these in the GradientspaceUI module. These are basically just widget “interfaces”, eg a “slider” is just a label and a range, and an Accessor object for an external “current value”. The app code creates Widgets and Containers and adds them to a WidgetScene, and then the UI renderer parses the scene and makes the ImGUI calls to actually instantiate the functional slider. All that happens in the file RuntimeToolsSystem\Private\FrameworkUI\ImGuiWidgetView.cpp. There is a passable widget library already.
Tools have ToolUI implementation classes, eg CTSectionTool has CTSectionToolUI, etc. This is where GradientspaceUI widgets are instantiated, hooked up to tool parameters, and added to panels that will (eg) be shown in the Tool Properties area. This system is pretty flexible but still has some major gaps. In particular there is nothing for localization currently — there is a concept of ‘Text’ that wraps most (some) UI strings, but no infrastructure for localization tables/etc.
GSUnrealEngine
As I mentioned above, Cotangent2 is built using a custom version of Unreal Engine 5.5. The actual changes are quite small, it adds a few features to UDynamicMeshComponent and tweaks a few other things in the UE codebase. However because Unreal Engine is not a fully public, I can’t make my fork public either. To give you access, you need to accept the UE5 EULA and link your github account to your UE account, as described here: https://www.unrealengine.com/ue-on-github. Then I can share the changes with you (you’ll have to email or ping me on twitter/etc).