I'm kinda tired of the usual AI hate post that appears often. So i'd like to bring something different to the table. Cannelloni... eh no. Some AI or ML tools that i think are good for a wide range of tasks regarding gamedev.
If you are a Traditional artist that may want to navigate the waters of this tumultuous present, you are welcome to make questions. If you are another AI-user and want to share cool tips or niche tools you know, please do.
:-)
Yeah, this is not advertising, promotions or etc. Just hoping it could help others to find a place to start or learn. That’s not just chatGPT or Gemini.
I like to use local models (Local means you don’t need internet because the thing runs inside your PC, no internet, no accounts).
## 🖥️ My PC specs are:
Ryzen 5, 16 GB RAM and a 16Gb RTX GPU (I had a 8GB AMD card before). Where I managed to test some things, some work well, some… meh and others, a plain no for me.
## 🎛️ Hubs & Launchers
ComfiUI is a tool for creating a wide range of content (images, videos, sound, text, etc) using a node system. Something like working with Blender’s shaders but for AI pipelines.
pinokio.co (I hope I don’t have the link wrong) is a tool to install and manage local AI systems
From there I have installed:
FaceFusion – A program that let’s you modify character’s faces in images or sprites. Either by doing a ‘DeepFakes’ or by using the option ‘face-edit’ that enables a wide of sliders to control lips curvature, head rotation, eyes openness and so on. (Doesn’t need GPU, but can help)
RMBG-2-Studio – This is basically a background removal tool, is easy to use, just upload an image and it just do it’s things, also it has an option to upload a wide batch if needed. (Doesn’t need GPU at all)
Other tools (outside pinokio).
## 🎨 Image Generation & Texturing
Style2Paints – https://style2paints.github.io/
If you already draw, are an illustrator or at least good at doing lineart, this tool can help to save time in coloring. What it does? Coloring. It analyze the images and apply color, you can control/indicate what areas should have what color and choose ‘coloring styles’ (or something like that) for the whole image.
(Doesn’t need GPU, but can help)
Krita AI Diffusion –https://kritaaidiffusion.com/
Good if you already draw, is a unofficial plugin for the free and open-source drawing program Krita. (https://krita.org/es/) The plugin can manage connections to cloud models or work as an assistant to install local Stable Diffusion (SD), models. Also in the last versions I noticed options to install Flux and I think z-image.
Is not just text2image. I like to use it to draw textures, basically drawing ‘by hand’ in Krita and using the plugin as a refiner. (There are also other tools like upscaling, Loras managers, ControlNets, live drawing and so on).
(Can run in CPU but I heavily recommend a GPU 4 to +8 GB VRAM for SD 1.5 or 16 or more for SDXL).
FastSD CPU – https://github.com/rupeshs/fastsdcpu
Is just a text2image diffusion model (that means images generator) made to run in CPUs, it can take a while to generate images but I think it takes less than normal SD in CPU.
(Doesn’t need GPU at all)
StableGen –https://github.com/sakalond/StableGen
A Blender plugin to do ‘automatic texturing’ using SDXL models (need a good PC, I manage to make it work in a 16gb Nvidia GPU with 16gb RAM). The more detailed/topology heavy is the mesh, the better job it does. (For me was a pain in the arse to set up, and I have to run more tests on it).
(Requires +12GB VRAM GPU)
## 🎬 Video Generation
FramePack –https://github.com/lllyasviel/FramePack
A model/system/program, whatever to generate video clips by using a picture as starting point. It can run in really small GPUs, like 6GB CUDA GPUs but requires a lot of RAM because it do a lot of offloading. My 16 BG RAM PC couldn’t handle it, I think I may need 32 GB of ram minimum.
Also can be installed via pinokio.
In ComfiUI you can also install other video tools like WAN, Hunyuan or LTX. Notice those requires really powerful PC’s.
## 🧊 3D Generation
In pinokio and ComfiUI there are 3D generation models like Trellis, Hunyuan and WAN. Soo far I have tried Online version of Trellis (a free demo form hugginface) and hunyuan locally. And in both cases I think they are not ready to be used in game dev. While they more or less do a good job in generating meshes from images, they tend to create an awfully heavy mesh that requires a lot or retopology work in order to optimize. Perhaps they can be good in 3D animation for movies/video production.
## 🎵 Music Production
Never used a lot in here, I know HeartMula (https://github.com/HeartMuLa/heartlib) and DiffRymth (https://github.com/ASLP-lab/DiffRhythm) exists. They both can be installed via pinokio. As far as I know HeartMula is like a local suno or local udio. (Generates songs from scratch/text prompts) while with DiffRymth you need to add a sound sample and written Lyrics for your music, I think this one is good if you area already a musician.
##📝 Text & Programming (Local LLMs)
From text we came and to text we return. Haha. But if you wanna use a Large Language Model (LLM) without sharing your dark secrets to a big corpo. (Like you are probably doing right now in Twitter or Facebook XD). There some tools to easily run LLM models in average consumer hardware.
GPT4ALL –https://www.nomic.ai/gpt4all
With this you can download and run a wide range of LLM’s locally. You can indicate it a folder with txt files to use as a RAG. (Basically, you can indicate where it can find extra info you think the model should know.) Also you can use it as backend for AI powered projects since it can run in server mode.
(Can run in CPU but I heavily recommend a GPU +8gb VRAM)
LMStudio - https://lmstudio.ai/
Same as before. You can download and run models, or use it as backend. I use it to play in SillyTavern.
(Can run in CPU but I heavily recommend a GPU +8gb VRAM)
KoboldCPP –https://github.com/LostRuins/koboldcpp
This one is more for running, it doesn’t have an assistant to find and download models. You may need to visit huggingface or Civitai to download and add models manually.
(Can run in CPU but I heavily recommend a GPU +8gb VRAM)
## 📚 Some Resources for Tutorials & Docs I know.
https://stable-diffusion-art.com/
https://www.youtube.com/@nodonmai
https://www.reddit.com/r/StableDiffusion/
### ⚠️ A Final Reminder:
Again. I made this post with the intention of sharing tools and info. If you are a traditional artist that may wanna explore this new weird world, you are welcome to make questions. If you are another AI user that knows a cool niche tool/trick or etc. Please share it.
This thread is mainly intended as a resource list and discussion space for workflows, tips, and tool recommendations.
:-)
Also, also check the licenses and terms of any model, dataset, or generated asset before using it in any project. (And i mean ANY, it doesn’t matter if it’s a free game.)