Table of Contents
What Stable Diffusion Actually Does
Stable Diffusion is open-source AI image generation you can run locally for free, customize with community models and fine-tuning, and integrate via API. Unlike Midjourney or DALL-E, it's not a single hosted product — it's a model you run yourself (or through Stability AI's pay-per-image cloud API if you don't want local setup).
Composition control no closed platform offers
ControlNet lets you guide image structure with a reference pose, depth map, or edge detection input — precise control that text prompts alone can't achieve, and that Midjourney or DALL-E don't support.
Getting Started: Local vs Cloud
Local generation is free and unlimited but requires an NVIDIA GPU with 8GB+ VRAM and real setup time (installing an interface like Automatic1111 or ComfyUI, downloading models). The cloud alternative — Stability AI's API — removes the hardware requirement entirely at $0.006-0.009 per image.
Start with the cloud API if you're unsure whether Stable Diffusion fits your workflow; move to local generation once you know you'll be generating enough volume to justify the setup time.
Choosing a Model
| Model | Best For | VRAM Needed |
|---|---|---|
| SD 3.5 Large | Highest quality, complex prompts | 16GB+ |
| SDXL 1.0 | Most community models/LoRAs | 8GB |
| SD 1.5 | Widest LoRA compatibility, fastest | 4GB |
Start with SDXL 1.0 unless you have a specific reason to choose otherwise — it has the largest community model and LoRA library at a manageable 8GB VRAM requirement.
Basic Prompting
How to Generate Your First Image
- In your interface (Automatic1111, ComfyUI, or the Stability API), enter a text prompt describing your desired image
- Add a negative prompt listing what to avoid (common: "blurry, low quality, distorted")
- Set resolution and sampling steps (20-30 steps is a reasonable starting point)
- Generate and iterate — refine your prompt based on results rather than expecting the first attempt to be final
LoRA Fine-Tuning
LoRA (Low-Rank Adaptation) models are small add-ons that steer generation toward a specific style, subject, or character — the Civitai community hosts over 100,000 free LoRAs covering everything from art styles to specific characters.
How to Use a LoRA
- Download a LoRA file matching your base model version (SD 1.5 LoRAs generally don't work with SDXL, and vice versa)
- Place it in your interface's LoRA folder
- Reference it in your prompt using your interface's LoRA syntax, with a weight value controlling its influence
- You can stack multiple LoRAs in one generation, mixing influences
Match LoRA weight to your base model version exactly
A LoRA trained for the wrong base model version produces broken or unrecognizable output — always confirm compatibility before troubleshooting a "bad" generation.
ControlNet: Composition Control
How to Use ControlNet
- Install the ControlNet extension for your interface
- Upload a reference image — a pose skeleton, depth map, or edge-detection line art
- Select the matching ControlNet model type for your reference input
- Generate — the output follows your reference structure while the text prompt controls style and content
This is transformative for product photography, character design, or anything where composition consistency across multiple generations matters more than pure prompt-driven randomness.
img2img: Consistent Characters
How to Use img2img
- Select an approved reference image as your starting point
- Set the denoising strength — lower values preserve more of the original, higher values allow more change
- Generate variations that preserve the core subject while changing pose, lighting, or setting
Combined with ControlNet, img2img gives more compositional control across a multi-image series than any single-prompt generation system.
Full Workflow: A Consistent Character Series
Here's how to produce multiple images of the same character in different scenes:
Generate or select an approved reference image of your character
This becomes your img2img source for every subsequent variation.
Train or find a matching LoRA if the character needs to recur across many images
For heavy reuse, a dedicated LoRA is more reliable than img2img alone.
Use ControlNet with pose references for each new scene
Guide the character's pose precisely rather than relying on text description alone.
Adjust denoising strength per image to balance consistency vs variation
Iterate until the character stays recognizable while the scene changes as intended.
Stable Diffusion Costs 2026
| Option | Cost |
|---|---|
| Local generation | Free (with compatible GPU) |
| Stability AI cloud API | $0.006-0.009/image |
Try Stability AI's cloud API if you don't have a compatible GPU or want to test before setting up local generation.
Common Beginner Mistakes
- Mismatching LoRA and base model versions. A LoRA trained for SD 1.5 won't work correctly with SDXL — always confirm compatibility.
- Skipping ControlNet for composition-critical work. Hoping a text prompt alone produces the right pose or layout wastes many generation attempts versus using a reference input directly.
- Setting denoising strength too high in img2img. This loses the character consistency that was the point of using img2img in the first place.