How to Fine-Tune Open AI Models for Decks

Frontier models have a house style problem. Ask them to build ten slide decks, and you get ten variations of the exact same tasteful defaults, complete with the familiar aesthetic of AI-generated design. When you need strict adherence to your corporate templates, proprietary typography, and confidential workflows, hosted frontier models fail you. They drift back to their defaults, and your raw data leaves your local network.

Palette solves this by pairing a small open-weight 20B model with a specialized LoRA adapter, trained on synthetic reasoning data that encodes the IBM Design Language. Instead of trusting a single model pass to handle both editorial structure and precise geometry, Palette splits the process into two distinct agent roles. A designer plans the layout and writes a structured JSON brief, while a coder translates each slide spec into clean pptxgenjs JavaScript. This architectural breakdown lets you run the coding phase in parallel across slides, accelerating generation speeds by up to 7x while eliminating overlapping text boxes and broken positioning.

How the Dual-Role LoRA Architecture Works

Small open-weight models crash when forced to handle high-level editorial decisions and precise x, y coordinate arithmetic simultaneously in a single generation pass. When a model guesses how a 75-character title will wrap at 56pt without seeing the rendered output, text inevitably overflows boundaries or collides with graphics.

Palette solves this by separating concerns through a single shared LoRA adapter split into two explicit behavioral modes:

• The Designer Role: Reads the raw source material or outline, decides the presentation genre and length, and outputs a structured JSON brief containing color palettes, typography rules, slide layouts, exact text content, and explicit design rationales.
• The Coder Role: Iterates through the designer's structured brief one slide at a time, generating explicit reasoning traces followed by executable pptxgenjs JavaScript code that places elements precisely on the canvas.

Because the designer brief is fully serialized to JSON before any code generation begins, you can review or modify individual slides manually or programmatically. Plus, the coder phase requires zero cross-slide state, allowing you to fire off asynchronous generation jobs for every slide simultaneously. Below is a structural representation of the JSON brief format passed between the two roles.

{
  "meta": {
    "theme": "ibm-carbon-dark",
    "slide_width": 13.33,
    "slide_height": 7.5
  },
  "slides": [
    {
      "slide_number": 1,
      "layout": "title-split",
      "rationale": "High-impact opening requires maximum contrast and sparse typography.",
      "elements": [
        {
          "type": "heading",
          "text": "Q4 Enterprise Infrastructure Migration",
          "font_size": 44,
          "x": 1.0,
          "y": 2.5
        }
      ]
    }
  ]
}

What Hardware and Software Prerequisites You Need

Training and running a 20B parameter model locally requires serious hardware. Do not attempt this on a standard laptop unless you enjoy thermal throttling and out-of-memory kernel panics.

Ensure your development environment meets these exact specifications before pulling dependencies:

• OS: Ubuntu 22.04 LTS or Debian 12 (Windows via WSL2 is supported, but native Linux minimizes CUDA overhead).
• GPU: Minimum single NVIDIA A100 (80GB VRAM) or H100 for fine-tuning the 20B base model. For inference and parallel rendering, an RTX 4090 (24GB) is sufficient using 4-bit quantization.
• Software: Python 3.11+, Node.js 20.x, CUDA 12.3+, and PyTorch 2.4+ with FlashAttention-2 enabled.
• Storage: At least 150GB of fast NVMe storage for base model weights, synthetic training datasets, and checkpoints.

Install your core Python dependencies in a fresh virtual environment using pip:

python3 -m venv venv
source venv/bin/activate
pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121
pip install transformers==4.44.0 peft==0.12.0 trl==0.9.6 accelerate==0.33.0 pptxgenjs

How to Synthesize Training Data and Format Your Dataset

You cannot download a public corpus of slide decks that includes the underlying editorial reasoning traces, rejected counterfactuals, and precise rendering code. You have to synthesize your training data using a larger open teacher model. We generate roughly 1,000 decks spanning 10,000 individual slides to ensure solid coverage of corporate genres.

Create a python script named generate_synthetic_corpus.py to orchestrate your data pipeline. This script ingests raw topic recipes and outputs structured JSON training pairs.

import json
import os
from transformers import AutoModelForCausalLM, AutoTokenizer

def build_deck_recipe(topic, genre):
    return {
        "topic": topic,
        "genre": genre,
        "slide_count": 10,
        "design_system": "IBM Design Language v10"
    }

def synthesize_training_sample(recipe):
    prompt = f"Generate a designer brief and coder steps for a {recipe['genre']} presentation about {recipe['topic']}."
    # In production, pass this to your large open teacher model (e.g., Llama 4 Maverick)
    mock_response = {
        "designer_output": "{\"layout\": \"grid-2x2\", \"rationale\": \"Used grid to compare metrics.\"}",
        "coder_output": "let pptx = new pptxgen(); let slide = pptx.addSlide(); slide.addText('Metrics', {x:1, y:1}); pptx.writeFile('output.pptx');"
    }
    return mock_response

if __name__ == "__main__":
    recipe = build_deck_recipe("Cloud Cost Optimization", "technical-review")
    sample = synthesize_synthetic_sample(recipe)
    os.makedirs("./dataset", exist_ok=True)
    with open("./dataset/train_sample.json", "w") as f:
        json.dump(sample, f, indent=2)
    print("Successfully generated synthetic training datum.")

One critical lesson learned during training: coder output examples are roughly 5x longer in token length than designer examples. If you feed raw generated pairs directly into your training loop, the model will severely undertrain on designer rationale. You must oversample your designer examples by a factor of 5 to maintain balanced loss convergence across both roles.

How to Configure and Execute the LoRA Training Script

With your synthetic dataset prepared and balanced, configure your LoRA fine-tuning parameters. Full fine-tuning of a 20B model is unnecessary and cost-prohibitive; LoRA lets you inject adapters into attention projections while freezing the base model weights.

Create your training runner script as train_palette_adapter.py:

import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer

model_id = "meta-llama/Llama-4-20b-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = prepare_model_for_kbit_training(model)

peft_config = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, peft_config)
dataset = load_dataset("json", data_files="./dataset/train_sample.json")

training_args = TrainingArguments(
    output_dir="./palette-lora-output",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    logging_steps=10,
    save_strategy="epoch",
    fp16=False,
    bf16=True,
    num_train_epochs=3
)

trainer = Trainer(
    model=model,
    train_dataset=dataset["train"],
    args=training_args,
    data_collator=None
)

trainer.train()
model.save_pretrained("./palette-adapter-final")
print("LoRA adapter training completed successfully.")

Execute the training job from your terminal with accelerate to distribute across available CUDA devices:

accelerate launch train_palette_adapter.py --config_file accelerate_config.yaml

How to Run End-to-End Slide Generation in Parallel

Once your LoRA adapter is saved, you can spin up your inference pipeline. The script loads your base model, attaches the trained adapter, accepts a raw prompt, generates the designer JSON brief, and immediately spawns parallel coder processes to output the final presentation files.

Create your execution script as run_palette.py:

import asyncio
import json
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "meta-llama/Llama-4-20b-Instruct"
adapter_path = "./palette-adapter-final"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(base_model_id, device_map="auto", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(model, adapter_path)

def generate_designer_brief(prompt):
    inputs = tokenizer(f"[DESIGNER MODE] {prompt}", return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=1024)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

async def render_slide_async(slide_spec):
    print(f"Rendering slide {slide_spec['slide_number']} in parallel..."
    # Simulate execution of pptxgenjs compiler
    await asyncio.sleep(0.5)
    return f"slide_{slide_spec['slide_number']}.pptx compiled."

async def main():
    brief_raw = generate_designer_brief("Create a deck on Q4 infrastructure performance.")
    print("Designer Brief Generated.")
    
    # Mock parsing of slides from brief
    slides = [{"slide_number": 1}, {"slide_number": 2}, {"slide_number": 3}]
    results = await asyncio.gather(*(render_slide_async(s) for s in slides))
    for res in results:
        print(res)

if __name__ == "__main__":
    asyncio.run(main())

Run your generation script to verify the parallel pipeline:

python3 run_palette.py

What Common Errors Occur and How to Fix Them

Fine-tuning and running custom reasoning adapters on consumer or enterprise hardware will expose edge cases. Keep these fixes handy when your pipeline throws exceptions.

• CUDA Out of Memory during LoRA training: Caused by excessive batch sizes or long context lengths colliding with activation checkpoints. Fix by lowering your batch size or enabling bitsandbytes 4-bit quantization in your PEFT configuration:

  from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
  # Ensure load_in_4bit=True is passed to AutoModelForCausalLM

• Text collision and layout overflow in generated slides: Happens when the coder role hallucinates font sizes larger than available canvas dimensions. Fix by tightening your synthetic dataset validation checks to reject any designer brief where title character lengths exceed container width limits at specified point sizes.
• Asynchronous race conditions in parallel slide compilation: Occurs when multiple Node.js subprocesses attempt to write to the exact same temporary file path simultaneously. Fix by appending a unique UUID hash to every intermediate script file generated by the coder role:

  import uuid
  temp_filename = f"slide_{uuid.uuid4().hex}.js"

How to Manage Production Security and Performance Tuning

Deploying Palette into an enterprise environment requires locking down infrastructure and optimizing throughput. Because your models process confidential corporate data internally, ensure that all weights, training sets, and generation caches reside on encrypted volumes formatted with LUKS.

For high-throughput slide generation, decouple your inference server using vLLM or TGI (Text Generation Inference) rather than native Hugging Face pipelines. Wrap your LoRA adapter directly into the vLLM serving engine to enable continuous batching and PagedAttention, which reduces memory fragmentation when multiple users request concurrent deck generation across different design templates.

Monitor your token generation latency closely. If your designer phase begins outputting verbose, rambling rationales that bloat context windows, enforce strict JSON schema constraints using grammar-guided generation tools like Outlines or SGLang. This guarantees that the designer role outputs valid JSON structures on the first pass, cutting down wasted compute cycles and keeping generation times under two seconds per slide.

Bottom Line

Stop accepting generic layouts and compromised brand guidelines from hosted frontier models. Fine-tuning a 20B open-weight model with a split designer and coder LoRA architecture gives you complete control over your presentation pipelines while keeping sensitive data entirely on-premise. Implement the synthetic generation pipeline, enforce strict JSON briefs, and scale your rendering throughput with parallel execution today.