Agentic Video Workflows

Agent-Directed vs. Fixed Pipelines

Fixed Pipeline	Agentic Workflow
Steps predetermined	Agent decides next step
Same process for all videos	Adapts to each video
Processes everything	Stops when answer is found
Predictable cost	Variable cost (usually lower)
Simple to debug	Requires observability

Defining Agent Tools

Give the agent tools to interact with video:

TOOLS = [
    {
        "name": "enhance_video",
        "description": "Improve video quality (upscale, sharpen, restore faces). "
                      "Use when video is blurry, dark, or low resolution.",
        "parameters": {
            "video_url": "URL of video to enhance",
            "resolution": "Target resolution: 1080p or 4k"
        }
    },
    {
        "name": "extract_frame",
        "description": "Get a specific frame from the video.",
        "parameters": {
            "video_url": "URL of video",
            "timestamp": "Time in seconds"
        }
    },
    {
        "name": "detect_objects",
        "description": "Find and locate objects in a frame.",
        "parameters": {
            "frame": "Frame to analyze"
        }
    },
    {
        "name": "read_text",
        "description": "Extract text from a frame using OCR.",
        "parameters": {
            "frame": "Frame to analyze"
        }
    },
    {
        "name": "describe_frame",
        "description": "Get a detailed description of what's in a frame.",
        "parameters": {
            "frame": "Frame to analyze",
            "focus": "What to focus on (optional)"
        }
    }
]

Agent Loop

class VideoAgent:
    def __init__(self, tools: List[Tool]):
        self.tools = {t.name: t for t in tools}
        self.llm = get_llm()

    def analyze(self, video_url: str, question: str) -> str:
        """Analyze video to answer a question."""
        messages = [{
            "role": "system",
            "content": "You analyze video to answer questions. Use tools to examine the video. "
                      "Stop when you have enough information to answer confidently."
        }, {
            "role": "user",
            "content": f"Video: {video_url}\n\nQuestion: {question}"
        }]

        while True:
            response = self.llm.complete(messages, tools=self.tools)

            if response.tool_calls:
                # Execute tools
                for call in response.tool_calls:
                    result = self.tools[call.name].execute(**call.args)
                    messages.append({
                        "role": "tool",
                        "name": call.name,
                        "content": str(result)
                    })
            else:
                # Agent is done
                return response.content

Example: Insurance Claim Analysis

question = "What damage is visible to the vehicle and when did the impact occur?"

# Agent's internal reasoning:
# 1. Video quality looks low, should enhance first
→ enhance_video(video_url, "1080p")

# 2. Need to find the moment of impact
→ detect_scenes(enhanced_url)
# Found scene change at 00:12:34

# 3. Extract frames around impact
→ extract_frame(enhanced_url, 12.0)  # Before
→ extract_frame(enhanced_url, 12.5)  # During
→ extract_frame(enhanced_url, 13.0)  # After

# 4. Describe the damage
→ describe_frame(after_frame, focus="vehicle damage")

# 5. Agent conclusion:
"The impact occurred at 12:34. Visible damage includes:
front bumper cracked, headlight broken, hood dented.
The collision appears to be a front-end impact at low speed."

Controlling Agent Behavior

Agents can do too much or too little. Control with:

Budgets

MAX_TOOL_CALLS = 10
MAX_COST = 5.00  # dollars

if tool_calls > MAX_TOOL_CALLS:
    force_conclusion()

Guardrails

# Only allow certain tools
ALLOWED_TOOLS = ["extract_frame", "describe_frame"]

# Require approval for expensive operations
if tool.cost > 1.00:
    require_approval()

Objectives

SYSTEM_PROMPT = """
You are analyzing video evidence. Your objectives:
1. Answer the user's question
2. Minimize tool usage (each call costs money)
3. Enhance video only if quality is clearly insufficient
4. Stop as soon as you have a confident answer
"""