OpenAI Agent + Hindsight Memory Integration

Complete Application

This is a complete, runnable application demonstrating Hindsight integration. View source on GitHub →

A fitness coach example demonstrating how to use OpenAI Agents with Hindsight as a memory backend.

What This Demonstrates

This example showcases:

• OpenAI Assistants handling conversation logic
• Hindsight providing sophisticated memory storage & retrieval
• Function calling to bridge them together
• Streaming responses for real-time interaction (enabled by default)
• Bidirectional memory - both user data AND coach observations stored
• System-level post-processing - automatic opinion storage for reliability
• Temporal-semantic memory queries via function tools
• Enhanced preference learning - coach learns and respects user likes/dislikes
• Real-world integration pattern for adding memory to AI agents

Architecture

User: "I ran 5K today, don't like tempo runs"
    |
OpenAI Assistant
    |
Function Call: store_memory(workout + preference)
    |
Hindsight API (stores as world/agent)
    |
OpenAI Assistant: "What should I focus on?"
    |
Function Call: retrieve_memories("workouts and preferences")
    |
Hindsight API (returns workouts + preferences)
    |
OpenAI Assistant (analyzes, gives advice)
    |
Function Call: store_memory(advice as opinion)
    |
Hindsight API (stores coach's observation)
    |
Personalized Answer

Key Difference from Standard Demo

Component	Standard Demo	OpenAI Integration
Conversation	Hindsight /think endpoint	OpenAI Assistant API
Memory	Hindsight (built-in)	Hindsight (via function calling)
LLM	Configured in Hindsight	OpenAI GPT-4
Opinion Formation	Automatic in /think	Explicit via store_memory(type="opinion")
Best For	Hindsight-native apps	Integrating memory into existing OpenAI agents

Quick Start

Prerequisites

1. OpenAI API Key

   export OPENAI_API_KEY=your_openai_api_key

2. Hindsight API running

   # Follow Hindsight setup instructions to start the API
   # Default: http://localhost:8888

3. Install dependencies

   pip install openai requests

Run the Conversational Demo

cd openai-fitness-coach
export OPENAI_API_KEY=your_key_here
python demo_conversational.py

The demo showcases:

1. Natural language workout logging - Tell the coach what you did conversationally
2. Preference learning - Express likes/dislikes and watch the coach adapt
3. Goal tracking - Set goals, track progress, achieve milestones
4. Bidirectional memory - Both your activities AND coach's advice are stored
5. Streaming responses - See responses appear in real-time
6. 7 interactive phases - From goal setting to achievement recognition

The demo uses a separate agent (fitness-coach-demo) to avoid mixing with real data.

Usage

Chat with Your Coach

Interactive mode:

python openai_coach.py

Single question:

python openai_coach.py "What did I do for training this week?"

How It Works

1. Memory Tools (memory_tools.py)

Defines function tools that the OpenAI Agent can call:

retrieve_memories(query, fact_types, top_k)
search_workouts(after_date, before_date, workout_type)
get_nutrition_summary(after_date, before_date)
get_user_goals()
get_coach_opinions(about)

Each function makes API calls to Hindsight to fetch relevant memories.

2. OpenAI Agent (openai_coach.py)

Creates an OpenAI Assistant with:

• Fitness coaching instructions
• Access to memory function tools
• Conversation management

When you ask a question:

1. User message is sent to OpenAI Assistant
2. Assistant decides which memory functions to call
3. Functions fetch data from Hindsight
4. Assistant generates response using retrieved context

3. Function Calling Flow

# User asks: "What did I run this week?"

# OpenAI Assistant decides to call:
search_workouts(
    after_date="2024-11-18",
    workout_type="running"
)

# Function retrieves from Hindsight:
{
  "results": [
    {"text": "User completed 45-minute cardio workout: running..."},
    {"text": "User completed 60-minute cardio workout: running..."}
  ]
}

# OpenAI Assistant generates response:
"This week you've done two runs: a 45-minute run on Monday
and a longer 60-minute run on Wednesday. Great consistency!"

Example Questions

Try asking:

python openai_coach.py "What does my training look like this week?"
python openai_coach.py "Based on my workouts, should I rest today?"
python openai_coach.py "How is my nutrition supporting my goals?"
python openai_coach.py "What's my progress toward my goal?"
python openai_coach.py "Compare my training this month to last month"

The agent will automatically:

1. Identify what memories it needs
2. Call the appropriate function tools
3. Retrieve data from Hindsight
4. Generate a personalized response

Memory Types Retrieved

The OpenAI Agent can retrieve different memory types from Hindsight:

• World Facts (fact_type: "world"): Workouts, meals, activities
• Agent Facts (fact_type: "agent"): Goals, intentions
• Opinions (fact_type: "opinion"): Coach's observations about patterns

Customization

Add New Function Tools

Edit memory_tools.py to add new capabilities:

def get_weekly_summary(week_offset: int = 0):
    """Get a summary of a specific week."""
    # Implementation
    pass

# Add to MEMORY_TOOLS list
MEMORY_TOOLS.append({
    "type": "function",
    "function": {
        "name": "get_weekly_summary",
        "description": "Get training summary for a specific week",
        # ... parameters
    }
})

# Add to FUNCTION_MAP
FUNCTION_MAP["get_weekly_summary"] = get_weekly_summary

Modify Assistant Instructions

Edit openai_coach.py to change the coach's personality or behavior:

assistant = client.beta.assistants.create(
    name="Your Custom Coach",
    instructions="Your custom instructions here...",
    model="gpt-4o-mini",
    tools=MEMORY_TOOLS
)

Use Cases

This pattern works for any application that needs memory:

1. Customer Support Agents - Remember past conversations and issues
2. Personal Assistants - Remember preferences, schedules, past decisions
3. Educational Tutors - Track learning progress over time
4. Health Coaches - Monitor habits, progress, goals (like this example)
5. Sales Assistants - Remember customer interactions and preferences

Integration Pattern

To add Hindsight memory to your own OpenAI Agent:

1. Define function tools that call Hindsight API
2. Register them with your OpenAI Assistant
3. Implement function handlers to execute Hindsight queries
4. Let OpenAI Assistant decide when to retrieve memories

The key benefit: Separation of concerns

• OpenAI = Conversation logic
• Hindsight = Memory storage, retrieval, temporal queries, entity linking

When to Use This vs. Standard Hindsight

Use OpenAI + Hindsight (this example) when:

• You want OpenAI's conversation capabilities
• You're already using OpenAI Agents
• You want explicit control over when to retrieve memories
• You want to combine Hindsight with other OpenAI features

Use Hindsight directly when:

• You want a complete memory-first solution
• You want automatic memory retrieval and opinion formation
• You want to use different LLM providers (not just OpenAI)
• You want the /think endpoint's integrated approach

Learning Points

After running this demo, you'll understand:

1. How to add sophisticated memory to any OpenAI Agent
2. How function calling bridges LLMs and memory systems
3. How temporal-semantic queries work via function tools
4. Real-world pattern for LLM + memory integration

Core Files

• demo_conversational.py - Conversational demo showcasing preference learning and goal tracking
• openai_coach.py - OpenAI Assistant wrapper with streaming and memory integration
• memory_tools.py - Function calling tools that bridge to Hindsight API
• .openai_assistant_id - Saved assistant ID (auto-generated, gitignored)

Common Issues

"OPENAI_API_KEY not set"

export OPENAI_API_KEY=your_api_key_here

"Agent not found"

• Make sure the Hindsight fitness-coach agent exists

"Connection refused"

• Make sure Hindsight API is running on localhost:8888

Next Steps

1. Run the demo to see it in action
2. Try chatting with the coach: python openai_coach.py
3. Log your own workouts and meals
4. Experiment with different questions
5. Add custom function tools for your use case

---

Built with:

• OpenAI Assistants API
• Hindsight (temporal-semantic memory)
• Function calling for integration