Skip to main content

Operations

Background tasks that Hindsight executes asynchronously.

Prerequisites

Make sure you've completed the Quick Start and understand how retain works.

How Operations Work​

Hindsight processes several types of tasks in the background to maintain memory quality and consistency. These operations run automatically—you don't need to trigger them manually.

By default, all background operations are executed in-process within the API service.

Kafka Integration

Support for external streaming platforms like Kafka for scale-out processing is planned but not available out of the box in the current release.

Operation Types​

OperationTriggerDescription
batch_retainretain_batch with async=TrueProcesses large content batches in the background
consolidateAfter retainConsolidates new facts into observations

Async Retain Example​

When retaining large batches of memories, use async=true to process in the background. The response includes an operation_id that you can use to poll for completion.

1. Submit async retain request​

curl -X POST "http://localhost:8000/v1/default/banks/my-bank/memories" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"content": "Alice joined Google in 2023"},
      {"content": "Bob prefers Python over JavaScript"}
    ],
    "async": true
  }'

Response:

{
  "success": true,
  "bank_id": "my-bank",
  "items_count": 2,
  "async": true,
  "operation_id": "550e8400-e29b-41d4-a716-446655440000"
}

2. Poll for operation status​

curl "http://localhost:8000/v1/default/banks/my-bank/operations"

Response:

{
  "bank_id": "my-bank",
  "operations": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "task_type": "retain",
      "items_count": 2,
      "document_id": null,
      "created_at": "2024-01-15T10:30:00Z",
      "status": "completed",
      "error_message": null
    }
  ]
}

Operation Status Values​

StatusDescription
pendingOperation is queued and waiting to be processed
completedOperation finished successfully
failedOperation failed (check error_message for details)

Managing Operations​

Cancel a pending operation​

curl -X DELETE "http://localhost:8000/v1/default/banks/my-bank/operations/550e8400-e29b-41d4-a716-446655440000"

Retry a failed operation​

If an operation fails, you can manually re-queue it for execution:

curl -X POST "http://localhost:8000/v1/default/banks/my-bank/operations/550e8400-e29b-41d4-a716-446655440000/retry"

Response:

{
  "success": true,
  "message": "Operation 550e8400-e29b-41d4-a716-446655440000 queued for retry",
  "operation_id": "550e8400-e29b-41d4-a716-446655440000"
}

The operation status resets to pending and the worker picks it up again. Returns 409 if the operation is not in failed state.

Next Steps​