LLM And Cognitive Layer Plan

Planned provider registry, credential ownership, memory, and reasoning layers.

LLM And Cognitive Layer Plan

This page is planning material. It describes how Gigi should grow from the current foundation into a Discord-native agent runtime without weakening the existing permission, audit, memory, and plugin safety model.

Design Principles

• Keep Discord as an adapter. The cognitive layer should not depend on discordgo.
• Keep internal/llm as the low-level provider port for text and embedding calls.
• Add agent orchestration as a separate package, such as internal/agent, above provider calls, memory, plugins, permissions, and audit.
• Keep deterministic plugin prefix matching before LLM planning.
• Treat @Gigi guild mentions as conversational agent requests, not as a broad deterministic natural-language command parser.
• Keep exact slash commands as the deterministic source for explicit behavior and syntax.
• Treat LLM output and plugin descriptions as untrusted hints, never executable authority.
• Build final tool plans from registered tool schemas, stored manifests, capability checks, confirmation policy, and audit rules.
• Keep deterministic tools for exact work such as memory counts, message search, plugin dispatch, permission checks, and usage summaries.
• Use the LLM for interpretation, planning, clarification, and answer composition, not as the source of truth for counts, permissions, or action execution.
• Never store raw prompts, completions, provider responses, planned commands, or secrets in audit metadata.

Agent Runtime Core

The new core model is:

Discord intake
  -> deterministic trigger handlers
  -> agent planner
  -> context broker
  -> tool executor
  -> answer composer
  -> confirmation executor for risky actions
  -> audit

This avoids writing one custom parser for every natural-language phrasing. Gigi still keeps small deterministic tools, but arbitrary user language maps onto those tools through a typed planning step.

Guild mentions should enter that planner path after only narrow deterministic gates that are already part of the documented transition: enabled external app prefix matching and exact current-channel memory-count pre-handling. Those gates should stay small compatibility paths, not grow into a second command language. When behavior needs stable syntax, strict arguments, or admin-grade guarantees, it belongs in a slash command and should be documented through the command guide.

Example:

@Gigi how often do we talk about postgres here?

Planner proposal:

{
  "intent": "answer_with_context",
  "tool_calls": [
    {"name": "memory.count", "args": {"text": "postgres", "scope": "this-channel"}},
    {"name": "memory.search", "args": {"query": "postgres", "scope": "this-channel", "limit": 5}}
  ]
}

Gigi then validates the proposal, checks memory.read.guild, enforces channel visibility, runs SQL-backed memory tools, packs a small context bundle, asks the answer model to explain the result, and audits the request without storing raw prompt or raw snippets by default.

Core packages should move toward these boundaries:

Package	Responsibility
internal/agent	Surface-independent request orchestration and agent loop
internal/agent/planner	Typed JSON planning against registered tools and available context
internal/contextbroker	Chooses and retrieves scoped context such as recent channel messages, memory search results, and plugin catalog summaries
internal/tools	Registry for native tools and enabled external app tools
internal/agent/policy	Capability checks, channel visibility checks, confirmation rules, redaction, and audit classification
internal/agent/answer	Final answer composition from prompt, tool results, citations, and policy notes

The important invariant: the LLM proposes; Gigi validates and executes deterministic code.

Provider Registry Direction

Gigi should support a provider registry with first-class entries for:

• OpenAI
• Anthropic
• Gemini
• future custom or compatible providers

Provider support should be purpose-aware instead of hard-coded to one chat model. Planned purposes:

Purpose	Use
chat	Normal DM or mention conversation
reasoning	Higher-effort planning and tool proposal
embedding	Message history and retrieval vectors
routing	Lightweight intent and plugin candidate selection

Model IDs should remain flexible strings validated by the provider service. Provider model catalogs change faster than Discord command definitions, so slash commands should not require a fixed model enum for every provider model.

Credential Ownership Model

The storage and resolver should support multiple credential owners from the start:

Owner type	Meaning	v0 behavior
guild	A Discord server owns the credential and model profiles.	Active UX
user	A Discord user owns a personal BYOK credential.	Schema/resolver-ready only
tenant	The deployment/operator owns a fallback credential.	Optional single-tenant fallback only

V0 should be multi-owner architecture with guild-only product behavior. That means the schema, usage events, and resolver should understand guild, user, and tenant, but the first shipped commands should only let authorized guild admins configure guild credentials.

Revised V0

V0 should establish the Discord agent runtime core, not only isolated chat or one-off semantic routes.

Current and planned V0 LLM/provider foundation includes:

• internal/llm/provider registry, credential store, secret sealing, model profile selection, provider testing, and credential resolution.
• Provider-backed text clients for OpenAI, Anthropic, and Gemini using resolved model credentials.
• Runtime generation wrapper that resolves active models and records token usage.
• Guild-scoped provider credentials configured by users with llm.provider.write.
• Guild-scoped model profiles selected by users with llm.provider.select.
• Provider test commands gated by llm.provider.test.
• Encrypted write-only secrets using a swappable SecretSealer.
• Usage events that record actor, guild, provider, model, purpose, billing owner type, billing owner ID, token counts, status, and classified error.
• /llm usage guild aggregate reporting with no raw prompt, completion, provider response, or secret storage.
• Guild-mention cognitive fallback chat after deterministic external app matching fails.
• Metadata-only conversation turn storage with no raw prompt or completion columns.
• Semantic routing that treats LLM output as a proposal, validates it against native tools and enabled manifests, applies capability checks, and returns dry-run or allowed read-only/public-safe results only.
• /ask command routing through the agent runtime with explicit context:none or dynamic current-channel context fetching through context:channel.
• Initial internal/agent runtime contracts, planner handler, native tool registry, memory count/search/recent tools, plugin plan/list tools, permission check tool, LLM usage summary tool, and chat fallback adapter.
• Initial internal/contextbroker context fetcher/packer seam for current-channel retained memory and bounded snippet packing, now extended into a typed pack builder with stable source IDs, citation labels, stale/invalidation markers, pinned snippets, omitted-unchanged records, restore handles, and next-state fingerprints.
• Initial durable agent run storage with SQL agent_runs, agent_run_steps, and agent_run_confirmations tables, runner lifecycle status, termination reasons, budget fields, cancellation checks, pending confirmation records, usage counters, and sanitized step observations.
• Initial internal/evalharness package for repeatable agent golden cases that capture responses, audit events, durable steps, run lifecycle records, pending confirmations, and observability redaction failures without a database or network.
• Guild policy storage that defaults personal keys to off.
• No per-user BYOK for guild mentions, shared memory, embeddings, or semantic routing.
• No silent fallback between billing owners.

Agent-runtime V0 should add:

• A single AgentHandler fallback behind deterministic external app matching. Started with agent-backed chat fallback and /ask.
• A typed planner contract for intents, context requests, tool calls, clarification questions, and confirmation requirements. Guild mention planning now starts with the general LLM planner over registered tool schemas, with the older LLM-backed memory planner as a compatibility fallback.
• A general AgentRunner with MaxSteps, MaxToolCalls, LLM-call budget, and redacted trace. Started with a compatibility runner that preserves current mention and /ask behavior while moving orchestration out of one-shot planning handlers, and now persists run lifecycle and step observations when a run store is available. Discord responses carry run IDs for confirmation-required runs, and /agent can trace, cancel, inspect, confirm, or reject records.
• A split policy/executor/answerer core. Started with policy-owned routing mode and capability checks, executor-owned tool lookup/execution/masked errors, deterministic write-tool confirmation guards, LLM answer synthesis over tool results, bounded short-term per-user/channel follow-up context, safe in-process /agent trace last diagnostics, and audit trace metadata for run_id, step_index, tool kind, and capability.
• A native tool registry with at least memory.count, memory.search, memory.recent, plugins.enabled, plugins.plan, permissions.check, llm.usage.guild, and llm.chat. Started with the read-only tools plus chat fallback; llm.chat remains an answerer/fallback concern instead of an LLM-calling tool.
• A context broker that decides whether a run needs no context, current-channel context, permitted guild memory, or enabled plugin catalog context. Started with bounded context pack building, restore-safe status metadata, and a first permitted recent-channel memory provider; cross-channel guild context, plugin catalog context, and richer retrieval-provider wiring remain incremental.
• A token packer that reserves budget for tool results, fetched context snippets, citations/source IDs, restore handles, and output, then trims context deterministically. Current implementation uses deterministic character budgets, pinned snippet priority, citation labels, and omitted-unchanged restore handles until token accounting lands.
• Read-only tool execution for memory, plugin planning, permission checks, and usage-style queries after capability and channel visibility checks. Started with current-channel memory tools, cited memory evidence payloads with message IDs/source IDs/restore handles/retention proof, plugins.enabled, plugins.plan, permissions.check, and llm.usage.guild.
• Public-safe external app dispatch only through existing manifest-grounded send_message rules; restricted actions remain dry-run or confirmation-required.
• Audits, durable steps, and safe in-process traces for agent.context, agent.plan, agent.tool, agent.answer, and agent.confirmation using request IDs, run IDs, ordered step indexes, status, scopes, tool names, tool kind, capability, token counts, termination reasons, and redacted metadata only. Debug trace output is actor/channel-scoped and safe-metadata-only; it does not expose raw prompts, memory snippets, provider responses, planned commands, or secrets.
• Compatibility migration from current wrappers: keep exact fast paths while agent-native tools replace SemanticMemoryHandler, AssistantFallbackHandler, and later semantic plugin handling.

Dynamic context fetching must not widen storage or audit exposure. Trace metadata may record run_id, step_index, context scope, fetch status, source type, result count, truncation flag, tool name, tool kind, capability, and message/source IDs. It must not store raw user prompts, raw fetched snippets, provider payloads, planned commands, embeddings, or secrets.

Deferred beyond V0:

• Rich semantic retrieval over vectors beyond current count/search scaffolding.
• /llm key personal BYOK behavior.
• Tenant/operator fallback credentials.
• Automatic resume after confirmed pending actions.
• Restricted dispatch and multi-step external app actions.

The important product rule: an admin can personally pay for a provider key, but if that key powers shared server behavior, Gigi treats it as a guild credential governed by guild policy and audit.

Implementation Slice Order

Each slice should be implemented as a small, reviewable project. For any non-trivial slice, prefer spawning caveman-style subagents so investigation, coding, review, testing, and documentation stay focused while keeping token use low. A typical slice should use:

• planning subagent: inspect current files, identify exact write scope, list risks, and return a compact implementation outline.
• builder subagent: own a narrow file set and make the bounded code change.
• reviewer subagent: review the diff for bugs, regressions, permission gaps, privacy leaks, and missing tests.
• tester subagent: run focused tests, add or propose missing harness coverage, and report exact failures.
• docs subagent: update user/admin docs after behavior is implemented and verified.

Recommended slice order:

Order	Slice	Goal	Caveman subagent split
1	Safety rails	Harden memory visibility, plugin import trust, audit redaction, provider-secret handling, and public/private response defaults before adding more agent power.	planning: map safety matrix. builder: implement one guard at a time. reviewer: fail-closed review. tester: denied-path tests. docs: admin-facing safety notes.
2	Context broker	Promote internal/contextbroker from a snippet packer into a Grape-inspired Discord context pack builder with source IDs, citations, stale markers, pinned snippets, omitted unchanged context, and restore handles.	planning: define pack schema. builder: implement pack/provider interfaces. reviewer: check privacy and truncation. tester: golden pack fixtures. docs: explain context behavior.
3	Durable agent runs	Move from one-shot plan execution toward persisted run state, step records, observations, termination reasons, budget accounting, cancellation, and resumable confirmations.	planning: schema and state-machine outline. builder: run store plus runner changes. reviewer: idempotency and resume review. tester: loop/confirmation tests. docs: operator/debug docs.
4	Memory retrieval	Add exact, FTS/BM25, and later vector retrieval over opted-in memory with provenance, retention proof, delete handling, backfill checkpoints, and safe citation output.	planning: retrieval contracts. builder: store/retriever worker slices. reviewer: channel visibility and retention review. tester: ingest-to-answer harness. docs: memory commands and policy.
5	Plugin manifest v1	Evolve plugin manifests from plugin-level prefix matching into action-level typed tool contracts with arg schemas, safety class, per-action capability, config validation, dispatch adapter, and explicit public-dispatch consent.	planning: manifest compatibility plan. builder: parser/store/matcher slices. reviewer: trust and permission review. tester: manifest corpus tests. docs: plugin author/admin guide.
6	Discord UX	Add component routing, buttons, modals, task status, cancel/retry, permission previews, plugin install review, and confirmation flows.	planning: interaction flow map. builder: component router and commands. reviewer: auth/ephemeral review. tester: fake Discord interaction tests. docs: command guide refresh.
7	Eval and observability harness	Add repeatable golden evals and integration harnesses before broad semantic behavior expands. Started with internal/evalharness for agent runner response/status/trace/confirmation/redaction cases.	planning: eval matrix. builder: fake Discord/provider/audit harness. reviewer: signal quality review. tester: CI/race/integration jobs. docs: test and ops guide.

The rule for every slice: safety, tests, and docs travel with code. Do not ship restricted plugin dispatch, guild-wide memory retrieval, arbitrary plugin code, or multi-agent Discord personas until durable runs, confirmation records, channel visibility, retention purge, and audit/query views are in place.

Revised V1

V1 can enable controlled personal BYOK:

• /llm key add, test, prefer, and remove for personal credentials.
• Personal keys allowed in DMs that do not use guild memory or guild actions.
• Guild policy for personal keys: off, dm-only, or guild-allowed.
• Explicit per-request billing choice where appropriate: server or me.
• Clear disclosure before a personal key can process guild context.
• No personal-key usage for shared guild embeddings by default.
• No automatic fallback from a failed personal key to a guild key, or from a failed guild key to a personal key.
• BYOK pays for reasoning but does not grant capabilities. The existing capability engine still decides whether actions can run.

Guild Memory Retrieval Plan

Guild memory should be explicit, bounded, and asynchronous. New guilds start with memory off. A guild admin must enable memory per channel before Gigi indexes messages from that channel. No DM, private thread, or group DM content should enter guild memory.

The target behavior has two retrieval paths:

• deterministic aggregate queries for questions such as @Gigi how many times did @sam mention "postgres"?
• semantic recall for questions that need similar-message retrieval and citations

Count questions should not depend on an LLM answer. Gigi should resolve the mentioned user when present, normalize the requested text, apply guild/channel/retention/deletion filters, and answer from SQL counts. The agent planner may map flexible language onto memory.count, but the count itself comes from deterministic SQL. Semantic recall can use embeddings after deterministic filters and capability checks.

Efficient ingestion should never block Discord event handling:

Discord MESSAGE_CREATE
  -> policy gate
  -> lightweight message upsert
  -> enqueue ingest job
  -> return

Memory worker
  -> redact
  -> chunk
  -> batch embed
  -> write segments and vectors
  -> checkpoint

Backfill should use /memory sync to queue a resumable job. The worker fetches Discord history page by page with rate limits, saves checkpoints, batch-inserts messages, and processes embeddings later. If the worker lags or fails, the bot should keep serving HTTP and Discord interactions.

Planned data model:

guild_memory_policies (
  guild_id text primary key,
  default_retention_days integer not null,
  raw_storage_mode text not null,
  embeddings_enabled boolean not null,
  updated_by_user_id text,
  updated_at timestamptz not null default now()
);

guild_memory_channels (
  guild_id text not null,
  channel_id text not null,
  mode text not null,
  retention_days integer,
  updated_by_user_id text,
  updated_at timestamptz not null default now(),
  primary key (guild_id, channel_id)
);

guild_memory_messages (
  message_id text primary key,
  guild_id text not null,
  channel_id text not null,
  author_user_id text not null,
  normalized_text text,
  content_ciphertext bytea,
  content_hash text,
  created_at timestamptz not null,
  edited_at timestamptz,
  deleted_at timestamptz,
  retention_until timestamptz not null
);

guild_memory_segments (
  id text primary key,
  message_id text not null,
  guild_id text not null,
  channel_id text not null,
  token_count integer not null,
  search_text text,
  created_at timestamptz not null default now()
);

guild_memory_embeddings (
  segment_id text primary key,
  provider_id text not null,
  model_id text not null,
  embedding vector,
  embedded_at timestamptz not null default now()
);

The exact schema can change during implementation, but these boundaries should stay:

• message ledger for exact count and audit-safe source metadata
• segment table for search units
• vector table for embeddings
• policy table for guild and channel opt-in
• queue/checkpoint tables for live ingest, backfill, retries, edits, deletes, and retention purge

Memory policy:

• default off
• per-channel opt-in
• ignore bot messages by default
• no shared-memory embeddings from personal BYOK
• raw content encrypted if retained
• retention required before ingest
• edit events supersede old segments
• delete events tombstone messages and purge segments/vectors
• audit events store metadata only, never raw message text, raw query text, snippets, embeddings, prompts, completions, provider responses, or secrets

Planned capabilities:

Capability	Use
memory.read.guild	Count, search, and ask over permitted guild memory
memory.manage.guild	Enable channels, set retention, queue sync, purge memory
memory.export.guild	Future bulk export path, if allowed by policy

Channel visibility should be enforced in addition to Gigi capabilities. A user should not retrieve memory from channels they cannot normally see.

Later Extensions

Later slices can add:

• tenant/operator hosted credentials for single-tenant deployments
• per-channel and per-role model policies
• per-user, per-guild, and per-provider budgets
• provider failover policies
• provider-specific model discovery
• persistent confirmation records
• Discord buttons and modals for confirmation and credential entry
• memory export and user deletion workflows
• webhook dispatch and multi-step plugin pipelines

Planned Data Shape

The exact SQL can evolve during implementation, but the first schema should preserve these ownership concepts:

llm_credentials (
  id text primary key,
  owner_type text not null,
  guild_id text,
  user_id text,
  provider_id text not null,
  label text not null,
  credential_ciphertext bytea not null,
  credential_nonce bytea not null,
  credential_key_id text not null,
  credential_fingerprint text not null,
  status text not null,
  last_test_status text,
  last_tested_at timestamptz,
  last_error_code text,
  created_by_user_id text,
  updated_by_user_id text,
  revoked_at timestamptz,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  unique (id, provider_id)
);

Active credential labels should be unique per owner. A label should identify one active credential in a guild, user, or tenant scope, not one credential per provider, so command UX can safely address credentials by label.

llm_model_profiles (
  id text primary key,
  owner_type text not null,
  guild_id text,
  user_id text,
  purpose text not null,
  credential_id text not null,
  provider_id text not null,
  model_id text not null,
  params jsonb not null default '{}'::jsonb,
  enabled boolean not null default true,
  selected_by_user_id text,
  updated_at timestamptz not null default now(),
  foreign key (credential_id, provider_id) references llm_credentials(id, provider_id)
);

llm_usage_events (
  id text primary key,
  request_id text not null,
  guild_id text,
  channel_id text,
  actor_user_id text not null,
  billing_owner_type text not null,
  billing_owner_id text not null,
  provider_id text not null,
  model_id text not null,
  purpose text not null,
  input_tokens integer,
  output_tokens integer,
  status text not null,
  error_class text,
  created_at timestamptz not null default now()
);

Command Shape

Live guild/admin commands:

/llm provider list
/llm provider add provider:<openai|anthropic|gemini|custom> label:<label>
/llm provider test label:<label>
/llm provider rotate label:<label>
/llm provider delete label:<label> confirm:true
/llm model show purpose:<chat|reasoning|embedding|routing>
/llm model set purpose:<chat|reasoning|embedding|routing> label:<label> model:<model-id>
/llm usage guild

Planned guild/admin policy command:

/llm policy set personal-keys:<off|dm-only|guild-allowed>

Personal BYOK commands are V1:

/llm key add provider:<openai|anthropic|gemini|custom> label:<label>
/llm key test label:<label>
/llm key prefer surface:<dm|guild> label:<label>
/llm key remove label:<label>

Credential entry should use a Discord modal or private flow, not a normal slash command string option. Gigi should never echo, log, or audit raw key material.

Planned memory commands:

/memory count user:<user> text:<string> scope:<this-channel|server> since:<date?> until:<date?> visibility:<private|channel>
/memory search query:<string> user:<user?> channel:<channel?> since:<date?> limit:<5|10|25>
/memory ask question:<string> scope:<this-channel|server> since:<date?> include-sources:<true|false>
/memory status
/memory sync channel:<channel?> since:<date?> confirm:true
/memory settings show
/memory settings set channel:<channel> mode:<off|metadata|full> retention-days:<number?>

Planned agent commands:

/ask question:<text> visibility:<public|private> context:<none|channel>
/gigi settings view
/gigi settings set ask-enabled:<true|false> default-context:<none|channel> default-visibility:<public|private>
/gigi audit recent target:<request|user|channel|plugin>
/gigi confirm request:<request-id>

/ask should route through the same agent runtime as guild mentions. context:none means normal chat without memory. context:channel can fetch bounded current-channel context and use current-channel memory tools when policy and capability checks pass. context:guild is future behavior until channel visibility filtering is strong enough across multiple channels.

Slash responses should default to ephemeral. If visibility:channel is supported, it should post only safe summaries, not raw hidden/private content.

Agent Runtime Direction

The first agent handler should be a fallback behind deterministic external app matching:

Discord MessageRouter
  -> ExternalAppDryRunHandler
     -> AgentHandler
        -> provider resolver
        -> planner
        -> context broker
        -> native tool registry
        -> capability evaluator
        -> answer composer
        -> audit recorder

V0 agent behavior should answer guild mentions after a provider is configured, route flexible memory questions through native tools, and produce plugin/tool dry-runs when execution is not clearly public-safe. Rich DM chat can reuse the same agent core with context:none until a guild context selector exists. Restricted dispatch and multi-step actions should wait until confirmation records, durable jobs, and tighter action schemas exist.