# llm-exe Documentation

> Complete documentation for llm-exe — a lightweight TypeScript toolkit for building LLM-powered applications.


# Install

Install llm-exe using npm.
```txt
npm i llm-exe
```

```ts
import * as llmExe from "llm-exe"

// or 

import { /* specific modules */ } from "llm-exe"
```

## Basic Example
Below is a simple example:
```ts
import {
  useLlm,
  createChatPrompt,
  createLlmExecutor
} from "llm-exe";

const llm = useLlm("openai.gpt-4o-mini");

// The first argument is the system message.
// Add a user message that references your input variable.
const prompt = createChatPrompt<{ input: string }>("Talk like a pirate.")
  .addUserMessage("{{input}}");

const executor = createLlmExecutor({
  llm,
  prompt,
});

const response = await executor.execute({ input: "Hello!" });
/**
 *
 * The prompt sent to the LLM would be:
 *
 * [
 *   { role: 'system', content: 'Talk like a pirate.' },
 *   { role: 'user',   content: 'Hello!' }
 * ]
 *
 */

/**
 *
 * Output from LLM executor:
 * Arrr, matey! Speak up, fer me ears be as keen as a sea serpent's!
 * Avast ye, what be your query?
 */
```

::: tip
Passing a value as `{ input }` does **not** automatically append a user message — your prompt must reference it (for example with `{{input}}` in a template or by calling `.addUserMessage("{{input}}")`). The chat prompt always renders exactly the messages you build.
:::
---


# Introduction

When writing llm-powered functions, you'll end up repeating a lot of code, and will end up needing some structure. A prompt may seem simple, but as your instructions grow, you may end up needing to add some more advanced abstractions. Likewise, if you're calling an llm in various functions in your code, you'll end up writing some wrapper around the llm so that you can share functionality like logging, collecting metrics, handling failures/timeouts/retry/etc. Furthermore, the LLM will be returning a string that you may need to validate or parse into a usable data type. Will you be duplicating code? Or creating sharable output parsers.

This package aims to be those lightweight abstractions.

Take the example below. In the first block, there is a function that takes an input, and responds yes or no

##  Example: A llm-powered function - Yes/No bot.
::: danger NOTE
This example does not use llm-exe
:::

This example uses the OpenAi nodejs package directly. Its likely where you'd start. It works, but as I'll explain below, its more of a proof of concept.
```typescript
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export async function YesOrNoBot(
  question: string
): Promise<{ response: string }> {
  const model = "gpt-4o-mini";

  const messages: OpenAI.ChatCompletionMessageParam[] = [
    {
      role: "system",
      content: `You are not an assistant, I need you to reply with only
  'yes' or 'no' as an answer to the question below. Do not explain yourself
  or ask questions. Answer with only yes or no.`,
    },
    {
      role: "user",
      content: question,
    },
    {
      role: "system",
      content: `yes or no:`,
    },
  ];

  const response = await openai.chat.completions.create({
    model: model,
    messages,
    temperature: 0,
    max_tokens: 160,
  });

  const [choice] = response.choices;
  const content = choice.message?.content ?? "";
  if (choice.finish_reason !== "stop") {
    console.log("error finish reason");
  }
  let cleanResponse = content.trim();
  return { response: cleanResponse };
}
```

##  Example (with llm-exe) : A llm-exe-powered function Yes/No bot.
::: tip
This example does use llm-exe!
:::
This example uses llm-exe to accomplish the same task.
```ts
import * as llmExe from "llm-exe";

export async function YesOrNoBot<I extends string>(input: I) {
  const llm = llmExe.useLlm("openai.gpt-4o-mini");

  const instruction = `You are not an assistant, I need you to reply with only 
  'yes' or 'no' as an answer to the question below. Do not explain yourself 
  or ask questions. Answer with only yes or no.`;

  const prompt = llmExe
    .createChatPrompt(instruction)
    .addUserMessage(input)
    .addUserMessage(`yes or no:`);

  const parser = llmExe.createParser("stringExtract", { enum: ["yes", "no"] });
  return llmExe.createLlmExecutor({ llm, prompt, parser }).execute({ input });
}
```

This is a simple example, but to highlight some key differences:
- The example uses OpenAI, but you could use a different model from a different vendor.
- The llm-exe version is structured in a way that each component could be testable. 
- Extra configuration details are hidden away.
- Parsing the output - the llm-exe version enforces the output we desire, and makes sure it is well-typed, without needing to deal with the response.

---


# What is an LLM function?

As the usefulness of LLMs grows, you have the ability to replace (at least consider replacing) complex functionality in your application with simple calls to an LLM.

Consider the following pseudo-code for a function that detects Personal Identifiable Information (PII) within the provided input:

```javascript:no-line-numbers
// The existing way to write a function to detect PII. Not using LLM.
function checkIfDocumentContainsPii(input){
  // regex for PII
  // more regex for PII
  // more regex for PII
  // more regex for PII
  // more regex for PII
  // etc
}

// A LLM-powered function
function checkIfDocumentContainsPiiWithLlm(input){
  // ask LLM if the content contains PII
}
```

As you can see, the LLM-powered version allows us to abstract away complex, brittle logic like regex patterns and instead rely on the reasoning abilities of the model—dramatically reducing the amount of code, improving flexibility, and often increasing accuracy.

Here is how you could implement example above with an LLM executor:

```ts
import {
  useLlm,
  createChatPrompt,
  createParser,
  createLlmExecutor,
} from "llm-exe";

export function piiDetector(input: string) {
  const instruction = `You need to check the text below for any of the PII listed below.
  
## PII you are looking for:
email addresses: anything that matches an email address
social security number: a personal social security number or business EIN
credit card number: a credit card number

## You should respond with the template below:
email addresses: <true or false if this type of PII is included>
social security number: <true or false if this type of PII is included>
credit card number: <true or false if this type of PII is included>`;

  const llm = useLlm("openai.gpt-4o-mini");
  const prompt = createChatPrompt(instruction).addUserMessage(input);
  const parser = createParser("listToJson");

  const executor = createLlmExecutor({
    llm,
    prompt,
    parser,
  });

  return executor.execute({ input });
}
```

```ts
/**
 * Example usage
 * Somewhere else in your codebase
 */
const input = "Hello! can you bill me? my cc is 4242-4242-4242-4242!!"
const response = await piiDetector(input)
/**
 *
 * Output:
 *
 * {
 *   emailAddresses: false,
 *   socialSecurityNumber: false,
 *   creditCardNumber: true
 * }
 */
```

The input and output of the `piiDetector` function are strongly typed, providing reliable structure and safety when integrating with the rest of your application.

For a complete, step-by-step build of a function like this — schema, prompt, parser, and inferred types — see [Write a Type-Safe LLM Function](/examples/concepts/type-safe-llm-function).

---


# LLM

LLM is a wrapper around various LLM providers, making your function implementations LLM-agnostic.

Note: llm-exe utilizes the underlying API's from the various providers. This means that you must have an account (and usually an API key) with them if you want to call those models using llm-exe.

**LLM Features:**

- Built-in timeout mechanism for better control when a provider takes too long.
- Automatic retry with configurable back-off for errors.
- Use different LLM's with different configurations for different functions.
Note: You can use and call methods on LLM's directly, but they are usually passed to an LLM executor and then called internally.

## Using `useLlm`

The `useLlm` function creates an LLM instance. It takes a provider key and an optional options object.

### Provider Shorthand (by model)

Use a provider-specific model shorthand to get full type support for that model's options:

```ts
import { useLlm } from "llm-exe";

const llm = useLlm("openai.gpt-4o-mini");
```

### Provider Key + Model Option

Use a generic provider key and specify the model in the options. This lets you use any model the provider supports without needing a dedicated shorthand:

```ts
const llm = useLlm("openai.chat.v1", {
  model: "gpt-4o",
});
```

### Options

All providers accept the [generic options](/llm/generic) (timeout, retries, temperature, maxTokens, etc.). Each provider may also accept provider-specific options — see the individual provider pages below.

### Deprecation Warnings

Deprecated provider/model shorthands continue to resolve for compatibility, but emit a Node `DeprecationWarning` with code `LLM_EXE_DEPRECATED` on first use. See [Deprecation Warnings](/llm/deprecations).

### Authentication

Each provider requires an API key. You can provide it in three ways:

1. **Environment variable** — set the provider's default env var (e.g., `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`)
2. **Setup options** — pass the key when creating the LLM (e.g., `{ openAiApiKey: "sk-..." }`)
3. **Execute options** — pass the key at execution time

See the individual provider pages for the exact option names and env var names.

### Direct Usage

While you'll typically pass the LLM instance to an [executor](/executor/), you can also call it directly:

```ts
const llm = useLlm("openai.gpt-4o-mini");
const response = await llm.call(prompt);
console.log(response.getResultText());
```

## Currently Supported Providers

Currently, llm-exe supports calling LLM's from:

- [OpenAi](/llm/openai)
- [Anthropic](/llm/anthropic)
- [xAI](/llm/xai)
- [Google](/llm/gemini)
- [AWS Bedrock](/llm/bedrock/)
- [Ollama](/llm/ollama)
- [Deepseek](/llm/deepseek)
- [Custom Providers](/llm/custom)

## Adding Custom LLM's

You can create custom LLM configurations using `useLlmConfiguration`. This allows you to:

- Connect to OpenAI-compatible APIs
- Use local models
- Work with corporate proxies
- Add support for new providers

See the [Custom Provider Configuration](/llm/custom) guide for details.

---


# Generic Options

llm-exe attempts to normalize the inputs for various llm vendors, providing a single interface that can be used to interact with different models. While this is not always possible, since certain vendors may implement features that others don't support, either way only the allowed options make it to the respective api calls.

## Options

For a worked example of `timeout`, `numOfAttempts`, and `maxDelay` in a production setup, see [Add Retries and Timeouts to LLM Calls](/examples/concepts/retries-and-timeouts).

| Option        | Type             | Default   | Description                                                                                                |
| ------------- | ---------------- | --------- | ---------------------------------------------------------------------------------------------------------- |
| timeout       | number           | 30000     | Max execution time of API call to the LLM, in milliseconds.                                                |
| maxDelay      | number           | 5000      | Used for retry back-off. Max time to wait between attempts when timeout has been reached, in milliseconds. |
| numOfAttempts | number           | 2         | Used for retry. How many attempts should be made before throwing error                                     |
| jitter        | "none" \| "full" | none      | Used for retry back-off.                                                                                   |
| temperature   | number           | undefined | Maps to provider-specific temperature parameter.                                                           |
| maxTokens     | number           | undefined | Maps to provider-specific max tokens parameter.                                                            |
| topP          | number           | undefined | Maps to provider-specific top_p parameter.                                                                 |
| stopSequences | string[]         | undefined | Maps to provider-specific stop sequences parameter.                                                        |
| effort        | string           | undefined | Maps to reasoning effort. Valid values: `"minimal"`, `"low"`, `"medium"`, `"high"`. Only supported by providers/models that support reasoning effort (e.g. OpenAI gpt-5, Google Gemini 2.5). |
| stream        | boolean \| null  | null      | Note: Not supported yet.                                                                                   |

> [!NOTE]
> Different vendors will allow (and may require) additional options.
> - [OpenAI Chat Model Options](/llm/openai#openai-specific-options)
> - [Anthropic Chat Model Options](/llm/anthropic#anthropic-specific-options)
> - [Google Gemini Chat Model Options](/llm/gemini#gemini-specific-options)
> - [xAI Chat Model Options](/llm/xai#xai-specific-options)
> - [Deepseek Chat Model Options](/llm/deepseek#deepseek-specific-options)
> - [Ollama Chat Model Options](/llm/ollama#ollama-specific-options)
> - [AWS Bedrock Chat Model Options](/llm/bedrock/)


---


# OpenAI

When using OpenAi models, llm-exe will make POST requests to `https://api.openai.com/v1/chat/completions`. All models are supported if you pass `openai.chat.v1` as the first argument, and then specify a model in the options.

## Basic Usage

### OpenAi Chat

```ts
const llm = useLlm("openai.chat.v1", {
  model: "gpt-4o", // specify a model
});
```

### OpenAi Chat By Model

```ts
const llm = useLlm("openai.gpt-4o", {
  // other options,
  // no model needed, using gpt-4o
});
```

<ImportModelNames provider="openai" />

## Authentication

To authenticate, you need to provide an OpenAi API Key. You can provide the API key various ways, depending on your use case.

1. Pass in as execute options using `openAiApiKey`
2. Pass in as setup options using `openAiApiKey`
3. Use a default key by setting an environment variable of `OPENAI_API_KEY`

Generally you pass the LLM instance off to an LLM Executor and call that. However, it is possible to interact with the LLM object directly, if you wanted.

```ts
// call the LLM directly with a prompt
await llm.call(prompt);
```

## OpenAi-Specific Options

In addition to the [generic options](/llm/generic), the following options are OpenAi-specific and can be passed in when creating a llm function.

| Option           | Type    | Default     | Description                                                    |
| ---------------- | ------- | ----------- | -------------------------------------------------------------- |
| model            | string  | —           | The model to use. Must be specified when using `openai.chat.v1`. Can be any valid chat model. See OpenAI Docs |
| openAiApiKey     | string  | undefined   | API key for OpenAi. See [authentication](/llm/openai#authentication)   |
| topP             | number  | undefined   | Maps to `top_p`. See OpenAI Docs                               |
| stopSequences    | array   | undefined   | Maps to `stop`. See OpenAI Docs                                |
| frequencyPenalty | number  | undefined   | Maps to `frequency_penalty`. See OpenAI Docs                   |
| logitBias        | object  | undefined   | Maps to `logit_bias`. See OpenAI Docs                          |
| useJson          | boolean | undefined   | When `true`, sets `response_format` to `json_object`           |
| effort           | string  | undefined   | Maps to `reasoning_effort`. Valid values: `"minimal"`, `"low"`, `"medium"`, `"high"`. Only supported with reasoning models (e.g. gpt-5). |

See [OpenAI API Reference](https://platform.openai.com/docs/api-reference/chat) for details on these parameters.

---


# Anthropic

When using Anthropic models, llm-exe will make POST requests to `https://api.anthropic.com/v1/messages`.

## Setup

### Anthropic Chat

```ts
const llm = useLlm("anthropic.chat.v1", {
  model: "claude-sonnet-4-6", // specify a model
});
```

### Anthropic Chat By Model

```ts
const llm = useLlm("anthropic.claude-sonnet-4-6", {
  // other options,
  // no model needed, using claude-sonnet-4-6
});
```

<ImportModelNames provider="anthropic" />


## Authentication

To authenticate, you need to provide an Anthropic API Key. You can either provide the API key various ways, depending on your use case.

- Pass in as execute options using `anthropicApiKey`
- Pass in as setup options using `anthropicApiKey`
- Use a default key by setting an environment variable of `ANTHROPIC_API_KEY`

## Basic Usage

Generally you pass the LLM instance off to an LLM Executor and call that. However, it is possible to interact with the LLM object directly, if you wanted.

```ts
// given array of chat messages, calls chat completion
await llm.call([]);

// given string prompt, calls completion
await llm.call("");
```

## Anthropic-Specific Options

<!--@include: ./anthropic.options.part.md-->
---


# Google Gemini

When using Google Gemini models, llm-exe will make POST requests to `https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent`. All models are supported if you pass `google.chat.v1` as the first argument, and then specify a model in the options.

## Basic Usage

### Gemini Chat

```ts
const llm = useLlm("google.chat.v1", {
  model: "gemini-2.5-flash", // specify a model
});
```

### Gemini Chat By Model

```ts
const llm = useLlm("google.gemini-2.5-flash", {
  // other options,
  // no model needed, using gemini-2.5-flash
});
```

<ImportModelNames provider="google" />

## Authentication

To authenticate, you need to provide a Google Gemini API Key. You can provide the API key various ways, depending on your use case.

1. Pass in as execute options using `geminiApiKey`
2. Pass in as setup options using `geminiApiKey`
3. Use a default key by setting an environment variable of `GEMINI_API_KEY`

Generally you pass the LLM instance off to an LLM Executor and call that. However, it is possible to interact with the LLM object directly, if you wanted.

```ts
// call the LLM directly with a prompt
await llm.call(prompt);
```

## Gemini-Specific Options

In addition to the generic options, the following options are Gemini-specific and can be passed in when creating a llm function.

| Option       | Type   | Default          | Description                                                          |
| ------------ | ------ | ---------------- | -------------------------------------------------------------------- |
| model        | string | —                | The model to use. Must be specified when using `google.chat.v1`. See Google Gemini Docs |
| geminiApiKey | string | undefined        | API key for Google. See [authentication](/llm/gemini#authentication) |
| effort       | string | undefined        | Maps to `thinkingConfig.thinkingBudget`. Valid values: `"minimal"`, `"low"`, `"medium"`, `"high"`. Only supported with reasoning models (e.g. gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite). |

> [!NOTE]
> The Gemini provider currently maps `model`, `geminiApiKey`, and `effort`. Generic options like `temperature`, `maxTokens`, and `topP` are not mapped to the Gemini API at this time.

See [Google Gemini API Reference](https://ai.google.dev/gemini-api/docs) for details.

---


# AWS Bedrock

## Setup

### AWS Bedrock Chat
```ts
const llm = useLlm("amazon:anthropic.chat.v1", {
  model: "anthropic.claude-sonnet-4-v2:0", // a valid model from bedrock

  // optional, see `Authentication` below
  awsRegion: "<Your aws Region>",
  awsSecretKey: "<Your AWS secret key.",
  awsAccessKey: "<Your AWS access key. Starts with AKIxxxx>",

  // rest of options
});
```

#### AWS Bedrock Chat Model Options
- amazon:meta.chat.v1
- amazon:anthropic.chat.v1



## Authentication
To authenticate, you need to provide a way to authenticate with AWS. You can provide the API key various ways, depending on your use case. 
- Pass in as execute options using `awsRegion` | `awsSecretKey` | `awsAccessKey`
- Pass in as setup options using `awsRegion` | `awsSecretKey` | `awsAccessKey`
- Use a default key by setting an environment variable of `AWS_REGION` | `AWS_SECRET_ACCESS_KEY` | `AWS_ACCESS_KEY_ID`, respectively.


::: tip
If you are in AWS environment, the above environment variables could be set automatically. 
:::

::: warning
Make sure the respective role has the necessary permissions to access Bedrock.
:::

---


# Anthropic

When using Anthropic models via AWS Bedrock, llm-exe will make POST requests to `https://bedrock-runtime.us-west-2.amazonaws.com/model/{MODEL_ID}/invoke`.

## Setup

### Anthropic Chat

```ts
const llm = useLlm("amazon:anthropic.chat.v1", {
  model: "anthropic.claude-sonnet-4-v2:0",  // This is the model id from Bedrock
});
```

## Bedrock Anthropic Options

In addition to the [generic options](/llm/generic), the following options are available for Anthropic models on Bedrock.

| Option      | Type   | Default   | Description                                                              |
| ----------- | ------ | --------- | ------------------------------------------------------------------------ |
| model       | string | —         | The Bedrock model id. Must be specified. See AWS Bedrock Docs            |
| maxTokens   | number | 10000     | Maps to `max_tokens`. See Anthropic Docs                                 |
| topP        | number | undefined | Maps to `top_p`. See Anthropic Docs                                      |
| awsRegion   | string | undefined | AWS Region. Can be set via `AWS_REGION` environment variable             |
| awsSecretKey| string | undefined | AWS Secret Key. Can be set via `AWS_SECRET_ACCESS_KEY` environment variable |
| awsAccessKey| string | undefined | AWS Access Key. Can be set via `AWS_ACCESS_KEY_ID` environment variable  |

> [!NOTE]
> The Bedrock Anthropic provider maps a subset of the direct [Anthropic provider](/llm/anthropic) options. Options like `temperature`, `topK`, `stopSequences`, `metadata`, and `serviceTier` are not mapped for the Bedrock variant at this time.
---


# Meta

When using Meta models via AWS Bedrock, llm-exe will make POST requests to `https://bedrock-runtime.us-west-2.amazonaws.com/model/{MODEL_ID}/invoke`.

## Setup

### Meta LLama

```ts
const llm = useLlm("amazon:meta.chat.v1", {
    model: "llama3-8b-instruct-v1:0", // This is the model id from Bedrock
});
```

## LLama-Specific Options

In addition to the generic options, the following options are available for Meta LLama models on Bedrock.

| Option      | Type   | Default   | Description                                                              |
| ----------- | ------ | --------- | ------------------------------------------------------------------------ |
| model       | string | —         | The Bedrock model id. Must be specified. See AWS Bedrock Docs            |
| temperature | number | undefined | Maps to temperature.                                                     |
| maxTokens   | number | 2048      | Maps to max_gen_len. See AWS Bedrock Docs                                |
| topP        | number | undefined | Maps to top_p. See AWS Bedrock Docs                                      |
| awsRegion   | string | undefined | AWS Region. Can be set via `AWS_REGION` environment variable             |
| awsSecretKey| string | undefined | AWS Secret Key. Can be set via `AWS_SECRET_ACCESS_KEY` environment variable |
| awsAccessKey| string | undefined | AWS Access Key. Can be set via `AWS_ACCESS_KEY_ID` environment variable  |
---


# xAI

When using xAI models, llm-exe will make POST requests to `https://api.x.ai/v1/chat/completions`. All models are supported if you pass `xai.chat.v1` as the first argument, and then specify a model in the options.

llm-exe ships typed shorthands for the most common xAI models so you do not have to remember the exact model strings:

| Shorthand                  | Default model                       |
| -------------------------- | ----------------------------------- |
| `xai.chat.v1`              | _none — set `model`_                |
| `xai.grok-2`               | `grok-2-latest`                     |
| `xai.grok-3`               | `grok-3`                            |
| `xai.grok-3-mini`          | `grok-3-mini`                       |
| `xai.grok-4`               | `grok-4`                            |
| `xai.grok-4-fast`          | `grok-4-fast-non-reasoning`         |
| `xai.grok-4-1-fast`        | `grok-4-1-fast-non-reasoning`       |
| `xai.grok-4.3`             | `grok-4.3` (reasoning model)        |
| `xai.grok-4.20`            | `grok-4.20-0309-non-reasoning`      |
| `xai.grok-4.20-reasoning`  | `grok-4.20-0309-reasoning`          |

## Basic Usage

### xAI Chat

```ts
const llm = useLlm("xai.chat.v1", {
  model: "grok-4", // specify a model
});
```

### x.ai Chat By Model

```ts
const llm = useLlm("xai.grok-4", {
  // other options,
  // no model needed, using grok-4
});
```

```ts
const llm = useLlm("xai.grok-3-mini", {
  // other options,
  // no model needed, using grok-3-mini
});
```

<ImportModelNames provider="xai" />

## Authentication

To authenticate, you need to provide an xAI API Key. You can provide the API key various ways, depending on your use case.

1. Pass in as execute options using `xAiApiKey`
2. Pass in as setup options using `xAiApiKey`
3. Use a default key by setting an environment variable of `XAI_API_KEY`

Generally you pass the LLM instance off to an LLM Executor and call that. However, it is possible to interact with the LLM object directly, if you wanted.

```ts
// call the LLM directly with a prompt
await llm.call(prompt);
```

## xAI-Specific Options

In addition to the generic options, the following options are xAI-specific and can be passed in when creating a llm function.

| Option           | Type    | Default     | Description                                                    |
| ---------------- | ------- | ----------- | -------------------------------------------------------------- |
| model            | string  | —           | The model to use. Must be specified when using `xai.chat.v1`.  |
| xAiApiKey        | string  | undefined   | API key for xAI. Optionally can be set using process.env.XAI_API_KEY |
| topP             | number  | undefined   | Maps to `top_p`. See xAI Docs                                 |
| stopSequences    | array   | undefined   | Maps to `stop`. See xAI Docs                                  |
| frequencyPenalty | number  | undefined   | Maps to `frequency_penalty`. See xAI Docs                     |
| logitBias        | object  | undefined   | Maps to `logit_bias`. See xAI Docs                            |
| useJson          | boolean | undefined   | When `true`, sets `response_format` to `json_object`           |
| effort           | string  | undefined   | Maps to `reasoning_effort`. Valid values: `"minimal"`, `"low"`, `"medium"`, `"high"`. Currently not supported by any available xAI model and will be silently ignored. |

See [xAI API Reference](https://docs.x.ai/docs/overview) for details on these parameters.

---


# Ollama

When using Ollama models, llm-exe will make POST requests to `http://localhost:11434`. You can also override this by setting the environment variable `OLLAMA_ENDPOINT`. All models are supported if you pass `ollama.chat.v1` as the first argument, and then specify a model in the options (assuming you have downloaded this model).

## Basic Usage

### Ollama Chat

```ts
const llm = useLlm("ollama.chat.v1", {
  model: "deepseek-r1", // specify a model
});
```

### Ollama Chat By Model

```ts
const llm = useLlm("ollama.deepseek-r1", {
  // other options,
  // no model needed, using deepseek-r1
});
```

<ImportModelNames provider="ollama" />

## Configuration

By default, llm-exe connects to `http://localhost:11434`. To use a different Ollama endpoint, set the `OLLAMA_ENDPOINT` environment variable.

Generally you pass the LLM instance off to an LLM Executor and call that. However, it is possible to interact with the LLM object directly, if you wanted.

```ts
// call the LLM directly with a prompt
await llm.call(prompt);
```

## Ollama-Specific Options

| Option | Type   | Default   | Description                                                    |
| ------ | ------ | --------- | -------------------------------------------------------------- |
| model  | string | —         | The model to use. Must be specified when using `ollama.chat.v1`. |

> [!NOTE]
> The Ollama provider currently maps `model` and `prompt` only. Generic options like `temperature`, `maxTokens`, and `topP` are not mapped to the Ollama API at this time. The Ollama request body is sent directly to the `/api/chat` endpoint.

See [Ollama Docs](https://ollama.com/) for details.

---


# Deepseek

When using Deepseek models, llm-exe will make POST requests to `https://api.deepseek.com/v1/chat/completions`. All models are supported if you pass `deepseek.chat.v1` as the first argument, and then specify a model in the options.

llm-exe ships typed shorthands for the most common Deepseek models so you do not have to remember the exact model strings:

| Shorthand              | Default model        |
| ---------------------- | -------------------- |
| `deepseek.chat.v1`     | _none — set `model`_ |
| `deepseek.chat`        | `deepseek-chat`      |
| `deepseek.v4-flash`    | `deepseek-v4-flash`  |
| `deepseek.v4-pro`      | `deepseek-v4-pro`    |

## Basic Usage

### Deepseek Chat

```ts
const llm = useLlm("deepseek.chat.v1", {
  model: "deepseek-chat", // specify a model
});
```

### Deepseek Chat By Model

```ts
const llm = useLlm("deepseek.chat", {
  // other options,
  // no model needed, using deepseek-chat model
});
```

```ts
const llm = useLlm("deepseek.v4-flash", {
  // other options,
  // no model needed, using deepseek-v4-flash model
});
```

```ts
const llm = useLlm("deepseek.v4-pro", {
  // other options,
  // no model needed, using deepseek-v4-pro model
});
```

<ImportModelNames provider="deepseek" />

## Authentication

To authenticate, you need to provide a Deepseek API Key. You can provide the API key various ways, depending on your use case.

1. Pass in as execute options using `deepseekApiKey`
2. Pass in as setup options using `deepseekApiKey`
3. Use a default key by setting an environment variable of `DEEPSEEK_API_KEY`

Generally you pass the LLM instance off to an LLM Executor and call that. However, it is possible to interact with the LLM object directly, if you wanted.

```ts
// call the LLM directly with a prompt
await llm.call(prompt);
```

## Deepseek-Specific Options

In addition to the generic options, the following options are Deepseek-specific and can be passed in when creating a llm function.

| Option           | Type    | Default       | Description                                                    |
| ---------------- | ------- | ------------- | -------------------------------------------------------------- |
| model            | string  | —             | The model to use. Must be specified when using `deepseek.chat.v1`. Can be any valid chat model. See Deepseek Docs |
| deepseekApiKey   | string  | undefined     | API key for Deepseek. See [authentication](/llm/deepseek#authentication) |
| topP             | number  | undefined     | Maps to `top_p`. See Deepseek Docs                             |
| stopSequences    | array   | undefined     | Maps to `stop`. See Deepseek Docs                              |
| frequencyPenalty | number  | undefined     | Maps to `frequency_penalty`. See Deepseek Docs                 |
| logitBias        | object  | undefined     | Maps to `logit_bias`. See Deepseek Docs                        |
| useJson          | boolean | undefined     | When `true`, sets `response_format` to `json_object`           |
| effort           | string  | undefined     | Maps to `reasoning_effort`. Valid values: `"minimal"`, `"low"`, `"medium"`, `"high"`. Currently not supported by Deepseek models and will be silently ignored. |

See [Deepseek API Reference](https://api-docs.deepseek.com/) for details on these parameters.

---


# Custom Provider Configuration

The `useLlmConfiguration` function allows you to define custom LLM provider configurations. This is useful for:

- Using OpenAI-compatible APIs (local models, third-party providers)
- Working behind corporate proxies
- Testing with mock servers
- Supporting new providers before they're officially added

## Basic Usage

```typescript
import { useLlmConfiguration } from "llm-exe";

const customLlm = useLlmConfiguration({
  key: "my-custom-provider",
  provider: "openai.chat",
  endpoint: "https://api.mycompany.com/v1/chat/completions",
  method: "POST",
  headers: `{"Authorization": "Bearer {{apiKey}}", "Content-Type": "application/json"}`,
  
  // Define available options and their defaults
  options: {
    apiKey: {
      required: [true, "API key is required"],
    },
    model: {
      default: "gpt-4o-mini",
    },
    temperature: {
      default: 0.7,
    },
  },
  
  // Map options to request body
  mapBody: {
    prompt: {
      key: "messages",
      transform: (messages) => messages, // Pass through
    },
    model: {
      key: "model",
    },
    temperature: {
      key: "temperature",
    },
  },
  
  // Transform the response to standard format
  transformResponse: (result) => ({
    id: result.id || `custom-${Date.now()}`,
    name: result.model || "custom-model",
    created: result.created || Date.now(),
    usage: result.usage || {
      input_tokens: 0,
      output_tokens: 0,
      total_tokens: 0,
    },
    content: result.choices?.[0]?.message?.content
      ? [{ type: "text", text: result.choices[0].message.content }]
      : [{ type: "text", text: "" }],
    stopReason: result.choices?.[0]?.finish_reason || "stop",
  }),
});

// Use it like any other LLM
const llm = customLlm({
  apiKey: process.env.CUSTOM_API_KEY,
  model: "gpt-4o",
} as any); // Note: Use 'as any' for custom options that aren't in base types

const response = await llm.call("Hello, how are you?");
console.log(response.getResultText());
```

## Examples

### OpenAI-Compatible Local Model

Many local model servers (Ollama, LM Studio, etc.) provide OpenAI-compatible endpoints. Use the built-in helper for these:

```typescript
import { useLlmConfiguration, createOpenAiCompatibleConfiguration } from "llm-exe";

// Simple and correct - includes prompt sanitization and response handling
const localLlm = useLlmConfiguration(
  createOpenAiCompatibleConfiguration({
    key: "local-llama",
    provider: "local.llama",
    endpoint: "http://localhost:11434/v1/chat/completions",
    apiKeyMapping: ["apiKey", "LOCAL_API_KEY"], // or ["apiKey", ""] if no key needed
  })
);

const llm = localLlm({ model: "llama-3.3-70b" });
const response = await llm.call("Explain quantum computing");
```

Or if you need to customize further:

```typescript
const localLlm = useLlmConfiguration({
  ...createOpenAiCompatibleConfiguration({
    key: "local-custom",
    provider: "local.custom",
    endpoint: "http://localhost:8080/v1/chat/completions",
    apiKeyMapping: ["apiKey", ""],
  }),
  // Override specific options if needed
  options: {
    model: {
      default: "mixtral-8x7b",
    },
    temperature: {
      default: 0.7,
    },
  },
});
```

### Corporate Proxy

Route requests through a corporate proxy with custom headers:

```typescript
const proxiedLlm = useLlmConfiguration({
  key: "proxied-openai",
  provider: "openai.chat",
  endpoint: "https://proxy.company.com/openai/v1/chat/completions",
  method: "POST",
  headers: `{
    "Authorization": "Bearer {{openAiApiKey}}",
    "X-Company-Auth": "{{companyToken}}",
    "X-Trace-Id": "{{traceId}}",
    "Content-Type": "application/json"
  }`,
  
  options: {
    openAiApiKey: {
      default: () => process.env.OPENAI_API_KEY,
    },
    companyToken: {
      default: () => process.env.COMPANY_TOKEN,
      required: [true, "Company token is required for proxy"],
    },
    traceId: {
      default: () => crypto.randomUUID(),
    },
    model: {
      default: "gpt-4o-mini",
    },
  },
  
  mapBody: {
    prompt: {
      key: "messages",
      transform: (messages) => {
        // Ensure messages are in OpenAI format
        if (Array.isArray(messages)) {
          return messages.map(m => ({
            role: m.role || "user",
            content: m.content || ""
          }));
        }
        return messages;
      },
    },
    model: {
      key: "model",
    },
  },
  
  transformResponse: (result) => ({
    id: result.id || `proxy-${Date.now()}`,
    name: result.model || "proxy-model",
    created: result.created || Date.now(),
    usage: result.usage || {
      input_tokens: 0,
      output_tokens: 0,
      total_tokens: 0,
    },
    content: result.choices?.[0]?.message?.content
      ? [{ type: "text", text: result.choices[0].message.content }]
      : [{ type: "text", text: "" }],
    stopReason: result.choices?.[0]?.finish_reason || "stop",
  }),
});
```

### Custom API with Different Format

For APIs that don't follow OpenAI/Anthropic conventions:

```typescript
const customApi = useLlmConfiguration({
  key: "custom-api",
  provider: "openai.chat", // Use as base for type compatibility
  endpoint: "https://api.custom-llm.com/generate",
  method: "POST",
  headers: `{
    "X-API-Key": "{{apiKey}}",
    "Content-Type": "application/json"
  }`,
  
  options: {
    apiKey: {
      required: [true, "API key is required"],
    },
    maxLength: {
      default: 1000,
    },
  },
  
  mapBody: {
    prompt: {
      key: "input_text",
      transform: (messages) => {
        // Convert chat messages to single string
        if (Array.isArray(messages)) {
          return messages
            .map(m => `${m.role}: ${m.content}`)
            .join("\n");
        }
        return messages;
      },
    },
    maxLength: {
      key: "max_tokens",
    },
  },
  
  transformResponse: (result) => {
    // Map custom response format to standard format
    return {
      id: result.request_id || "custom-" + Date.now(),
      name: "custom-model",
      created: Date.now(),
      usage: {
        input_tokens: result.stats?.input_tokens || 0,
        output_tokens: result.stats?.output_tokens || 0,
        total_tokens: result.stats?.total_tokens || 0,
      },
      content: [
        {
          type: "text",
          text: result.generated_text || "",
        },
      ],
      stopReason: result.stop_reason || "stop",
    };
  },
});
```

## OpenAI-Compatible APIs

### createOpenAiCompatibleConfiguration

For OpenAI-compatible endpoints, use the built-in `createOpenAiCompatibleConfiguration` helper to avoid boilerplate. This is perfect for services like Ollama, LM Studio, Together AI, and any other OpenAI-compatible API.

```typescript
import { useLlmConfiguration, createOpenAiCompatibleConfiguration } from "llm-exe";

const config = createOpenAiCompatibleConfiguration({
  key: "my-provider",           // Unique identifier
  provider: "my-provider.chat",  // Provider name  
  endpoint: "https://api.example.com/v1/chat/completions",
  apiKeyMapping: ["apiKey", "MY_API_KEY_ENV"]  // [option name, env var]
});

const llm = useLlmConfiguration(config);
```

#### Parameters

- **key** (string): Unique identifier for this configuration
- **provider** (string): Provider identifier (e.g., "local.ollama", "custom.service")
- **endpoint** (string): Full URL to the OpenAI-compatible `/chat/completions` endpoint
- **apiKeyMapping** ([string, string]): Tuple of `[optionName, envVarName]`
  - First element: the option name users will use (e.g., `apiKey`)
  - Second element: environment variable to read from (or empty string if no default)
- **transformResponse** (optional): Custom response transformer (defaults to OpenAI format)

#### What's Included

This helper automatically provides:
- Proper prompt sanitization for OpenAI format
- Response transformation to standard format
- Function calling support
- JSON schema support
- Proper Authorization and Content-Type headers
- All required parameter mappings (model, temperature, top_p, etc.)

#### Examples

**Local Ollama Server:**
```typescript
const ollamaConfig = createOpenAiCompatibleConfiguration({
  key: "ollama-local",
  provider: "ollama.local",
  endpoint: "http://localhost:11434/v1/chat/completions",
  apiKeyMapping: ["apiKey", ""] // No API key needed
});

const ollama = useLlmConfiguration(ollamaConfig);
const llm = ollama({ model: "llama3.3:70b" });
```

**Together AI:**
```typescript
const togetherConfig = createOpenAiCompatibleConfiguration({
  key: "together-ai",
  provider: "together.ai",
  endpoint: "https://api.together.xyz/v1/chat/completions",
  apiKeyMapping: ["apiKey", "TOGETHER_API_KEY"]
});

const together = useLlmConfiguration(togetherConfig);
const llm = together({ 
  apiKey: process.env.TOGETHER_API_KEY,
  model: "meta-llama/Llama-3.3-70B-Instruct-Turbo"
});
```

**Extending the Configuration:**
```typescript
const baseConfig = createOpenAiCompatibleConfiguration({
  key: "custom",
  provider: "custom.service",
  endpoint: "https://api.custom.com/v1/chat/completions",
  apiKeyMapping: ["apiKey", "CUSTOM_API_KEY"]
});

// Add or override options
const extendedConfig = {
  ...baseConfig,
  options: {
    ...baseConfig.options,
    maxTokens: { default: 2048 }
  },
  mapBody: {
    ...baseConfig.mapBody,
    maxTokens: { key: "max_tokens" }
  }
};

const custom = useLlmConfiguration(extendedConfig);
```

## Configuration Reference

### Config Object Properties

| Property | Type | Description |
|----------|------|-------------|
| `key` | `string` | Unique identifier for this configuration |
| `provider` | `string` | Base provider type (e.g., "openai.chat") |
| `endpoint` | `string` | API endpoint URL (supports `{{variable}}` templates) |
| `method` | `"POST" \| "GET" \| "PUT"` | HTTP method |
| `headers` | `string` | JSON string of headers (supports `{{variable}}` templates) |
| `options` | `object` | Available parameters and their defaults/requirements |
| `mapBody` | `object` | Maps parameters to request body fields |
| `mapOptions?` | `object` | Maps executor options (functions, schemas) to provider format |
| `transformResponse` | `function` | Transforms provider response to standard format |

### Options Configuration

Each option can have:
- `default`: Default value or function that returns the value
- `required`: `[boolean, string]` tuple for required fields with error message

### MapBody Configuration

Each mapping can have:
- `key`: Target field name in request body (supports dot notation)
- `default`: Default value if not provided
- `transform`: Function to transform the value before sending

### Response Transform Function

The `transformResponse` function must return an object with this structure:

```typescript
{
  id: string;
  name: string;
  created: number;
  usage: {
    input_tokens: number;
    output_tokens: number;
    total_tokens: number;
  };
  content: Array<{
    type: "text";
    text: string;
  }>;
  stopReason: string;
}
```

## Use with Executors

Custom providers work seamlessly with executors:

```typescript
import { useLlmConfiguration, createLlmExecutor, createChatPrompt, createParser } from "llm-exe";

const customLlm = useLlmConfiguration({
  // ... your config
});

const executor = createLlmExecutor({
  llm: customLlm({ apiKey: process.env.API_KEY } as any),
  prompt: createChatPrompt(`Analyze: {{text}}`),
  parser: createParser("json"),
});

const result = await executor.execute({ text: "..." });
```

## Testing

Use custom configurations to create mock providers for testing:

```typescript
const mockLlm = useLlmConfiguration({
  key: "test-mock",
  provider: "openai.chat",
  endpoint: "http://localhost:3000/mock",
  method: "POST",
  headers: `{}`,
  
  options: {},
  mapBody: {
    prompt: { key: "messages" },
  },
  
  transformResponse: (result) => ({
    id: "test-1",
    name: "mock",
    created: Date.now(),
    usage: { input_tokens: 0, output_tokens: 0, total_tokens: 0 },
    content: [{ type: "text", text: result.response || "Mock response" }],
    stopReason: "stop",
  }),
});
```

## Important Notes

1. **Custom options require type assertion**: When passing custom options that aren't in the base types, use `as any`:
   ```typescript
   const llm = customLlm({ customOption: "value" } as any);
   ```

2. **Transform functions are required**: You must implement your own `transformResponse` function to map the provider's response format to llm-exe's standard format.

3. **Headers must be valid JSON**: The `headers` property expects a JSON string, not an object.

4. **Template variables**: Both `endpoint` and `headers` support `{{variable}}` placeholders that will be replaced with values from the options.
---


# Prompt

The prompt is the instruction for the LLM, usually sent in plain-text or an array of chat-style messages. When working with certain models, the prompt is formatted like chat messages, allowing you to control a system message, user message, and assistant message.

llm-exe provides a prompt interface to simplify working with prompts. Ultimately a prompt is a string, but building elaborate prompts can quickly get complicated. The prompt utility provides a foundation for building complex prompts. For the recommended way to organize templates in a real app, see [Separate Prompts from Business Logic](/examples/prompt/separate-prompts-from-code).

- Support for text-based or chat-based prompts.
- Uses Handlebars as template engine, allowing you to use features such as custom templates, partials, functions, etc. See [handlebars documentation](https://handlebarsjs.com/guide/) for everything you can do.
- Infers types when they are provided.

Note: You can use and call methods on prompts directly, but they are usually passed to an LLM executor and then called internally.

There are 2 types of prompts included, along with a `BasePrompt` class that can be extended, if needed.

See:

- [Text Prompt](/prompt/text)
- [Chat Prompt](/prompt/chat)
- [Prompt Validation](/prompt/validation)

## Basic Replacements

The object that you pass to `prompt.format` (or `.execute` when a prompt is part of an LLM executor) gets passed to the template engine, making all those variables available to you in your prompt template.

<GenericOutput example="prompt.basic.exampleOne">

<<< ../../examples/prompt/basic.ts#exampleOne
</GenericOutput>

For advanced uses and working with custom helpers/partials, [see here](/prompt/advanced).

## Validating Template Inputs

Prompts can validate that required template variables and helpers are available before rendering. Use `prompt.validate(input)` directly, or set `validateInput: "strict"` / `"warn"` on a prompt to check calls to `format()`.

See [Prompt Validation](/prompt/validation).

## Using Types with Prompts

<GenericOutput example="prompt.basic.exampleTwo">

<<< ../../examples/prompt/basic.ts#exampleTwo
</GenericOutput>

---


# Chat Prompt

The other type of prompt is a chat prompt. The chat prompt can be used with chat-based models from any supported provider (OpenAI, Anthropic, Google, etc.).

You create a chat prompt using `createPrompt("chat")` or `createChatPrompt()`.

<GenericOutput example="prompt.chat.exampleCreateChatPrompt">

<<< ../../examples/prompt/chat.ts#exampleCreateChatPrompt
</GenericOutput>

When creating a chat prompt, you can optionally set an initial system message.

<GenericOutput example="prompt.chat.exampleCreateChatPromptWithSystem">

<<< ../../examples/prompt/chat.ts#exampleCreateChatPromptWithSystem
</GenericOutput>

To use the prompt as input to an LLM, you can call the `format()` method on the prompt. The format method accepts an object, which is used to supply the prompt template with replacement values.

<GenericOutput example="prompt.chat.exampleCreateChatPromptUseFormat">

<<< ../../examples/prompt/chat.ts#exampleCreateChatPromptUseFormat
</GenericOutput>

Chat prompts support more than just a basic text-based message. You can also add assistant and user content.

<GenericOutput example="prompt.chat.exampleCreateChatPromptWithAssistant">

<<< ../../examples/prompt/chat.ts#exampleCreateChatPromptWithAssistant
</GenericOutput>

See [prompt templates](/prompt/advanced) for more advanced prompt usage.

## Chat Prompt Methods

`addUserMessage`
Appends a user message to the prompt.

`addAssistantMessage`
Appends an assistant message to the prompt.

`addSystemMessage`
Appends a system message to the prompt.

`addFromHistory`
Appends an array of existing chat history messages to the prompt.

`format`
Format the prompt for LLM. This processes the template as a handlebars template.

<!-- `validate`
Validate the prompt. Makes sure there are no unresolved tokens. -->

---


# Text Prompt
The text prompt returns a single formatted string, useful for simple completion-style prompts or when you need the prompt as a plain string rather than structured chat messages.

You create a prompt using `createPrompt()`.

<GenericOutput example="prompt.text.exampleCreateTextPrompt">

<<< ../../examples/prompt/text.ts#exampleCreateTextPrompt
</GenericOutput>


When creating a prompt, you can optionally set an initial message.

<GenericOutput example="prompt.text.exampleCreateTextPromptWithInitial">

<<< ../../examples/prompt/text.ts#exampleCreateTextPromptWithInitial
</GenericOutput>


You can also add additional content to the prompt by calling `addToPrompt()` on the prompt.

<GenericOutput example="prompt.text.exampleCreateTextPromptAddToPrompt">

<<< ../../examples/prompt/text.ts#exampleCreateTextPromptAddToPrompt
</GenericOutput>


To use the prompt as input to an LLM, you can call the `format()` method on the prompt. The format method accepts an object, which is used to supply the prompt template with replacement values.

<GenericOutput example="prompt.text.exampleCreateTextPromptFormat">

<<< ../../examples/prompt/text.ts#exampleCreateTextPromptFormat
</GenericOutput>


Prompt methods are chainable
<GenericOutput example="prompt.text.exampleCreateTextPromptChainable">

<<< ../../examples/prompt/text.ts#exampleCreateTextPromptChainable
</GenericOutput>


By default, formatted text prompt messages are separated using 2 line breaks (\\n\\n). You can override this by defining a custom separator.

<GenericOutput example="prompt.text.exampleCreateTextPromptCustomDelimiter">

<<< ../../examples/prompt/text.ts#exampleCreateTextPromptCustomDelimiter
</GenericOutput>

See [prompt templates](/prompt/advanced) for more advanced prompt usage.

#### Text Prompt Methods

**.addToPrompt()**
Adds content to the prompt.
@param `content` {string} The content to be added to the prompt.

**.registerPartial()**
`partials` {<{template: string; name: string;}>} Additional partials that can be made available to the template parser.

**.registerHelpers()**
`helpers` {<{handler: function; name: string;}>} Additional helper functions that can be made available to the template parser.

**.format()**
`format`
Processes the prompt template and returns prompt ready for LLM.
---


# Prompts with Templates

Handlebars is used as a template engine when generating the prompt, so you can take advantage of advanced template features in the prompt. See full capabilities [here](https://handlebarsjs.com/guide/).

Below is a simple example showing syntax for replacing simple variables in the template.

<GenericOutput example="prompt.advanced.withReplacements">

<<< ../../examples/prompt/advanced.ts#withReplacements
</GenericOutput>

Here is a more advanced example showing a template that uses the `if` and `each` helpers supplied by Handlebars.
<GenericOutput example="prompt.advanced.withReplacementsTwo">

<<< ../../examples/prompt/advanced.ts#withReplacementsTwo
</GenericOutput>

Below is a robust example showing multiple variables, and defining types.

<GenericOutput example="prompt.advanced.withReplacementsAndTypes">

<<< ../../examples/prompt/advanced.ts#withReplacementsAndTypes
</GenericOutput>

## Prompt Template Default Helpers

Prompts are powered by handlebars, and you are able to register your own custom helpers, adding super powers to your prompt templates. Some core helpers are included by default.

- pluralize
- eq
- neq
- ifCond

## Prompt Template Default Partials

Some core partials are included by default:

### MarkdownCode

::: code-group

```[Template]
{{> MarkdownCode code='const name="Greg";' language='typescript' }}
```

```[Parsed]
'''typescript
const name="Greg";
'''
```

:::

### DialogueHistory

::: code-group

```txt [Raw Template]
// Basic example
{{> DialogueHistory key='keyOfTheChatHistory' }}

//  With title
{{> DialogueHistory title='The conversation is below:' key='keyOfTheChatHistory' }}

// Setting user name
{{> DialogueHistory key='keyOfTheChatHistory' user='Greg' }}

// Setting user and assistant - be creative!
{{> DialogueHistory key='keyOfTheChatHistory' assistant='Thought' user='Observation' }}


// Assuming you passed the following:
{
  keyOfTheChatHistory: [{
    role: "user",
    content: "Hello?",
  },{
    role: "user",
    content: "Hi! How can I help you?",
  },{
    role: "user",
    content: "I was wondering if you were open",
  },{
    role: "user",
    content: "We sure are!",
  }]
}
```

```txt [Parsed]
// Basic example
User: Hello?
Assistant: Hi! How can I help you?
User: I was wondering if you were open
Assistant: We sure are!

//  With title
The conversation is below:
User: Hello?
Assistant: Hi! How can I help you?
User: I was wondering if you were open
Assistant: We sure are!

// Setting user name
Greg: Hello?
Assistant: Hi! How can I help you?
Greg: I was wondering if you were open
Assistant: We sure are!

// Setting user and assistant - be creative!
Thought: Hello?
Observation: Hi! How can I help you?
Thought: I was wondering if you were open
Observation: We sure are!

```

:::

<!-- ### JsonSchema
```:no-line-numbers
```
### JsonSchemaExampleJson
```:no-line-numbers
``` -->

## Template Custom Partials & Helpers

You can load custom Handlebars partials and helpers a few different ways:

### 1. Pass them in when initializing the prompt

```ts
// these could be managed elsewhere and imported here
const helpers = [
  {
    handler: (date: Date) => "morning",
    name: "getTimeOfDay",
  },
];

const partials = [
  {
    template: `Phone: 1-800-000-0000
Support Email: support@example.com
Website: www.example.com`,
    name: "partialContactInformation",
  },
];

// pass in `{ helpers, partials }` when creating prompt.
const prompt = createPrompt("text", "You are a cowboy.", { helpers, partials });
```

### 2. Register them globally

If you want helpers or partials available to all prompts in your application, use the standalone `registerHelpers` and `registerPartials` utility functions. These register with the global Handlebars instance, so any prompt created afterward can use them.

```ts
import { registerHelpers, registerPartials } from "llm-exe";

// Register custom helpers globally
registerHelpers([
  {
    name: "uppercase",
    handler: (str: string) => str.toUpperCase(),
  },
]);

// Register custom partials globally
registerPartials([
  {
    name: "greeting",
    template: "Hello {{name}}!",
  },
]);

// Now any prompt can use them
const prompt = createChatPrompt("{{> greeting name='World'}} {{uppercase 'hello'}}");
```

---


# Why Handlebars?

When preparing your prompts, it may seem easier to use native Javascript template literals within your prompts instead of utilizing Handlebars. It is suggested you embrace the Handlebars approach.

### 1. Separation of Logic and Data

Handlebars templates keep presentation (template) separate from code and data, promoting clearer separation of concerns. With template literals, you often end up mixing logic and string construction in your JavaScript.

✅ **Using Handlebars the llm-exe way**

```ts
// the prompt is just a string!
const instruction = `You are answering a user's question.

{{#if user.isFirstTime}}
Give a detailed beginner-friendly answer.
{{else}}
Give a short expert-level summary.
{{/if}}

Question:
{{question}}`;

const prompt = createPrompt("text", instruction);

// data gets passed in later, when we need to render the template
// these values do not need to be defined when crafting the prompt
const formatted = prompt.format({
  user: { isFirstTime: true },
  question: "What is machine learning?",
});
```

❌ **The template literal way**

The data you are referencing in your prompt must be in the same scope - it must be defined when the prompt is defined.

```ts
const user = { isFirstTime: true };
const question = "What is machine learning?";

const output = `
You are answering a user's question.

${
  user.isFirstTime
    ? "Give a detailed beginner-friendly answer."
    : "Give a short expert-level summary."
}

Question:
${question}`;
```

::: tip
Since your template is disconnected from your data, you can even store your prompts in separate files - for example import them from text files - like from an s3 bucket.
:::

### 2. Logic Helpers

Handlebars supports built-in helpers like `if`, `each`, and even custom helpers, all directly inside the template. This lets you use loops, conditions, and transformations which are not possible with vanilla template literals.

✅ **Using Handlebars the llm-exe way**

The prompt is just a string. So clean!

```
{{#if user.isResearcher}}
Summarize the following studies in a formal tone.
{{else}}
Summarize the following studies in simple language.
{{/if}}

Studies to summarize:
{{#each studies}}
- {{this}}
{{/each}}
```

❌ **The template literal way**

🤮

```js
const output = `
${user.isResearcher
  ? "Summarize the following studies in a formal tone."
  : "Summarize the following studies in simple language."
}

Studies to summarize:
${studies.map((study) => `- ${study}`).join("\n")}`;
```


### 3. Reusability via Partials

Handlebars allows you to define **partials** reusable chunks of template. This is great for shared layouts or repeated sections, and is not natively supported by template literals.

```ts
utils.registerPartials([{ name: 'greeting', template: `Hello {{name}}!` }])

const instruction = `
{{> greeting name=userName}}

Please answer the following questions:
- What are your goals for the next 6 months?
- How can we support you best?`
```

With template literals - have to manually build and call functions with js/ts:

```ts
function greeting(name) {
  return `Hello ${name}!`;
}

const output = `
  ${greeting("Greg")}

Please answer the following questions:
  - What are your goals for the next 6 months?
  - How can we support you best?`;
```

### 4. Harness llm-exe prompt types

llm-exe prompts are designed with Handlebars in mind - by allowing you to declare types when defining your prompt. The types get inferred when calling the `format` method on the prompt and when using the prompt in an [llm-executor](/executor/) and calling `execute`.

```ts
interface SomePromptInput {
  user: {
    isAdmin: boolean;
    name: string;
  };
  tasks: string[];
}

const prompt = createChatPrompt<SomePromptInput>(`Your name is {{agentName}}.

You will help the user learn about {{topic}}.`);
```

![Screenshot of a code editor showing the createChatPrompt generic accepting a typed input object, with autocomplete suggesting agentName, topic, user, and tasks fields](/images/prompt-showing-types-being-passed-in.jpg)

And an error if you pass in the wrong input:

![Screenshot of the same prompt call with the wrong input shape, showing a TypeScript compiler error highlighting the missing required properties](/images/prompt-showing-types-being-passed-in-with-error.jpg)

### More Readable and Maintainable

Whether you are working with simple, or complex templates, Handlebars syntax is often more readable, especially for non-developers or when templates are maintained separately from code.

---


# Parser

When calling LLM's the response is ultimately a string. While you can (and will) instruct the LLM to respond with a number, or formatted as JSON... the response will still be a string.

Parsers are used to take the output from the LLM, and format it into a data type that is usable by your application.

There are various default parsers included, and the parser class is easily extendable.

When combined with an LLM executor, the parser is responsible for providing type hints to the Typescript compiler on the expected output for the LLM executor.

**Note**: You can use and call methods on parsers directly, but they are usually passed to an [LLM executor](/executor/) and then called internally.

## Getting Started

### Working with Parsers

When working with output parsers, you have two options:

1. Use a [default parser](/parser/included-parsers).
2. Create a [custom parser](/parser/custom) for full control over output transformation.

#### Use a Default Parser

```ts
const parser = createParser("listToArray"); // see list of included parsers

// example output string from LLM
const exampleOutputFromLlm = `First, hover the services menu
Wait for the dropdown menu to appear
Click on the development link`;

// listToArray will split a string on line breaks
const parsed = parser.parse(exampleOutputFromLlm);

/**
 *
 * console.log(parsed);
 *
 * [
 *   "First, hover the services menu",
 *   "Wait for the dropdown menu to appear",
 *   "Click on the development link"
 * ]
 *
 */
```

#### Using a Parser with Schema

When instructing the LLM to respond with json or a format that can be parsed to json, it can be helpful to define schema. This allows you to validate, provide default values, and have a fully-typed response. In fact, the JSON Schema you define can be really useful (and re-used!) in your prompt. [See tips](/examples/concepts/working-with-json) for working with JSON.

##### `defineSchema`

The `defineSchema` helper narrows a JSON Schema definition to its literal type so TypeScript can infer the exact shape of the parsed output. It also sets `additionalProperties: false` to ensure strict validation. Use it whenever you define schemas for JSON parsers.

```ts
import { defineSchema, createParser } from "llm-exe";

const schema = defineSchema({
  type: "object",
  properties: {
    statement: { type: "string", default: "" },
    answer: { type: "string", default: "" },
    confidence: { type: "integer", default: 0 },
  },
  required: ["statement", "answer", "confidence"],
});

const parser = createParser("listToJson", { schema });

const exampleOutputFromLlm = `Statement: The included document contains PII\nAnswer: No\nConfidence: 90`;

const parsed = parser.parse(exampleOutputFromLlm);
/**
 *
 * console.log(parsed);
 *
 * {
 *   "statement": "The included document contains PII",
 *   "answer": "No",
 *   "confidence": 90
 * }
 *
 */
```

::: warning Schema validation is on by default
When you provide a `schema`, the parser enforces it on the LLM's output by default (`validateSchema` defaults to `true`). Missing `required` fields or constraint violations throw `parser.schema_validation_failed` instead of returning a partial object, and defaults are applied **after** validation. Pass `validateSchema: false` to opt into filter/default-only behavior. See the [JSON parser reference](/parser/included-parsers.html#json) for details.
:::

---


# Default Included Parsers

[[toc]]

## String Parser

`string`
The default parser. Doesn't really parse, it passes through the string response with no modification.
Returns: string

```ts
const parser = createParser("string");
```

::: code-group

```[Output]
This is an example input message.
```

```[Response]
This is an example input message.
```

:::

## Number Parser

`number`
Extracts exactly one number from the LLM response.
Returns: number

```ts
const parser = createParser("number");
```

Options:
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `match` | `"extract" \| "exact"` | `"extract"` | `"extract"` finds one numeric token in the response. `"exact"` requires the entire trimmed response to be one numeric token. |

::: code-group

```[Output]
42
```

```[Response]
The answer is 42.
```

:::

## Boolean Parser

`boolean`
Parses a boolean value from the LLM response. Recognizes common truthy/falsy patterns like "true", "false", "yes", "no".
Returns: boolean

```ts
const parser = createParser("boolean");
```

Options:
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `match` | `"exact" \| "extract"` | `"exact"` | `"exact"` requires the entire trimmed response to be one boolean literal. `"extract"` finds one boolean literal in surrounding text. |

See a complete recipe using this parser: [Get a Yes/No Decision from an LLM](/examples/bots/yes-no).

::: code-group

```[Output]
true
```

```[Response]
yes
```

:::

## String Extractor Parser

`stringExtract`
Use this parser to ensure the response is one of many specific strings you define. This parser does not return the LLM's actual response, but works through the `enum` you provide and looks for a match. When it finds one, it returns the `enum` value, ensuring the response is exactly as expected.

Returns: string

> **Example Prompt:** <br>You need to reply with one of three options. Either stop, go forward, turn left, turn right.

```ts
const parser = createParser("stringExtract", {
  enum: ["stop", "go forward", "turn left", "turn right"],
});
```

Options:
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `enum` | `string[]` | `[]` | The list of allowed values to match against. |
| `ignoreCase` | `boolean` | `true` | When `true`, matching is case-insensitive. |
| `match` | `"word" \| "substring" \| "exact"` | `"word"` | `"word"` matches enum values on word boundaries. `"substring"` uses contains matching. `"exact"` requires the entire trimmed response to equal one enum value. |

::: code-group

```[Output]
go forward
```

```[Response]
Go Forward.
```

:::

## List to Array

`listToArray`
Converts a list (separated by \n) to an array of strings.
Returns: string[]

> **Example Prompt:** <br>You need to reply with a list of test cases that should be written for the code I included below. You must reply in an unordered list.

```ts
const parser = createParser("listToArray");
```

::: code-group

```[Output]
[
 "Should return the default name if the function argument is null or undefined",
 "Should return function's name if function has a name property",
 "Should return the correct name if function is bound to an object",
 "Should return the correct name if function is anonymous",
 "Should return the correct name if the function's source has additional space",
 "Should return empty string if function has no applicable name.",
]
```

```[Response]
- Should return the default name if the function argument is null or undefined
- Should return function's name if function has a name property
- Should return the correct name if function is bound to an object
- Should return the correct name if function is anonymous
- Should return the correct name if the function's source has additional space
- Should return empty string if function has no applicable name.
```

:::

## List to Key/Value[]

`listToKeyValue`
Converts a list of `key: value` pairs (separated by newlines) to an array of key/value objects.
Returns `Array<{ key: string; value: string; }>`

::: code-group

```[Output]
[{
    "key": "Getting Started",
    "value": "To get started, we need to..."
},{
    "key": "Setting Up Your Account",
    "value": "To set up your account, you need to..."
}]
```

```[Response]
Getting Started: To get started, we need to...
Setting Up Your Account: To set up your account, you need to...
```

:::

## Markdown Code Block

`markdownCodeBlock`
Extracts the **first** code block from the LLM response, including the language identifier. If the response contains no code blocks, returns `{ code: "", language: "" }`. For extracting all code blocks, see [`markdownCodeBlocks`](#markdown-code-blocks) below.
Returns: `{ code: string; language: string; }`

```ts
const parser = createParser("markdownCodeBlock");
```

::: code-group

```[Output]
{
    "code": "function add(a: number, b: number){\nreturn a + b;\n}",
    "language": "typescript"
}
```

````[Response]
Below is the generated code:

```typescript
function add(a: number, b: number){
    return a + b;
}
```
````

:::

## Markdown Code Blocks

`markdownCodeBlocks`
Extracts **all** code blocks from the LLM response, returning them as an array. Use this when the response may contain multiple code blocks. For extracting only the first code block, see [`markdownCodeBlock`](#markdown-code-block) above.
Returns: `Array<{ code: string; language: string; }>`

```ts
const parser = createParser("markdownCodeBlocks");
```

::: code-group

```[Output]
[{
    "code": "function add(a: number, b: number){\nreturn a + b;\n}",
    "language": "typescript"
},
{
    "code": "function subtract(a: number, b: number){\nreturn a - b;\n}",
    "language": "typescript"
}]
```

````[Response]
Below is the generated code:

```typescript
function add(a: number, b: number){
    return a + b;
}
```

And the next:

```typescript
function subtract(a: number, b: number){
    return a - b;
}
```
````

:::

## Replace String Template

`replaceStringTemplate`
Runs Handlebars substitution on the LLM's output, using the executor's input attributes as template data. This lets the LLM return a template string that gets filled in with the original input variables before being returned to the caller.

Returns: string

```ts
const parser = createParser("replaceStringTemplate");
```

**Use case:** The LLM generates a response containing placeholders, and the parser fills them in with known values from the executor input — useful for personalized messages, dynamic templates, or mail-merge patterns.

::: code-group

```[Output]
Hello Alice! Your order #12345 has shipped.
```

```[Response]
Hello {{name}}! Your order #{{orderId}} has shipped.
```

```[Attributes passed to parser]
{ name: "Alice", orderId: "12345" }
```

:::

## List to JSON

`listToJson`
Parses key:value lines (separated by newlines) into a **single flat object**. Each line becomes one property — the key is derived from the text before the first colon, and the value is the text after it. Keys are camelCased by default.

Returns: object (flat key-value pairs)

::: warning
This parser produces a single object, not an array. If the LLM output contains duplicate keys, later values overwrite earlier ones. If you need to preserve multiple records with the same keys, use [`listToKeyValue`](#list-to-key-value) instead (which returns an array of `{ key, value }` pairs).
:::

> **Example Prompt:** <br>Extract the following information. Reply only with: Color: the color\nName: the name\nType: the type

```typescript
const parser = createParser("listToJson");
```

Options:
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `keyTransform` | `"camelCase" \| "preserve"` | `"camelCase"` | How to transform keys. `"camelCase"` converts keys like "First Name" to "firstName". `"preserve"` keeps the original key text (trimmed). |
| `schema` | `JSONSchema` | `undefined` | Optional JSON Schema to validate and enforce types on the output. |
| `validateSchema` | `boolean` | `true` (when `schema` is set) | When a schema is provided, `required` fields and type/constraint checks are enforced by default. Set to `false` to opt out into filter/default-only behavior (strips unknown keys and applies defaults, but does **not** check `required`). |

::: code-group

```[Output]
{
    "color": "red",
    "name": "apple",
    "type": "fruit"
}
```

```[Response]
Color: Red
Name: Apple
Type: Fruit
```

:::

**Choosing between `listToJson` and `listToKeyValue`:**

| Parser | Returns | Best for |
| --- | --- | --- |
| `listToJson` | `{ key: value, ... }` (flat object) | Extracting a fixed set of unique fields from a response |
| `listToKeyValue` | `Array<{ key, value }>` | Preserving order, handling duplicate keys, or iterating over pairs |

## JSON

`json`
Parse an expected stringified JSON object or array into a valid object. Schema can be passed in to enforce schema and provide default values.
Returns: object | array

```ts
const parser = createParser("json");
```

Options:
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `schema` | `JSONSchema` | `undefined` | Optional JSON Schema to validate and enforce types on the output. Use with [`defineSchema`](/parser/#defineschema) for full type inference. |
| `validateSchema` | `boolean` | `true` (when `schema` is set) | When a schema is provided, `required` fields and type/constraint checks are enforced by default — invalid or incomplete input throws `parser.schema_validation_failed`. Set to `false` to opt out into filter/default-only behavior (strips unknown keys and applies defaults, but does **not** check `required`). |
| `match` | `"exact" \| "extract"` | `"exact"` | `"exact"` parses the entire response as JSON. `"extract"` finds one JSON object or array in surrounding text. |

::: warning
Providing a `schema` enforces `required` fields by default. Input missing a required field throws rather than returning a partial object. Defaults are applied **after** validation, so a `default` does not satisfy a `required` field. Pass `validateSchema: false` only if you intentionally want the legacy strip-and-default behavior without `required` enforcement.
:::

::: code-group

```[Output]
{
    "name": "Greg",
    "age": "89"
}
```

```[Response]
\`\`\`json
{ "name": "Greg", "age": "89" }
\`\`\`

or

{ \"name\": \"Greg\", \"age\": \"89\" }
```

:::


---


# Custom Parser

You can define a custom parser to have full control over transforming the LLM output into the format you expect.

A custom parser is a function that receives two arguments — the LLM's text response and an executor context object — and returns a transformed result.

When using TypeScript, the return type of your parser function flows through to the executor, so `executor.execute()` returns the correct type automatically.

## Defining a custom parser

```ts
import { createCustomParser } from "llm-exe";

// The return type of the handler becomes the parser's output type.
const customParser = createCustomParser(
  "extractEmails",
  (text: string, context) => {
    // text = the raw LLM response string
    // context = the executor context (input values, metadata)
    const emails = text.match(/[\w.-]+@[\w.-]+\.\w+/g) || [];
    return emails; // string[]
  }
);
```

## Using a custom parser

```ts
// Use .parse() directly
const emails = customParser.parse("Contact us at help@example.com", {} as any);
// emails: string[]

// Or use it in an executor — the executor infers the return type
const executor = createLlmExecutor({
  llm: useLlm("openai.chat.v1", { model: "gpt-4o-mini" }),
  prompt: createChatPrompt(`Extract all email addresses from: {{text}}`),
  parser: customParser,
});

const result = await executor.execute({ text: "Email me at hello@test.com" });
// result is typed as string[] — inferred from the parser
```

## Type inference

The key benefit of custom parsers is that their return type propagates to the executor:

```ts
// Parser returns a specific interface
interface SentimentResult {
  sentiment: "positive" | "negative" | "neutral";
  confidence: number;
}

const sentimentParser = createCustomParser(
  "sentiment",
  (text: string): SentimentResult => {
    const parsed = JSON.parse(text);
    return {
      sentiment: parsed.sentiment,
      confidence: parsed.confidence,
    };
  }
);

const executor = createLlmExecutor({
  llm: useLlm("openai.chat.v1", { model: "gpt-4o-mini" }),
  prompt: createChatPrompt(`Analyze sentiment of: {{text}}`),
  parser: sentimentParser,
});

const result = await executor.execute({ text: "I love this!" });
// result is typed as SentimentResult
```

## Using the `CustomParser` class directly

You can also instantiate the `CustomParser` class directly instead of using `createCustomParser`:

```ts
import { CustomParser } from "llm-exe";

const parser = new CustomParser("myParser", (text: string, context) => {
  return text.toUpperCase();
});
```

Both approaches produce the same result — `createCustomParser` is a convenience wrapper around `new CustomParser()`.

---


# State

When calling an LLM from your code, the LLM only knows the history you provide it.

When using llm-exe, the state is something you need to manage. To help provide a concept of memory to your LLM's, we provide a simple state module. 

The state module consists of 3 concepts:
- Dialogues
- Context
- Attributes

Dialogues are a place to store conversation history, internal dialogues, really any conversation that is taking place with an LLM. You can have one or many dialogues. When you create a new dialogue, you should provide a key, which allows you to access the dialogue from the state later if needed.

**Context** items are instances of `BaseStateItem` — typed classes with `getValue()`, `setValue()`, and `resetValue()` methods. Use context for structured, typed data that needs its own lifecycle (e.g., extracted entities, session config). Create context items with `createStateItem(name, defaultValue)` and add them via `state.createContextItem(item)`.

```ts
import { createState, createStateItem } from "llm-exe";

const state = createState();

// Create a typed context item with a name and default value
const userIntent = createStateItem("userIntent", "unknown");

// Add it to the state
state.createContextItem(userIntent);

// Use getValue/setValue to manage the item
userIntent.getValue();    // "unknown"
userIntent.setValue("booking");
userIntent.getValue();    // "booking"
userIntent.resetValue();  // resets to "unknown"
```

**Attributes** are a simple key-value store for lightweight metadata. Use `state.setAttribute(key, value)`, `state.deleteAttribute(key)`, and `state.clearAttributes()`.

## Creating State

Initializing a state object is simple:
```ts
import { createState } from "llm-exe";

const state = createState();
```

## Dialogues

If you want to store a chat conversation dialogue, create one on the state:
```ts
const state = createState();

// this creates a new dialogue in the state, and returns it
const chatHistory = state.createDialogue("chatHistory");

// add messages directly
chatHistory.setUserMessage("Hey anyone there?");

// or retrieve from state later
state.getDialogue("chatHistory").setAssistantMessage("Yep! What's up?");
```

You can also create a standalone dialogue without state using `createDialogue`. See the [Dialogue](/state/dialogue) page for full details.

## Saving State

The `DefaultState` class implements `saveState()` with a warning log by default. To persist state, extend `DefaultState` and override it with your own save logic:

```ts
import { DefaultState } from "llm-exe";

class MyState extends DefaultState {
  async saveState() {
    const data = {
      dialogues: this.dialogues,
      attributes: this.attributes,
    };
    await db.save("state", JSON.stringify(data));
  }
}
```
---


# Dialogue

The `Dialogue` class manages conversation history for multi-turn LLM interactions. It's used internally by `state`, but you can also use it independently with `createDialogue`.

## Creating a dialogue

```ts
import { createDialogue } from "llm-exe";

const dialogue = createDialogue("chat");
```

## Adding messages

All message methods return `this` for chaining:

```ts
dialogue
  .setSystemMessage("You are a helpful assistant.")
  .setUserMessage("What is TypeScript?")
  .setAssistantMessage("TypeScript is a typed superset of JavaScript.");
```

### Available methods

| Method | Description |
|--------|-------------|
| `setUserMessage(content, name?)` | Add a user message. Content can be a string or detailed content array. |
| `setAssistantMessage(content)` | Add an assistant message. Accepts a string or `OutputResultsText`. |
| `setSystemMessage(content)` | Add a system message. |
| `setToolMessage(content, name, id?)` | Add a tool/function result message. |
| `setToolCallMessage({ name, arguments, id? })` | Add a tool call (function invocation) message. |
| `setFunctionCallMessage(input)` | Add a function call message (legacy format also supported). |
| `setMessageTurn(user, assistant, system?)` | Add a complete turn (user + assistant + optional system). |
| `setHistory(messages)` | Load an array of messages into the dialogue. |
| `getHistory()` | Get all messages as an `IChatMessages` array. |
| `addFromOutput(output)` | Add messages from an LLM `OutputResult` or `BaseLlCall`. |

## Adding LLM output directly

After making an LLM call, you can add the response to dialogue history automatically:

```ts
const output = await llm.call(prompt);
dialogue.addFromOutput(output);
```

This handles both text responses and tool/function calls, adding them in the order they were returned.

## Loading existing history

Use `setHistory` to load a full conversation from an array:

```ts
dialogue.setHistory([
  { role: "user", content: "Hi" },
  { role: "assistant", content: "Hello! How can I help?" },
  { role: "user", content: "Tell me a joke" },
]);
```

## Using with state

Dialogues integrate with the state module:

```ts
import { createState } from "llm-exe";

const state = createState();
const chatHistory = state.createDialogue("chatHistory");

chatHistory.setUserMessage("Hello!");
chatHistory.setAssistantMessage("Hi there!");

// Retrieve later
const history = state.getDialogue("chatHistory").getHistory();
```

## Examples

<GenericOutput example="state.dialogue.basic.exampleOne">

<<< ../../examples/state/dialogue/basic.ts#exampleOne
</GenericOutput>


<GenericOutput example="state.dialogue.functions.dialogueWithFunctionCall">

<<< ../../examples/state/dialogue/functions.ts#dialogueWithFunctionCall
</GenericOutput>

---


# LLM Executor

The LLM executor takes an [llm](/llm/), a [prompt](/prompt/), optionally a [parser](/parser/), and wraps in a well-typed executor function. An LLM executor is a _container_ that can be used to call an LLM with a pre-defined input and output; with additional values provided at the time of execution. The executor is responsible for combining and calling the provided components, with added tracing, metadata, and extendable hooks.

An LLM executor's input and output types are determined by the prompt and parser respectively. These are inferred automatically when working with Typescript. See working with types in prompts and parsers.

## Llm Executor Input

**llm** (required) `instance of BaseLlm`. Use `useLlm()` to create one.

**prompt** (required) `instance of BasePrompt` Either text or chat, respective of the LLM.

**parser** (optional) defaults to string if not provided.

**hooks** (optional) allows you to hook into various stages of the execution. Most useful for logging, but may be used for additional plugins.

## Basic Example

```typescript
import { useLlm, createChatPrompt, createLlmExecutor } from "llm-exe";

const llm = useLlm("openai.gpt-4o-mini");
const instruction = `You are a customer support agent. Reply to the user's message.\n{{input}}`;
const prompt = createChatPrompt<{ input: string }>(instruction);

const executor = createLlmExecutor({
  llm,
  prompt,
});

const response = await executor.execute({ input: "Hello!" });
```

## Basic Example with All Options

```typescript
import { useLlm, createChatPrompt, createParser, createLlmExecutor } from "llm-exe";

const llm = useLlm("openai.gpt-4o-mini");
const instruction = `You are a customer support agent. Reply to the user's message as JSON.\n{{input}}`;
const prompt = createChatPrompt<{ input: string }>(instruction);
const parser = createParser("json");

const hooks = {
  onComplete() {},
  onSuccess() {},
  onError() {},
};

const executor = createLlmExecutor(
  {
    llm,
    prompt,
    parser,
  },
  { hooks }
);

executor.on("onComplete", () => {});
executor.once("onComplete", () => {});

const response = await executor.execute({ input: "Hello!" });
```

`createLlmExecutor` Returns an instance of LlmExecutor.

## Function Executor

If you need tool/function calling support, use `createLlmFunctionExecutor`. It extends the standard LLM executor with the ability to define functions the LLM can call. This works with any provider that supports tool calling (OpenAI, Anthropic, Google, etc.).

```typescript
import { useLlm, createChatPrompt, createLlmFunctionExecutor } from "llm-exe";

const llm = useLlm("openai.gpt-4o-mini");
const prompt = createChatPrompt("You are a helpful assistant.");

const executor = createLlmFunctionExecutor({
  llm,
  prompt,
});

const response = await executor.execute(
  { input: "What's the weather?" },
  {
    functionCall: "auto",
    functions: [
      {
        name: "get_weather",
        description: "Get the current weather",
        parameters: { /* JSON Schema */ },
      },
    ],
  }
);
```

`createLlmFunctionExecutor` Returns an instance of LlmExecutorWithFunctions.

See [Tool Calling Executor](/executor/openai-functions) for full documentation and examples.

## Core Executor

If you need a typed executor that wraps a plain function (no LLM involved), use `createCoreExecutor`. This is useful for composing non-LLM steps alongside LLM executors in a pipeline — the core executor provides the same `execute` interface, tracing, and hooks as an LLM executor.

```typescript
import { createCoreExecutor } from "llm-exe";

const executor = createCoreExecutor(
  async (input: { text: string }) => {
    return { wordCount: input.text.split(" ").length };
  }
);

const result = await executor.execute({ text: "Hello world from llm-exe" });
// result: { wordCount: 4 }
```

`createCoreExecutor` Returns an instance of CoreExecutor.

---


# Tool Calling Executor

To take advantage of tool calling with OpenAI, Anthropic, and other providers that support it, you can use `createLlmFunctionExecutor` or the `LlmExecutorWithFunctions` class directly. It works exactly like a regular [llm executor](/executor/) — it extends the class and adds options with some additional type constraints.

::: warning Deprecated Export
`LlmExecutorOpenAiFunctions` is deprecated and will be removed in a future major version. Use `LlmExecutorWithFunctions` or `createLlmFunctionExecutor` instead — they support tool calling across all providers, not just OpenAI.
:::

## Basic Example

```ts{13,14,15,16,17,18,19,20,21,26,27}
const llm = useLlm("openai.gpt-4o-mini");
const instruction = `You are walking through a maze.
You must take one step at a time.
Pick a direction to move.`;

const prompt = createChatPrompt(instruction);

// Using the factory function (recommended)
const executor = createLlmFunctionExecutor({
  llm,
  prompt,
})

// Or using the class directly
// const executor = new LlmExecutorWithFunctions({ llm, prompt })

const functions = [{
    name: "move_left",
    description: "move one block to the left",
    parameters: {/* options, as JSON Schema */}
},{
    name: "move_right",
    description: "move one block to the right",
    parameters: {/* options, as JSON Schema */}
}]

const response = await executor.execute({
  input: "Hello!"
}, {
  functionCall: "auto",
  functions: functions,
})
```

---


## LLM Hooks

**LLM hooks are lifecycle callbacks that fire at specific points around an LLM call** — before results are returned, when a call fails, and when execution finishes. Instead of wrapping every LLM call in try/catch blocks and timing code, you register a hook once and it runs on every execution. This keeps cross-cutting concerns — logging, metrics, alerting, auditing — out of your business logic.

Typical things you'd use LLM hooks for:

- **Logging** every prompt input and parsed output for debugging or audit trails
- **Latency tracking** — measure how long each LLM call takes and send it to your metrics system
- **Error monitoring** — capture failures (timeouts, parse errors, provider errors) and alert on them
- **Usage analytics** — count executions per function to see which LLM features get used

In llm-exe, every executor supports hooks. They are optional, and you can register more than one function per hook (meaning there can be many functions listening on the same hook).

The following hooks are available:

- `onSuccess` — runs after a successful execution, once the output has been produced
- `onError` — runs when the executor throws, before the error is re-thrown
- `onComplete` — runs after `onSuccess` or `onError`, regardless of outcome

### Common Patterns

Log the duration of every LLM call:

```typescript:no-line-numbers
const classifier = createLlmExecutor({ name: "classify-ticket", llm, prompt, parser });

classifier.on("onComplete", (exec, meta) => {
  logger.info(`${meta.name} finished in ${exec.end - exec.start}ms`, {
    executions: meta.executions,
  });
});
```

Alert on failures without touching the call site:

```typescript:no-line-numbers
classifier.on("onError", (exec, meta) => {
  alerting.notify(`LLM call ${meta.name} failed: ${exec.errorMessage}`, {
    code: exec.errorCode,
    input: exec.input,
  });
});
```

The executor still throws on failure — `onError` observes the error, it does not swallow it. Your calling code handles the failure; the hook handles the telemetry. For a complete production setup combining hooks with retries and timeouts, see [Add Retries and Timeouts to LLM Calls](/examples/concepts/retries-and-timeouts).

### Hook Signature

Every hook receives the same two arguments:

```typescript
type Hook = (
  executionMetadata: ExecutorExecutionMetadata,
  executorMetadata: ExecutorMetadata
) => void;
```

`executionMetadata` contains information about this specific execution (input, output, timings, and — for `onError` — the thrown error). `executorMetadata` contains information about the executor instance itself (id, type, name, total executions).

| Field                              | Available on              | Description                                      |
| ---------------------------------- | ------------------------- | ------------------------------------------------ |
| `executionMetadata.input`          | all hooks                 | The input passed to `.execute()`                 |
| `executionMetadata.output`         | `onSuccess`, `onComplete` | The parsed output returned by the executor       |
| `executionMetadata.handlerInput`   | all hooks                 | The transformed input passed to the internal handler |
| `executionMetadata.handlerOutput`  | `onSuccess`, `onComplete` | The raw handler output before parsing            |
| `executionMetadata.error`          | `onError`, `onComplete`   | The thrown `Error` instance                      |
| `executionMetadata.errorMessage`   | `onError`, `onComplete`   | Shortcut for `error.message`                     |
| `executionMetadata.errorCategory`  | `onError`, `onComplete`   | Structured category for `LlmExeError` failures  |
| `executionMetadata.errorCode`      | `onError`, `onComplete`   | Structured code for `LlmExeError` failures      |
| `executionMetadata.errorContext`   | `onError`, `onComplete`   | Structured context for `LlmExeError` failures   |
| `executionMetadata.hookErrors`     | later hooks               | Failures captured from earlier hook callbacks   |
| `executionMetadata.start`          | all hooks                 | Execution start time (ms since epoch)            |
| `executionMetadata.end`            | `onComplete`              | Execution end time (ms since epoch)              |
| `executorMetadata.id`              | all hooks                 | Stable id of the executor                        |
| `executorMetadata.name`            | all hooks                 | Name of the executor, if set                     |
| `executorMetadata.type`            | all hooks                 | Type of executor (e.g. `"llm-executor"`)         |
| `executorMetadata.created`         | all hooks                 | Timestamp when the executor was created (ms since epoch) |
| `executorMetadata.executions`      | all hooks                 | Number of times this executor has run            |

Hooks should be synchronous and lightweight. Errors thrown inside a hook are caught and collected by llm-exe. They do not affect the executor result.

### Hook Errors

If an `onSuccess` or `onError` hook throws, the failure is captured in `executionMetadata.hookErrors` for later hooks such as `onComplete`.

```typescript:no-line-numbers
executor.on("onSuccess", () => {
  throw new Error("logging failed");
});

executor.on("onComplete", (exec) => {
  console.log(exec.hookErrors?.[0]?.errorMessage);
});
```

Each hook error record includes the raw thrown value and `errorMessage`. If the thrown value is an `LlmExeError`, `errorCategory`, `errorCode`, `errorContext`, and `errorCause` are included.

`onComplete` runs last. If an `onComplete` hook throws, there is no later hook where that failure can be surfaced in execution metadata.

You can attach hooks:

- During initialization
- After initialization using `on` / `off` / `once`

#### Adding hooks during initialization

```typescript{5}:no-line-numbers
const hooks = {
  onSuccess: (exec) => console.log("output:", exec.output),
  onError: (exec) => console.error("failed:", exec.errorMessage),
};
const llm = useLlm("openai.chat-mock.v1", {});
const prompt = createChatPrompt("This is a prompt.");
const executor = createLlmExecutor({ llm, prompt }, { hooks });
```

#### Adding hooks after initialization

Use `.on()` to attach a hook, and `.off()` to remove it.

```typescript{10,11,14}:no-line-numbers
function logSuccess(exec) {
  console.log("output:", exec.output);
}

function logError(exec) {
  console.error("failed:", exec.errorMessage);
}

const executor = createLlmExecutor({ llm, prompt });
executor.on("onSuccess", logSuccess);
executor.on("onError", logError);

// Later, detach a specific listener:
executor.off("onSuccess", logSuccess);
```

#### Listening exactly once

Use `.once()` to add a hook that fires on the next execution only and then detaches itself.

```typescript{3}:no-line-numbers
const executor = createLlmExecutor({ llm, prompt });

executor.once("onComplete", (exec) => {
  console.log(`first run took ${exec.end - exec.start}ms`);
});

await executor.execute({});
await executor.execute({}); // the once handler no longer fires
```

::: tip
Default behavior is to not add the same handler to the same hook more than once. There is also a per-event hook limit to prevent unbounded memory growth — if you hit it, remove unused hooks with `.off()` rather than piling on new ones.
:::

#### Clearing hooks

Use `.clearHooks()` to remove all hooks for a specific event, or all events at once. This is useful for preventing memory leaks in long-running processes.

```typescript{3,6}:no-line-numbers
const executor = createLlmExecutor({ llm, prompt });

// Clear hooks for a specific event
executor.clearHooks("onSuccess");

// Clear all hooks across all events
executor.clearHooks();
```

#### Inspecting hook count

Use `.getHookCount()` to check how many hooks are registered, either for a specific event or across all events.

```typescript{3,6}:no-line-numbers
const executor = createLlmExecutor({ llm, prompt });

// Get count for a specific event
const count = executor.getHookCount("onSuccess"); // number

// Get counts for all events
const counts = executor.getHookCount(); // { onSuccess: 0, onError: 0, onComplete: 0 }
```

#### Trace ID

Use `.withTraceId()` to associate an executor with a trace identifier for observability. The trace ID is included in `executorMetadata` passed to hooks.

```typescript{3}:no-line-numbers
const executor = createLlmExecutor({ llm, prompt });

executor.withTraceId("req-abc-123");

executor.on("onComplete", (exec, meta) => {
  console.log(meta.traceId); // "req-abc-123"
});
```

---


# Executor Options

## Creation Options

When creating an executor with `createLlmExecutor`, you pass two arguments:

1. **Configuration object** — the LLM, prompt, parser, and optional state
2. **Options object** (optional) — hooks and other settings

```typescript
import { useLlm, createChatPrompt, createParser, createLlmExecutor } from "llm-exe";

const executor = createLlmExecutor(
  {
    llm: useLlm("openai.gpt-4o-mini"),
    prompt: createChatPrompt("Summarize: {{text}}"),
    parser: createParser("string"),
  },
  {
    hooks: {
      onSuccess: (exec) => console.log("Result:", exec.output),
      onError: (exec) => console.error("Error:", exec.errorMessage),
      onComplete: (exec) => console.log(`Done in ${exec.end - exec.start}ms`),
    },
  }
);
```

### Configuration

| Option   | Type          | Required | Description                                                    |
| -------- | ------------- | -------- | -------------------------------------------------------------- |
| llm      | `BaseLlm`    | Yes      | LLM instance created with `useLlm()`                          |
| prompt   | `BasePrompt`  | Yes      | Prompt instance, or a function that returns one                |
| parser   | `BaseParser`  | No       | Parser instance. Defaults to string parser if not provided     |
| state    | `BaseState`   | No       | State instance for managing dialogue and context               |
| name     | `string`      | No       | Name for the executor, used in tracing and metadata            |

### Hooks

See [Hooks](/executor/hooks) for full documentation on available hooks.

## Execute Options

When calling `executor.execute()`, you can optionally pass a second argument with execution-time options:

```typescript
const result = await executor.execute(
  { text: "Hello world" },
  { jsonSchema: mySchema }
);
```

| Option     | Type                      | Description                                         |
| ---------- | ------------------------- | --------------------------------------------------- |
| jsonSchema | `Record<string, any>`     | JSON schema to pass to the LLM for structured output |

---


# Embeddings

Embeddings is a wrapper around various embeddings providers, making your function implementations vendor-agnostic.

**Embeddings Features:**

- Built-in timeout mechanism for better control when a provider takes too long.
- Automatic retry with configurable back-off for errors.
- Use different LLM's with different configurations for different functions.

## Basic Usage

Use `createEmbedding` to create an embedding instance for a supported provider, then call it with the text you want to embed:

```ts
import { createEmbedding } from "llm-exe";

const embeddings = createEmbedding("openai.embedding.v1", {
  model: "text-embedding-3-small",
});

const embedding = await embeddings.call("The text you want to embed");
const vector = embedding.getEmbedding();
// Returns a number[] representing the embedding vector
```

### Parameters

`createEmbedding(provider, options)` accepts:

| Parameter | Type | Description |
|-----------|------|-------------|
| `provider` | `EmbeddingProviderKey` | The embedding provider key (see supported providers below) |
| `options` | `object` | Provider-specific options including `model` |

The returned object has a `.call(input)` method that returns a promise. The resolved result has a `.getEmbedding()` method that returns the embedding vector as `number[]`.

## Supported Embedding Providers

| Provider | Key | Details |
|----------|-----|---------|
| OpenAI (and OpenAI-compatible) | `openai.embedding.v1` | [OpenAI Embeddings](./openai.md) |
| Amazon Titan | `amazon.embedding.v1` | [Amazon Embeddings](./amazon.md) |
| Cohere (via Bedrock) | `amazon:cohere.embedding.v1` | [Cohere Embeddings](./cohere.md) |

The `openai.embedding.v1` provider accepts a `baseUrl` option, so it can also be used with any OpenAI-compatible embeddings endpoint (Baseten, Together AI, vLLM, TEI, local servers, etc.). See [OpenAI Embeddings → OpenAI-Compatible Endpoints](./openai.md#openai-compatible-endpoints).

## Adding Custom Providers
Custom embedding providers are not currently supported. If you need an embedding provider that isn't listed above, please open an issue.

---


# OpenAI Embeddings

When using OpenAI embeddings, llm-exe will make POST requests to `https://api.openai.com/v1/embeddings` by default. Override `baseUrl` to point at any OpenAI-compatible embeddings endpoint (Baseten, Together, vLLM, TEI, etc.) — see [OpenAI-Compatible Endpoints](#openai-compatible-endpoints) below.

## Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `model` | `string` | — | The OpenAI embedding model to use (e.g., `text-embedding-3-small`) |
| `dimensions` | `number` | `1536` | The number of dimensions for the output embedding |
| `encodingFormat` | `string` | — | The encoding format (e.g., `float`, `base64`) |
| `openAiApiKey` | `string` | `OPENAI_API_KEY` env var | Your OpenAI (or compatible provider) API key. Sent as `Authorization: Bearer <key>` |
| `baseUrl` | `string` | `https://api.openai.com/v1` | Base URL for the embeddings request. The final endpoint is `{baseUrl}/embeddings` |

## Basic Usage

```ts
import { createEmbedding } from "llm-exe";

const embeddings = createEmbedding("openai.embedding.v1", {
    model: "text-embedding-3-small",
});

const str = "The string of text you would like as vector";
const embedding = await embeddings.call(str);
const vector = embedding.getEmbedding();
console.log(vector);
// [
//   -0.014564549,   0.026690058,  -0.021338109,  -0.042174473,   -0.05775645,
//    -0.04404208, -0.0035819034,  -0.012320633,   -0.02112905,   0.030355586,
// ...etc
// ]
```

## Custom Dimensions

```ts
const embeddings = createEmbedding("openai.embedding.v1", {
    model: "text-embedding-3-small",
    dimensions: 512,
});
```

## OpenAI-Compatible Endpoints

The `openai.embedding.v1` provider can talk to any service that implements the OpenAI `/embeddings` API shape. Pass `baseUrl` to redirect the request, and pass your provider's API key via `openAiApiKey` (it is sent as a bearer token regardless of which provider it belongs to).

### Baseten

```ts
const embeddings = createEmbedding("openai.embedding.v1", {
  baseUrl: "https://model-xyz.api.baseten.co/environments/production/sync/v1",
  openAiApiKey: process.env.BASETEN_API_KEY,
  model: "Qwen/Qwen3-Embedding-8B",
});
```

### Together AI

```ts
const embeddings = createEmbedding("openai.embedding.v1", {
  baseUrl: "https://api.together.xyz/v1",
  openAiApiKey: process.env.TOGETHER_API_KEY,
  model: "BAAI/bge-large-en-v1.5",
});
```

### Local servers (vLLM, TEI, Ollama, LM Studio)

For local OpenAI-compatible servers, point `baseUrl` at the server and pass a placeholder key if the server doesn't require auth:

```ts
const embeddings = createEmbedding("openai.embedding.v1", {
  baseUrl: "http://localhost:8000/v1",
  openAiApiKey: "not-needed",
  model: "BAAI/bge-base-en-v1.5",
});
```

Notes:

- `{baseUrl}` is concatenated with `/embeddings`, so omit any trailing `/embeddings` from the URL.
- If your compatible provider doesn't support `dimensions` or returns a fixed vector size, the default `dimensions: 1536` may need to be overridden or left to the provider — consult the provider's docs.

---


# Amazon Embeddings

When using Amazon embeddings, llm-exe will make POST requests to the AWS Bedrock endpoint for your configured region.

## Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `model` | `string` | — | The Bedrock model ID (e.g., `amazon.titan-embed-text-v2:0`) |
| `dimensions` | `number` | `512` | The number of dimensions for the output embedding |
| `awsRegion` | `string` | `AWS_REGION` env var | The AWS region for the Bedrock endpoint (required) |
| `awsSecretKey` | `string` | — | AWS secret key (if not using default credentials) |
| `awsAccessKey` | `string` | — | AWS access key (if not using default credentials) |

## Basic Usage

```ts
import { createEmbedding } from "llm-exe";

const embeddings = createEmbedding("amazon.embedding.v1", {
  model: "amazon.titan-embed-text-v2:0",
});

const str = "The string of text you would like as vector";
const embedding = await embeddings.call(str);
const vector = embedding.getEmbedding();
console.log(vector);
// [
//  -0.08704914152622223,   0.062177956104278564,     0.0284775048494339,
//    0.0569550096988678,   0.021762285381555557,   0.046509113162755966,
// ...etc
// ]
```

---


## Callable Executor Wrapper

When building LLM agents that need to call tools or functions, `createCallableExecutor` wraps your executors (or plain functions) into a standardized interface. Use `useExecutors` to group them into a collection you can query and invoke by name.

### Creating a callable executor

Use `createCallableExecutor` to wrap a handler function or an existing executor.

**Config properties:**

| Property | Type | Required | Description |
|----------|------|----------|-------------|
| `name` | `string` | Yes | Unique name used to look up and invoke the function |
| `description` | `string` | Yes | Description of what the function does (shown to the LLM) |
| `input` | `string` | Yes | JSON-stringified schema describing the expected input shape |
| `handler` | `function \| BaseExecutor` | No | The function to execute, or an existing executor instance |
| `parameters` | `Record<string, any>` | No | Additional static parameters passed alongside the input |
| `attributes` | `Record<string, any>` | No | Metadata attributes returned with the result |
| `visibilityHandler` | `function` | No | Controls whether this function is visible in a given context |
| `validateInput` | `function` | No | Validates input before execution |

```ts
import { createCallableExecutor, createLlmExecutor, useLlm, createChatPrompt, createParser } from "llm-exe";

// Wrap a plain function
const searchCallable = createCallableExecutor({
  name: "searchInternet",
  description: "Search the internet for information",
  input: JSON.stringify({
    type: "object",
    properties: {
      query: { type: "string", description: "The search query" },
    },
  }),
  handler: async (input: { query: string }) => {
    // your search logic here
    return { results: [`Result for: ${input.query}`] };
  },
});

// Or wrap an existing LlmExecutor
const summarizeExecutor = createLlmExecutor({
  llm: useLlm("openai.chat.v1", { model: "gpt-4o-mini" }),
  prompt: createChatPrompt("Summarize: {{input}}"),
  parser: createParser("string"),
});

const summarizeCallable = createCallableExecutor({
  name: "summarize",
  description: "Summarize the given text",
  input: JSON.stringify({
    type: "object",
    properties: {
      input: { type: "string", description: "Text to summarize" },
    },
  }),
  handler: summarizeExecutor,
});
```

### Grouping with `useExecutors`

`useExecutors` groups callables into a collection with lookup and invocation methods:

```ts
import { useExecutors } from "llm-exe";

const executors = useExecutors([
  searchCallable,
  summarizeCallable,
]);

// Check if a function exists
executors.hasFunction("searchInternet"); // true

// Get a function by name
const fn = executors.getFunction("searchInternet");

// Call a function by name — input is parsed from a JSON string
const result = await executors.callFunction(
  "searchInternet",
  JSON.stringify({ query: "llm-exe docs" })
);
// result: { result: any, attributes: any }

// Get all functions (useful for building tool lists)
const allFunctions = executors.getFunctions();
```

### Optional features

**Visibility handler** — conditionally show/hide functions based on context:

```ts
const adminCallable = createCallableExecutor({
  name: "deleteUser",
  description: "Delete a user account",
  input: JSON.stringify({
    type: "object",
    properties: {
      userId: { type: "string", description: "The ID of the user to delete" },
    },
  }),
  handler: async (input: { userId: string }) => { /* ... */ },
  visibilityHandler: (input, context, attributes) => {
    return attributes?.role === "admin";
  },
});

// Only returns functions visible for the given context
const visible = executors.getVisibleFunctions(input, { role: "admin" });
```

**Input validation** — validate input before execution:

```ts
const callable = createCallableExecutor({
  name: "sendEmail",
  description: "Send an email",
  input: JSON.stringify({
    type: "object",
    properties: {
      to: { type: "string", description: "Recipient email address" },
      subject: { type: "string", description: "Email subject line" },
      body: { type: "string", description: "Email body content" },
    },
  }),
  handler: async (input: { to: string; subject: string; body: string }) => { /* ... */ },
  validateInput: async (input) => {
    if (!input.to) {
      return { result: false, attributes: { error: "Missing 'to' field" } };
    }
    return { result: true, attributes: {} };
  },
});

const validation = await executors.validateFunctionInput(
  "sendEmail",
  JSON.stringify({ subject: "Hi" })
);
// validation: { result: false, attributes: { error: "Missing 'to' field" } }
```

### Callable Errors

Callable executor failures use structured `LlmExeError` codes:

| Code | Meaning |
| --- | --- |
| `callable.invalid_handler` | The callable was created without a valid function or executor handler |
| `callable.handler_not_found` | `useExecutors` could not find a callable with the requested name |
| `callable.validation_failed` | Callable input validation failed |

Use `isLlmExeError` to check callable failure codes.

```ts
try {
  await executors.callFunction("sendEmail", input);
} catch (error) {
  if (isLlmExeError(error, "callable.handler_not_found")) {
    // Ask the model to choose one of the available functions.
  }
}
```

### Using with an LLM agent loop

A typical agent pattern: the LLM decides which function to call, and you execute it:

```ts
const executors = useExecutors([
  finalAnswerCallable,
  interactWithUserCallable,
  searchInternetCallable,
  takeNotesCallable,
]);

const functionWithCallableExecutors = async (input: {
  action: string;
  input: string;
  thought: string;
}) => {
  return executors.callFunction(input.action, input.input);
};
```

---


# LLM Executor Function Syntax

llm-exe aims to be building blocks, allowing you to use the pieces as you wish. Below are suggestions on how to contain an LLM executor in a way that can be used throughout your application. Ultimately, do what you want.

## LLM Executor Function

All examples below will share this prompt instruction and input interface.

<<< ../../examples/basicFunctions.ts#IntroduceSharedInstructions

### Return an Executor

One way to structure the function is to have the function return the executor its self. This is useful in that it returns an instance of LlmExecutor. You can attach hooks, listen for events, etc.

<<< ../../examples/basicFunctions.ts#llmExecutorExample

When using this approach, whenever you import this function to use, you will need to call execute on it. For example:

```ts{5}
import { llmExecutorExample } from "example-above"

const executor = llmExecutorExample()

executor.on("onSuccess", (result) => {
  console.log("Classification result:", result);
});

executor.on("onError", (error) => {
  console.error("Classification error:", error);
});

const result = await executor.execute({
    userInput: "",
    timeOfDay: ""
})
```

### Return a Value

With this approach, you encapsulate the executor and execution.

<<< ../../examples/basicFunctions.ts#llmExecutorExampleExecute

```ts
import { llmExecutorExample } from "example-above";

const result = await llmExecutorExample({
  userInput: "",
  timeOfDay: "",
});
```

Note the main differences:

1. The function is now an async function
2. The function requires an input (which gets passed into `.execute()`)
3. The LLM executor is executed and the value is returned on every call, rather than the executor.
4. Unless you bind hooks/events within the function, you lose that ability because you're returning the result instead of the executor.

## Structuring Files

When dealing with LLM Executors that have elaborate prompts, custom output parsers, or both, it may be useful to break the components out into different files. For example:

```
llms
- intent-bot
  - index.ts // executor lives here
  - parser.ts // export your parser
  - prompt.ts // export your prompt
  - types.ts // types
```

index.ts

```ts
import { CustomInputType } from "./types";
import { parser } from "./parser";
import { prompt } from "./prompt";

export async function myLlmExecutorExecute(input: CustomInputType) {
  return createLlmExecutor({
    prompt,
    parser,
    llm,
  }).execute(input);
}
```

::: tip
Allowing `prompt` and `parsers` to be imported/exported makes them testable components!
:::

::: tip
You can even load the prompt from a third-party resource. This means your prompt content/logic does not need to be bundled into your app code.
:::

## Other Notes

Its reasonable to pass an llm around.

<<< ../../examples/basicFunctions.ts#llmExecutorExampleNeedsLlm{2}

---


# Tips: Working with JSON

Instructing an LLM to work with JSON can be difficult.  Below are some tricks to working with JSON.

Use JSON Schema in your instructions
One useful is taking advantage of JSON Schema structure for explaining details about the expected response. Not only can you use the schema to tell the LLM which properties you want back, you can utilize properties such as 'required' 'name', description', and other properties to provide structured instructions.

Take the following example:

Here is a prompt which is attempting to tell the LLM which properties it expects, with some additional info.


The following example demonstrates how you could attempt to instruct the LLM to respond with a particular JSON format.

```
...rest of prompt

I need you to reply with valid JSON containing the following properties:

thought: this is where you explain your thoughts. This is required.
direction: the direction you chose to move. Muse be one of: forward, back, left, right. This is required.

For Example:
{
    "thought":  "explanation of your thoughts",
    "direction": "the direction you chose to move"
}
```


Here we can provide the same information, but this time using JSON Schema within our instruction. 

```
...rest of prompt

Your response must EXACTLY follow the JSON Schema specified below:
{
    type: "object",
    properties: {
      thought: {
        type: "string",
        description: "explanation of your thoughts" 
        },
      direction: {
        type: "string",
        description: "the direction you chose to move",
        enum: ["forward", "back", "left", "right"] 
      },
    },
    required: ["thought", "direction"],
    additionalProperties: false,
}

For Example:
{
    "thought":  "explanation of your thoughts",
    "direction": "the direction you chose to move"
}
```

Now, we have instructed the LLM without directly telling it that:
- We expect the response to be an object (we could use type: array syntax if we wanted!)
- We were able to hint at the data type.
- We were able to provide a well-marked description
- We were able to provide the options when there are specific choices
- We were able to tell it which fields were required without repeating ourselves over an over (which could stray the prompt)
- We are able to hint that we don't want additional properties.

You can also:
- Set defaults with the `default` keyword on a property.

## Enforcing the schema at parse time

The schema is not only a prompting aid — when you pass it to the JSON parser it is enforced on the LLM's response:

```ts
import { defineSchema, createParser } from "llm-exe";

const schema = defineSchema({
  type: "object",
  properties: {
    thought: { type: "string" },
    direction: { type: "string", enum: ["forward", "back", "left", "right"] },
  },
  required: ["thought", "direction"],
  additionalProperties: false,
});

const parser = createParser("json", { schema });
```

Once a `schema` is provided, `required` fields and type/constraint checks are enforced **by default** (`validateSchema` defaults to `true` when a schema is set). Input that is missing a required field — or violates the schema — throws `parser.schema_validation_failed` rather than silently returning a partial object:

```ts
parser.parse(`{"thought": "I should go"}`);
// throws: parser.schema_validation_failed — requires property "direction"
```

A couple of things to keep in mind:

- **Defaults are applied _after_ validation.** A `default` on a `required` property does **not** satisfy that requirement — if the LLM omits the field, validation still fails.
- **Opt out with `validateSchema: false`.** This switches to filter/default-only behavior: unknown keys are stripped and defaults are applied, but `required` and constraints are **not** checked. Use it only if you intentionally want that looser behavior.

See the [JSON parser reference](/parser/included-parsers.html#json) for the full option list.

### Related

- [Write a Type-Safe LLM Function](/examples/concepts/type-safe-llm-function) — schema-driven prompt, validation, and return type in one
- [Extract Structured Data](/examples/bots/extract) — caller-supplied schemas for slot filling
- [Included Parsers](/parser/included-parsers) — the JSON parser reference

---


# Replicating Lex

This is a non-exhaustive overview on how to replace some of the core concepts of Amazon Lex with a series of LLM executor functions.

We will aim to replace the following key components with individual LLM calls:
- Identify Intent
- Extract Slots
- Confirm Intent
- Response Generator

The challenge will come down to how you choose to:

1. Manage the intents/slots
2. Manage state
3. Handle the flow

## Identify Intent
To identify intent, we ask an LLM to identify what the intent of the conversation is. Simple and powerful! There is a little more to it than that, but not much. To start, we pre-define the intents that we expect (or want) to match. These get fed into the prompt and become the options the LLM can choose from. If the LLM can not identify the intent, the fallback is 'unknown'. This way, we can easily classify if the conversation matches something you planned for, or not.

We can also use some tricks to help get the best match:
- The prompt can also contain a description to help the LLM match to the prompt.
- Like Lex, you can tell the LLM to respond with its 'confidence'.
- To build on this, we tell the LLM to go through all the intents we give it, and rate each by confidence, and bring us back to top 3.

These are all prompt tricks. They are tips, not rules. Experiment yourself with what works.

Note: See [intent](/examples/bots/intent) for detailed example of the intent executor.


## Extract Slots
We'd use the result of the intent LLM to know which slots are available/required for the next step. Then, we'll ask the LLM if the specific information is included in the conversation.

Note: See [extract](/examples/bots/extract) for detailed example of the confirm executor.


## Confirm
To confirm the intent, you guessed it, we'll be asking an LLM if the user has confirmed! There are various ways to go about this, as it comes down to engineering a prompt that satisfies the requirement. For this example, I will use the validator LLM executor from this example.

The validator LLM executor takes a conversation, and a series of true/false statements, and asks the LLM to work through the statements and identify if they are true or false. Our LLM Executor has a custom output parser which will summarize this output into something easily usable.


Note: See [intent](/examples/bots/intent) for detailed example of the confirm executor.

## Response Generator
You can use a single LLM executor with some prompt templates to generate responses when you need to elicit intent, confirm, or carry on conversation.


## Putting it together

With these pieces, we can put together a proof of concept:

```typescript
import { identifyIntent } from "../other-example"
import { extractInformation } from "../another-example";
import { checkPolicy } from "policy-example"
import {  createDialogue } from "llm-exe";

// if you use history. See state/Dialogue
const chatHistory = createDialogue("chat");

// An example. 
const intentMap = {
    rent_car: {
        slots: {
            // slot schema
        }
    },
    book_hotel: {
        slots: {
            // slot schema
        } 
    },
    book_flight: {
        slots: {
            // slot schema
        }
    }
}


// get the intent
const { intent } = await identifyIntent({
    input,
    chatHistory: chatHistory.getHistory()
});


let fulfilled = false;
let confirmed = false;

if(intent === "unknown"){
    // return llm executor that deals with normal conversation, or unknown intents
    // This can just be a call to GPT4 with the whole conversation. It'll do okay to start.
}


// ok, we have a known intent.
// lets use it to tell the extractor what we're looking for
const { slots } = intentMap[intent];

// We provide the extractor the input, history, and slot schema
const extraction = await extractInformation({
    input,
    chatHistory: chatHistory.getHistory()
}, slots);

// now you should check slot values, validate slots, etc

// need a slot? Return an elicit slot LLM call

// all slots valid? Great! 

// this is another example of where it may help to manage state

// lets che
const didConfirm = await checkPolicy({
    mostRecentMessage: input,
    chatHistory: chatHistory.getHistory()
    statements: [
        "did the user explicitly confirm they wanted to rent a xxx on xxx?",
        "Did the assistant ask the user to confirm?"
    ]
});

// if did not confirm, ask them to confirm


// if did confirm, maybe its time to fulfill?


```

Tips:
- Will update with more.
---