Skip to main content

How to return structured data from a model

It is often useful to have a model return output that matches some specific schema. One common use-case is extracting data from arbitrary text to insert into a traditional database or use with some other downstrem system. This guide will show you a few different strategies you can use to do this.

Prerequisites
This guide will assume familiarity with the following concepts:

The .withStructuredOutput() method​

There are several strategies that models can use under the hood. For some of the most popular model providers, including OpenAI, Anthropic, and Mistral, LangChain implements a common interface that abstracts away these strategies called .withStructuredOutput.

By invoking this method (and passing in JSON schema or a Zod schema) the model will add whatever model parameters + output parsers are necessary to get back structured output matching the requested schema. If the model supports more than one way to do this (e.g., function calling vs JSON mode) - you can configure which method to use by passing into that method.

You can find the current list of models that support this method here.

Let’s look at some examples of this in action! We’ll use Zod to create a simple response schema.

Pick your chat model:

Install dependencies

tip

See this section for general instructions on installing integration packages.

yarn add @langchain/openai

Add environment variables

OPENAI_API_KEY=your-api-key

Instantiate the model

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
  model: "gpt-3.5-turbo-0125",
  temperature: 0
});
// | output: false
// | echo: false

import { ChatOpenAI } from "@langchain/openai";

const model = new ChatOpenAI({
  model: "gpt-4-0125-preview",
  temperature: 0,
});
import { z } from "zod";

const Joke = z.object({
  setup: z.string().describe("The setup of the joke"),
  punchline: z.string().describe("The punchline to the joke"),
  rating: z.number().optional().describe("How funny the joke is, from 1 to 10"),
});

const structuredLlm = model.withStructuredOutput(Joke);

await structuredLlm.invoke("Tell me a joke about cats");
{
  setup: "Why was the cat sitting on the computer?",
  punchline: "Because it wanted to keep an eye on the mouse!",
  rating: 8
}

The result is a JSON object.

We can also pass in an OpenAI-style JSON schema dict if you prefer not to use Zod. This object should contain three properties:

  • name: The name of the schema to output.
  • description: A high level description of the schema to output.
  • parameters: The nested details of the schema you want to extract, formatted as a JSON schema dict.

In this case, the response is also a dict:

const structuredLlm = model.withStructuredOutput({
  name: "joke",
  description: "Joke to tell user.",
  parameters: {
    title: "Joke",
    type: "object",
    properties: {
      setup: { type: "string", description: "The setup for the joke" },
      punchline: { type: "string", description: "The joke's punchline" },
    },
    required: ["setup", "punchline"],
  },
});

await structuredLlm.invoke("Tell me a joke about cats");
{
  setup: "Why was the cat sitting on the computer?",
  punchline: "Because it wanted to keep an eye on the mouse!"
}

If you are using JSON Schema, you can take advantage of other more complex schema descriptions to create a similar effect.

You can also use tool calling directly to allow the model to choose between options, if your chosen model supports it. This involves a bit more parsing and setup. See this how-to guide for more details.

Specifying the output method (Advanced)​

For models that support more than one means of outputting data, you can specify the preferred one like this:

const structuredLlm = model.withStructuredOutput(Joke, {
  method: "json_schema",
});

await structuredLlm.invoke(
  "Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys"
);
{
  setup: "Why was the cat sitting on the computer?",
  punchline: "To keep an eye on the mouse!"
}

In the above example, we use OpenAI’s alternate JSON mode capability along with a more specific prompt.

For specifics about the model you choose, peruse its entry in the API reference pages.

Prompting techniques​

You can also prompt models to outputting information in a given format. This approach relies on designing good prompts and then parsing the output of the models. This is the only option for models that don’t support .with_structured_output() or other built-in approaches.

Using JsonOutputParser​

The following example uses the built-in JsonOutputParser to parse the output of a chat model prompted to match a the given JSON schema. Note that we are adding format_instructions directly to the prompt from a method on the parser:

import { JsonOutputParser } from "@langchain/core/output_parsers";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
  name: string;
  height_in_meters: number;
};

type People = {
  people: Person[];
};

const formatInstructions = `Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.
`;

// Set up a parser
const parser = new JsonOutputParser<People>();

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
  [
    "system",
    "Answer the user query. Wrap the output in `json` tags\n{format_instructions}",
  ],
  ["human", "{query}"],
]).partial({
  format_instructions: formatInstructions,
});

Let’s take a look at what information is sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Wrap the output in `json` tags
Respond only in valid JSON. The JSON object you return should match the following schema:
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}

Where people is an array of objects, each with a name and height_in_meters field.

Human: Anna is 23 years old and she is 6 feet tall

And now let’s invoke it:

const chain = prompt.pipe(model).pipe(parser);

await chain.invoke({ query });
{ people: [ { name: "Anna", height_in_meters: 1.83 } ] }

For a deeper dive into using output parsers with prompting techniques for structured output, see this guide.

Custom Parsing​

You can also create a custom prompt and parser with LangChain Expression Language (LCEL), using a plain function to parse the output from the model:

import { AIMessage } from "@langchain/core/messages";
import { ChatPromptTemplate } from "@langchain/core/prompts";

type Person = {
  name: string;
  height_in_meters: number;
};

type People = {
  people: Person[];
};

const schema = `{{ people: [{{ name: "string", height_in_meters: "number" }}] }}`;

// Prompt
const prompt = await ChatPromptTemplate.fromMessages([
  [
    "system",
    `Answer the user query. Output your answer as JSON that
matches the given schema: \`\`\`json\n{schema}\n\`\`\`.
Make sure to wrap the answer in \`\`\`json and \`\`\` tags`,
  ],
  ["human", "{query}"],
]).partial({
  schema,
});

/**
 * Custom extractor
 *
 * Extracts JSON content from a string where
 * JSON is embedded between ```json and ``` tags.
 */
const extractJson = (output: AIMessage): Array<People> => {
  const text = output.content as string;
  // Define the regular expression pattern to match JSON blocks
  const pattern = /```json(.*?)```/gs;

  // Find all non-overlapping matches of the pattern in the string
  const matches = text.match(pattern);

  // Process each match, attempting to parse it as JSON
  try {
    return (
      matches?.map((match) => {
        // Remove the markdown code block syntax to isolate the JSON string
        const jsonStr = match.replace(/```json|```/g, "").trim();
        return JSON.parse(jsonStr);
      }) ?? []
    );
  } catch (error) {
    throw new Error(`Failed to parse: ${output}`);
  }
};

Here is the prompt sent to the model:

const query = "Anna is 23 years old and she is 6 feet tall";

console.log((await prompt.format({ query })).toString());
System: Answer the user query. Output your answer as JSON that
matches the given schema: ```json
{{ people: [{{ name: "string", height_in_meters: "number" }}] }}
```.
Make sure to wrap the answer in ```json and ``` tags
Human: Anna is 23 years old and she is 6 feet tall

And here’s what it looks like when we invoke it:

import { RunnableLambda } from "@langchain/core/runnables";

const chain = prompt
  .pipe(model)
  .pipe(new RunnableLambda({ func: extractJson }));

await chain.invoke({ query });
[
  { people: [ { name: "Anna", height_in_meters: 1.83 } ] }
]

Next steps​

Now you’ve learned a few methods to make a model output structured data.

To learn more, check out the other how-to guides in this section, or the conceptual guide on tool calling.

Help us out by providing feedback on this documentation page: