> ## Documentation Index
> Fetch the complete documentation index at: https://agno-v2.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Code Chunking

> Split source code at AST boundaries with CodeChunking, powered by Chonkie.

`CodeChunking` wraps Chonkie's AST-based code chunker. Configure its tokenizer, token limit, and source language.

Code chunking supports several built-in tokenizers or a custom `Tokenizer` instance.

**1. Create a Python file**

```python Code Chunking theme={null}
from agno.agent import Agent
from agno.knowledge.chunking.code import CodeChunking
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.text_reader import TextReader
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    vector_db=PgVector(table_name="python_code_chunking", db_url=db_url),
)

knowledge.insert(
    url="https://raw.githubusercontent.com/agno-agi/agno/v2.7.2/libs/agno/agno/workflow/workflow.py",
    reader=TextReader(
        chunking_strategy=CodeChunking(
            tokenizer="character",
            chunk_size=500,
            language="python",
        ),
    ),
)

agent = Agent(knowledge=knowledge, search_knowledge=True)
agent.print_response("How does Workflow run its steps?", markdown=True)
```

```python Code Chunking with Custom Tokenizer theme={null}
from typing import Sequence

from agno.agent import Agent
from agno.knowledge.chunking.code import CodeChunking
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.text_reader import TextReader
from agno.vectordb.pgvector import PgVector
from chonkie.tokenizer import Tokenizer

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"


class LineTokenizer(Tokenizer):
    """Custom tokenizer that counts lines of code."""

    def __init__(self):
        self.vocab = []
        self.token2id = {}

    def __repr__(self) -> str:
        return f"LineTokenizer(vocab_size={len(self.vocab)})"

    def tokenize(self, text: str) -> Sequence[str]:
        if not text:
            return []
        return text.split("\n")

    def encode(self, text: str) -> Sequence[int]:
        encoded = []
        for token in self.tokenize(text):
            if token not in self.token2id:
                self.token2id[token] = len(self.vocab)
                self.vocab.append(token)
            encoded.append(self.token2id[token])
        return encoded

    def decode(self, tokens: Sequence[int]) -> str:
        try:
            return "\n".join([self.vocab[token] for token in tokens])
        except Exception as e:
            raise ValueError(
                f"Decoding failed. Tokens: {tokens} not found in vocab."
            ) from e

    def count_tokens(self, text: str) -> int:
        if not text:
            return 0
        return len(text.split("\n"))


knowledge = Knowledge(
    vector_db=PgVector(table_name="code_custom_tokenizer", db_url=db_url),
)

knowledge.insert(
    url="https://raw.githubusercontent.com/agno-agi/agno/v2.7.2/libs/agno/agno/workflow/workflow.py",
    reader=TextReader(
        chunking_strategy=CodeChunking(
            tokenizer=LineTokenizer(),
            chunk_size=500,
            language="python",
        ),
    ),
)

agent = Agent(knowledge=knowledge, search_knowledge=True)
agent.print_response("How does Workflow run its steps?", markdown=True)
```

<Snippet file="create-venv-step.mdx" />

**2. Install dependencies**

```bash theme={null}
uv pip install -U agno sqlalchemy psycopg pgvector "chonkie[code]" openai
```

**3. Export your OpenAI API key**

<Snippet file="set-openai-key.mdx" />

<Snippet file="run-pgvector-step.mdx" />

**4. Run the script**

```bash theme={null}
python code_chunking.py
```

## Code Chunking Params

<Snippet file="chunking-code.mdx" />

## Developer Resources

* [Chonkie Code Chunker](https://docs.chonkie.ai/oss/chunkers/code-chunker)
* [Chunking overview](https://agno-v2.mintlify.site/knowledge/concepts/chunking/overview.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.