Notes on the CrewAI knowledge feature

CrewAI recently released a new feature called Knowledge that allows you to set up a knowledge base for your crews and agents. Compared to traditional RAG pipelines, it's simple and fast to set up. Here's how it works:

• Create a folder called knowledge in your project's root directory
• Load your documents into the folder (supports txt, pdf, word, excel, json, and more)
• Create a knowledge source based on your document types. For example, use PDFKnowledgeSource for PDF files
• Add the knowledge source to your crew or agent
• That's it!

This is one of the simplest ways I've found to set up a basic RAG pipeline for AI agents and applications.

The Knowledge feature offers two main configuration options: chunking and embedding. Knowledge sources automatically chunk content for better processing. You can customize the chunk size and overlap based on your needs:

source = StringKnowledgeSource(
    content="Your content here",
    chunk_size=4000,      # Maximum size of each chunk (default: 4000)
    chunk_overlap=200     # Overlap between chunks (default: 200)
)

You can also configure the embedding provider, which is particularly useful if you want to use different embeddings than those used by your agents:

crew = Crew(
    agents=[agent],
    tasks=[task],
    verbose=True,
    process=Process.sequential,
    knowledge_sources=[string_source],
    embedder={
        "provider": "google",
        "config": {
            "model": "models/text-embedding-004",
            "api_key": GEMINI_API_KEY,
        }
    }
)

For more advanced use cases, CrewAI supports custom knowledge sources, allowing you to create knowledge sources for any data type by extending BaseKnowledgeSource.

I'm excited about this feature as it makes creating practical AI agents much simpler. For example, you could quickly build a customer support chatbot with access to your product documentation. What do you think about this feature? Reach out to me on X (formerly Twitter) or LinkedIn if you have any questions or comments.

AI should drive results, not complexity. AgentemAI helps businesses build scalable, efficient, and secure AI solutions. See how we can help.