Can you make a llms.txt file for latest .md files

[Kjdragan]

As an AI focused scraper, it would be great if you could add an llms.txt file a la Jeremy Howard so we can easily feed a LLM for AI driven use context. Or a MCP server? :)
Great repo! Thx!

[unclecode]

@Kjdragan You’re absolutely right, it’s a shame, haha :)). Funny enough, I’m currently working on a new Crawl4AI feature to generate clean llm.txt files, not just in Jeremy’s format but in a better, more customizable one based on my experience. I’ll definitely release this llm.txt feature to help people make better use of the library with their favorite AI chat models.

On another note, I’ve also created an MCP server, which I shared on my X. It’s still under testing and not finalized yet, but feel free to take a look and let me know if you encounter any issues! https://github.com/unclecode/crawl4ai_mcp_service

[profplum700]

The link to your MCP server is broken 😢

[dwstevens]

any way I can use the llms.txt functionality in a dev build?