AnymarkTurn a folder of documents into a knowledge base an agent can search
Every “chat with your documents” tool, from the simplest local search to a full agent framework, starts with the same step: the documents have to be text. Not pictures of text, not binary office formats, text with enough structure that a search hit lands on a section rather than a page. This article is about that step, because it is the one most projects stall on.
Why Markdown, specifically
Plain text would do for search. Markdown does better because it keeps the headings, and headings are how an indexer splits a long document into chunks that make sense. A hit inside “4. Payment terms” is more useful than a hit at character 18,240. Lists and tables stay recognisable, so a question about a price finds the price table rather than a sentence that mentions prices. And Markdown is what the language model on the other end reads best.
The sequence
1. Gather, do not move. Leave the originals where they are. You will mirror the Markdown into a separate folder; the originals stay the source of truth and nobody’s Finder gets rearranged.
2. Convert the folder. Anymark: File > Batch Convert Folder, destination mirrored into another folder, subfolders on, existing files skip. Pictures to asset folders. Start. For a few thousand mixed documents this takes an hour or two, most of it OCR on scans.
3. Read the report. It lists every file as converted, skipped or failed. Failures are usually corrupt files or unsupported formats; decide whether they matter.
4. Sample the output. Open ten converted files of different kinds. Look at headings and tables. If a whole class of document converts badly (a particular form, say), that class may need the original kept beside the Markdown with a note.
5. Add a little metadata. A line at the top of each file with the source path and date is cheap and pays back constantly. Anymark does not add one; a short script over the mirrored folder can.
6. Point the indexer at the mirror folder. Whatever you use, a local search tool, an embedding index, an agent’s file tool, give it the Markdown folder only. It will be fast, because text is small, and precise, because the structure is there.
7. Re-run when documents change. Because existing Markdown is skipped, a re-run only converts new or changed files. Schedule it, or run anymark --batch ./documents --out ./markdown from a script.
What to leave out
Not everything belongs in a knowledge base. Drafts superseded by finals, duplicates, and anything you would not want surfaced in an answer. Prune the originals folder or the mirror before indexing; an agent cannot tell a draft from a final.
Privacy, again
If the documents are private, the conversion step should happen on your machine, and the index should too, or at least you should know exactly where it goes. The point of converting locally is that the only thing that ever leaves, if anything does, is text you have read and chosen to share.