Why it matters for your business
Chunking is unglamorous and has a large effect on RAG quality. Chunks that are too small lose context, such as a table separated from its heading or an exception separated from its rule. Chunks that are too large dilute the match and waste tokens. Policies, contracts and manuals usually retrieve better when chunks follow the document's own sections and carry the section title with them. When an assistant gives a partial or wrong answer, reviewing the retrieved chunks is often the fastest way to find the cause.
Example
An HR team loads its employee handbook into a policy assistant. The first version split the handbook every 500 tokens, so the PTO accrual table landed in a chunk without its heading and answers mixed up full-time and part-time rules. Re-chunking by handbook section, with the section title added to each chunk, fixed the confusion.