Demo source code

The core of demo.cs, the C# source of the multi-file document Q&A demo shipped with the toolkit, arranged by stage. Windows Forms plumbing lives in a separate demo_ui.cs and is omitted here; the complete project is included in the evaluation download. The tuning values shown (retrieval percentage, enrichment budget, coverage share, confidence threshold) are examples and should be adjusted for your documents and token budget.

The demo is provided as sample code for building your own product. See the license agreement in the Documentation tab regarding distribution of the demo itself.

Setup: fields, per-file state and model initialization

The class-level state of the demo: collection totals, tuning percentages, the per-file ClsFiles record that holds each document's chunks, metadata and Ven object, and the initialization of the embedding and chat clients.

using LangChain.Databases;      // comes from LangChain.Databases.InMemory
using LangChain.Databases.Sqlite;
using LangChain.DocumentLoaders;
using LangChain.Extensions; // Required for the unified VectorStore type
using LangChain.Providers;  // For OpenAI provider specific types
using LangChain.Providers.OpenAI;
using Ollama;
using OpenAI.Chat;   // official OpenAI SDK to bypass LangChain when sending the chunks to LLM
using OpenAI.VectorStores;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Diagnostics;
using System.Diagnostics.Eventing.Reader;
using System.Diagnostics.SymbolStore;
using System.Drawing.Drawing2D;
using System.IO;
using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using System.Security.Policy;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading.Tasks;
using System.Windows.Forms;
using tryAGI.OpenAI;
using static System.Windows.Forms.VisualStyles.VisualStyleElement.ProgressBar;

using SubSystems.RagDocumentToolkit.Dan;
using SubSystems.RagDocumentToolkit.Ven;


namespace Demo
{
    public partial class Demo : Form
    {
string msg = "";
IReadOnlyCollection<Document>? AllDocs = null;   // Document object containing all documents - used for
                                                 // retrieval across all documents in the collection
string TokenMessage = "";            // number of tokens used message
int TotalTokenCount = 0;             // token count for all files in the collection
int TotalChunkCount = 0;             // chunk count for all files in the collection
int TokenThreshold = 20000;          // when total token count is below this threshold, we send all file chunks to LLM for each user query.
                                     // Above this threshold, we apply user-query-dependent retrieval and Vector Enrichment to
                                     // determine only the relevant chunks to send to LLM
int retrievalPct = 15;               // retrieval search limit as the percentage of total chunks in the file collection
int enrichmentPct = 10;              // enrichment budget as the percentage of total tokens in the file collection
int globalSpreadPct = 25;            // for broad (global) query, percent of retrieval chunks to distribute to each file in file-size proportion
int filterTokenPct = 10;             // Filter routing token allowance
int filterTokenCount = 0;            // filter token count

int TotalRetrievalTokens = 0;        // number of tokens in the retrieved set for the query

string IndexDoc = "";
int TotalFilterTokens = 0;           // number of tokens in the combined phrase document
bool AnswerSummaryQueryUsingPhraseDoc = true;  // true: use the phrase doc to answer a collection-wide summary query
bool GlobalQueryExecutedUsingPhraseDoc = false;  // true if the global query is executed using the phrase document

string newUserQuestion = "";         // current user question

bool globalQuery = false;              // true if the current query is of a broad nature


string fullContent;
LangChain.Providers.Message userMessage;

// LangChain parameters for strategy evaluation and search
OpenAiProvider provider;

OpenAI.Chat.ChatClient? fullChat = null;
OpenAI.Chat.ChatClient? miniChat = null;


private List<LangChain.Providers.Message> chatHistory = new List<LangChain.Providers.Message>();
private List<LangChain.Providers.Message> filterChatHistory = new List<LangChain.Providers.Message>();

string htmlResponse = "", PrevHtmlResponse = "";
//private SqLiteVectorDatabase vectorDatabase;
private IVectorCollection? vectorCollection = null;   // collection containing all documents

private OpenAiEmbeddingModel embeddingsProvider;
private const string EmbeddingModelName = "text-embedding-3-small"; // if we switch to another AI, say Claude, both the
                                                                    // EmbeddingModelName and the Dimensions (below)
                                                                    // need to change
int Dimensions = 1536;    // dimension of each LLM vector point.  1536 = recommended for ChatGPT
String CollectionName = "Dan_Docs_" + EmbeddingModelName.Replace("-", "_");  // in case of a persisted vector (This demo
                                                                             // uses in-memory vectors), adding the embedding model to the
                                                                             // collection name will ensure that any old persisted
                                                                             // vector is not used for AI query

private OpenAiChatSettings Settings;
private int ChunkCount = 0;



private bool UserQueryCheckedForRevisions = false;   // true=checked the user query for Revisions
private bool QueryIsAboutRevisions = false;          // true=the query is about revisions (redlined text)
private bool UserQueryCheckedForComments = false;    // true=checked the user query for Commented text
private bool QueryIsAboutComments = false;           // true=the query is about commented text

private bool AiWorking = false;
private bool AiInitialized = false;

internal class ClsFiles
{
    internal string FilePath = "";        // file path of this document
    internal string DocId = "";           // unique document id of this document
    internal string DocTitle = "";        // document title

    internal Ven? ven = null;             // Instance of Vector Enrichment Library for this document
    internal int ChunkCount = 0;          // size of the chunk/MetaRec arrays
    internal string[]? chunk = null;      // chunk array
    internal Dictionary<string, object>[]? MetaRec = null;
    internal string KeyPhrases;           // key phrases for the document

    internal SortedSet<int> RetrieveSeq;  // retrieved sequences
    internal SortedSet<int> EnrichedSeq;  // enriched sequences
    internal int TokenBudget;             // Enrichment token budget for this document

    internal IReadOnlyCollection<Document>? docs = null; // chunk/meta-rec collection for this document - used for retrieval at the document level
    internal SqLiteVectorDatabase? vectorDatabase = null;
    internal IVectorCollection? vectorCollection = null;  // per-file vector collection

    internal int FirstIdx = 0;            // index of the first chunk/MetaRec record position within the combined document.
                                          // It would be the sum of ChunkCounts of the preceding files.

    internal bool HasTables = false;      // true=this file includes tables
    internal bool HasRevisions = false;   // true=this file includes revised text
    internal bool HasComments = false;    // true=this file includes comments
    internal int TokenCount = 0;          // token count for this file

    internal int RetrievalLimit = 0;      // number of records to retrieve for this document
    internal int ErichmentBudget = 0;     // Enrichment token count budget for this document

    internal int RetrievalTokenCount;     // The number of tokens in the retrieved set
    internal int EnrichmentTokenCount;    // The number of tokens added for enrichment

    internal int quota;                   // this file's quota for vector search pass #2

    internal bool InFilter;               // true if filtering is enabled and this file is included after filtering for a user query
    internal string PhraseDoc = "";       // phrase document - key values/phrases from the meta information about this file, in suitable chunks to send to AI
    internal int FilterTokens = 0;        // number of tokens for the PhraseDoc
}

List<ClsFiles> files = new List<ClsFiles>();   // current file collection

Dictionary<string, ClsFiles>? filesDict = null;

internal class ClsFilterResult
{
    internal bool FilterApplied = false;        // the filter query set the InFilter flag for those documents found relevant after the filter query
    internal List<LangChain.Providers.Message>? payload = null;   // payload already created and ready to be sent to AI
}

public Demo()
{
    InitializeComponent();
}

private async void Demo_Load(object sender, EventArgs e)
{
    await InitUi(sender, e);

    if (Dan.DasHasSubSystemsOpenAIEvalKeyExpired())     // Sub Systems' OpenAI key expired for your eval, please provide your own key
    {
        LblStatus.Text = "Sub Systems' OpenAI key expired for your eval.  Please enter your OpenAI key to proceed:";
        FldKey.Visible = true;
        FldAcceptKey.Visible = true;
        FldSend.Enabled = false;
        FldAddDoc.Enabled = false;
        FldRestore.Enabled = false;
        return;
    }

    await InitAi();
}

private async Task InitAi()
{
    string fullModel = "gpt-5.6-terra";   // model for main chat
    string miniModel = "gpt-5.6-luna";    // model for index lookup or classification queries

    if (Dan.DasHasSubSystemsOpenAIEvalKeyExpired())  // you are using your own key because Sub Systems' OpenAI key has expired for your eval
    {
        string YourOpenAiKey = FldKey.Text.Trim();
        provider = new OpenAiProvider(YourOpenAiKey);

        fullChat = new OpenAI.Chat.ChatClient(fullModel, YourOpenAiKey);
        miniChat = new OpenAI.Chat.ChatClient(miniModel, YourOpenAiKey);
    }
    else {   // Apply Sub Systems' OpenAI key, which is valid during your eval period
        Dan.DasSetSubSystemsOpenAIEvalKey(out provider,
              out fullChat, out miniChat, fullModel, miniModel);
    }

    // Initialize the embedding model (used to mathematically index the document text)
    // Instantiate OpenAiEmbeddingModel using the provider and the target model
    embeddingsProvider = new OpenAiEmbeddingModel(provider, EmbeddingModelName);

    Settings = new OpenAiChatSettings
    {
        // OpenAI renamed 'MaxTokens' to 'MaxCompletionTokens' recently
        MaxCompletionTokens = 1000,    // don't let AI answers get too long.  Also works as a circuit breaker if AI gets caught in an internal loop interpreting our data
        Temperature = 1   // don't let AI get too general, stick to my document
    };

    // check if the OpenAI key is valid
    try
    {
        // Make a tiny, minimal request to verify the key
        await miniChat.CompleteChatAsync("ping");
        // If it reaches here without throwing a 401 exception, the key is correct!
    }
    catch (System.ClientModel.ClientResultException ex) when (ex.Status == 401)
    {
        // Key is explicitly bad or unauthorized
        LblStatus.Text = "Invalid OpenAI API Key provided, Please enter your correct key";
        return;
    }
    catch (Exception ex)
    {
        // Handle other errors (like network/timeout issues)
        LblStatus.Text = $"Connection error: {ex.Message}";
        return;
    }

    FldSend.Enabled = true;
    AiInitialized = true;
    FldKey.Visible = false;
    FldAcceptKey.Visible = false;
    LblStatus.Text = "";
    FldAddDoc.Enabled = true;
    FldRestore.Enabled = true;
}

Adding documents with Dan and building the vector collections

Importing each file, calling DasGetMarkdown, creating the per-document Ven object, and loading chunks with their metadata into per-file and collection-wide vector stores.

/**************************************************************
 *  AddDocument:
 *  Add the given file to the document collection
 *  ***********************************************************/
async Task<bool> AddDocument(string NewFile)
{
    msg = "";  // reset any error message

    ClsFiles file = new ClsFiles();
    file.FilePath = NewFile;

    // Create markdown object
    Dan dan = new Dan();
    dan.LogMsg += LogDanMsg;
    dan.MdoNames += MdoNames;  // The library sends plain text for each chunk,
                               // so if your application needs to, it can detect the names used in the chunk text and
                               // return those names.  This can significantly increase the document index performance.

    SetStatusText($"Reading file: {NewFile}");

    // import the file into the Dan object
    if (!dan.DasImportFile(NewFile, GetDocType(NewFile)))
    {
        if (msg == "") LblStatus.Text = "Error getting markdown for: " + NewFile;
        else LblStatus.Text = msg;
        return false;
    }

    SetStatusText("Creating markdown...");


    // Get the markdown for the imported file
    Dan.DanResult mdo = dan.DasGetMarkdown(-1, -1); // -1 = get markdown for the entire document
    if (mdo == null)
    {
        if (msg == "") LblStatus.Text = "Error getting markdown for: " + NewFile;
        else LblStatus.Text = msg;
        return false;
    }

    SetStatusText("");

    file.DocId = mdo.DocId;    // Unique id for the document
    file.DocTitle = mdo.DocTitle;  // document title retrieved from the document. If the document title is not found, this field contains the document file path
    file.chunk = mdo.chunks;
    file.MetaRec = mdo.MetaRecs;
    file.ChunkCount = file.chunk.Length;

    // create the Vector Enrichment object for this markdown
    file.ven = null;
    try
    {
        file.ven = new Ven(file.MetaRec);
    }
    catch (Exception ex)
    {
        LblStatus.Text = "Error creating Vector Enrichment object: " + ex.ToString();
        return false;
    }
    file.HasTables = file.ven.VesHasTables();
    file.HasRevisions = file.ven.VesHasRevisions();
    file.HasComments = file.ven.VesHasComments();
    file.TokenCount = file.ven.VesGetTokenCount();

    file.PhraseDoc = "";   // the phrase document contains a summary of the meta records in a format suitable for AI query
    file.FilterTokens = 0;

    // build List<Document>() for this document for retrieval
    // specific to this document
    List<Document> docs = new List<Document>();
    int ChunkCount = file.ChunkCount;
    for (int i = 0; i < ChunkCount; i++)
    {
        docs.Add(new Document(
            content: file.chunk[i],
            metadata: file.MetaRec[i]
        ));
    }

    file.docs = docs;   // assign to read-only collection

    file.vectorDatabase = new SqLiteVectorDatabase(dataSource: ":memory:");
    file.vectorCollection = await file.vectorDatabase.GetOrCreateCollectionAsync(
        collectionName: CollectionName + files.Count,  // added count to make a unique name, and differentiate from the global CollectionName
        dimensions: Dimensions     // 1536-dimensional vectors, this is a standard chosen by OpenAI; needs to change for other providers
    );

    await file.vectorCollection.AddDocumentsAsync(embeddingsProvider, docs);


    files.Add(file);

    ResetChat();   // reset chat related variables

    FldSave.Enabled = true;

    return true;
}

/**************************************************************
 * RecreateVectorStore:
 * Recreate the combined document object
 * ************************************************************/
async Task RecreateVectorStore()
{
    if (!AiInitialized) await InitAi();

    if (files.Count == 0)   // for an empty file collection
    {
        AllDocs = null;
        TotalTokenCount = 0;
        TotalChunkCount = 0;
        LblTokenCount.Text = "";
        return;
    }

    vectorCollection = await GetVectorCollection(true);   // collection containing all documents in the filter

    //clear chat history
    chatHistory = new List<LangChain.Providers.Message>();
    FldHtml.NavigateToString("");

    LblTokenCount.Text = $"Total Tokens: {(TotalTokenCount / 1000):N0}K";
    LblChunkCount.Text = $"Total Chunks: {TotalChunkCount}";
}

/*****************************************************************************
 * GetVectorCollection:
 * Get the vector collection for all files, or
 * for only the files in the preceding filter set.
 * When collecting all files, the method also updates TotalChunkCount, TotalTokenCount
 * and each file's beginning chunk index
 * **************************************************************************/
private async Task<IVectorCollection> GetVectorCollection(bool all)
{

    var docList = new List<Document>();
    int FirstIdx = 0;  // index of a document's first chunk/MetaRec in the combined collection

    // check if all files are selected
    if (!all)
    {
        ClsFiles file = files.FirstOrDefault(x => (!x.InFilter));   // find one file not in the filter
        if (file == null) all = true;
    }

    if (all && vectorCollection != null && filesDict != null) return vectorCollection; // we already have it

    if (all)
    {
        TotalTokenCount = 0;
        TotalChunkCount = 0;

        filesDict = files.ToDictionary(f => f.DocId);  // look up file by DocId
    }

    foreach (ClsFiles file in files)
    {
        if (file.chunk == null || file.MetaRec == null || file.docs == null) continue;
        if (!all && !file.InFilter) continue;   // only include files in the filter

        List<Document> FileDoc = file.docs as List<Document>;
        if (all || file.InFilter) docList.AddRange(FileDoc);

        if (all)  // save to class variables
        {
            file.FirstIdx = FirstIdx;
            FirstIdx += file.ChunkCount;  // FirstIdx of the next file

            TotalTokenCount += file.TokenCount;
            TotalChunkCount += file.ChunkCount;
        }
    }

    IReadOnlyCollection<Document> docs = docList;   // Document object containing all/filtered documents


    // Load the document markdown chunks directly into in-memory vector storage.
    // This takes our documents and builds a searchable index out of them.
    // Create the database in-memory (using the ":memory:" data source)
    SqLiteVectorDatabase vectorDatabase = new SqLiteVectorDatabase(dataSource: ":memory:");

    // Add the pre-split documents to the collection using the embedding provider
    // Note: OpenAI embeddings standard width is 1536 dimensions
    IVectorCollection vCollection = await vectorDatabase.GetOrCreateCollectionAsync(
        collectionName: CollectionName,
        dimensions: Dimensions     // 1536-dimensional vectors, this is a standard chosen by OpenAI; needs to change for other providers
    );

    await vCollection.AddDocumentsAsync(embeddingsProvider, docs);

    if (all)
    {
        AllDocs = docs;   // save to class variables
        vectorCollection = vCollection;   // If this included all files, save the collection for future queries
    }

    return vCollection;
}

/*******************************************************************
 * ResetChat:
 * Reset the chat related variables
 * *****************************************************************/
void ResetChat()
{
    AllDocs = null;    // nullify because it needs to be recreated using the updated set of files
    vectorCollection = null;
    chatHistory = new List<LangChain.Providers.Message>();
    filesDict = null;

    IndexDoc = "";  // combined phrase doc - needs to be recalculated
}
/********************************************************************

Name extraction hook

The MdoNames handler Dan calls with each chunk's plain text. This concept version uses regular expressions and word lists; a production application would use an NER model or its own domain vocabulary.

/********************************************************************
 * Here is a quick concept of extracting names of persons, places, and
 * organizations.
 * In this concept check, we detect names and return them.
 * In your application you would use Microsoft.ML.OnnxRuntime, or another
 * such library, to detect names and return them.
 * The 'ImportantTerms' can include checks for all important terms
 * for your organization or industry.
 * This step is not mandatory, but recommended.
 * *******************************************************************/
// Persons: honorific followed by 1-3 capitalized words ("Dr. Anita Patel")
static readonly Regex PersonRx = new Regex(
    @"\b(?:Dr|Mr|Mrs|Ms|Prof)\.?\s+((?:[A-Z][a-z]+\s?){1,3})",
    RegexOptions.Compiled);

// Orgs: 1-4 capitalized words ending in a corporate suffix ("CranePoint Manufacturing Co.")
static readonly Regex OrgRx = new Regex(
    @"\b((?:[A-Z][A-Za-z&]+\s+){1,4}(?:Inc\.?|LLC|Ltd\.?|Corp\.?|Co\.?|Group|Medical Center|Manufacturing))(?=[\s,.;:]|$)",
    RegexOptions.Compiled);

static readonly string[] Places = { "Austin", "St. Petersburg", "Port Haven", "Rm 118" };

static readonly HashSet<string> Terms = new HashSet<string>(StringComparer.OrdinalIgnoreCase)
    { "Lisinopril", "Metformin", "Amlodipine", "Net-45", "Spindle bearing", "President" };

static bool ContainsWord(string text, string term) =>
    Regex.IsMatch(text, $@"\b{Regex.Escape(term)}\b", RegexOptions.IgnoreCase);

Dan.ClsMdoNames MdoNames(object sender, string DocTitle, int ChunkIndex, String text)
{
    var names = new Dan.ClsMdoNames();
    var persons = new HashSet<string>();
    var orgs = new HashSet<string>();
    var places = new List<string>();
    var terms = new List<string>();

    // Persons — capture group 1 is the name without the honorific
    foreach (Match m in PersonRx.Matches(text))
        persons.Add(m.Groups[1].Value.Trim());
    persons.Add("Jonathan");  // we add this just for our testing

    // Organizations — capture group 1 is the full org name
    foreach (Match m in OrgRx.Matches(text))
        orgs.Add(m.Groups[1].Value.Trim());

    // Places and domain terms — simple whole-word lookup against your lists
    foreach (var p in Places)
        if (ContainsWord(text, p)) places.Add(p);

    foreach (var t in Terms)
        if (ContainsWord(text, t)) terms.Add(t);

    names.PersonNames = persons.ToArray();
    names.OrgNames = orgs.ToArray();
    names.PlaceNames = places.ToArray();
    names.ImportantTerms = terms.ToArray();

    return names;
}

Answering a question: threshold check and routing

The send handler chooses between sending every chunk (small collections), the AI document filter, and retrieval with enrichment. IsGlobalQuery asks a small model whether the question is broad or pointed.

private async void FldSend_Click(object sender, EventArgs e)
{
    if (!AiInitialized) await InitAi();
    if (AiWorking) return;  // not done with the previous question

    AiWorking = true;   // don't take the next request until we are done with this one

    await FldHtml.EnsureCoreWebView2Async();

    // add the new question to chatHistory
    newUserQuestion = FldQuery.Text;
    chatHistory.Add(new LangChain.Providers.Message(newUserQuestion, LangChain.Providers.MessageRole.Human));

    PrevHtmlResponse = htmlResponse; // save the previous response
    UpdateChatBox(newUserQuestion, "<span id=\"progress\"></span>");
    LblStatus.Text = "";

    // get the payload to send to AI
    List<LangChain.Providers.Message>? payload = null;
    if (TotalTokenCount < TokenThreshold) payload = GetAllChunksContext();
    else
    {
        GlobalQueryExecutedUsingPhraseDoc = false;  // will be true if the user query is answered from the filter phrase document instead of the file collection.
                                                    // This could happen if filtering is enabled and AI determines that the user query is of a broad summary nature.
        payload = await GetContextForAboveThresholdQuery();
        if (GlobalQueryExecutedUsingPhraseDoc)
        {
            AiWorking = false;
            return;   // user query already executed using the phrase document as the source
        }
    }

    if (payload == null)
    {
        LblStatus.Text = "Error generating full AI payload.";
        AiWorking = false;
        return;
    }

    // send the payload to AI
    await SendToAI(fullChat, payload, chatHistory, true);
    AiWorking = false;
}

/************************************************************************
 * GetAllChunksContext:
 * Return the content of the chunks of the whole collection, plus the system prompt
 * and chatHistory
 * ********************************************************************/
private List<LangChain.Providers.Message> GetAllChunksContext()
{
    string chunks = "";

    foreach (ClsFiles file in files)
    {
        if (chunks.Length > 0) chunks += "\n\n";
        chunks += String.Join("\n\n", file.chunk);
    }

    // to take full advantage of caching, we place the static content before the chat history
    string dynamicSystemPrompt = Dan.DasGetSystemMessage() +
                 "\n\nUSE THIS EXTRACTED DOCUMENT CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                 chunks;

    // create a combined list: system message followed by the chat history
    LangChain.Providers.Message dynamicMessage = new LangChain.Providers.Message(dynamicSystemPrompt,
                                                                         LangChain.Providers.MessageRole.System);
    List<LangChain.Providers.Message> payload = [dynamicMessage, .. chatHistory];

    TokenMessage = $"Number of document tokens sent: {TotalTokenCount}";

    return payload;
}

/************************************************************************
 * GetContextForAboveThresholdQuery:
 * Get the content when the file collection has a token count above the threshold
 * below which we send all contents.
 * For this method we let LLM decide if the query is of a broad nature or if
 * it is a pointed query. We use this for further routing.
 * ********************************************************************/
private async Task<List<LangChain.Providers.Message>>? GetContextForAboveThresholdQuery()
{
    //var sw=StartStopwatch();

    // the following flags are set to true after evaluating the new question for revisions and comments
    UserQueryCheckedForRevisions = false;
    UserQueryCheckedForComments = false;

    GlobalQueryExecutedUsingPhraseDoc = false;

    // check if we need to do filtration routing
    bool filterApplied = false;
    if (FldUseFilter.Checked)
    {
        ClsFilterResult result = await ApplyFilter();

        if (result.payload != null) return result.payload;

        if (result.FilterApplied)
        {  //
            filterApplied = true;
            globalQuery = true;   // now treat this as a broad query over a narrow subset of files after filtration
        }

    }

    if (!filterApplied)
    {
        foreach (ClsFiles file in files) file.InFilter = true;  // assume all files in filter
        globalQuery = await IsGlobalQuery(newUserQuestion);     // Let AI determine the nature of the user query
    }
    //LogTime(sw, "after global query");

    return await GetRetrievalEnrichmentContext();
}

/******************************************************************************
    IsGlobalQuery:
    Determine the scope of the user question.
    This could be an expensive call.  We recorded it at 1600 ms
*******************************************************************************/
private async Task<bool> IsGlobalQuery(string userQuestion)
{
    try
    {
        // A quick, low-cost system prompt to categorize the question intent
        string routingPrompt = @"
       Analyze the user's question about a document. Categorize it into one of two strategies:
       - 'GLOBAL': The question requires aggregation, math, summaries across the whole file, or looking at data over many different pages (e.g., 'What is the average revenue?', 'Summarize the whole profile', 'List all names mentioned').
       - 'LOCAL': The question looks for a specific rule, fact, step, or localized piece of text (e.g., 'How do I modify the budget?', 'What is the subject's birthdate?').
       When in doubt, categorize as 'GLOBAL'.

       Respond with ONLY the word 'GLOBAL' or 'LOCAL'.";

        var activeChat = miniChat;
        if (FldUseFilter.Checked) activeChat = fullChat;   // when using the filter, leave miniChat just for filtering so filtering can execute with discounted cache pricing

        ChatCompletion decision = await activeChat.CompleteChatAsync(
            new List<ChatMessage> {
               new SystemChatMessage(routingPrompt),
               new UserChatMessage(userQuestion)
            },
            new ChatCompletionOptions { MaxOutputTokenCount = 50 });
        string answer = decision.Content[0].Text.Trim().ToUpperInvariant();
        return answer.StartsWith("GLOBAL");
    }
    catch (Exception)
    {
        LblStatus.Text = "Exception in IsGlobalQuery";
        return true;  // in the unlikely case of a crash, use the global strategy
    }
}

AI document filter: building the index and routing on it

Creating the collection index with VesCreateDocumentIndex, sending it with the question to the routing model using VesGetFilterSystemPrompt, and applying the confidence threshold to decide between whole-document, filtered and global modes.

/****************************************************************************
 * ApplyFilter:
 * When the filter is enabled, we use the combined phrase document to let AI
 * determine the subset of files that are needed to satisfy the user query.
 * If AI determines that all files are necessary, then if the
 * 'Use filter Phrase Document to answer broad query' checkbox is checked,
 * we use the filter phrase document as the context to answer the broad
 * query. When this checkbox is not checked, the context for the global
 * query is built as normal using retrieval/enrichment.
 * You may want to present the user with the answers from both routes.
 * **************************************************************************/
private async Task<ClsFilterResult> ApplyFilter()
{
    string dynamicSystemPrompt = "";
    LangChain.Providers.Message dynamicMessage;

    ClsFilterResult result = new ClsFilterResult();

    // ensure that all files have a phrase document
    if (IndexDoc == "")
    {
        filterTokenCount = 0;
        foreach (ClsFiles file in files)
        {
            if (file.PhraseDoc == "") file.PhraseDoc = file.ven.VesCreateDocumentIndex(filterTokenPct, out file.FilterTokens);

            if (IndexDoc.Length > 0) IndexDoc += "\n\n";
            IndexDoc += file.PhraseDoc;
            filterTokenCount += file.FilterTokens;
        }
    }
    foreach (ClsFiles file in files) file.InFilter = false;  // reset

    // to take full advantage of caching, we place the static content before the chat history
    dynamicSystemPrompt = Ven.VesGetFilterSystemPrompt() +
                 "\n\nUSE THIS DOCUMENT INDEX CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                 IndexDoc;

    // create a combined list: system message followed by the filter chat history
    dynamicMessage = new LangChain.Providers.Message(dynamicSystemPrompt, LangChain.Providers.MessageRole.System);

    filterChatHistory.Add(new LangChain.Providers.Message(newUserQuestion, LangChain.Providers.MessageRole.Human));
    List<LangChain.Providers.Message> payload = [dynamicMessage, .. filterChatHistory];  // .. appends all elements of the filterChatHistory list

    // When the filter is enabled, miniChat (Luna) is reserved for
    // only examining the user question against the collection-wide document index.
    // This ensures that we are not only charged the lower miniChat cost, but also that such calls
    // are highly discounted due to an assured cache hit.
    string answer = await SendToAI(miniChat, payload, filterChatHistory, false);

    answer = answer.Trim().ToUpper();

    // Does AI think that no document is relevant, or that all documents are needed
    // to answer the user question?
    // If so, we follow the usual route of retrieval/enrichment over the entire
    // collection.
    if (answer == "NONE" || answer == "ALL") return result;
    else if (answer == "INDEX")
    {  // the answer is found in the index document, so build a payload using the index document
        if (!FldUsePhraseDocForBroadQuery.Checked) return result;   // The user does not want to use the index, go back to using full retrieval/enrichment
        // In your app, you might present the user the answers from both sources, indicating the source: file collection or index document

        dynamicSystemPrompt = Ven.VesGetIndexSystemPrompt() + IndexDoc;

        // skip adding the user prompt because it was already added to chatHistory at the beginning of the FldSend event handler

        // create a combined message and the payload
        dynamicMessage = new LangChain.Providers.Message(dynamicSystemPrompt, LangChain.Providers.MessageRole.System);
        result.payload = [dynamicMessage, .. chatHistory];

        TokenMessage = $"Answered from collection index ({files.Count} documents). " + $"Number of index document tokens sent: {filterTokenCount}";

        result.FilterApplied = true;
        return result;
    }
    else
    {    // list of doc-id and confidence level, example:  TE582281:92|TE851927:70|TE642026:35   (expect spaces between the elements)
        // check if we have at least 70% confidence level
        int RequiredConfidenceLevel = 70;  // you can change this to any level needed by your application
        bool WeHaveConfidence = false;

        // extract doc-ids and confidence levels
        List<string> DocIds = new List<string>();
        string[] items = answer.Split('|');
        int NumItems = items.Length;
        for (int i = 0; i < NumItems; i++)
        {
            items[i] = items[i].Trim();
            string[] subitems = items[i].Split(":");
            if (subitems.Length == 2)
            {
                string id = subitems[0].Trim();
                if (string.IsNullOrEmpty(id)) continue;
                if (!int.TryParse(subitems[1].Trim(), out int lvl)) lvl = 0;
                if (lvl >= RequiredConfidenceLevel) WeHaveConfidence = true;
                DocIds.Add(id);
            }
        }

        if (!WeHaveConfidence) return result;  // can't use the filter, resort to the regular retrieval/enrichment

        int tokens = 0;
        if (filesDict == null) return result;  // this should not happen
        foreach (string DocId in DocIds)
        {
            if (filesDict.TryGetValue(DocId, out ClsFiles? file))
            {
                file.InFilter = true;
                tokens += file.TokenCount;
            }
        }

        if (tokens == 0) return result;  // no files found.  This should not happen

        // check if the token count of the filtered file set is less than the user allowance. In that case, send the entire context
        int AveTokenPerChunk = TotalTokenCount / TotalChunkCount;
        int RetrievalChunkLimit = TotalChunkCount * retrievalPct;
        int RetrievalTokenLimit = AveTokenPerChunk * RetrievalChunkLimit / 100;
        int TotalUserAllowance = RetrievalTokenLimit + TotalTokenCount * enrichmentPct / 100;  // the user is okay with sending up to this many tokens

        // Exceeded: go through the regular retrieval/enrichment over the filtered set.
        // The fewer files in the filter, the stronger the context sent to AI, because the token allowance gets spread over fewer files.
        // Filtration here provides strong context for the filtered files to send to AI.
        if (tokens > TotalUserAllowance)
        {
            result.FilterApplied = true;
            return result;
        }

        // Great success!  We can send each one of these files completely
        // and still stay within the user token allowance

        // get a single flattened collection of all chunks:
        string[] allChunksArray = files
            .Where(f => f.InFilter)
            .SelectMany(f => f.chunk)
            .ToArray();

        // combined together:
        string allChunksJoined = string.Join("\n\n", allChunksArray);

        dynamicSystemPrompt = Dan.DasGetSystemMessage() +
                             "\n\nUSE THIS EXTRACTED DOCUMENT CONTEXT TO ANSWER THE USER'S QUESTION:\n" + allChunksJoined;

        // No need to add the user prompt because it was already added to chatHistory at the beginning of the FldSend event handler

        // create a combined message and the payload
        dynamicMessage = new LangChain.Providers.Message(dynamicSystemPrompt, LangChain.Providers.MessageRole.System);
        result.payload = [dynamicMessage, .. chatHistory];

        TokenMessage = $"Answered from filtered file collection ({DocIds.Count} documents). " + $"Number of document tokens sent: {tokens}";

    }


    return result;
}

Retrieval: LOCAL and GLOBAL strategies

One collection-wide search for pointed questions; for broad questions, a coverage share is spread across every file and the remainder is distributed in proportion to each file's hits, then each file is searched with its own quota.

/************************************************************************
 * GetRetrievalEnrichmentContext:
 * The user asked a broad question.  To get the content, we will:
 * 1. Do retrieval over the entire collection.
 * 2. Isolate the chunks in the retrieval for each file.
 *    For global query:
 *    2a. Calculate the proportion of each file in the hits.  Say file #2 gets 60% of the hits, file #4 gets the remaining chunks,
 *        and the other files are not represented in the retrieval search.
 *    2b. Find the chunk quota for each file this way:
 *        Distribute the 'coverage share' percentage of the total allowed retrieval chunks equally to all files
 *        whether they were found in the retrieval set or not.  This is what makes the search global.
 *        However, some files may not get any if there are not enough to share.
 *        The filtration routing step remedies this gap. The filtration reduces the number of files of
 *        interest for the current query.
 *    2c. Find the retrieved chunks for each file using the quota calculated in step 2b.
 *        (steps 2a through 2c are performed in the DoGlobalDistribution method below)
 * 3. Do enrichment for each file with retrieval, thus expanding the chunks of interest.
 * 4. Accumulate the chunks of interest for each file in a common array.
 * 5. Add the system message and chat history to make the final payload.
 * ********************************************************************/
private async Task<List<LangChain.Providers.Message>>? GetRetrievalEnrichmentContext()
{
    var sw = StartStopwatch();

    // initialize the retrieved and enriched sequences before adding new retrieved sequences
    foreach (ClsFiles file in files)
    {
        file.RetrieveSeq = null;
        file.EnrichedSeq = null;
        file.RetrievalTokenCount = 0;
        file.EnrichmentTokenCount = 0;
    }

    float[] queryVector = await GetQueryVector();

    int searchLimit = (TotalChunkCount * retrievalPct + 99) / 100;   // +99 to round up
    if (searchLimit < 1) searchLimit = 1;

    var searchSettings = new VectorSearchSettings
    {
        NumberOfResults = searchLimit   // our search limit
    };

    IVectorCollection? CurVectorCollection = await GetVectorCollection(false);   // collection containing documents that are in filter

    LogTime(sw, "GetVectorCollection");

    VectorSearchResponse searchResponse =
        await CurVectorCollection.SearchAsync(queryVector, searchSettings);
    LogTime(sw, "SearchAsync");

    // Build RetrieveSeq for each file from the common searchResponse result
    // Create a dictionary lookup map: O(M) time complexity
    // Assumes DocId is unique across files. If not unique, use ToLookup() instead.
    TotalRetrievalTokens = 0;   // will be calculated below
    foreach (ClsFiles file in files) file.RetrievalTokenCount = 0;

    foreach (var item in searchResponse.Items)
    {
        if (item.Metadata == null) continue;

        // Safely extract metadata (avoids potential KeyNotFoundException)
        if (!item.Metadata.TryGetValue("ssDocId", out var docIdObj) ||
            !item.Metadata.TryGetValue("ssSeq", out var seqObj))
            continue;

        string docId = (string)docIdObj;
        int seq = Convert.ToInt32(seqObj);

        // O(1) constant-time lookup instead of scanning the whole list
        if (filesDict.TryGetValue(docId, out ClsFiles? file))
        {
            // Null-coalescing assignment ensures initialization happens only once
            file.RetrieveSeq ??= new SortedSet<int>();
            file.RetrieveSeq.Add(seq);

            int SeqTokenCount = file.ven.VesGetChunkTokenCount(seq);

            TotalRetrievalTokens += SeqTokenCount;  // for the collection
            file.RetrievalTokenCount += SeqTokenCount;  // for this file
        }
    }
    LogTime(sw, "build retrieval seq");

    if (globalQuery) await DoGlobalDistribution();
    LogTime(sw, "DoGlobalDistribution");

    // do Vector Enrichment for the retrieved sequences for each file

    int QueryRetrievalTokens = 0;

    int QueryEnrichmentTokens = 0;
    foreach (ClsFiles file in files)
    {
        if (!file.InFilter || file.RetrieveSeq == null) continue;

        await DoVectorEnrichment(file);

        QueryRetrievalTokens += file.RetrievalTokenCount;
        QueryEnrichmentTokens += file.EnrichmentTokenCount;
    }
    LogTime(sw, "DoVectorEnrichment");

    // collect the chunks for the query
    StringBuilder sb = new StringBuilder();

    foreach (ClsFiles file in files)
    {
        if (file.chunk == null || file.RetrieveSeq == null || !file.InFilter) continue;

        if (sb.Length > 0) sb.Append("\n\n");

        if (file.EnrichedSeq != null)
        {
            if (file.EnrichedSeq.Count == file.chunk.Length) // all chunks in the enriched sequence
            {
                sb.Append(string.Join("\n\n", file.chunk));
            }
            else
            {
                // LINQ: Pulls only the chunks at the specified indices and joins them
                var selectedChunks = file.EnrichedSeq.Select(seq => file.chunk[seq]);
                sb.Append(string.Join("\n\n", selectedChunks));
            }
        }
        else
        {
            if (file.RetrieveSeq.Count == file.chunk.Length)   // all chunks in the retrieved sequence
            {
                sb.Append(string.Join("\n\n", file.chunk));
            }
            else
            {
                // LINQ: Pulls only the chunks at the specified indices and joins them
                var selectedChunks = file.RetrieveSeq.Select(seq => file.chunk[seq]);
                sb.Append(string.Join("\n\n", selectedChunks));
            }
        }
    }

    string chunks = sb.ToString();

    LogTime(sw, "collect chunks");

    // To take full advantage of caching, we place the static content before the chat history
    string dynamicSystemPrompt = Dan.DasGetSystemMessage() +
                 "\n\nUSE THIS EXTRACTED DOCUMENT CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                 chunks;

    // create a combined list: system message followed by the chat history
    LangChain.Providers.Message dynamicMessage = new LangChain.Providers.Message(dynamicSystemPrompt,
                                                                         LangChain.Providers.MessageRole.System);
    List<LangChain.Providers.Message> payload = [dynamicMessage, .. chatHistory];

    TokenMessage = $"Number of document tokens sent: {filterTokenCount + QueryRetrievalTokens + QueryEnrichmentTokens}";
    if (FldUseFilter.Checked) TokenMessage += $", Filter tokens: {filterTokenCount} (uses cache)";
    TokenMessage += $", Retrieval tokens: {QueryRetrievalTokens}, Enrichment tokens: {QueryEnrichmentTokens}";
    if (globalQuery) TokenMessage += ", Broad query.";
    else TokenMessage += ", Pointed query.";

    return payload;
}

/*****************************************************************************
 * DoGlobalDistribution:
 *    1. Calculate the proportion of each file in the hits.  Say file #2 gets 60% of the hits, file #4 gets the remaining chunks,
 *       and the other files are not represented in the retrieval search.
 *    2. Find the chunk quota for each file this way:
 *       Distribute the 'coverage share' percentage of the total allowed retrieval chunks equally to all files
 *       whether they were found in the retrieval set or not.  This is what makes the search global.
 *       However, some files may not get any if there are not enough to share.
 *       The filtration routing step remedies this gap. The filtration reduces the number of files of
 *       interest for the current query.
 *    3. Find the retrieved chunks for each file using the quota calculated in step 2.
 ****************************************************************************/
private async Task DoGlobalDistribution()
{
    int TotalRetrievalQuota = TotalChunkCount * retrievalPct / 100;   // number of chunks for equal distribution among the files
    int TotalGlobalQuota = TotalRetrievalQuota * globalSpreadPct / 100;

    int FileCount = files.Count;
    int ActualGlobalQuota = 0;
    var sw = StartStopwatch();

    foreach (ClsFiles file in files)
    {
        file.quota = 0;
        if (!file.InFilter) continue;

        file.quota = TotalGlobalQuota / FileCount;
        if (file.quota < 1) file.quota = 1;
        ActualGlobalQuota += file.quota;
    }

    int RemainingQuota = TotalRetrievalQuota - ActualGlobalQuota;   // left over to spread over the files with hits

    // apportion the remaining quota to the files with hits
    foreach (ClsFiles file in files)
    {
        if (!file.InFilter) continue;

        if (file.RetrieveSeq != null)    // file.RetrieveSeq comes from the previous step, when we did the first vector search over the entire collection to find the files with hits
        {
            file.quota += (RemainingQuota * file.RetrieveSeq.Count) / TotalRetrievalQuota;
        }
        file.RetrievalTokenCount = 0;  // reinitialize for global query
    }

    // Calculate RetrieveSeq for each file using its quota
    TotalRetrievalTokens = 0;  // total retrieval for the collection

    float[] queryVector = await GetQueryVector();

    foreach (ClsFiles file in files)
    {
        if (!file.InFilter) continue;

        file.RetrieveSeq = new SortedSet<int>();

        var searchSettings = new VectorSearchSettings
        {
            NumberOfResults = file.quota   // get this many hits
        };

        VectorSearchResponse searchResponse =
            await file.vectorCollection.SearchAsync(queryVector, searchSettings);

        // Build RetrieveSeq for this file from the searchResponse result
        foreach (var item in searchResponse.Items)
        {
            if (item.Metadata == null) continue;  // shouldn't happen

            // Safely extract metadata (avoids potential KeyNotFoundException)
            if (!item.Metadata.TryGetValue("ssSeq", out var seqObj)) continue;

            int seq = (int)seqObj;
            file.RetrieveSeq.Add(seq);

            int SeqTokenCount = file.ven.VesGetChunkTokenCount(seq);
            file.RetrievalTokenCount += SeqTokenCount;  // for this file
        }
    }
    LogTime(sw, "do global inner");
}

/******************************************************************************
    GetQueryVector:
    Get the vector for the latest query
*******************************************************************************/
private async Task<float[]>? GetQueryVector()
{
    // This will return an array containing your single numerical coordinate array.
    float[][] embeddingBatch = await embeddingsProvider.CreateEmbeddingsAsync(new[] { newUserQuestion });

    // Extract the first (and only) vector element from the returned batch
    return embeddingBatch[0];
}

Enriching retrieved chunks with Ven

For each document with hits: VesBeginExpansion, VesAddRelatedChunks to complete tables, revisions and comments under a token budget, optional revision and comment additions, then VesGetExpandedSeq. The two classifiers are available when an application wants to gate revision and comment enrichment on the question.

/*****************************************************************************
 * DoVectorEnrichment:
 * Use the Vector Enrichment Library to add structurally related chunks to
 * the retrieved set of sequences.
******************************************************************************/
private async Task DoVectorEnrichment(ClsFiles file)
{
    // In this method, we are doing 'completion' enrichment if the document
    // has tables, revisions, or comments.
    // However, if you wish to perform other types of enrichment,
    // please remove this 'if' condition and use the various methods from the
    // Vector Enrichment Library (vei package) to perform enrichment
    // suitable for your application.
    Stopwatch sw = StartStopwatch();

    if (!file.HasTables && !file.HasRevisions && !file.HasComments)
    {
        file.EnrichedSeq = file.RetrieveSeq;
        file.EnrichmentTokenCount = 0;  // no enrichment
        return;
    }

    bool CheckForRevisionCommentUsingAI = false;   // this could be an expensive call, about 900 ms for the first call
    if (CheckForRevisionCommentUsingAI)
    {
        if (file.HasRevisions && !UserQueryCheckedForRevisions)
        {
            QueryIsAboutRevisions = await IsQueryAboutRevisions(newUserQuestion);
            UserQueryCheckedForRevisions = true;
        }

        if (file.HasComments && !UserQueryCheckedForComments)
        {
            QueryIsAboutComments = await IsQueryAboutComments(newUserQuestion);
            UserQueryCheckedForComments = true;
        }
    }
    else QueryIsAboutRevisions = QueryIsAboutComments = true; // Enrichment Library calls are much more efficient,
                                                              // so let's enrich for revisions/comments if the file has them,
                                                              // without using AI to first check the query type as above

    LogTime(sw, "after AI query");

    if (TotalTokenCount == 0) return;

    int TotalEnrichmentBudget = TotalTokenCount * enrichmentPct / 100;  // number of tokens budgeted to add for structural enrichment
    float EnrichmentBudgetPerRetrievalToken = (float)TotalEnrichmentBudget / TotalRetrievalTokens;

    int EnrichmentBudget = (int)(EnrichmentBudgetPerRetrievalToken * file.RetrievalTokenCount);  // allocate the budget in proportion to the number of tokens retrieved for the file

    // The Vector Enrichment Library allows you to carry out enrichment at a granular level.
    // Your application can choose to add the enrichment relevant to you.
    // For the sake of simplicity in this demo, we will use the big-hammer method VesAddRelatedChunks, forsaking the granular control
    // offered by the other methods.
    // The Vector Enrichment Library includes two types of enrichment:
    // 1. Completion type:  This includes adding surrounding chunks so that tables, revisions, and comments are presented to AI
    //    in a structurally complete manner.  An incomplete table is worse than no table at all.
    // 2. Addition type: This type includes the addition of pages, and spread-out revisions and comments, to present comprehensive
    //    information to AI.
    // Of the two above, the completion types are the most critical to incorporate.


    Object exp = null;   // Expansion object - the object that carries out enrichment expansion
    try
    {
        exp = file.ven.VesBeginExpansion(file.RetrieveSeq);  // For each query, create the expansion object passing the zero-based retrieved sequences
    }
    catch (ArgumentException ex)
    {
        LblStatus.Text = $"Error creating expansion object: {ex.Message}";
    }
    LogTime(sw, "after creating exp object");

    if (file.ven != null && exp != null)
    {
        //int InitialTokenCount=file.ven.VesGetTokenCount(exp);  // example of finding the number of tokens in the retrieved sequences


        int TokensAdded = file.ven.VesAddRelatedChunks(exp,       // current expansion object
                                file.HasRevisions && QueryIsAboutRevisions,  //    True to include all contiguous chunks containing revisions, to complete the revisions for the reviewer authors
                                                                             //    referred to in the original set
                                file.HasComments && QueryIsAboutComments,    //    True to include all contiguous chunks containing comments, to complete the comments by the authors
                                                                             //    referred to in the original set

                     /*AllTables*/ false,       //    Set to 'true' to include all tables in the document.
                                                //    If set to false, this method ensures that all tables in the original set and
                                                //    partial adjacent tables are included in the final set.
                                                //    Both options ensure no orphan or incomplete table is included in the final set.
                                                //    It is important that AI is given complete context to decipher
                                                //    the contextual relationship between table columns, and the significance of
                                                //    the spanned rows and columns.
                                                //    This parameter is ignored if the document contains no tables.

               /*AdjacentTables*/   true,       //    When AllTables is false but AdjacentTables is true, the method not only completes
                                                //    the tables in the original set, but also completes adjacent tables found in the expanded chunks.
                                                //    This ensures that AI does not receive any incomplete tables.

                              EnrichmentBudget);//    Token Budget: try to limit the token count of the selected chunks to this limit. Internally,
                                                //    when this limit is reached, this method
                                                //    turns off all 'addition' such as adding pages or sections.  Only the 'completion' operations
                                                //    are performed, such as completing the selected table, revision, or comment chunks.
                                                //    A partial table confuses AI, and could be worse than no table at all.
                                                //    Pass a very large value for no limit (not recommended).

        LogTime(sw, $"tokens added: {TokensAdded}");

        file.EnrichmentTokenCount = TokensAdded;


        if (TokensAdded < EnrichmentBudget)
        {
            // If the question is about commented text, comment authors or document revisions
            if (file.HasComments && QueryIsAboutComments)
            {
                if (file.EnrichmentTokenCount < EnrichmentBudget) TokensAdded = file.ven.VesAddCommentChunks(exp, "");  // if the budget allows, do this addition
                file.EnrichmentTokenCount = TokensAdded;
            }

            if (file.HasRevisions && QueryIsAboutRevisions)
            {
                if (file.EnrichmentTokenCount < EnrichmentBudget) TokensAdded = file.ven.VesAddRevisionChunks(exp, "");
            }
        }

        file.EnrichmentTokenCount = TokensAdded; // TokensAdded returned from Ven methods is the cumulative enrichment tokens added

        //Ven.LogPrintf("initial/budget/added", InitialTokenCount, EnrichmentBudget,file.EnrichmentTokenCount);

        file.EnrichedSeq = file.ven.VesGetExpandedSeq(exp);  // get the expanded set of sequences

        file.ven.VesEndExpansion(exp);  // end the expansion

        LogTime(sw, "ves exp end, file: " + file.DocTitle);
    }
    else
    {
        file.EnrichedSeq = file.RetrieveSeq;
        file.EnrichmentTokenCount = 0;  // no enrichment
    }

    //LogTime(sw, "ves exp end ");

}

/******************************************************************************
    IsQueryAboutRevisions:
    Is the user querying about inserted and deleted text (redlining)?
*******************************************************************************/
private async Task<bool> IsQueryAboutRevisions(string userQuestion)
{
    try
    {
        // A quick, low-cost system prompt to categorize the question intent
        string routingPrompt = @"
       Analyze the user's question about a document. Categorize it into one of two strategies:
       - 'YES': The question is related to the revisions made by one or more reviewers to the document, such as 'What changes were made by Mary Hoffins?', 'Who made revisions to the document?'.
       - 'NO': The question is not about revisions made to the document, example: 'What is the warranty period?'.
       When in doubt, categorize as 'YES'.

       Respond with ONLY the word 'YES' or 'NO'.";

        var activeChat = miniChat;
        if (FldUseFilter.Checked) activeChat = fullChat;   // when using the filter, leave miniChat just for filtering so filtering can execute with discounted cache pricing

        ChatCompletion decision = await activeChat.CompleteChatAsync(
            new List<ChatMessage> {
               new SystemChatMessage(routingPrompt),
               new UserChatMessage(userQuestion)
            },
            new ChatCompletionOptions { MaxOutputTokenCount = 50 });
        return decision.Content[0].Text.Trim().ToUpperInvariant().StartsWith("YES");
    }
    catch (Exception)
    {
        LblStatus.Text = "Exception in IsQueryAboutRevisions";
        return false;  // in the unlikely case of a crash, assume the query is not about revisions
    }

}

/******************************************************************************
    IsQueryAboutComments:
    Is the user querying about document comments?
*******************************************************************************/
private async Task<bool> IsQueryAboutComments(string userQuestion)
{
    try
    {
        // A quick, low-cost system prompt to categorize the question intent
        string routingPrompt = @"
       Analyze the user's question about a document. Categorize it into one of two strategies:
       - 'YES': The question is related to the commented text and comment authors in this document, such as 'Which changes were made by John?', 'Which text did he comment on?', 'What were Mary's comments?'.
       - 'NO': The question is not about comments or commented text, example: 'What are the revenue items?'.
       When in doubt, categorize as 'YES'.

       Respond with ONLY the word 'YES' or 'NO'.";

        var activeChat = miniChat;
        if (FldUseFilter.Checked) activeChat = fullChat;   // when using the filter, leave miniChat just for filtering so filtering can execute with discounted cache pricing

        ChatCompletion decision = await activeChat.CompleteChatAsync(
            new List<ChatMessage> {
               new SystemChatMessage(routingPrompt),
               new UserChatMessage(userQuestion)
            },
            new ChatCompletionOptions { MaxOutputTokenCount = 50 });

        return decision.Content[0].Text.Trim().ToUpperInvariant().StartsWith("YES");
    }
    catch (Exception)
    {
        LblStatus.Text = "Exception in IsQueryAboutComments";
        return false;  // in the unlikely case of a crash, assume the query is not about comments
    }

}

Sending the payload to the model

The chat call uses the official OpenAI .NET SDK; LangChain messages are converted to SDK messages first. The system message from Dan.DasGetSystemMessage() plus the selected chunks is placed ahead of the chat history so the static part is served from cache.

/**********************************************************************
 * SendToAI:
 * Send the payload to AI
 * *******************************************************************/
private async Task<string> SendToAI(OpenAI.Chat.ChatClient activeChat, List<LangChain.Providers.Message> payload, List<LangChain.Providers.Message> history, bool UpdateUI)
{
    string responseText = "";

    try
    {
        // Using OpenAI for actual chat
        ChatCompletion completion = await activeChat.CompleteChatAsync(
            ConvertLangChainToOpenAiMessage(payload),
            new ChatCompletionOptions
            {
                MaxOutputTokenCount = 4000    // replaces MaxCompletionTokens = 1000; no Temperature
            });

        responseText = completion.Content[0].Text;

        // Track the text string in the history and update the HTML view
        history.Add(new LangChain.Providers.Message(responseText, LangChain.Providers.MessageRole.Ai));

        if (UpdateUI)
        {
            htmlResponse = PrevHtmlResponse;  // restore
            UpdateChatBox(newUserQuestion, responseText);

            LblStatus.Text = TokenMessage;
            FldQuery.Text = ""; // clear for the next question
        }
    }
    catch (tryAGI.OpenAI.ApiException apiEx)
    {
        history.RemoveAt(history.Count - 1);  // Since we have an exception, meaning no AI response, undo the unanswered question as well from the history
        MessageBox.Show($"Error(sta-end): {apiEx.Message}\n\nBody: {apiEx.ResponseBody}");
    }
    catch (Exception ex)
    {
        history.RemoveAt(history.Count - 1);  // Since we have an exception, meaning no AI response, undo the unanswered question as well from the history
        MessageBox.Show($"Error(sta2-end): {ex.Message}");    // short message
    }

    return responseText;
}

/*********************************************************************
   ConvertLangChainToOpenAiMessage:
   Convert a LangChain message list to OpenAI SDK messages
**********************************************************************/
static List<ChatMessage> ConvertLangChainToOpenAiMessage(IEnumerable<LangChain.Providers.Message> msgs)
{
    var list = new List<ChatMessage>();
    foreach (var m in msgs)
    {
        switch (m.Role)
        {
            case LangChain.Providers.MessageRole.System:
                list.Add(new SystemChatMessage(m.Content)); break;
            case LangChain.Providers.MessageRole.Ai:
                list.Add(new AssistantChatMessage(m.Content)); break;
            default:  // Human/user
                list.Add(new UserChatMessage(m.Content)); break;
        }
    }
    return list;
}

Documentation

The complete help for RAG Document Toolkit: license agreement, getting started, a step-by-step code example, and the full Dan and Ven API references.

using SubSystems.RagDocumentToolkit.Dan;
  using SubSystems.RagDocumentToolkit.Ven;
  

<PackageReference Include="dai" Version="1.0.0.6" />
  <PackageReference Include="vei" Version="1.0.0.5" />
  

<PackageReference Include="LangChain" Version="0.17.0" />
  <PackageReference Include="LangChain.Core" Version="0.17.0" />
  <PackageReference Include="LangChain.DocumentLoaders.Abstractions" Version="0.17.0" />
  <PackageReference Include="LangChain.Providers.OpenAI" Version="0.17.0" />
  <PackageReference Include="LangChain.Databases.InMemory" Version="0.17.0" />
  <PackageReference Include="LangChain.Databases.Sqlite" Version="0.17.0" />
  

int result = Dan.DasSetLicenseInfo("your-license-key", "your-license-number", "Your Company Name");
  

using SubSystems.RagDocumentToolkit.Dan;

  Dan dan = new Dan();
  dan.LogMsg   += LogDanMsg;    // optional: receive log messages
  dan.MdoNames += MdoNames;     // optional: supply names and terms for each chunk (see 1b)

  // import the file into the Dan object
  if (!dan.DasImportFile(NewFile, GetDocType(NewFile))) return false;

  // get the Markdown chunks and the metadata records (first page, last page; -1 = the entire document)
  Dan.DanResult mdo = dan.DasGetMarkdown(-1, -1);
  if (mdo == null) return false;

  // save these to your vector database
  file.DocId    = mdo.DocId;      // unique id for the document
  file.DocTitle = mdo.DocTitle;   // title found in the document, or the file path when there is none
  file.chunk    = mdo.chunks;     // string[]: the Markdown text of each chunk
  file.MetaRec  = mdo.MetaRecs;   // Dictionary<string,object>[]: one record per chunk
  

// PersonRx, OrgRx, Places, Terms and ContainsWord belong to your application
  Dan.ClsMdoNames MdoNames(object sender, string DocTitle, int ChunkIndex, string text)
  {
      var names   = new Dan.ClsMdoNames();
      var persons = new HashSet<string>();
      var orgs    = new HashSet<string>();

      // capture group 1 is the name without the honorific
      foreach (Match m in PersonRx.Matches(text)) persons.Add(m.Groups[1].Value.Trim());

      // capture group 1 is the full organization name
      foreach (Match m in OrgRx.Matches(text)) orgs.Add(m.Groups[1].Value.Trim());

      names.PersonNames    = persons.ToArray();
      names.OrgNames       = orgs.ToArray();
      names.PlaceNames     = Places.Where(p => ContainsWord(text, p)).ToArray();
      names.ImportantTerms = Terms.Where(t => ContainsWord(text, t)).ToArray();

      return names;
  }
  

using SubSystems.RagDocumentToolkit.Ven;

  // create the Vector Enrichment object from the document's complete metadata record set
  try
  {
      file.ven = new Ven(file.MetaRec);
  }
  catch (Exception ex)
  {
      LblStatus.Text = "Error creating Vector Enrichment object: " + ex.Message;
      return false;
  }

  // what does the document contain?
  file.HasTables    = file.ven.VesHasTables();
  file.HasRevisions = file.ven.VesHasRevisions();
  file.HasComments  = file.ven.VesHasComments();
  file.TokenCount   = file.ven.VesGetTokenCount();    // token count of the whole document

  // store each chunk with its metadata record in the vector database
  var docs = new List<Document>();
  for (int i = 0; i < file.chunk.Length; i++)
      docs.Add(new Document(content: file.chunk[i], metadata: file.MetaRec[i]));

  await vectorCollection.AddDocumentsAsync(embeddingsProvider, docs);

  files.Add(file);
  filesDict[file.DocId] = file;    // look up a file by its document id
  

// Build the index document once. Rebuild it when files are added or removed.
  if (IndexDoc == "")
  {
      foreach (ClsFiles file in files)
      {
          // index entry for one file, sized as a percentage of the file's tokens
          file.PhraseDoc = file.ven.VesCreateDocumentIndex(filterTokenPct, out file.FilterTokens);

          if (IndexDoc.Length > 0) IndexDoc += "\n\n";
          IndexDoc += file.PhraseDoc;
      }
  }

  // Static content goes first, so the index document is billed at the cached-input rate
  string systemPrompt = Ven.VesGetFilterSystemPrompt() +
                        "\n\nUSE THIS DOCUMENT INDEX CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                        IndexDoc;

  // AskModel is your own function. Use a low-cost model for this call.
  string answer = (await AskModel(miniChat, systemPrompt, newUserQuestion)).Trim().ToUpper();
  

foreach (ClsFiles f in files) f.InFilter = false;

  if (answer == "NONE" || answer == "ALL")
  {
      // no routing: regular retrieval and enrichment over the whole collection (steps 4 and 5)
  }
  else if (answer == "INDEX")
  {
      // answer the question from the index document itself
      systemPrompt = Ven.VesGetIndexSystemPrompt() + IndexDoc;
  }
  else
  {
      // example:  TE582281:92 | TE851927:70 | TE642026:35
      int  RequiredConfidenceLevel = 70;    // choose the level that suits your application
      bool confident = false;
      int  tokens = 0;

      foreach (string item in answer.Split('|'))
      {
          string[] parts = item.Split(':');
          if (parts.Length != 2 || !int.TryParse(parts[1].Trim(), out int level)) continue;

          if (level >= RequiredConfidenceLevel) confident = true;

          if (filesDict.TryGetValue(parts[0].Trim(), out ClsFiles? file))
          {
              file.InFilter = true;
              tokens += file.TokenCount;
          }
      }

      if (!confident || tokens == 0)
      {
          // no document stands out: regular retrieval and enrichment over the whole collection
      }
      else if (tokens <= TotalUserAllowance)
      {
          // the selected files fit within the token allowance: send them whole
          string context = string.Join("\n\n", files.Where(f => f.InFilter).SelectMany(f => f.chunk));

          systemPrompt = Dan.DasGetSystemMessage() +
                         "\n\nUSE THIS EXTRACTED DOCUMENT CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                         context;
      }
      else
      {
          // too large to send whole: run steps 4 and 5 over the selected files only,
          // treating the question as a broad query over this narrow set of files.
          // The token allowance is now spread over fewer files, so each gets stronger context.
          globalQuery = true;
      }
  }
  

VectorSearchResponse searchResponse = await vectorCollection.SearchAsync(queryVector, searchSettings);

  TotalRetrievalTokens = 0;
  foreach (var item in searchResponse.Items)
  {
      if (item.Metadata == null) continue;

      // ssDocId and ssSeq are written by Dan. A chunk without them did not come from Dan.
      if (!item.Metadata.TryGetValue("ssDocId", out var docIdObj) ||
          !item.Metadata.TryGetValue("ssSeq",   out var seqObj)) continue;

      if (filesDict.TryGetValue(Convert.ToString(docIdObj) ?? "", out ClsFiles? file))
      {
          int seq = Convert.ToInt32(seqObj);           // zero-based chunk sequence number

          file.RetrieveSeq ??= new SortedSet<int>();
          file.RetrieveSeq.Add(seq);

          int seqTokens = file.ven.VesGetChunkTokenCount(seq);
          file.RetrievalTokenCount += seqTokens;       // for this file
          TotalRetrievalTokens     += seqTokens;       // for the collection
      }
  }
  

// Share the enrichment budget among the files in proportion to their retrieved tokens
  int TotalEnrichmentBudget = TotalTokenCount * enrichmentPct / 100;
  int EnrichmentBudget = (int)((float)TotalEnrichmentBudget / TotalRetrievalTokens * file.RetrievalTokenCount);

  // Create an expansion object for this query from the retrieved sequence numbers
  object exp;
  try
  {
      exp = file.ven.VesBeginExpansion(file.RetrieveSeq);
  }
  catch (ArgumentException ex)
  {
      LblStatus.Text = "Error creating expansion object: " + ex.Message;
      file.EnrichedSeq = file.RetrieveSeq;    // use the chunks as retrieved
      return;
  }

  int TokensAdded = file.ven.VesAddRelatedChunks(exp,
          file.HasRevisions,    // complete each run of revisions found in the retrieved set
          file.HasComments,     // complete each run of comments found in the retrieved set
          false,                // AllTables: true = also add every other table, if the budget allows
          true,                 // AdjacentTables: also complete the tables met while expanding
          EnrichmentBudget);    // token budget for AllTables. Completions are always performed

  // If the budget allows, add comments and revisions found elsewhere in the document.
  // The second parameter selects an author. Pass "" for all authors.
  // Each method returns the cumulative number of enrichment tokens added.
  if (file.HasComments  && TokensAdded < EnrichmentBudget) TokensAdded = file.ven.VesAddCommentChunks(exp, "");
  if (file.HasRevisions && TokensAdded < EnrichmentBudget) TokensAdded = file.ven.VesAddRevisionChunks(exp, "");

  file.EnrichmentTokenCount = TokensAdded;
  file.EnrichedSeq = file.ven.VesGetExpandedSeq(exp);    // the expanded set of sequence numbers

  file.ven.VesEndExpansion(exp);
  

var sb = new StringBuilder();
  foreach (ClsFiles file in files)
  {
      if (!file.InFilter || file.RetrieveSeq == null) continue;

      var seqs = file.EnrichedSeq ?? file.RetrieveSeq;

      if (sb.Length > 0) sb.Append("\n\n");
      sb.Append(string.Join("\n\n", seqs.Select(seq => file.chunk[seq])));   // document order
  }

  // Static content goes before the chat history to take full advantage of caching
  string systemPrompt = Dan.DasGetSystemMessage() +
                        "\n\nUSE THIS EXTRACTED DOCUMENT CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                        sb.ToString();

  // send systemPrompt, the chat history and the new question to your main model
  

Back to menu

About RAG Document Toolkit

RAG Document Toolkit is a pair of .NET libraries for building retrieval-augmented generation (RAG) applications over word-processing documents. It covers the two steps where document structure is most often lost: preparing documents for a vector database, and assembling the context that is sent to the language model.

The toolkit does not include an embedding model, a vector database, or a language model. You keep your choice of each. Dan supplies the chunks and metadata to store, and Ven works on the chunk sequence numbers that your search returns.

Why structure-aware chunks matter

• Chunks break at document boundaries such as pages, paragraphs and table rows, not at an arbitrary character count, so a table row or an HTML tag is never cut in half.
• Chunk size is measured in tokens, so chunks fit the embedding model and the context budget predictably, for Latin and CJK text alike.
• Every chunk carries its document, section and page information, so answers can cite the document title, section and page number they came from.
• Tables, tracked revisions and comments are recorded in the metadata, so they can be retrieved completely instead of in fragments.
• A compact index document summarizes every file in the collection. A question can be routed to the few files that matter, and when those files are small enough they are sent whole, which gives the model complete context for fewer tokens than searching the entire collection.

Back to menu

Getting started

A RAG application built with the toolkit follows this sequence:

1. Set the license key.
2. For each document, use Dan to produce the Markdown chunks and their metadata records.
3. Create an embedding for each chunk and store the chunk, its embedding and its metadata record in your database.
4. Optionally, use Ven to build the index document for the collection, and store it. Rebuild it when files are added or removed.
5. For each user question, optionally send the index document and the question to a low-cost model to select the relevant documents.
6. Run your vector search, over the selected documents or the whole collection, to retrieve the best matching chunks.
7. Group the retrieved chunks by document id, and use Ven to expand each group with related chunks.
8. Send the expanded chunks to the language model with the question, and use the metadata to show source citations with the answer.

Relevant namespaces

using SubSystems.RagDocumentToolkit.Dan;
  using SubSystems.RagDocumentToolkit.Ven;
  

DLL names

DAN.DLL, VEN.DLL

NuGet package references

<PackageReference Include="dai" Version="1.0.0.6" />
  <PackageReference Include="vei" Version="1.0.0.5" />
  

Minimum .NET version

.NET 9.0

Other dependencies used in the demo program

The demo program uses these packages for embeddings, vector storage and chat. Your application is free to use any vector database and any model. DAN.DLL refers to the OpenAI provider and chat client types only for the evaluation method DasSetSubSystemsOpenAIEvalKey.

<PackageReference Include="LangChain" Version="0.17.0" />
  <PackageReference Include="LangChain.Core" Version="0.17.0" />
  <PackageReference Include="LangChain.DocumentLoaders.Abstractions" Version="0.17.0" />
  <PackageReference Include="LangChain.Providers.OpenAI" Version="0.17.0" />
  <PackageReference Include="LangChain.Databases.InMemory" Version="0.17.0" />
  <PackageReference Include="LangChain.Databases.Sqlite" Version="0.17.0" />
  

1. License key

Your license key is e-mailed to you after your order is processed. Set it once at program start, before creating a Dan or Ven object, using the static DasSetLicenseInfo method:

int result = Dan.DasSetLicenseInfo("your-license-key", "your-license-number", "Your Company Name");
  

This one call licenses both libraries. Ven does not need a license call of its own. The method returns 0 when the license is accepted. Without a valid license key the product runs in evaluation mode. You can check the status at any time with DasGetLicenseStatus.

2. Code example

The examples below are condensed from the multi-file demo program that is included with the toolkit. Please refer to the demo source for the complete code. In these examples, file (class ClsFiles) is the application's own record for one document, files is the list of these records, and filesDict finds a record by its document id. The demo uses LangChain with an SQLite vector database and OpenAI models. The toolkit works the same way with any vector database and any model.

Step 1. Convert a document

Import the file into a Dan object, then retrieve the Markdown chunks and one metadata record for each chunk. Save the four items shown at the end to your database. DasGetMarkdown(-1, -1) converts the entire document. To convert only a part of the document, pass the zero-based first and last page numbers instead.

using SubSystems.RagDocumentToolkit.Dan;

  Dan dan = new Dan();
  dan.LogMsg   += LogDanMsg;    // optional: receive log messages
  dan.MdoNames += MdoNames;     // optional: supply names and terms for each chunk (see 1b)

  // import the file into the Dan object
  if (!dan.DasImportFile(NewFile, GetDocType(NewFile))) return false;

  // get the Markdown chunks and the metadata records (first page, last page; -1 = the entire document)
  Dan.DanResult mdo = dan.DasGetMarkdown(-1, -1);
  if (mdo == null) return false;

  // save these to your vector database
  file.DocId    = mdo.DocId;      // unique id for the document
  file.DocTitle = mdo.DocTitle;   // title found in the document, or the file path when there is none
  file.chunk    = mdo.chunks;     // string[]: the Markdown text of each chunk
  file.MetaRec  = mdo.MetaRecs;   // Dictionary<string,object>[]: one record per chunk
  

1b. Names and important terms (optional). When a MdoNames handler is set, Dan calls it with the plain text of each chunk. Return the names and terms found in the text, using any detection method you prefer. Dan records them in the chunk's metadata, and they appear in the index document, where they can significantly improve document selection. The important terms may, for example, come from a list supplied by the author of the document.

// PersonRx, OrgRx, Places, Terms and ContainsWord belong to your application
  Dan.ClsMdoNames MdoNames(object sender, string DocTitle, int ChunkIndex, string text)
  {
      var names   = new Dan.ClsMdoNames();
      var persons = new HashSet<string>();
      var orgs    = new HashSet<string>();

      // capture group 1 is the name without the honorific
      foreach (Match m in PersonRx.Matches(text)) persons.Add(m.Groups[1].Value.Trim());

      // capture group 1 is the full organization name
      foreach (Match m in OrgRx.Matches(text)) orgs.Add(m.Groups[1].Value.Trim());

      names.PersonNames    = persons.ToArray();
      names.OrgNames       = orgs.ToArray();
      names.PlaceNames     = Places.Where(p => ContainsWord(text, p)).ToArray();
      names.ImportantTerms = Terms.Where(t => ContainsWord(text, t)).ToArray();

      return names;
  }
  

Step 2. Add the document to the collection

Create a Ven object from the document's metadata records and keep it for the life of the collection. It serves every question that follows. Then store the chunks and their metadata records in your vector database.

using SubSystems.RagDocumentToolkit.Ven;

  // create the Vector Enrichment object from the document's complete metadata record set
  try
  {
      file.ven = new Ven(file.MetaRec);
  }
  catch (Exception ex)
  {
      LblStatus.Text = "Error creating Vector Enrichment object: " + ex.Message;
      return false;
  }

  // what does the document contain?
  file.HasTables    = file.ven.VesHasTables();
  file.HasRevisions = file.ven.VesHasRevisions();
  file.HasComments  = file.ven.VesHasComments();
  file.TokenCount   = file.ven.VesGetTokenCount();    // token count of the whole document

  // store each chunk with its metadata record in the vector database
  var docs = new List<Document>();
  for (int i = 0; i < file.chunk.Length; i++)
      docs.Add(new Document(content: file.chunk[i], metadata: file.MetaRec[i]));

  await vectorCollection.AddDocumentsAsync(embeddingsProvider, docs);

  files.Add(file);
  filesDict[file.DocId] = file;    // look up a file by its document id
  

Step 3. Select documents with the index document (optional)

Ven creates an index entry for each file. Join the entries to form the index document for the collection, and send it with the question to a low-cost model. Ven supplies the system prompt for this call.

// Build the index document once. Rebuild it when files are added or removed.
  if (IndexDoc == "")
  {
      foreach (ClsFiles file in files)
      {
          // index entry for one file, sized as a percentage of the file's tokens
          file.PhraseDoc = file.ven.VesCreateDocumentIndex(filterTokenPct, out file.FilterTokens);

          if (IndexDoc.Length > 0) IndexDoc += "\n\n";
          IndexDoc += file.PhraseDoc;
      }
  }

  // Static content goes first, so the index document is billed at the cached-input rate
  string systemPrompt = Ven.VesGetFilterSystemPrompt() +
                        "\n\nUSE THIS DOCUMENT INDEX CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                        IndexDoc;

  // AskModel is your own function. Use a low-cost model for this call.
  string answer = (await AskModel(miniChat, systemPrompt, newUserQuestion)).Trim().ToUpper();
  

The model replies with one of the following:

foreach (ClsFiles f in files) f.InFilter = false;

  if (answer == "NONE" || answer == "ALL")
  {
      // no routing: regular retrieval and enrichment over the whole collection (steps 4 and 5)
  }
  else if (answer == "INDEX")
  {
      // answer the question from the index document itself
      systemPrompt = Ven.VesGetIndexSystemPrompt() + IndexDoc;
  }
  else
  {
      // example:  TE582281:92 | TE851927:70 | TE642026:35
      int  RequiredConfidenceLevel = 70;    // choose the level that suits your application
      bool confident = false;
      int  tokens = 0;

      foreach (string item in answer.Split('|'))
      {
          string[] parts = item.Split(':');
          if (parts.Length != 2 || !int.TryParse(parts[1].Trim(), out int level)) continue;

          if (level >= RequiredConfidenceLevel) confident = true;

          if (filesDict.TryGetValue(parts[0].Trim(), out ClsFiles? file))
          {
              file.InFilter = true;
              tokens += file.TokenCount;
          }
      }

      if (!confident || tokens == 0)
      {
          // no document stands out: regular retrieval and enrichment over the whole collection
      }
      else if (tokens <= TotalUserAllowance)
      {
          // the selected files fit within the token allowance: send them whole
          string context = string.Join("\n\n", files.Where(f => f.InFilter).SelectMany(f => f.chunk));

          systemPrompt = Dan.DasGetSystemMessage() +
                         "\n\nUSE THIS EXTRACTED DOCUMENT CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                         context;
      }
      else
      {
          // too large to send whole: run steps 4 and 5 over the selected files only,
          // treating the question as a broad query over this narrow set of files.
          // The token allowance is now spread over fewer files, so each gets stronger context.
          globalQuery = true;
      }
  }
  

Step 4. Retrieve and group the chunks

Run your vector search as usual. Then use the ssDocId and ssSeq metadata fields to collect the retrieved sequence numbers for each document. Ven needs only these sequence numbers.

VectorSearchResponse searchResponse = await vectorCollection.SearchAsync(queryVector, searchSettings);

  TotalRetrievalTokens = 0;
  foreach (var item in searchResponse.Items)
  {
      if (item.Metadata == null) continue;

      // ssDocId and ssSeq are written by Dan. A chunk without them did not come from Dan.
      if (!item.Metadata.TryGetValue("ssDocId", out var docIdObj) ||
          !item.Metadata.TryGetValue("ssSeq",   out var seqObj)) continue;

      if (filesDict.TryGetValue(Convert.ToString(docIdObj) ?? "", out ClsFiles? file))
      {
          int seq = Convert.ToInt32(seqObj);           // zero-based chunk sequence number

          file.RetrieveSeq ??= new SortedSet<int>();
          file.RetrieveSeq.Add(seq);

          int seqTokens = file.ven.VesGetChunkTokenCount(seq);
          file.RetrievalTokenCount += seqTokens;       // for this file
          TotalRetrievalTokens     += seqTokens;       // for the collection
      }
  }
  

Step 5. Enrich the retrieved chunks

Run this step for each file that has retrieved chunks. This example uses VesAddRelatedChunks, which performs all the completions in one call, because a partial table can mislead the model more than no table at all. It then adds the comments and revisions found elsewhere in the document when the budget allows. For finer control, and to add pages and sections, use the single-purpose methods described in the Ven API reference.

// Share the enrichment budget among the files in proportion to their retrieved tokens
  int TotalEnrichmentBudget = TotalTokenCount * enrichmentPct / 100;
  int EnrichmentBudget = (int)((float)TotalEnrichmentBudget / TotalRetrievalTokens * file.RetrievalTokenCount);

  // Create an expansion object for this query from the retrieved sequence numbers
  object exp;
  try
  {
      exp = file.ven.VesBeginExpansion(file.RetrieveSeq);
  }
  catch (ArgumentException ex)
  {
      LblStatus.Text = "Error creating expansion object: " + ex.Message;
      file.EnrichedSeq = file.RetrieveSeq;    // use the chunks as retrieved
      return;
  }

  int TokensAdded = file.ven.VesAddRelatedChunks(exp,
          file.HasRevisions,    // complete each run of revisions found in the retrieved set
          file.HasComments,     // complete each run of comments found in the retrieved set
          false,                // AllTables: true = also add every other table, if the budget allows
          true,                 // AdjacentTables: also complete the tables met while expanding
          EnrichmentBudget);    // token budget for AllTables. Completions are always performed

  // If the budget allows, add comments and revisions found elsewhere in the document.
  // The second parameter selects an author. Pass "" for all authors.
  // Each method returns the cumulative number of enrichment tokens added.
  if (file.HasComments  && TokensAdded < EnrichmentBudget) TokensAdded = file.ven.VesAddCommentChunks(exp, "");
  if (file.HasRevisions && TokensAdded < EnrichmentBudget) TokensAdded = file.ven.VesAddRevisionChunks(exp, "");

  file.EnrichmentTokenCount = TokensAdded;
  file.EnrichedSeq = file.ven.VesGetExpandedSeq(exp);    // the expanded set of sequence numbers

  file.ven.VesEndExpansion(exp);
  

Step 6. Send the context to the model

Collect the selected chunks of each file in document order, and place them after the system message supplied by Dan. This message tells the model how to read the Markdown produced by Dan and how to cite the document, section and page.

var sb = new StringBuilder();
  foreach (ClsFiles file in files)
  {
      if (!file.InFilter || file.RetrieveSeq == null) continue;

      var seqs = file.EnrichedSeq ?? file.RetrieveSeq;

      if (sb.Length > 0) sb.Append("\n\n");
      sb.Append(string.Join("\n\n", seqs.Select(seq => file.chunk[seq])));   // document order
  }

  // Static content goes before the chat history to take full advantage of caching
  string systemPrompt = Dan.DasGetSystemMessage() +
                        "\n\nUSE THIS EXTRACTED DOCUMENT CONTEXT TO ANSWER THE USER'S QUESTION:\n" +
                        sb.ToString();

  // send systemPrompt, the chat history and the new question to your main model
  

Back to menu

Dan - Document to AI Friendly Markdown

Dan loads a document and returns it as an array of Markdown chunks, together with one metadata record for each chunk. The chunks are ready to embed and the metadata records are ready to store alongside them.

Input formats

DOCX, RTF, HTML, Markdown (.md) and plain text.

Output

Chunk size

You set the maximum chunk size in tokens (default 1024). Dan counts tokens with the o200k encoding as the chunk is built, so the limit holds for any language. Chunks end early at natural boundaries, such as the end of a page, and an incomplete table row or HTML tag is moved whole to the next chunk. Expect chunks to average about 70% of the maximum. The actual token size of each chunk is recorded in its metadata record.

Document identity and your own metadata

You can supply a document id and a document title, or let Dan assign an id. You can also pass a dictionary of your own fields, such as a customer number or a file path. Dan copies these fields into every metadata record of the document so that you can filter on them in your database.

Names and important terms

Dan can call a handler that you supply with the plain text of each chunk as the conversion proceeds. Your handler may return person names, organization names, place names and important terms found in the text, using any extraction method you prefer. Dan adds them to the chunk's metadata record, where they are available for pre-filtering before the vector search.

Dan API reference

Namespace: SubSystems.RagDocumentToolkit.Dan. Class: Dan. Assembly: DAN.DLL.

Constructor | Properties | Events | Methods | Classes | Constants

Constructor

Creates a Dan object. One Dan object holds one document at a time. To convert a number of documents, you can create an object for each document, or use one object for all of them, one after the other.

Properties

Events

MdoNames

Dan fires this event for each chunk during DasGetMarkdown. It gives your application the opportunity to supply the person names, organization names, place names and important terms found in the chunk. Dan saves them in the metadata record of the chunk, and Ven includes them in the index document, where they can significantly improve document selection. Please see step 1b of the code example.

Return value: A ClsMdoNames object. Any of its arrays may be null. Return null when the chunk has no names to report.

LogMsg

Dan fires this event to pass on the log messages generated by this library and by the libraries that it uses. Handling this event is optional. It is useful for diagnostics.

Methods

DasImportFile

Imports a document file into the Dan object and paginates it. Call this method, or one of the two import methods that follow, before calling DasGetMarkdown. A Markdown file is converted to HTML internally and then imported as HTML.

Return value: True when successful. The method returns false when the file is not found, the document type is invalid, the file can not be read, or the evaluation period has expired. Use DasGetLastMessage to retrieve the reason.

DasImportDocFromString

Imports a document from a string. Use this method when the document is held in a database or arrives over a network, and no disk file exists.

Return value: True when successful.

DasImportDocFromBytes

Imports a DOCX document from a byte array.

Return value: True when successful.

DasGetMarkdown

Converts the imported document to AI-friendly Markdown chunks, and creates one metadata record for each chunk. Set the properties that control the conversion, such as TokensPerChunk, UserMetaDict, DocId and DocTitle, before calling this method. When a MdoNames handler is set, Dan calls it for each chunk during this call.

Return value: A DanResult object containing the chunks, the metadata records, the document id and the document title. The method returns null when no document has been imported or when the conversion fails. On return, the ChunkCount property is updated, and the DocId and DocTitle properties are cleared for the next document.

DasOverrideChunkFieldName

Replaces the default label of one chunk information field. Use this method when the default labels do not suit your documents or your language. Call it after importing a document and before calling DasGetMarkdown.

Return value: True when successful. False when the id is out of range.

DasGetSystemMessage

Returns the system message to send to the language model ahead of the chunks. The message tells the model how to read the Markdown produced by Dan, including its tables, revisions and comments, and how to cite the document, section and page. Please see step 6 of the code example. This is a static method, so it is called as Dan.DasGetSystemMessage(). The same text is available from the SystemMessage property of a Dan object.

Return value: The system message text.

DasGetLastMessage

Retrieves the last message generated by the library. Messages are saved for this method, instead of being displayed, when the InWebServer property is true (the default) or when the DAFLAG_RETURN_MSG_ID flag is set.

Return value: The message id. A value of 0 indicates that there is no message.

DasResetLastMessage

Clears the last message. Call this method before an operation when you wish to be certain that a message retrieved afterwards belongs to that operation.

Return value: This method always returns true.

DasSetFlags

Sets or resets one or more operation flags.

Return value: The new value of all flags.

DasDispose

Releases the document and the resources held by the Dan object. Call this method when you have finished with the object. The object can import a new document afterwards.

DasSetLicenseInfo

Sets the license key for the product. This is a static method. Call it once at program start, before creating a Dan or Ven object. Without a valid license key the product runs in evaluation mode.

Return value: 0 = the license is valid, 1 = evaluation mode, 2 = invalid license number, 3 = license limit exceeded.

DasGetLicenseStatus

Returns the current license status.

Return value: 0 = licensed, 1 = evaluation mode, 2 = invalid license number, 3 = license limit exceeded, 4 = the evaluation period has expired.

DasSetSubSystemsOpenAIEvalKey

For product evaluation only. This method lets you run the demo program during the evaluation period without an OpenAI account of your own. It creates the embedding provider and the two chat clients used by the demo, using an OpenAI key supplied by Sub Systems. The objects are created inside the library, so the key itself is never passed to the calling program. In your own application, create these objects with your own OpenAI key.

DasHasSubSystemsOpenAIEvalKeyExpired

For product evaluation only. Reports whether the OpenAI key supplied by Sub Systems is still available to the caller.

Return value: True when the key is no longer available, for example because the evaluation period has expired.

Classes

Dan.DanResult

Returned by DasGetMarkdown.

Dan.ClsMdoNames

Returned by your MdoNames event handler.

Constants

The constants are members of the Dan class, for example Dan.DOC_DOCX.

Document types

Operation flags

Used with DasSetFlags.

Chunk information fields

Dan collects the following information for each chunk. The constants identify a field for DasOverrideChunkFieldName.

The label is the name of the field in the metadata record, and your application can change it with DasOverrideChunkFieldName. In addition, each metadata record contains the following fields with fixed names. Your application can not rename them, so they can always be relied upon:

Use ssDocId and ssSeq to group the results of a vector search by document. Please see step 4 of the code example.

Every value in a metadata record is either a string or a whole number. A yes/no value is stored as the string "yes" or "no". A record contains no arrays or nested objects, so it can be saved in a flat metadata store, in a database row, or as JSON without any conversion.

Back to menu

Ven - Vector Enrichment Library

A vector search returns the chunks that look most like the question. The answer often needs more: the remaining rows of the table, the rest of the section, or the comment attached to the paragraph. Ven adds these related chunks, using the structure information that Dan recorded in the metadata.

Ven also works one level up, on the collection. It builds an index document, a compact summary of every file, which a low-cost model uses to pick the documents that are relevant to a question before any retrieval takes place.

Chunk expansion works on chunk sequence numbers only. It never needs the chunk text, the embeddings, or access to your database.

How it is used

1. Group the retrieved chunks by document id. Chunks that did not come from Dan are left out of this step and are used as retrieved.
2. For each document, load its complete set of metadata records and create a Ven object from them. The Ven object is immutable. It can be kept and used for any number of queries, including concurrent ones.
3. Call VesBeginExpansion with the retrieved sequence numbers. It returns an expansion object that holds the state of this one query.
4. Call one or more expansion methods. Each returns the cumulative number of enrichment tokens added so far.
5. Call VesGetExpandedSeq to read the expanded set, and replace the document's retrieved chunks with it.
6. Call VesEndExpansion to release the expansion object.

Expansion methods

The methods come in two kinds. Completion methods finish a structure that a retrieved chunk is part of: VesCompleteTables, VesCompleteRevisions and VesCompleteComments. Addition methods bring in neighboring or related content: VesAddPageChunks, VesAddSectPages, VesAddRevisionChunks and VesAddCommentChunks. The methods can be called in any order and stopped at any point. To begin, call VesAddRelatedChunks, which performs all the completions in one call, and then call the addition methods when your token budget allows. Please see step 5 of the code example.

Anchoring

Expansion methods work relative to the anchor set, not to the accumulated result, so chaining several methods does not compound. To expand further from the result so far, call VesAnchorExpandedSeq, which makes the expanded set the new anchor.

Staying within budget

Ven measures everything in tokens. The expansion methods return the enrichment tokens added so far, as does VesGetEnrichmentTokenCount. VesGetTokenCount returns the token count of the document or of the expanded set at any stage, and VesGetChunkTokenCount returns the token count of one chunk. These calls cost very little, so you can stop expanding as soon as the context budget is reached.

Document queries

VesHasTables, VesHasRevisions and VesHasComments report what a document contains, so that you call only the expansions that apply. VesGetRevisionAuthors and VesGetCommentAuthors list the reviewers and the comment authors, so that revision and comment expansion can be limited to one person. VesGetMdoSectPages and VesGetMdoSectChunks give the size of each section.

Index document

Vector search compares a question with individual chunks. It has no view of the collection as a whole, so it cannot tell that a question concerns only two files out of fifty, and it struggles with questions about the collection itself, such as "which contracts mention arbitration?" The index document fills this gap.

What it contains

The index document has one entry per file. Each entry begins with the document id and title, the document properties (whether the file has tables, revisions or comments), and the page and section counts. The key phrases of the document follow, grouped by type: reviewers and comment authors, field data, the names and important terms supplied by your MdoNames handler, headings, emphasized text, a sample of the body text, table column headings and list text. VesCreateDocumentIndex creates the entry for one file. You set its size as a percentage of the file's tokens, so the index remains a small fraction of the collection. Your application joins the entries to form the index document.

How it is used

1. Build the index document once for the collection and store it. It changes only when files are added or removed, so it benefits fully from cached-input pricing when it is sent with every question.
2. For each question, send the index document and the question to a low-cost model, using the system prompt returned by Ven.VesGetFilterSystemPrompt. The model replies with the relevant document ids, each with a confidence level, or reports that the index document itself answers the question.
3. Keep the documents that score above your threshold, and limit retrieval and chunk expansion to them. If no document stands out, or all do, search the whole collection as usual.

Benefits

• Better context. When the selected documents fit within your token budget, send them whole. The model then reads complete documents instead of fragments, and both retrieval and expansion can be skipped.
• Lower cost. The full retrieval and expansion budget is spent on the few documents that matter, and irrelevant files contribute no tokens. The routing call itself is inexpensive: a small model reading a small, cached document.
• Fewer false matches. Chunks from unrelated files that happen to share wording with the question are excluded before the search runs.
• Collection-level answers. Questions about the collection itself can often be answered from the index document directly, with no retrieval at all.

Observed results

We measured the index document option in the multi-file demo project that ships with the toolkit. The test collection holds about 167,000 tokens in a diverse set of files: short business and medical documents, a table-intensive document of about 16,000 tokens, and a document of about 200 pages and 115,000 tokens.

• More than half of the questions took the whole-document path. The routing model selected a few files that fit within the token budget, and those files were sent complete. For these questions no vector search and no chunk expansion were needed. This was typical of a question about a named person, a request to summarize one document, and a comparison of two documents, such as one year's budget with the next year's projection.
• A typical question used about 4,000 tokens. For a question asking who had treated a named patient, the routing model selected two files, one of them with a confidence level of 95. Both files were sent whole, at a cost of about 4,000 tokens out of the 167,000 in the collection.
• Fewer tokens per question, with more complete context. Sending a few whole documents used fewer tokens than retrieval and expansion across the full collection, while giving the model every table, section and comment of the documents that mattered. The usual trade between cost and context did not apply: both improved together. Regular retrieval costs more because a question that is judged to be broad must be given at least one chunk from every file, so that no file is overlooked. The cost of this coverage grows with the number of files, and it can exceed the retrieval budget. Routing removes the need for it, because the files that do not matter are excluded first.
• The index stayed small. The index document came to about 5% of the collection's tokens, half of the 10% we had allowed for it. Because it is identical for every question, it is billed at the cached-input rate after the first call.
• Nothing was lost on the remaining questions. When no document stood out, the application fell back to regular retrieval and expansion over the whole collection, so broad questions were answered as before.

Results depend on the collection. The option helps most when the collection has many files and a typical question concerns only a few of them. It helps least when every question draws on every file, in which case routing falls back to the regular search and costs only the small routing call.

Please see step 3 of the code example for the complete sequence.

Ven API reference

Namespace: SubSystems.RagDocumentToolkit.Ven. Class: Ven. Assembly: VEN.DLL.

Constructor | Events | Methods

Constructor

Creates a Ven object for one document from the metadata records created by DasGetMarkdown. Pass the complete set of records of the document, and of this document only. The records can be in any order, because Ven arranges them by their sequence numbers. Ven needs only the metadata. It never needs the text of the chunks.

The object does not change after it is created. Create it once when the document is added to the collection, and keep it for every question that follows. It can be used by a number of threads at the same time, each with its own expansion object.

Exceptions: The constructor throws an ArgumentException, with a message that names the record and the field, when:

• the array is null,
• a record does not contain all the ChunkId fields,
• a field is missing from a record, or a numeric field does not contain a number,
• a sequence number is outside the range of the array, which happens when the set of records is not complete, or
• two records have the same sequence number, which happens when the records of two documents are mixed.

Events

LogMsg

Ven fires this event to pass on its log messages. Handling this event is optional. It is useful for diagnostics.

Methods

Expansion object | Completion methods | Addition methods | Combined method | Token counts | Document information | Index document methods

Expansion object

An expansion object holds the state of one query for one document: the anchor set, which is the set of chunks that the expansion works from, and the expanded set, which is the anchor set plus the chunks added so far.

VesBeginExpansion

Creates an expansion object from the chunk sequence numbers that your vector search retrieved for this document. The retrieved set becomes the anchor set. Pass the returned object as the first parameter to the other expansion methods. The Ven object is not modified, so any number of expansion objects can be in use at the same time, including on different threads.

Return value: The expansion object. The method throws an ArgumentException when a sequence number is negative, or is not less than the number of chunks in the document.

VesGetExpandedSeq

Returns the expanded set: the anchor chunks and all the chunks added so far, in document order. Send the chunks with these sequence numbers to the language model. This method can be called at any stage, and more than once.

Return value: The zero-based sequence numbers of the expanded set.

VesAnchorExpandedSeq

Adopts the set expanded so far as the new anchor set. The completion methods, and the page and section methods, work from the anchor set and not from the accumulated result, so calling a number of them in a row does not compound. Call this method when you do want the next method to work from everything selected so far, for example to add the full page of each chunk that a table completion brought in.

VesEndExpansion

Ends the expansion. Call this method when you have retrieved the expanded set. In the .NET version the expansion object holds no system resources, so this method does no work at this time. Calling it keeps your code compatible with future versions.

Completion methods

A completion method finishes a structure that an anchor chunk is part of. These are the most important expansions, because a partial table, or half of a run of revisions, can mislead the language model more than its absence would.

VesCompleteTables

A large table is divided among a number of chunks. When a retrieved chunk begins or ends within a table, this method adds the chunks that contain the remaining rows, so that the model receives the whole table with its column headings. A table in the page body is completed from the page body chunks only, and a table in a page header or footer from the chunks of the same type.

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

VesCompleteRevisions

For each anchor chunk that contains tracked revisions, adds the neighboring chunks before and after it that also contain revisions, up to the first chunk without any. An edited passage that spans a number of chunks is then presented to the model as a whole.

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

VesCompleteComments

For each anchor chunk that contains comments, adds the neighboring chunks before and after it that also contain comments, up to the first chunk without any.

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

Addition methods

An addition method brings in content from the neighborhood of the anchor chunks, or from elsewhere in the document. Additions widen the context. They are optional, and they are the part to reduce when tokens are limited.

VesAddPageChunks

Adds page context around each anchor chunk. The method works in stages, and checks the token budget after each stage:

1. The remaining chunks of the page of each anchor chunk.
2. The first and the last page of the document, which often carry the title, the introduction and the conclusion.
3. The given number of pages before and after each anchor chunk.
4. The second and the second to last page of the document.

The method returns without any change when FullPage is false and NumPages is 0.

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

VesAddSectPages

Adds pages around each anchor chunk without leaving the document section that contains the chunk. Use this method for documents in which each section is a unit, such as the chapters of a manual or the articles of an agreement.

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

VesAddRevisionChunks

Adds every chunk in the document that contains tracked revisions, wherever it is located. Unlike VesCompleteRevisions, this method does not depend on the anchor set. Use it when the question is about the revisions themselves, such as "what did the reviewers change?"

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

VesAddCommentChunks

Adds every chunk in the document that contains comments, wherever it is located. Use it when the question is about the comments themselves.

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

Combined method

One call that performs the completions. It is the easiest way to begin, and it is the method used by the demo program.

VesAddRelatedChunks

Performs the completions in this order: the tables of the selected chunks, then the revisions, then the comments. These are always performed, whatever the budget, because an incomplete structure should not be sent to the model. When AllTables is true and the budget has not been reached, the method then adds the remaining tables of the document. This method does not add pages or sections, because such additions rarely fit within a modest enrichment budget. When a question calls for wider context, call the addition methods afterwards, and use the return value of this method to see how much of your budget remains. Please see step 5 of the code example.

Return value: The cumulative number of enrichment tokens added to the expansion object so far, by this method and by the methods called before it.

Token counts

Ven reads the token size of each chunk from the metadata, so these methods do no tokenizing and cost very little. Call them as often as you need.

VesGetTokenCount

Without a parameter, returns the token count of the whole document. Use it to decide whether the document can be sent complete. With an expansion object, returns the token count of the expanded set, which includes the anchor chunks. Before any expansion, this is the token count of the retrieved chunks.

Return value: The number of tokens.

VesGetEnrichmentTokenCount

Returns the token count of the chunks added by the expansion, without the anchor chunks. This is the same value that the expansion methods return.

Return value: The number of enrichment tokens.

VesGetChunkTokenCount

Returns the token count of one chunk. Please see step 4 of the code example, where it is used to total the retrieved tokens for each document.

Return value: The number of tokens, or 0 when the sequence number is out of range.

Document information

These methods report what the document contains. Use them to decide which expansions are worth calling, and to offer the user a choice of reviewers or comment authors.

VesHasTables

Reports whether any chunk of the document contains table rows.

Return value: True when the document has tables.

VesHasRevisions

Reports whether the document contains tracked revisions (redlining).

Return value: True when the document has revisions.

VesHasComments

Reports whether the document contains comments.

Return value: True when the document has comments.

VesGetRevisionAuthors

Returns the names of the reviewers who made the tracked revisions in the document. Each name appears once.

Return value: The reviewer names. The array is empty when the document has no revisions.

VesGetCommentAuthors

Returns the names of the authors of the comments in the document. Each name appears once.

Return value: The author names. The array is empty when the document has no comments.

VesGetMdoSectPages

Returns the number of pages occupied by each section of the document. Use it to choose a suitable value for VesAddSectPages.

Return value: An array with one element for each section, in document order.

VesGetMdoSectChunks

Returns the number of chunks occupied by each section of the document.

Return value: An array with one element for each section, in document order.

Index document methods

Please see the Index document topic for the purpose of these methods, and step 3 of the code example for their use.

VesCreateDocumentIndex

Creates the index entry for this document. Join the entries of all the documents, separated by a blank line, to form the index document for the collection. The entry begins with the document id and title, followed by the document properties (whether it has revisions, comments or tables), the page count and the section count. The key phrases follow, grouped by type, in the order of their value for document selection: reviewers, comment authors, field names and data, important terms, person names, organization names, place names, headings of levels 1 to 3, bold text, plain text, italic text, table column headings and list text.

Each chunk contributes phrases in proportion to its size, so every part of the document is represented. A phrase is included once, and a phrase consisting of a number only is left out, with the exception of a year. A long entry is divided into parts of about 900 tokens, marked with (cont.), so that the index document can itself be stored in parts when needed.

Return value: The index entry.

VesGetFilterSystemPrompt

Returns the system prompt for the document selection call. Append the index document to it, and send it with the user's question to a low-cost model. The prompt describes the format of the index to the model, and instructs it to reply with exactly one line in one of these forms:

This is a static method, called as Ven.VesGetFilterSystemPrompt().

Return value: The system prompt text.

VesGetIndexSystemPrompt

Returns the system prompt to use when the selection call replies INDEX. Append the index document to it, and send it with the user's question to your main model. The prompt instructs the model to answer from the index only, to list every document that matches, to refer to each document by its title followed by its id, and to name the documents that are likely to hold the answer when the question needs the full text. This is a static method.

Return value: The system prompt text.

Back to menu