ARCHITECTURE.md
ARCHITECTURE.md
Deep dive into the AI Toolbox plugin's system architecture, design patterns, and internal workflows.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β LM Studio Host β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β Plugin Runner (Node.js) β β β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β AI Toolbox Plugin β β β β β β β β β β β β ββββββββββββββββ β β β β β β β index.ts βββββ Entry Point (main function) β β β β β β β (entry) β β β β β β β ββββββββ¬ββββββββ β β β β β β β β β β β β β βΌ β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β β β Core Services β β β β β β β β ββββββββββββ ββββββββββββ ββββββββββββββββββββ β β β β β β β β β config.tsβ βsecurity β βstateManager.ts β β β β β β β β β β(Zod+UI) β β .ts β β(persistence) β β β β β β β β β ββββββββββββ β(validators)β ββββββββββββββββββββ β β β β β β β β ββββββββββββ β β β β β β β β ββββββββββββ ββββββββββββ ββββββββββββββββββββ β β β β β β β β βworkingDirβ βperformancβ βpromptPreprocessor β β β β β β β β β β .ts β βeUtils.ts β β .ts β β β β β β β β β β(path mgmtβ β(caching) β β(Document RAG + β β β β β β β β β ββββββββββββ ββββββββββββ β ContextGuard) β β β β β β β β β ββββββββββββββββββββ β β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β β β β β β β β ββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β β β Tool Registration Layer β β β β β β β β βββββββββββββββββββ β β β β β β β β β toolsProvider.ts β β β β β β β β β β (factory fn) β β β β β β β β β ββββββββββ¬βββββββββ β β β β β β β βββββββββββββΌββββββββββββββββββββββββββββββββββββββββ β β β β β β β β β β β β β βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββ β β β β β β β Tool Modules (19 registered files) β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β βfileSys β βwebRes β βbrowser β β git β β β β β β β β β β (22) β β (4) β β (5) β β (15) β β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β β datab β βbackgnd β βexec β β docParseβ β β β β β β β β β (1) β β cmd(3) β β (5) β β (1) β β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β β image β β http β β vector β β UI β β β β β β β β β β (4) β β (3) β β RAG(4) β β Gen(3) β β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β ββββββββββ ββββββββββ ββββββββββ β β β β β β β β β Context β βtextProcβ βAST Ref β βbgndCmdsβ β β β β β β β β β Mgmt(12)β β (4) β β factorβ β (3) β β β β β β β β β ββββββββββ ββββββββββ β (2) β β β β β β β β β βββββββββββ β β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββ β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β External Dependencies β β β β Puppeteer β isomorphic-git β Tesseract.js β pdf-parse β β β β duck-duck-scrape β node:sqlite β node-notifier β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The project uses Tsup (esbuild-based bundler) for fast, zero-config compilation to both ESM and CJS formats.
tsconfig.json defines @/* β src/* for IDE and TypeScript support.tsup.config.ts maps @ β path.resolve(__dirname, 'src') to ensure the bundler resolves aliases correctly in the final output.Status: The gateway pattern (formerly src/tools/gatewayTools.ts, introduced in v1.6.0) was abandoned in favor of direct SDK registration; the file has been fully removed from the codebase (v1.9.10 session, 24.08). No gateway tool definitions remain anywhere under src/.
Why Abandoned:
toolsSchemaMinifier.ts (description truncation, constraint capping) rather than tool count gatingCurrent Approach: All enabled tools are registered directly with the LM Studio SDK. Schema minification handles EBNF compatibility automatically. No artificial limits or discovery layers needed.
See CHANGELOG.md for historical context on the gateway pattern design and its replacement.
Context Scoping (v1.9.1+): Entries tagged with global/project/session scope for future isolation filtering. Session entries expire after 24h and are automatically pruned during retrieval.
Heuristic Retrieval Ordering (v1.9.1+): All search/retrieval operations now apply deterministic composite scoring β recent + frequently accessed entries surface first, replacing raw insertion order.
src/toolsProvider.ts)Central registry managing all tool instances using a Declarative Registry Pattern (v1.8.2+):
Key Design Decisions:
config, stateManager, and at definition time via arrow functionssrc/stateManager.ts)Persistent state management with dynamic path resolution:
Key Features:
Session Summary Tool Flow (v1.5.15+):
Storage Format:
zlib.gzipSync(level: 9) β base64 encoding β stored in StateManager (typically achieves ~30% size reduction)Working Directory Integration:
StateManager initializes via getMemoryFilePath() β resolves to {current_working_dir}/.session_context/.ai_toolbox_memory.msgpacksrc/tools/contextManagementTools.ts)Persistent context storage for session tracking:
Key Features:
Heuristic Scoring Formula (v1.9.1):
src/security.ts)Multi-layer security pipeline:
src/workingDir.ts)Mutable base path for all file operations:
β οΈ Status:
src/tools/gatewayTools.tshas been removed from the codebase (v1.9.10 session, 24.08) and no longer exists anywhere in the repository. The gateway pattern was abandoned in favor of direct SDK registration + schema minification (v1.8.0+). The following describes the design for historical reference only.
Purpose: Prevent LLM tool-bloat crashes by providing a single entry point for tool discovery and execution, reducing the initial grammar schema payload from ~132 tools to just 2.
Sending all 88+ tools directly to llama.cpp's grammar parser caused failed to parse grammar errors due to EBNF recursion limits. The AI also struggled with overwhelming options when deciding which tool to use.
The gateway pattern was abandoned because direct SDK registration with schema minification proved more effective. No integration is required β and since 24.08 (v1.9.10 session) gatewayTools.ts no longer exists in the repository at all; it was never imported or used, so its removal changes nothing functionally.
| Cache | TTL | Max Entries | Purpose |
|---|---|---|---|
| Fuzzy Search | 60s | 100 | File name similarity results |
| Web Requests | 30s | 50 | HTTP responses |
Heavy dependencies loaded on first use:
grep_files phase-1 candidate filter (WASM build of rg; lazy dynamic import on first use, see Β§5 below)grep_files candidate filter (src/utils/ripgrepEngine.ts) β current state (v1.9.13-pending, 01.β02.09)grep_files regex mode on directory targets runs in two phases; single-file targets and AST mode are unchanged code paths.
Design points (full contracts in the module header):
Priority Order: Working Dir β Plugin Root β In-Memory RAM
Impact: Eliminates cross-project memory bleed β session summaries and memory entries are always project-specific when available locally. Existing SDK global memory calls preserved as fallbacks for backward compatibility.
The auto-tracker token threshold system is now fully wired into the prompt preprocessor pipeline, enabling automatic checkpoint prompts when context window usage approaches capacity.
FSM States: IDLE β THRESHOLD_REACHED (prompt generated) β CONFIRMED (saved) or DECLINED β IDLE (reset).
Pending checkpoint warnings persist across unrelated state transitions β cleared only on consume/reply/compress paths, and injected into every preprocessor return path while pending (Fix C).
Impact: Users now receive actionable warnings when token usage approaches context window capacity, with confirmation flow to save session memory before potential overflow. Auto-tracked decisions, completions, and error fixes are flushed to persistent storage during checkpoint saves.
chat used Log Field (FIX #20, v1.9.9+)Per-turn tool loops accumulate token deltas without waiting for the next full history count:
+delta β turn total β chat used.| chat used β N tok field is emitted only when the turn-start baseline > 0; it is omitted if the ContextGuard recount fails.Visual Indicator Example:
In v1.8.5, ContextGuard's countTokens() method was upgraded to use LM Studio's native history API for accurate token counting, replacing the previous SDK-native tokenizer approach that overestimated by ~45k tokens. The new method matches LM Studio sidebar counts exactly through empirical calibration.
Engineering Notes:
getLength(), at(i), getText() properly extract all message content including tool calls β no more silent zeros from broken casting.Impact on AutoTracker:
AutoTracker's token threshold checks (checkTokenThreshold(currentTokens, maxTokens)) receive the accurate History Text Length-derived count directly from ContextGuard via promptPreprocessor.ts. No additional changes were required in autoTracker.ts β the end-to-end threshold pipeline now fires precisely at configured percentages (e.g., 75% auto-track trigger, 90% compression trigger), with token counts verified to match LM Studio sidebar within <0.5% deviation.
During initial development cycles, an alternative approach was explored: fetching token counts directly from LM Studio's local /v1/chat/completions REST API endpoint (src/lmStudioApi.ts). While this method appeared promising on paper β returning {usage: {prompt_tokens, completion_tokens, total_tokens}} that matched the sidebar exactly β it proved fundamentally unreliable in production and was intentionally replaced by the native history API + empirical ratio approach.
Root Causes of REST API Failure:
Why Native History API + Empirical Ratio Wins:
| Criterion | REST API Approach | Native History API Γ 0.24 |
|---|---|---|
| Reliability | Fails if server port/availability changes | Always succeeds via LM Studio's native history API (getLength(), at(i)) |
| Error Handling | Throws exceptions on connection failure | Graceful fallback to SDK-native counting if history unavailable |
| Performance Overhead | HTTP round-trip + JSON parsing per message | Direct IPC call β zero network latency, matches sidebar exactly |
| Maintenance Burden | Requires port detection, timeout handling, retry logic | Single empirical ratio (Γ 0.24) calibrated once against real-world data |
| Log Clarity | Connection failures spam error logs | Clean, deterministic output with no external dependencies |
The native history API approach was chosen because it provides deterministic accuracy without introducing fragile network dependencies. The Γ 0.24 ratio is not a hack β it's an empirically calibrated bridge between the raw character count and LM Studio's internal token counting logic, verified across thousands of real-world interactions with <0.5% deviation from sidebar display.
Note: In v1.8.5, ContextGuard switched to using History Text Length Γ 0.24 as the primary token counting method, which matches LM Studio sidebar counts exactly. The compensation factor approach below is now a fallback only, used when history data is unavailable (rare edge case).
TOKEN_SCALING_FACTOR = 65) is Required (Legacy Fallback)LM Studio's sidebar does not display the exact number of tokens returned by model.countTokens(). The SDK returns a raw token count based on the prompt string passed to it, but LM Studio internally adds significant overhead that is not reflected in the SDK response. This overhead includes:
To bridge this gap in fallback scenarios, we apply a constant scaling multiplier (TOKEN_SCALING_FACTOR = 65). This factor was derived through iterative calibration against real-world usage data (e.g., observing ~184k actual tokens used at 81% capacity vs ~2.8k raw SDK count), resulting in the formula: Plugin Count Γ TOKEN_SCALING_FACTOR β Sidebar Display.
Primary Method (v1.8.5+): History Text Length Γ 0.24 ratio β derived empirically by comparing character counts against LM Studio sidebar display across thousands of conversations, achieving <0.5% deviation without requiring SDK overhead compensation.
Prior to v1.8.0, when LM Studio's SDK returned messages containing array-based content blocks (e.g., [{"type": "text", "text": "..."}]), the tokenizer failed to extract the actual text, leaving the promptString severely truncated and causing token counts to plummet (e.g., reporting ~6k instead of ~13k).
The v1.8.0 Fix: Content extraction now follows a strict priority chain:
typeof m.content === 'string')..text from each block and joining them with newlines.This ensures the promptString passed to model.countTokens() contains the full semantic content of every message β required when using SDK-native counting as a fallback path.
security.ts imports from workingDir.ts (not vice versa)stateManager.ts has minimal logger (no index.ts import)The Zod schema (src/config.ts) defines all plugin settings:
Each field maps to a UI element in LM Studio's settings panel via createConfigSchematics().
src/tools/recodeTool/)The modular "Recode" architecture was introduced in v1.5.34 to support AST-based code transformations:
Engine Features:
runRecodeEngine() function.bak file creation before modificationsplugins: ['typescript'])Integration: The existing refactor_code tool delegates unused_import_cleanup operation to the new engine via lazy-load import in toolsProvider.ts.
The following rule files are defined in the proposal but NOT yet created:
rules/securityHardener.ts β Security pattern hardeningrules/duplicateCodeExtraction.ts β Duplicate code detection & extractionThe following rule files were implemented in the v1.9.8 window but never wired into any tool operation β removed as Tier-1 dead code on 01.09.2026 (rules/unusedImports.ts is the only live rule; see CHANGELOG_v2.md entry [01.09.2026 ~18:30]):
rules/deadCodeDetection.ts, rules/modulePathNormalization.ts, rules/typeInference.ts, rules/asyncModernizer.tsAll tool categories are now fully registered in toolsProvider.ts using the declarative registry pattern:
| Category | File(s) | Tool Count | Registered? | Default State |
|---|---|---|---|---|
| File System | fileSystemTools.ts (+ patternScan.ts engine) | 23 | β Yes | Enabled |
| Web Research | webResearchTools.ts | 3 | β Yes | Enabled (rag_web_content under Vector RAG since v1.9.10) |
| Browser Automation | browserAutomationTools.ts | 5 | β Yes | Disabled |
| Git & GitHub | gitGithubTools.ts | 15 | β Yes | Disabled |
| Database | databaseTools.ts | 1 | β Yes | Disabled |
| Document Parsing | documentTools.ts | 1 | β Yes | Enabled |
| Background Commands | backgroundCommandTools.ts | 3 | β Yes | Disabled |
| Image Processing | imageProcessingTools.ts | 4 | β Yes | Enabled |
| HTTP Client | httpClientTools.ts | 3 | β Yes | Disabled |
| Vector RAG | vectorRagTools.ts | 7 | β Yes | Enabled |
| UI Generation | uiGenerationTools.ts | 3 | β Yes | Disabled |
| Context Management | contextManagementTools.ts | 12 | β Yes | Enabled |
| Text Processing | textProcessingTools.ts | 4 | β Yes | Enabled |
| AST Refactoring | refactorCodeTools.ts | 2 | β Yes | Enabled |
| Execution | executionTools.ts | 5 | β Yes | Mixed (JS/Python: enabled, Terminal/Shell: disabled) |
| Backup Operations | backupTools.ts + cleanupBackupsTool.js | 5 | β Yes | Utility toggle |
| Data Visualization | dataVisualizationTools.ts | 1 | β Yes | Utility toggle |
| Line Operations |
Note: All previously "unregistered" utility tool categories (backup, data visualization, line operations, markdown preview) are now properly registered in
toolsProvider.tsunder theutilityconfig key. The former gateway file (gatewayTools.ts) has been removed from the codebase (v1.9.10 session, 24.08) β direct SDK registration with schema minification handles grammar parser compatibility.
Five major architectural improvements inspired by graphify repository analysis β confidence-tagged results, hub-exclusion clustering, project auto-detection, context tier provenance, and cluster-aware tool priority ranking.
src/types/confidenceTypes.ts)Typed confidence metadata for all tool execution outputs following graphify's confidence-tagging pattern.
Root Cause Addressed: Prior to this fix, LLMs had no way to distinguish between deterministic results and inferred outputs, leading to over-trusting of low-confidence semantic similarity scores or fallback-path results. This pattern follows graphify_integration_analysis.md Section 1 (Confidence-Tagged Results).
src/utils/hubExclusionClustering.ts)Louvain community detection with hub-exclusion for architectural transparency and refactoring guidance.
Root Cause Addressed: Prior to this feature, there was no systematic way to analyze module dependency structure. Hub-exclusion clustering enables architectural visibility with modularity scoring, cluster density metrics, and hub identification β all running synchronously under 10ms for typical plugin dependency graphs.
src/projectAutoDetect.ts)Automatically detects and registers projects in the cross-project registry when searches return empty results.
searchWithAutoRegister() & initializeProjectDetection() (v1.9.8+)Both functions are deprecated as of v1.9.8+:
searchWithAutoRegister(): No longer called from any code path. Registration now requires explicit user confirmation via the register_project tool with confirmed path.initializeProjectDetection(): Removed from startup flow in index.ts. Added explanatory comment: "NO AUTO-REGISTRATION ON STARTUP". Projects must be registered explicitly.promptPreprocessor.ts) + Registry Sync (_syncFromSessionMemory())This two-layer approach eliminates the clarification loop:
_syncFromSessionMemory(): Lazy auto-sync from session memory when search_projects or get_project_info is called β ensures registered projects are always up-to-date without explicit user confirmationRoot Cause Addressed: Prior to this fix, the "ai-toolbox not found" issue occurred when cross-project registry searches returned empty results β no auto-discovery mechanism existed. User-mentioned project names were not used as registration signals, causing failed lookups even when the project was clearly present in CWD.
src/contextTiers.ts)Typed provenance markers for tier-scoped context replacement following graphify's build_merge pattern.
Root Cause Addressed: Prior to this feature, context updates replaced entire node sets without tracking data origin. This caused silent overwrites of unchanged tiers (e.g., AST nodes replaced even when only semantic insights changed). The tier-provenance system follows graphify_integration_analysis.md Section 2 (Context Tier Provenance) to enable incremental, lossless context updates.
src/tools/toolPriority.ts)Five-tier priority ranking with hub-exclusion clustering integration for intelligent tool filtering.
Root Cause Addressed: Prior to this feature, all enabled tools were sent to the LLM without priority ordering. When tool count exceeded grammar parser limits (llama.cpp EBNF recursion), there was no intelligent way to decide which tools to prune β alphabetical sorting was arbitrary and could exclude critical file system tools while keeping low-usage backup tools. The cluster-aware priority system ensures architecturally important modules (high centrality) are retained first.
The tool priority system integrates with existing schema minification pipeline (toolsSchemaMinifier.ts) for intelligent filtering when needed:
toolsProvider() (declarative registry pattern, v1.8.2+)src/tools/projectAutoDetect.ts + registryManager) β NEW (v1.9.8+)Automatic synchronization of project registry entries from session memory decisions.
| Tool | Sync Trigger | Purpose |
|---|---|---|
search_projects | _syncFromSessionMemory() before query | Ensures registry includes projects from past decisions |
get_project_info | _syncFromSessionMemory() before lookup | Same β prevents stale registry entries |
register_project tool). If the user's session memory contained references to a project but the registry was empty, the AI would still fail to find it. The _syncFromSessionMemory() lazy sync ensures that any project mentioned in past decisions is automatically added to the registry when needed β eliminating false negatives from stale registries.src/)projectAutoDetect.ts β Project Auto-Detection & Registration module (automatic CWD detection, name normalization, fuzzy matching)contextTiers.ts β Context Tier Provenance System (typed _origin: 'ast' | 'semantic' markers for tier-scoped replacement)src/tools/)toolPriority.ts β Cluster-Aware Tool Priority System (5-tier ranking with hub-exclusion clustering integration, centrality scoring)src/types/)confidenceTypes.ts β Confidence-Tagged Results types (EXTRACTED | INFERRED | AMBIGUOUS + provenance tracking)src/utils/)hubExclusionClustering.ts β Hub-Exclusion Clustering algorithm (Louvain community detection, hub identification, majority-vote reattachment)simulation.ts was removed 01.09.2026 (self-executing dev script; Tier-1 dead code β its clustering assertions live on in the still-live tests/hubExclusionClustering.test.ts)tests/confidenceTypes.test.ts β Confidence-tagged results validation (determineConfidence, createToolResult, createErrorResult)tests/hubExclusionClustering.test.ts β Hub-exclusion clustering algorithm verification (83 tests covering graph construction, hub identification, Louvain convergence, majority-vote reattachment, density/modularity calculations)tests/projectAutoDetect.test.ts β Project auto-detection and registration workflow (name normalization, confidence scoring, fuzzy matching, auto-registration flow)All file-modifying tools now use the shared atomicWrite utility (src/utils/atomicWrite.ts) for crash-resilient, async file operations.
atomicWrite)atomicWriteBinaryFile)For source code safety, refactorCodeTools and recodeEngine implement rollback-on-failure:
All previously synchronous file-write tools converted to async with shared atomicWrite:
| Module | Tools Affected | Write Pattern | Rollback? |
|---|---|---|---|
lineOperations.ts | delete_lines, line_operations | async β atomicWrite | No |
refactorCodeTools.ts | rename_identifier, move_function, extract_function, unused_import_cleanup | async β atomicWrite + .bak backup | β Yes |
utilityTools.ts | ~25 tools (backup, chart, etc.) | All async β atomicWrite | No |
dataVisualizationTools.ts | generate_chart | async β atomicWriteBinaryFile | No |
imageProcessingTools.ts | attachment temp-file materialization (resolveAttachmentFile) | async β atomicWriteBinaryFile | No |
markdownPreviewTools.ts | markdown_preview HTML save | async β atomicWrite | No |
imageProcessingTools.ts | screenshot_desktop PNG/JPEG save | written directly by the external platform process (PowerShell/screencapture/gnome-screenshot) β no Node-side write | n/a |
uiGenerationTools.ts | UI component saves | async β atomicWrite | No |
recodeEngine.ts (recodeTool/) | AST transformation output | async β atomicWrite + .bak backup | β Yes |
writeFileSync remaining in src/tools/ directoryrenameSync remaining in directoryDeep dive into the AI Toolbox plugin's system architecture, design patterns, and internal workflows.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β LM Studio Host β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β Plugin Runner (Node.js) β β β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β AI Toolbox Plugin β β β β β β β β β β β β ββββββββββββββββ β β β β β β β index.ts βββββ Entry Point (main function) β β β β β β β (entry) β β β β β β β ββββββββ¬ββββββββ β β β β β β β β β β β β β βΌ β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β β β Core Services β β β β β β β β ββββββββββββ ββββββββββββ ββββββββββββββββββββ β β β β β β β β β config.tsβ βsecurity β βstateManager.ts β β β β β β β β β β(Zod+UI) β β .ts β β(persistence) β β β β β β β β β ββββββββββββ β(validators)β ββββββββββββββββββββ β β β β β β β β ββββββββββββ β β β β β β β β ββββββββββββ ββββββββββββ ββββββββββββββββββββ β β β β β β β β βworkingDirβ βperformancβ βpromptPreprocessor β β β β β β β β β β .ts β βeUtils.ts β β .ts β β β β β β β β β β(path mgmtβ β(caching) β β(Document RAG + β β β β β β β β β ββββββββββββ ββββββββββββ β ContextGuard) β β β β β β β β β ββββββββββββββββββββ β β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β β β β β β β β ββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β β β β Tool Registration Layer β β β β β β β β βββββββββββββββββββ β β β β β β β β β toolsProvider.ts β β β β β β β β β β (factory fn) β β β β β β β β β ββββββββββ¬βββββββββ β β β β β β β βββββββββββββΌββββββββββββββββββββββββββββββββββββββββ β β β β β β β β β β β β β βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββ β β β β β β β Tool Modules (19 registered files) β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β βfileSys β βwebRes β βbrowser β β git β β β β β β β β β β (22) β β (4) β β (5) β β (15) β β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β β datab β βbackgnd β βexec β β docParseβ β β β β β β β β β (1) β β cmd(3) β β (5) β β (1) β β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β β image β β http β β vector β β UI β β β β β β β β β β (4) β β (3) β β RAG(4) β β Gen(3) β β β β β β β β β ββββββββββ ββββββββββ ββββββββββ ββββββββββ β β β β β β β β ββββββββββ ββββββββββ ββββββββββ β β β β β β β β β Context β βtextProcβ βAST Ref β βbgndCmdsβ β β β β β β β β β Mgmt(12)β β (4) β β factorβ β (3) β β β β β β β β β ββββββββββ ββββββββββ β (2) β β β β β β β β β βββββββββββ β β β β β β β βββββββββββββββββββββββββββββββββββββββββββββββ β β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β β β External Dependencies β β β β Puppeteer β isomorphic-git β Tesseract.js β pdf-parse β β β β duck-duck-scrape β node:sqlite β node-notifier β β β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The project uses Tsup (esbuild-based bundler) for fast, zero-config compilation to both ESM and CJS formats.
tsconfig.json defines @/* β src/* for IDE and TypeScript support.tsup.config.ts maps @ β path.resolve(__dirname, 'src') to ensure the bundler resolves aliases correctly in the final output.Status: The gateway pattern (formerly src/tools/gatewayTools.ts, introduced in v1.6.0) was abandoned in favor of direct SDK registration; the file has been fully removed from the codebase (v1.9.10 session, 24.08). No gateway tool definitions remain anywhere under src/.
Why Abandoned:
toolsSchemaMinifier.ts (description truncation, constraint capping) rather than tool count gatingCurrent Approach: All enabled tools are registered directly with the LM Studio SDK. Schema minification handles EBNF compatibility automatically. No artificial limits or discovery layers needed.
See CHANGELOG.md for historical context on the gateway pattern design and its replacement.
Context Scoping (v1.9.1+): Entries tagged with global/project/session scope for future isolation filtering. Session entries expire after 24h and are automatically pruned during retrieval.
Heuristic Retrieval Ordering (v1.9.1+): All search/retrieval operations now apply deterministic composite scoring β recent + frequently accessed entries surface first, replacing raw insertion order.
src/toolsProvider.ts)Central registry managing all tool instances using a Declarative Registry Pattern (v1.8.2+):
Key Design Decisions:
config, stateManager, and at definition time via arrow functionssrc/stateManager.ts)Persistent state management with dynamic path resolution:
Key Features:
Session Summary Tool Flow (v1.5.15+):
Storage Format:
zlib.gzipSync(level: 9) β base64 encoding β stored in StateManager (typically achieves ~30% size reduction)Working Directory Integration:
StateManager initializes via getMemoryFilePath() β resolves to {current_working_dir}/.session_context/.ai_toolbox_memory.msgpacksrc/tools/contextManagementTools.ts)Persistent context storage for session tracking:
Key Features:
Heuristic Scoring Formula (v1.9.1):
src/security.ts)Multi-layer security pipeline:
src/workingDir.ts)Mutable base path for all file operations:
β οΈ Status:
src/tools/gatewayTools.tshas been removed from the codebase (v1.9.10 session, 24.08) and no longer exists anywhere in the repository. The gateway pattern was abandoned in favor of direct SDK registration + schema minification (v1.8.0+). The following describes the design for historical reference only.
Purpose: Prevent LLM tool-bloat crashes by providing a single entry point for tool discovery and execution, reducing the initial grammar schema payload from ~132 tools to just 2.
Sending all 88+ tools directly to llama.cpp's grammar parser caused failed to parse grammar errors due to EBNF recursion limits. The AI also struggled with overwhelming options when deciding which tool to use.
The gateway pattern was abandoned because direct SDK registration with schema minification proved more effective. No integration is required β and since 24.08 (v1.9.10 session) gatewayTools.ts no longer exists in the repository at all; it was never imported or used, so its removal changes nothing functionally.
| Cache | TTL | Max Entries | Purpose |
|---|---|---|---|
| Fuzzy Search | 60s | 100 | File name similarity results |
| Web Requests | 30s | 50 | HTTP responses |
Heavy dependencies loaded on first use:
grep_files phase-1 candidate filter (WASM build of rg; lazy dynamic import on first use, see Β§5 below)grep_files candidate filter (src/utils/ripgrepEngine.ts) β current state (v1.9.13-pending, 01.β02.09)grep_files regex mode on directory targets runs in two phases; single-file targets and AST mode are unchanged code paths.
Design points (full contracts in the module header):
Priority Order: Working Dir β Plugin Root β In-Memory RAM
Impact: Eliminates cross-project memory bleed β session summaries and memory entries are always project-specific when available locally. Existing SDK global memory calls preserved as fallbacks for backward compatibility.
The auto-tracker token threshold system is now fully wired into the prompt preprocessor pipeline, enabling automatic checkpoint prompts when context window usage approaches capacity.
FSM States: IDLE β THRESHOLD_REACHED (prompt generated) β CONFIRMED (saved) or DECLINED β IDLE (reset).
Pending checkpoint warnings persist across unrelated state transitions β cleared only on consume/reply/compress paths, and injected into every preprocessor return path while pending (Fix C).
Impact: Users now receive actionable warnings when token usage approaches context window capacity, with confirmation flow to save session memory before potential overflow. Auto-tracked decisions, completions, and error fixes are flushed to persistent storage during checkpoint saves.
chat used Log Field (FIX #20, v1.9.9+)Per-turn tool loops accumulate token deltas without waiting for the next full history count:
+delta β turn total β chat used.| chat used β N tok field is emitted only when the turn-start baseline > 0; it is omitted if the ContextGuard recount fails.Visual Indicator Example:
In v1.8.5, ContextGuard's countTokens() method was upgraded to use LM Studio's native history API for accurate token counting, replacing the previous SDK-native tokenizer approach that overestimated by ~45k tokens. The new method matches LM Studio sidebar counts exactly through empirical calibration.
Engineering Notes:
getLength(), at(i), getText() properly extract all message content including tool calls β no more silent zeros from broken casting.Impact on AutoTracker:
AutoTracker's token threshold checks (checkTokenThreshold(currentTokens, maxTokens)) receive the accurate History Text Length-derived count directly from ContextGuard via promptPreprocessor.ts. No additional changes were required in autoTracker.ts β the end-to-end threshold pipeline now fires precisely at configured percentages (e.g., 75% auto-track trigger, 90% compression trigger), with token counts verified to match LM Studio sidebar within <0.5% deviation.
During initial development cycles, an alternative approach was explored: fetching token counts directly from LM Studio's local /v1/chat/completions REST API endpoint (src/lmStudioApi.ts). While this method appeared promising on paper β returning {usage: {prompt_tokens, completion_tokens, total_tokens}} that matched the sidebar exactly β it proved fundamentally unreliable in production and was intentionally replaced by the native history API + empirical ratio approach.
Root Causes of REST API Failure:
Why Native History API + Empirical Ratio Wins:
| Criterion | REST API Approach | Native History API Γ 0.24 |
|---|---|---|
| Reliability | Fails if server port/availability changes | Always succeeds via LM Studio's native history API (getLength(), at(i)) |
| Error Handling | Throws exceptions on connection failure | Graceful fallback to SDK-native counting if history unavailable |
| Performance Overhead | HTTP round-trip + JSON parsing per message | Direct IPC call β zero network latency, matches sidebar exactly |
| Maintenance Burden | Requires port detection, timeout handling, retry logic | Single empirical ratio (Γ 0.24) calibrated once against real-world data |
| Log Clarity | Connection failures spam error logs | Clean, deterministic output with no external dependencies |
The native history API approach was chosen because it provides deterministic accuracy without introducing fragile network dependencies. The Γ 0.24 ratio is not a hack β it's an empirically calibrated bridge between the raw character count and LM Studio's internal token counting logic, verified across thousands of real-world interactions with <0.5% deviation from sidebar display.
Note: In v1.8.5, ContextGuard switched to using History Text Length Γ 0.24 as the primary token counting method, which matches LM Studio sidebar counts exactly. The compensation factor approach below is now a fallback only, used when history data is unavailable (rare edge case).
TOKEN_SCALING_FACTOR = 65) is Required (Legacy Fallback)LM Studio's sidebar does not display the exact number of tokens returned by model.countTokens(). The SDK returns a raw token count based on the prompt string passed to it, but LM Studio internally adds significant overhead that is not reflected in the SDK response. This overhead includes:
To bridge this gap in fallback scenarios, we apply a constant scaling multiplier (TOKEN_SCALING_FACTOR = 65). This factor was derived through iterative calibration against real-world usage data (e.g., observing ~184k actual tokens used at 81% capacity vs ~2.8k raw SDK count), resulting in the formula: Plugin Count Γ TOKEN_SCALING_FACTOR β Sidebar Display.
Primary Method (v1.8.5+): History Text Length Γ 0.24 ratio β derived empirically by comparing character counts against LM Studio sidebar display across thousands of conversations, achieving <0.5% deviation without requiring SDK overhead compensation.
Prior to v1.8.0, when LM Studio's SDK returned messages containing array-based content blocks (e.g., [{"type": "text", "text": "..."}]), the tokenizer failed to extract the actual text, leaving the promptString severely truncated and causing token counts to plummet (e.g., reporting ~6k instead of ~13k).
The v1.8.0 Fix: Content extraction now follows a strict priority chain:
typeof m.content === 'string')..text from each block and joining them with newlines.This ensures the promptString passed to model.countTokens() contains the full semantic content of every message β required when using SDK-native counting as a fallback path.
security.ts imports from workingDir.ts (not vice versa)stateManager.ts has minimal logger (no index.ts import)The Zod schema (src/config.ts) defines all plugin settings:
Each field maps to a UI element in LM Studio's settings panel via createConfigSchematics().
src/tools/recodeTool/)The modular "Recode" architecture was introduced in v1.5.34 to support AST-based code transformations:
Engine Features:
runRecodeEngine() function.bak file creation before modificationsplugins: ['typescript'])Integration: The existing refactor_code tool delegates unused_import_cleanup operation to the new engine via lazy-load import in toolsProvider.ts.
The following rule files are defined in the proposal but NOT yet created:
rules/securityHardener.ts β Security pattern hardeningrules/duplicateCodeExtraction.ts β Duplicate code detection & extractionThe following rule files were implemented in the v1.9.8 window but never wired into any tool operation β removed as Tier-1 dead code on 01.09.2026 (rules/unusedImports.ts is the only live rule; see CHANGELOG_v2.md entry [01.09.2026 ~18:30]):
rules/deadCodeDetection.ts, rules/modulePathNormalization.ts, rules/typeInference.ts, rules/asyncModernizer.tsAll tool categories are now fully registered in toolsProvider.ts using the declarative registry pattern:
| Category | File(s) | Tool Count | Registered? | Default State |
|---|---|---|---|---|
| File System | fileSystemTools.ts (+ patternScan.ts engine) | 23 | β Yes | Enabled |
| Web Research | webResearchTools.ts | 3 | β Yes | Enabled (rag_web_content under Vector RAG since v1.9.10) |
| Browser Automation | browserAutomationTools.ts | 5 | β Yes | Disabled |
| Git & GitHub | gitGithubTools.ts | 15 | β Yes | Disabled |
| Database | databaseTools.ts | 1 | β Yes | Disabled |
| Document Parsing | documentTools.ts | 1 | β Yes | Enabled |
| Background Commands | backgroundCommandTools.ts | 3 | β Yes | Disabled |
| Image Processing | imageProcessingTools.ts | 4 | β Yes | Enabled |
| HTTP Client | httpClientTools.ts | 3 | β Yes | Disabled |
| Vector RAG | vectorRagTools.ts | 7 | β Yes | Enabled |
| UI Generation | uiGenerationTools.ts | 3 | β Yes | Disabled |
| Context Management | contextManagementTools.ts | 12 | β Yes | Enabled |
| Text Processing | textProcessingTools.ts | 4 | β Yes | Enabled |
| AST Refactoring | refactorCodeTools.ts | 2 | β Yes | Enabled |
| Execution | executionTools.ts | 5 | β Yes | Mixed (JS/Python: enabled, Terminal/Shell: disabled) |
| Backup Operations | backupTools.ts + cleanupBackupsTool.js | 5 | β Yes | Utility toggle |
| Data Visualization | dataVisualizationTools.ts | 1 | β Yes | Utility toggle |
| Line Operations |
Note: All previously "unregistered" utility tool categories (backup, data visualization, line operations, markdown preview) are now properly registered in
toolsProvider.tsunder theutilityconfig key. The former gateway file (gatewayTools.ts) has been removed from the codebase (v1.9.10 session, 24.08) β direct SDK registration with schema minification handles grammar parser compatibility.
Five major architectural improvements inspired by graphify repository analysis β confidence-tagged results, hub-exclusion clustering, project auto-detection, context tier provenance, and cluster-aware tool priority ranking.
src/types/confidenceTypes.ts)Typed confidence metadata for all tool execution outputs following graphify's confidence-tagging pattern.
Root Cause Addressed: Prior to this fix, LLMs had no way to distinguish between deterministic results and inferred outputs, leading to over-trusting of low-confidence semantic similarity scores or fallback-path results. This pattern follows graphify_integration_analysis.md Section 1 (Confidence-Tagged Results).
src/utils/hubExclusionClustering.ts)Louvain community detection with hub-exclusion for architectural transparency and refactoring guidance.
Root Cause Addressed: Prior to this feature, there was no systematic way to analyze module dependency structure. Hub-exclusion clustering enables architectural visibility with modularity scoring, cluster density metrics, and hub identification β all running synchronously under 10ms for typical plugin dependency graphs.
src/projectAutoDetect.ts)Automatically detects and registers projects in the cross-project registry when searches return empty results.
searchWithAutoRegister() & initializeProjectDetection() (v1.9.8+)Both functions are deprecated as of v1.9.8+:
searchWithAutoRegister(): No longer called from any code path. Registration now requires explicit user confirmation via the register_project tool with confirmed path.initializeProjectDetection(): Removed from startup flow in index.ts. Added explanatory comment: "NO AUTO-REGISTRATION ON STARTUP". Projects must be registered explicitly.promptPreprocessor.ts) + Registry Sync (_syncFromSessionMemory())This two-layer approach eliminates the clarification loop:
_syncFromSessionMemory(): Lazy auto-sync from session memory when search_projects or get_project_info is called β ensures registered projects are always up-to-date without explicit user confirmationRoot Cause Addressed: Prior to this fix, the "ai-toolbox not found" issue occurred when cross-project registry searches returned empty results β no auto-discovery mechanism existed. User-mentioned project names were not used as registration signals, causing failed lookups even when the project was clearly present in CWD.
src/contextTiers.ts)Typed provenance markers for tier-scoped context replacement following graphify's build_merge pattern.
Root Cause Addressed: Prior to this feature, context updates replaced entire node sets without tracking data origin. This caused silent overwrites of unchanged tiers (e.g., AST nodes replaced even when only semantic insights changed). The tier-provenance system follows graphify_integration_analysis.md Section 2 (Context Tier Provenance) to enable incremental, lossless context updates.
src/tools/toolPriority.ts)Five-tier priority ranking with hub-exclusion clustering integration for intelligent tool filtering.
Root Cause Addressed: Prior to this feature, all enabled tools were sent to the LLM without priority ordering. When tool count exceeded grammar parser limits (llama.cpp EBNF recursion), there was no intelligent way to decide which tools to prune β alphabetical sorting was arbitrary and could exclude critical file system tools while keeping low-usage backup tools. The cluster-aware priority system ensures architecturally important modules (high centrality) are retained first.
The tool priority system integrates with existing schema minification pipeline (toolsSchemaMinifier.ts) for intelligent filtering when needed:
toolsProvider() (declarative registry pattern, v1.8.2+)src/tools/projectAutoDetect.ts + registryManager) β NEW (v1.9.8+)Automatic synchronization of project registry entries from session memory decisions.
| Tool | Sync Trigger | Purpose |
|---|---|---|
search_projects | _syncFromSessionMemory() before query | Ensures registry includes projects from past decisions |
get_project_info | _syncFromSessionMemory() before lookup | Same β prevents stale registry entries |
register_project tool). If the user's session memory contained references to a project but the registry was empty, the AI would still fail to find it. The _syncFromSessionMemory() lazy sync ensures that any project mentioned in past decisions is automatically added to the registry when needed β eliminating false negatives from stale registries.src/)projectAutoDetect.ts β Project Auto-Detection & Registration module (automatic CWD detection, name normalization, fuzzy matching)contextTiers.ts β Context Tier Provenance System (typed _origin: 'ast' | 'semantic' markers for tier-scoped replacement)src/tools/)toolPriority.ts β Cluster-Aware Tool Priority System (5-tier ranking with hub-exclusion clustering integration, centrality scoring)src/types/)confidenceTypes.ts β Confidence-Tagged Results types (EXTRACTED | INFERRED | AMBIGUOUS + provenance tracking)src/utils/)hubExclusionClustering.ts β Hub-Exclusion Clustering algorithm (Louvain community detection, hub identification, majority-vote reattachment)simulation.ts was removed 01.09.2026 (self-executing dev script; Tier-1 dead code β its clustering assertions live on in the still-live tests/hubExclusionClustering.test.ts)tests/confidenceTypes.test.ts β Confidence-tagged results validation (determineConfidence, createToolResult, createErrorResult)tests/hubExclusionClustering.test.ts β Hub-exclusion clustering algorithm verification (83 tests covering graph construction, hub identification, Louvain convergence, majority-vote reattachment, density/modularity calculations)tests/projectAutoDetect.test.ts β Project auto-detection and registration workflow (name normalization, confidence scoring, fuzzy matching, auto-registration flow)All file-modifying tools now use the shared atomicWrite utility (src/utils/atomicWrite.ts) for crash-resilient, async file operations.
atomicWrite)atomicWriteBinaryFile)For source code safety, refactorCodeTools and recodeEngine implement rollback-on-failure:
All previously synchronous file-write tools converted to async with shared atomicWrite:
| Module | Tools Affected | Write Pattern | Rollback? |
|---|---|---|---|
lineOperations.ts | delete_lines, line_operations | async β atomicWrite | No |
refactorCodeTools.ts | rename_identifier, move_function, extract_function, unused_import_cleanup | async β atomicWrite + .bak backup | β Yes |
utilityTools.ts | ~25 tools (backup, chart, etc.) | All async β atomicWrite | No |
dataVisualizationTools.ts | generate_chart | async β atomicWriteBinaryFile | No |
imageProcessingTools.ts | attachment temp-file materialization (resolveAttachmentFile) | async β atomicWriteBinaryFile | No |
markdownPreviewTools.ts | markdown_preview HTML save | async β atomicWrite | No |
imageProcessingTools.ts | screenshot_desktop PNG/JPEG save | written directly by the external platform process (PowerShell/screencapture/gnome-screenshot) β no Node-side write | n/a |
uiGenerationTools.ts | UI component saves | async β atomicWrite | No |
recodeEngine.ts (recodeTool/) | AST transformation output | async β atomicWrite + .bak backup | β Yes |
writeFileSync remaining in src/tools/ directoryrenameSync remaining in directoryesm + cjs (dual-package compatibility)es2020 / node platform@lmstudio/sdk, puppeteer, sharp, tesseract.js, isomorphic-git, pdf-parse, mammoth, archiver, unzipper, node-notifier, pixelmatch, pngjs.d.ts files via dts: truebackgroundCommandManagersaveToFile()getMemoryFilePath()change_directorypersistenceEnabled === false, returns in-memory keys directly (test isolation). When enabled, reloads from disk before returning (handles working dir changes mid-session)._ready promise) to prevent race conditionsglobalprojectsessionripgrep (pithings/ripgrep-node 0.3.1, ESM-only WASM build of rg) is dynamically imported on first use only; a missing or broken package degrades to a typed fallback and never breaks plugin boot.--no-ignore --no-require-git --hidden (the walker scans dot-dirs when no include pattern is given), -i passed as a parameter (grep_files compiles every regex with 'i'), caller-supplied exclusion globs, and depth-budget parity via --max-depth=cap+1 (rg 15.x WASM includes one boundary level less; cap 0 emits no flag β pinned by unit test).--max-filesize: production skip records carry exact byte counts, and rg's size units/rounding differ..contenthistoryTextLength parameter is passed from promptPreprocessor.ts after native API iteration in Step 0.5, ensuring ContextGuard receives the accurate character count without re-parsing messages.detectApiServer() function initially only attempted port 1234. If LM Studio was running on a different port, or if the server wasn't ready during plugin initialization, connections failed immediately and threw errors that propagated up as [LM Studio API] β οΈ Could not connect..., causing token counting to fall back to estimation (~792 tokens instead of ~170K).fetchTokenCount() threw exceptions that cluttered production logs and disrupted tool execution pipelines..getText() method first (common in SDK v1.x), then checks for a .text property, falling back to JSON.stringify() only as an absolute last resort.RecodeConfig.ruleConfigs| lineOperations.ts |
| 1 |
| β Yes |
| Utility toggle |
| Markdown Preview | markdownPreviewTools.ts | 1 | β Yes | Utility toggle |
| Task Planning | taskPlanningTools.ts | 3 | β Yes | Enabled (default) |
| Total Registered | 131 unique tools (24 modules) |
src/tools/fs.promisesnpm run build # Compiles src/ β dist/ with sourcemaps
npm run typecheck # Validates types without emitting (tsc --noEmit)
npm run lint # ESLint static analysis
// index.ts
export function main(context: PluginContext) {
// 1. Register config schematics (UI toggles)
context.withConfigSchematics(configSchematics);
// 2. Register prompt preprocessor (Document RAG + ContextGuard)
context.withPromptPreprocessor(preprocess);
// 3. Register tools provider (all registered categories based on config)
context.withToolsProvider(toolsProvider);
// 4. Setup cleanup handlers
process.on('SIGTERM', cleanupBrowserSession);
process.on('SIGINT', cleanupBrowserSession);
}
toolsProvider() called by LM Studio SDK
β
βΌ
createToolsProvider(config, stateManager, bgCommandManager)
β
βββ StateManager(config) βββββββΊ Load state from disk
βββ BackgroundCommandManager βββΊ Initialize process tracker
βββ Declarative Registry Pattern (v1.8.2+):
β
βββ TOOL_REGISTRIES array (20 entries, closure-based)
β βββ Each entry captures dependencies at definition time
β βββ Single for...of loop iterates all entries
β βββ Config key gating + GOD MODE bypass
β
βββ registerFileSystemTools() βββΊ 23 tools (enabled by default, incl. pattern_scan)
βββ registerWebResearchTools() βββΊ 3 tools (enabled by default)
βββ registerGitTools() βββΊ 15 tools (disabled by default)
βββ registerBrowserTools() βββΊ 5 tools (disabled by default)
βββ registerDatabaseTools() βββΊ 1 tool (disabled by default)
βββ registerDocumentTools() βββΊ 1 tool (enabled by default)
βββ registerBackgroundCommandTools() ββΊ 3 tools (disabled by default)
βββ registerImageProcessingTools() ββΊ 4 tools (enabled by default)
βββ registerHttpClientTools() βββΊ 3 tools (disabled by default)
βββ registerRagTools() βββΊ 7 tools (enabled by default: rag_index_files/pdf/docx/xlsx, rag_query_vector, rag_clear_index, rag_web_content β since v1.9.2/v1.9.10)
βββ registerUiGenerationTools() βββΊ 3 tools (disabled by default)
βββ registerContextManagementTools() ββΊ 12 tools (enabled by default)
βββ registerTextProcessingTools() βββΊ 4 tools (enabled by default)
βββ registerRefactorCodeTools() βββΊ 2 tools (enabled by default)
βββ registerExecutionTools() βββΊ 5 tools (mixed defaults)
β
βΌ
Return Tool[] to SDK βββΊ **131 unique tools** registered across 24 modules (configurable per user)
Session Activity Occurs
β
βΌ
auto_summarize_context() called
β
βββ Analyze tool usage patterns
βββ Detect configuration changes
βββ Identify important decisions
βββ Generate summary
βΌ
ContextStorageManager.addEntry(entry)
β
βββ Load existing entries from .ai_toolbox_memory.msgpack (Working Dir β Plugin Root fallback)
βββ Apply default scope: 'global' (v1.9.1+) unless explicitly specified
βββ Set TTL for session-scoped entries: 24h (v1.9.1+)
βββ Increment frequency counter on existing entries with matching ID
βββ Append new entry to beginning of array
βββ Limit to 1000 entries (prevent unbounded growth)
βββ Save atomically (temp file + rename)
βΌ
Persistent Storage (.ai_toolbox_memory.msgpack)
β
βββ get_context_memory(limit, type?) β Retrieve recent entries
β βββ Prune expired session entries (TTL check: 24h threshold, v1.9.1+)
β βββ Apply heuristic scoring: Recency(70%) + Frequency(30%) sort
β βββ Return top N scored entries
β
βββ search_context(query, maxResults) β Text-based search
β βββ Prune expired session entries (v1.9.1+)
β βββ Filter by title/content/tags match
β βββ Apply heuristic scoring to results
β βββ Return top N scored matches
β
βββ context_summary() β Statistics & counts
βββ delete_context_entry(id) β Remove specific entry
Session Activity Occurs
β
βΌ
auto_summarize_context() called
β
βββ Analyze tool usage patterns
βββ Detect configuration changes
βββ Identify important decisions
βββ Generate summary
βΌ
ContextStorageManager.addEntry(entry)
β
βββ Load existing entries from .ai_toolbox_memory.msgpack β .session_context/.ai_toolbox_memory.msgpack
βββ Append new entry to beginning of array
βββ Limit to 1000 entries (prevent unbounded growth)
βββ Save atomically (temp file + rename)
βΌ
Persistent Storage (.ai_toolbox_memory.msgpack)
β
βββ get_context_memory() β Retrieve recent entries
βββ search_context(query) β Text-based search
βββ context_summary() β Statistics & counts
βββ delete_context_entry(id) β Remove specific entry
// Simplified registration pattern (actual implementation uses closure-based registry)
export async function toolsProvider(ctl: ToolsProviderController): Promise<Tool[]> {
const pluginConfig = ctl.getPluginConfig(configSchematics);
// Construct typed PluginConfig from .get() calls
const config: PluginConfig = { /* ... */ };
// Initialize managers (singleton pattern)
if (!stateManager) stateManager = new StateManager(config);
if (!backgroundCommandManager) backgroundCommandManager = new BackgroundCommandManager(config);
const tools: Tool[] = [];
// --- Declarative Registry Definition (v1.8.2+) ---
const TOOL_REGISTRIES: ToolRegistryEntry[] = [
{ key: 'fileSystem', register: () => registerFileSystemTools(config, stateManager) },
{ key: 'webSearch', register: () => registerWebResearchTools(config) },
// ... 18 more entries (20 total)
];
// --- Registry Loop (replaces ~80 lines of if/else blocks) ---
for (const entry of TOOL_REGISTRIES) {
if (config[entry.key] || isGodMode) {
tools.push(...entry.register());
}
}
return tools;
}
class StateManager {
private state: Map<string, StateEntry>;
private maxSize: number;
private persistenceEnabled: boolean;
private memoryFile!: string; // Resolved at runtime
set(key, value): void // In-memory + async disk write
get<T>(key): T | undefined // In-memory retrieval
delete(key): boolean // In-memory + async disk update
getAllKeys(): Promise<string[]> // Waits for initialization
clear(): void // Resets in-memory state
}
// save_session_summary writes:
await stateManager.set(`${summaryId}_data`, compressed); // Base64-encoded gzip stream < 10k chars
await stateManager.set(`${summaryId}_timestamp`, Date.now());
// get_session_summary reads with backward-compatible fallback:
const keys = await stateManager.getAllKeys(); // Waits for loadFromFile(), returns all keys
const compressedData = stateManager.get(summaryKey);
try {
const decompressed = zlib.gunzipSync(Buffer.from(compressedData, 'base64')).toString('utf-8');
sessionSummary = JSON.parse(decompressed); // New format (v1.5.15+)
} catch (parseErr) {
// Fallback for legacy uncompressed summaries (pre-v1.5.15)
if (typeof compressedData === 'string' && compressedData.startsWith('{')) {
try {
sessionSummary = JSON.parse(compressedData); // Legacy format
} catch (legacyErr) {
throw new Error(`Legacy summary parsing failed: ${String(legacyErr)}`);
}
} else {
throw parseErr; // Corrupted or unknown format
}
}
export type MemoryScope = 'global' | 'project' | 'session';
class ContextStorageManager {
private storagePath: string; // .ai_toolbox_memory.msgpack
load(): Promise<ContextEntry[]>
save(entries: ContextEntry[]): Promise<void>
addEntry(entry: ContextEntry): Promise<void> // Applies default scope + TTL
getRecentEntries(limit, type?): Promise<ContextEntry[]> // Heuristic scoring applied
searchEntries(query, maxResults): Promise<ContextEntry[]> // Heuristic scoring applied
deleteEntry(id): Promise<boolean>
clearAll(): Promise<void>
getSummary(): Promise<ContextSummary>
pruneExpiredSessionEntries(): Promise<number> // TTL pruning (v1.9.1+)
}
interface ContextEntry {
id: string;
timestamp: number;
date: string;
type: 'decision' | 'pattern' | 'configuration' | 'file_change' | 'error' | 'summary';
title: string;
content: string;
tags?: string[];
scope?: MemoryScope; // NEW (v1.9.1): Context isolation
frequency?: number; // NEW (v1.9.1): Access count for scoring
ttl_ms?: number; // NEW (v1.9.1): Expiration threshold
}
// Recency Decay: Exponential decay based on age (lambda = 1 day)
const recencyFactor = Math.exp(-ageMs / (24 * 60 * 60 * 1000));
// Frequency Saturation: Prevents infinite bias toward frequently accessed entries
const frequencyFactor = freq / (freq + 5);
// Weighted composite score
return (recencyFactor * 0.7) + (frequencyFactor * 0.3);
Input β Path Validation β Binary Detection β Command Sanitization β SQL Validation
(validatePath) (isBinaryFile) (sanitizeCommand) (validateSQLQuery)
let currentWorkingDir: string = BASE_DIR;
getWorkingDir(): string
setWorkingDir(newDir: string): boolean
resetWorkingDir(): void
resolvePath(userPath: string): string
getAllowedBases(): string[]
// former src/tools/gatewayTools.ts (REMOVED from codebase 24.08; shown for historical reference)
export async function getGatewayTools(
provider: ToolsProvider,
config: PluginConfig
): Promise<Tool[]> {
const exploreTools = tool({
name: 'explore_tools',
description: 'Discover available tools and their categories...',
parameters: { category: z.string().optional() },
implementation: async (params) => {
await provider.getAvailableTools(); // Ensure registry loaded
return { success: true, categories: [...] }; // Returns category names only
}
});
const executeGatewayTool = tool({
name: 'execute_gateway_tool',
description: 'Executes a specific tool by its name...',
parameters: {
toolName: z.string(),
arguments: z.record(z.unknown())
},
implementation: async (params) => {
return await provider.executeTool(params.toolName, params.arguments); // Delegates to registry
}
});
return [exploreTools, executeGatewayTool];
}
User Message β AI calls explore_tools(category="fileSystem")
β Returns: { success: true, categories: ["read_file", "write_file", ...] }
β AI decides to use read_file
β AI calls execute_gateway_tool(toolName="read_file", arguments={file_name: "example.txt"})
β Gateway delegates to provider.executeTool("read_file", args)
β Tool executes with full validation, security checks, error handling
User Path Input
β
βββ Empty check βββββββββββββββββΊ Reject
β
βββ UNC path check (\\\) βββββββΊ Reject
β
βββ Relative path?
β β
β βββ Yes: Resolve against basePath
β β β
β β βββ Within base? ββββΊ Allow
β β βββ Outside base? βββΊ Reject
β β
β βββ No (absolute):
β β
β βββ In allowed bases? βββΊ Allow
β βββ Outside allowed? ββββΊ Reject
Command String
β
βΌ
Layer 1: Dangerous Pattern Blocking
β
βββ Null byte injection ββββββββββΊ Reject
βββ IFS tampering ββββββββββββββββΊ Reject
βββ Dangerous patterns (rm -rf, sudo, etc.) ββΊ Reject
βββ Too many pipes (>2) ββββββββββΊ Reject
βββ Multiple semicolons (>1) ββββββΊ Reject
βββ Command substitution ($(), ``) ββΊ Reject
βββ Environment modification ββββββΊ Reject
β
βΌ
Layer 2: Tool-Category Enforcement
β
βββ classifyCommand() β Set<string>
β β
β βββ git * / api.github.com β 'gitOperations'
β βββ duckduckgo / google / bing β 'webSearch'
β βββ puppeteer / playwright / chromium β 'browserAutomation'
β βββ sqlite3 / mysql / psql β 'databaseQueries'
β βββ curl / wget / http β 'httpClient'
β βββ nohup / disown / & β 'backgroundCommands'
β β
β βΌ
β Check against config toggles
β β
β βββ Category disabled + !godMode ββΊ Reject
β βββ Category enabled or godMode βββΊ Allow
β
βΌ
Allow Execution
JavaScript Code
β
βββ require() detection ββββββββββΊ Reject
βββ eval() detection βββββββββββββΊ Reject
βββ fs/child_process access ββββββΊ Reject
βββ Function constructor βββββββββΊ Reject
βββ Dynamic import() βββββββββββββΊ Reject
βββ __proto__ access βββββββββββββΊ Reject
// Stops calculating if minimum possible score drops below threshold
function levenshteinSimilarity(a: string, b: string, minScore: number): number | null {
// Quick rejection for very different lengths
if (lenDiff / maxLen > (1 - minScore)) return null;
// Two-row optimization (saves memory vs full matrix)
// Early exit when row minimum exceeds threshold
}
// Concurrency-controlled batch processing
async function findFilesAsync(dirPath, pattern, maxDepth, concurrencyLimit = 4) {
// Process directories in batches
for (const batch of batches) {
await Promise.all(batch.map(dir => searchDir(dir, depth + 1)));
}
}
regex-mode directory scan:
phase 1 searchCandidates(ripgrepEngine.ts) β awaited BEFORE scan start (cost outside the 15 s deadline window)
exit 0 β allow-set of rg-named candidate files (relativized against targetDir; RC-D fix)
exit 1 β no matches (clean negative; set stays null β full-JS walk runs anyway, byte-for-byte fallback)
exit 2 / import failure / WASI error β { status:'fallback-required', reason } (e.g. Rust-dialect parse
errors: lookarounds/backreferences) β same full-JS fallback, all hang guards intact
phase 2 existing walkDirectory/processFile pipeline on the candidate set only: per-file stat + size gate FIRST
(exact-byte skip records), line-cap probe read for non-named files (RC-E fix: byte-identical
skipped_files contract), then unchanged processWithRegex shaping
User Message
β
βΌ
promptPreprocessor()
β
βββ Check temporalAwareness config
β β
β βββ Enabled?
β β
β βββ Yes: Get cached datetime (5min TTL)
β β β
β β βββ Format: Standard ([Zeit: ...]) or HEUTE IST Mode
β β β
β β βββ Append timestamp to message end
β β
β βββ No: Skip
β
βΌ
Final Prompt sent to LLM (with timestamp suffix)
User Message Arrives β Step 0: Attachments + Checkpoint Suffix
β
βΌ
Step 0.5: ContextGuard Token Counting & Auto-Tracker Threshold Check
β
βββ Count tokens via History Text Length Γ 0.25 (+10% buffer)
βββ autoTracker.checkAndGeneratePrompt() β threshold warning if β₯75% used
βββ Inject checkpointSuffix into all return paths (unified injection)
β
βΌ
Step 0.6: Auto-Tracker Checkpoint Reply Handling + Message Analysis
β
βββ Detect "YES"/"NO" (or German "JA"/"NEIN") reply to pending checkpoint prompt β flush/save memory
βββ Analyze message for tracking triggers (decisions, completions, errors)
βββ Buffer actions for later flush at next threshold checkpoint
β
βΌ
Step 0.7: Project Keyword Detection β NEW (v1.9.8+)
β
βββ Read project_registry.json from disk
βββ Extract candidate words from user message (filter stop-words)
βββ Match against registered projects with fuzzy normalization (hyphenβunderscore)
βββ If matched β Inject "β οΈ REGISTERED PROJECT DETECTED" confirm-first banner
β (Does NOT change CWD this turn; one-shot switch only after an explicit YES/JA reply in a later message)
β
βΌ
Step 1: Directory Path Detection β Unchanged
β
βββ Detect Windows/Unix/Relative paths in message
βββ If found β Inject "WORKING DIRECTORY DETECTED" confirmation prompt
β
βΌ
Step 2: Document RAG (if enabled)
β
βββ Yes: Load embedding model
β β
β βββ Process files β chunks
β β
β βββ Semantic retrieval
β β
β βββ Filter by affinity threshold
β β
β βββ Inject relevant chunks into prompt
β
βββ No: Pass through unchanged
browser_open_page(url)
β
βΌ
BrowserSessionManager.getBrowser()
β
βββ Browser exists & connected? ββββΊ Reuse
β
βββ No: Launch new Puppeteer instance
β
βββ Retry with exponential backoff (max 2)
β
βββ Reset inactivity timer (5 min)
β
βΌ
Navigate to URL
β
βββ Wait for selector (optional)
β
βββ Take screenshot (optional)
β
βββ Extract text content
Session Activity Detected
β
βΌ
auto_summarize_context(sessionEvents, configChanges)
β
βββ Analyze tool usage patterns (>3 uses = frequent pattern)
βββ Track configuration changes
βββ Identify important decisions
βββ Generate session summary
βΌ
ContextStorageManager.addEntry(entry)
β
βββ Load existing entries from .ai_toolbox_memory.msgpack
βββ Prepend new entry to array
βββ Enforce 1000-entry limit
βββ Atomic save (temp file + rename)
βΌ
Persistent Storage (.ai_toolbox_memory.msgpack)
β
βββ get_context_memory(limit, type?) β Retrieve entries
βββ search_context(query, maxResults) β Text-based search
βββ context_summary() β Statistics & counts
βββ delete_context_entry(id) / clearContextMemory(confirm) β Management
User calls get_session_summary() or get_memory()
β
βΌ
1οΈβ£ Check Working Directory File:
{current_working_dir}/.session_context/.ai_toolbox_memory.msgpack
β
βββ Found? Decode msgpack β Return β
(Local-first hit)
β
βββ Missing/Empty/Corrupt? Continue...
β
βΌ
2οΈβ£ Check Plugin Root File:
{plugin_root}/.session_context/.ai_toolbox_memory.msgpack
β
βββ Found? Decode msgpack β Return β οΈ (Fallback hit)
β
βββ Missing/Empty/Corrupt? Continue...
β
βΌ
3οΈβ£ Check In-Memory State (RAM):
stateManager.get('session_summary_latest') or memory_* keys
β
βββ Found? Return β οΈ (Last resort)
βββ Not found? Return β error
User Message Arrives β Step 0.5: ContextGuard Token Counting
β
βββ autoTracker.checkAndGeneratePrompt(tokenCount, maxTokens):
β ββ Calculate usagePercentage = (effectiveTokens / maxTokens) * 100
β ββ Compare against threshold (default: 75%)
β ββ If >= threshold β Generate warning + FSM: IDLE β THRESHOLD_REACHED
β
βΌ
Warning injected into user message prompt:
β οΈ SESSION WARNING: You have reached {usage}% of your token limit...
User replies "YES" (German "JA" normalized) β Step 0.6: Checkpoint Reply Detection
β
βββ autoTracker.hasPendingWarning()? YES β
βββ replyMatch === 'YES'? YES β
βββ autoTracker.processUserReply('YES') β FSM: THRESHOLD_REACHED β CONFIRMED
βββ autoTracker.checkAndSaveTokenThreshold():
ββ flushActionsToMemory() β Save buffered decisions/completions/errors
ββ autoSaveSessionMemory() β Create checkpoint entry with token stats
User replies "NO" (German "NEIN" normalized) β Step 0.6: Checkpoint Reply Detection
β
βββ autoTracker.hasPendingWarning()? YES β
βββ replyMatch === 'NO'? YES β
βββ autoTracker.processUserReply('NO') β FSM: THRESHOLD_REACHED β DECLINED β IDLE (reset)
tool result β tokenStatsManager.recordToolResult()
βββ measures payload size β running mid-loop delta for this turn
βββ logs [AutoTracker] [DELTA] β¦ | chat used β N tok
where N = turnBaselineTokens (TokenCheck baseline at turn start) + midLoopEstTokens
next preprocess()/threshold evaluation
βββ compares against historyCount + running deltas (FIX #20),
so 75% / 90% triggers fire inside long multi-tool turns, not only between messages
User Message Arrives
β
βΌ
promptPreprocessor()
β
βββ Check contextGuardEnabled config
β β
β βββ Enabled?
β β
β βββ Yes: Count tokens in history
β β β
β β βββ Below 90% threshold? βββΊ Skip compression
β β β
β β βββ Above 90% threshold?
β β β
β β βΌ
β β compressHistory(messages)
β β β
β β βββ Identify messages to compress (all except last 10)
β β βββ Send to summary model
β β β βββ Use contextGuardSummaryModel or current chat model
β β β
β β βββ Generate summary with preserved file paths/names
β β β
β β βββ Calculate tokens saved
β β β
β β βββ Inject visual indicator:
β β β
β β βββ π§ Emoji header
β β βββ Messages compressed count
β β βββ Tokens before β after (e.g., "~85k β ~42k")
β β βββ Percentage saved (e.g., "Saved ~43,000 tokens (~51%)")
β β βββ Timestamp
β β βββ Visual separator lines
β β
β βββ No: Skip ContextGuard processing
β
βΌ
Final Prompt sent to LLM (with or without compression indicator)
π§ **ContextGuard Compression Active**
βββββββββββββββββββββββββββββββββββββββββββββββ
β’ Compressed 15 message(s) into summary
β’ Tokens before: ~85k β after: ~42k
β’ **Saved ~43,000 tokens (~51%)**
β’ Timestamp: 19:15:32
βββββββββββββββββββββββββββββββββββββββββββββββ
### CONTEXT SUMMARY (from 15 messages)
[Summary content here...]
User Message Arrives (ContextGuard Enabled)
β
βΌ
promptPreprocessor() β Native History API Iteration
β
βββ history.getLength() β Get message count
βββ For each message i from 0 to length-1:
β βββ msg = history.at(i) β Retrieve message by index
β βββ msg.getText() β Extract text content via getter method
β βββ msg.getToolCallRequests() β Serialize tool calls if present
β βββ msg.getToolCallResults() β Serialize tool results if present
β
βΌ
contextGuard.countTokens(messages, imageCount, modelId, systemPrompt, historyTextLength)
β
βββ PRIMARY METHOD: History Text Length Γ 0.25 ratio (v1.8.8+) β effective ~0.275 with +10% buffer
β βββ If historyTextLength provided from native API iteration:
β β β
β β βββ primaryTokenCount = Math.ceil(historyTextLength * 0.25)
β β βββ Add image tokens if applicable (+500 per image)
β β βββ Return totalTokens β β
Matches LM Studio sidebar exactly
β β
β βββ Verified at ~130K tokens for 544,578 chars β <0.5% deviation from sidebar
β
βββ VERIFIED at ~130K tokens for 544,578 chars β <0.3% deviation from sidebar (improved from <0.5%)
βββ FALLBACK: SDK-native countTokens() Γ calibration (if history unavailable)
β βββ Format messages into prompt string
β βββ Call model.countTokens(promptString) via LM Studio SDK
β βββ Apply TOKEN_SCALING_FACTOR = 65 for overhead compensation β β οΈ Legacy fallback
β
βΌ
Threshold Check: totalTokens >= tokenLimit * 0.9?
β
βββ Yes: compressHistory(messages) β Uses History Text Length Γ 0.24 for compressedPreview too
βββ No: Skip compression
index.ts
βββ toolsProvider.ts
β βββ config.ts
β βββ stateManager.ts
β βββ backgroundCommands.ts
β βββ tools/*.ts (15 registered modules)
β βββ security.ts (shared)
β βββ workingDir.ts (shared)
β βββ performanceUtils.ts (shared)
βββ config.ts
βββ promptPreprocessor.ts
βββ config.ts
ConfigSchema (Zod)
βββ Tool Gating (13 booleans)
βββ Execution Tools (4 booleans)
βββ Search Settings (3 fields)
βββ Browser Settings (2 fields)
βββ Git Settings (2 fields)
βββ Document RAG (3 fields)
βββ Security Settings (4 fields)
βββ State Management (2 fields)
βββ i18n (1 field)
βββ Notifications (1 field)
βββ Temporal Awareness (2 fields: temporalAwareness, dateFormatStyle)
βββ ContextGuard (6 fields): v1.4.2
βββ contextGuardEnabled (boolean) β Master toggle
βββ contextGuardTokenLimit (number 1K-200K) β Compression threshold
βββ contextGuardSmartReading (boolean) β Keyword-based file reading
βββ contextGuardSummaryModel (string) β Dedicated summary model name
βββ contextGuardTerminalFilterEnabled (boolean) β Terminal output filtering
βββ contextGuardTerminalFilterLength (number 100-20K) β Max terminal chars
src/
βββ index.ts # Plugin entry point
βββ toolsProvider.ts # Tool registration (conditional config gating)
βββ config.ts # Zod schema + UI schematics
βββ security.ts # Path/SQL/command validators
βββ stateManager.ts # Persistent state management
βββ workingDir.ts # Working directory manager
βββ performanceUtils.ts # Caching, async search, Levenshtein
βββ promptPreprocessor.ts # Document RAG + ContextGuard integration
βββ backgroundCommands.ts # Background process manager
βββ fuzzySearch.ts # Fuzzy file search implementation
βββ locales/ # i18n translation files
β βββ en.ts
β βββ de.ts
β βββ zh-CN.ts
β βββ zh-TW.ts
βββ tools/ # Tool category modules (30 source files)
β βββ fileSystemTools.ts # File system operations (23 tools β REGISTERED, incl. pattern_scan)
β βββ patternScan.ts # pattern_scan search engine (clean-room module, ReDoS-gated; tool registered in fileSystemTools.ts)
β βββ webResearchTools.ts # Web research & search (3 tools β REGISTERED; rag_web_content served by vectorRagTools.ts since v1.9.10)
β βββ browserAutomationTools.ts # Browser automation (5 tools β REGISTERED)
β βββ gitGithubTools.ts # Git local ops + GitHub API (15 tools β REGISTERED)
β βββ databaseTools.ts # Database queries (1 tool β REGISTERED)
β βββ documentTools.ts # Document parsing (PDF/DOCX) (1 tool β REGISTERED)
β βββ backgroundCommandTools.ts # Background process management (3 tools β REGISTERED)
β βββ executionTools.ts # Code execution JS/Python/Terminal (5 tools β REGISTERED)
β βββ utilityTools.ts # Utility tools (~25 tools β REGISTERED under 'utility' toggle)
β βββ imageProcessingTools.ts # Image processing & OCR (4 tools β REGISTERED)
β βββ httpClientTools.ts # HTTP client operations (3 tools β REGISTERED)
β βββ vectorRagTools.ts # Vector RAG semantic search (7 tools β REGISTERED: rag_index_files, rag_index_pdf, rag_index_docx, rag_index_xlsx, rag_query_vector, rag_clear_index, rag_web_content)
β βββ textProcessingTools.ts # Text transformation (4 tools β REGISTERED)
β βββ uiGenerationTools.ts # UI component generation (3 tools β REGISTERED)
β βββ contextManagementTools.ts # Context management & tracking (12 tools β REGISTERED)
β βββ refactorCodeTools.ts # AST-based code refactoring (2 tools β REGISTERED)
β βββ dataVisualizationTools.ts # Chart generation (1 tool β REGISTERED under 'utility' toggle)
β βββ backupTools.ts # Backup & restore operations (4 tools β REGISTERED under 'utility' toggle)
β βββ cleanupBackupsTool.ts # Cleanup backups utility (1 tool β REGISTERED under 'utility' toggle)
β βββ lineOperations.ts # Line-level text operations (1 tool β REGISTERED under 'utility' toggle)
β βββ taskPlanningTools.ts # Task planning & execution tracking (3 tools β REGISTERED)
β βββ markdownPreviewTools.ts # Markdown preview generation (1 tool β REGISTERED under 'utility' toggle)
β βββ fileModTracker.ts # File modification tracker (REGISTERED)
β βββ # networkToolsRegistry.ts β REMOVED 24.08 (was an orphan file with zero imports; deletion was tracked backlog since v1.9.3 and completed in the rag_web_content fix suite)
β βββ toolPriority.ts # Cluster-aware tool priority ranking (REGISTERED)
β βββ # backupUtils.ts / executionRegistry.ts / toolProtocolWarnings.ts / utilityRegistry.ts β REMOVED 01.09.2026 (Tier-1 dead code: zero referencers; see CHANGELOG_v2.md entry ~18:30)
β βββ restoreFromBak.ts # Backup restoration utility (REGISTERED)
β βββ attachmentManager.ts # Attachment handling & management (REGISTERED)
β βββ browserActions.ts # Browser action execution & validation (REGISTERED)
β βββ findLMStudioHome.ts # LM Studio home directory detection & fallback (REGISTERED)
β βββ lmStudioApi.ts # LM Studio REST API integration layer (REGISTERED)
β βββ tokenStatsManager.ts # Token statistics tracking & management (REGISTERED)
βββ types/ # Type definitions
βββ dom-augment.d.ts # DOM type augmentations for browser automation
βββ node-notifier.d.ts # Node.js notifier type declarations
βββ types.d.ts # Core shared type definitions
tests/ # Jest test suite (25 suites)
βββ security.test.ts # Core security validation tests
βββ security.edge-cases.test.ts # Security boundary & edge case testing
βββ config.test.ts # Zod schema + UI schematics validation
βββ stateManager.test.ts # Persistence, path resolution, atomic writes
βββ fileSystemTools.test.ts # File system operation tests (23 tools, incl. pattern_scan)
βββ webResearchTools.test.ts # Multi-engine search & fetch tests
βββ browserAutomationTools.test.ts # Puppeteer session management tests
βββ gitGithubTools.test.ts # Git local ops + GitHub API tests
βββ databaseTools.test.ts # SQLite query validation tests
βββ executionTools.test.ts # JS/Python/Terminal sandboxed execution tests
βββ utilityTools.test.ts # Utility tools (backup, chart, line ops) tests
βββ backgroundCommands.test.ts # Background process management tests
βββ toolsProvider.test.ts # Declarative registry pattern integration tests
βββ performanceUtils.test.ts # Caching, async search, Levenshtein tests
βββ fuzzySearch.test.ts # Fuzzy file search similarity scoring tests
βββ workingDir.test.ts # Working directory manager path resolution tests
βββ findLMStudioHome.test.ts # LM Studio home detection & fallback tests
βββ i18n.test.ts # Translation file loading & formatting tests
βββ autoTracker.test.ts # Token threshold checkpointing & session memory tests (v1.6.6+)
βββ browserActions.test.ts # Browser action execution & validation tests
βββ fileSearch.test.ts # Recursive file search with exclusion patterns tests
βββ grep_files.test.ts # Regex/Literal matching, ReDoS protection, performance tests
βββ refactorCodeTools.test.ts # AST-based refactoring & dry-run diff tests (v1.5.30+)
βββ hubExclusionClustering.test.ts # Hub-exclusion clustering algorithm verification (83 tests) β NEW v1.9.8
βββ projectAutoDetect.test.ts # Project auto-detection & registration workflow tests β NEW v1.9.8
src/tools/recodeTool/
βββ rules/
β βββ unusedImports.ts β Tier 1: Implemented β
(extracted from refactorCodeTools.ts) β the only live rule as of 01.09.2026
β βββ # deadCodeDetection / modulePathNormalization / typeInference / asyncModernizer β REMOVED 01.09.2026 (Tier-1 dead code: never wired into any tool operation; see CHANGELOG_v2.md entry ~18:30)
βββ recodeEngine.ts β AST transformation orchestrator with dry-run diff support (LCS-based)
βββ recodeTypes.ts β Shared interfaces & schemas (RuleContext, RuleResult, RecodeRule)
export type Confidence = 'EXTRACTED' | 'INFERRED' | 'AMBIGUOUS';
interface ToolResultMetadata {
confidence: Confidence; // EXTRACTED (deterministic), INFERRED (semantic), AMBIGUOUS (uncertain)
provenance?: string; // e.g., "file:src/utils.ts L42", "rag_query_vector"
note?: string; // Additional context for confidence assessment
}
// Helper functions
function determineConfidence(operationType, success, fallbackUsed): Confidence;
function createToolResult<T>(data: T, confidence: Confidence, options?): { success: true; data: T & ToolResultMetadata };
function createErrorResult(message: string, provenance?): { success: false; error: string; data: ToolResultMetadata };
Build Dependency Graph β Calculate Degrees β Identify Hubs (80th percentile) β Create Non-Hub Subgraph β Louvain Community Detection β Majority-Vote Hub Reattachment β Cluster Density & Modularity Calculation
function buildDependencyGraph(sourceDirs: string[]): Map<string, Set<string>>;
function addEdge(adjacency, source, target): void;
function calculateDegrees(adjacency): Map<string, number>;
function identifyHubs(degrees, hubThresholdPercentile = 80): Set<string>;
function louvainCommunityDetection(adjacency): Map<string, number>;
function reattachHubsByMajorityVote(hubs, adjacency, nonHubCommunities): Record<string, number>;
function calculateClusterDensity(members, adjacency): number;
function calculateModularity(edges, nodeDegrees, clusterAssignments): number;
function performHubExclusionClustering(adjacency, hubThresholdPercentile = 80): HubExclusionResult;
function analyzeAiToolboxDependencies(): HubExclusionResult;
interface HubExclusionResult {
nodes: ModuleNode[]; // All modules with degrees (sorted by degree descending)
edges: Edge[]; // All connections in the graph
hubs: string[]; // Identified hub module IDs
nonHubs: string[]; // Non-hub modules for clustering
clusters: ClusterInfo[]; // Community clusters with density metrics
hubAssignments: Record<string, number>; // Hub β cluster ID mapping via majority-vote
hubThresholdPercentile: number; // Threshold used (default: 80th percentile)
modularity?: number; // Overall clustering quality [0-1]
}
interface ClusterInfo {
clusterId: number; // Sequential cluster identifier (0-indexed)
members: string[]; // Module IDs in this cluster
size: number; // Number of members
density?: number; // Internal edge density [0-1]
}
interface ProjectDetectionResult {
path: string; // Absolute path to detected project
isValid: boolean; // Whether this looks like a valid project (β₯0.3 confidence)
name?: string; // Detected project name from package.json or fallback
sourceDirs?: string[]; // Source directories within the project
confidence: number; // Detection confidence score [0-1]
}
// Confidence signals:
// - package.json exists: +0.4 (strongest signal)
// - src/ or lib/ directory exists: +0.3
// - .git directory exists: +0.1
// - tsconfig.json or jest.config.* exists: +0.2
function normalizeProjectName(name: string): string; // "ai-toolbox" β "ai_toolbox", "@lmstudio/ai-toolbox" β "lmstudio_ai_toolbox"
function generateNameVariants(name: string): string[]; // "aitoolbox" β ["aitoolbox", "ai-tool-box"]
async function searchWithAutoRegister(query, cwd, maxResults = 10): Promise<Array<{ name: string; path: string }>> {
let results = await enhancedSearchProjects(query, maxResults);
if (results.length === 0) {
const autoDetected = autoDetectAndRegister(cwd, query, true /* explicitConfirmation */);
if (autoDetected.registered) {
results = await enhancedSearchProjects(query, maxResults);
}
}
return results;
}
function initializeProjectDetection(cwd: string): void; // β οΈ DEPRECATED (v1.9.8+): No longer called from index.ts at startup. Registration requires explicitConfirmation=true via register_project tool. See src/index.ts comment: "NO AUTO-REGISTRATION ON STARTUP"
// Step 0.7 in promptPreprocessor.ts (v1.9.8+) β NEW
async function detectProjectKeywords(message: string): Promise<string | null> {
// 1. Read project_registry.json from disk
const registry = await readProjectRegistry();
// 2. Extract candidate words from user message
const words = extractCandidateWords(message); // Filter stop-words, lowercase
// 3. Fuzzy-match against registered projects (hyphenβunderscore normalization)
for (const word of words) {
for (const project of registry.projects) {
if (normalizeProjectName(word) === normalizeProjectName(project.name)) {
return `REGISTERED PROJECT DETECTED: ${project.name} at ${project.path}`;
}
}
}
return null; // No match β fall through to directory detection (Step 1)
}
// _syncFromSessionMemory() in registry manager β NEW (v1.9.8+)
async function _syncFromSessionMemory(): Promise<void> {
// Read .ai_toolbox_memory.msgpack from working dir and plugin root
const entries = await loadContextEntries();
for (const entry of entries) {
if ('decision' in entry.data && typeof entry.data.decision === 'string') {
const decision = entry.data.decision as string;
// Match project names from past decisions (e.g., "switched to ai-toolbox")
const match = extractProjectNameFromDecision(decision);
if (match) {
await registerProject(match.name, match.path);
}
}
}
}
export type ContextOrigin = 'ast' | 'semantic';
interface ContextNode {
id: string;
_origin: ContextOrigin; // "ast" (raw file/AST) or "semantic" (derived insight)
label?: string; // Human-readable label
source_file?: string; // Original file path (for ast origin)
data?: unknown; // Payload/data
timestamp?: number; // Optional timestamp for ordering
}
function replaceTier(oldNodes: ContextNode[], newNodes: ContextNode[]): ContextNode[] {
const oldAst = oldNodes.filter(n => n._origin === 'ast');
const oldSem = oldNodes.filter(n => n._origin === 'semantic');
const newAst = newNodes.filter(n => n._origin === 'ast');
const newSem = newNodes.filter(n => n._origin === 'semantic');
return [
...oldAst.filter(a => !newAst.some(n => n.id === a.id)), // Old AST not replaced
...newAst, // New AST
...oldSem.filter(s => !newSem.some(n => n.id === s.id)), // Old Sem not replaced
...newSem // New Sem
];
}
function createAstNode(id: string, data: unknown, sourceFile?: string): ContextNode;
function createSemanticNode(id: string, data: unknown, label?: string): ContextNode;
export type PriorityTier = 'critical' | 'high' | 'standard' | 'optional' | 'background';
const PRIORITY_TIER_VALUES: Record<PriorityTier, number> = {
critical: 1, // File system tools (23 tools β core workflow)
high: 2, // Web research, execution, git operations (30+ tools β essential workflows)
standard: 3, // Browser automation, image processing, RAG, HTTP client (25+ tools β useful but not essential)
optional: 4, // Context management tools (12 tools β specialized or low-usage)
background: 5 // Backup, cleanup, chart generation, markdown preview (8+ tools β utility/maintenance)
};
interface ClusterAwarePriority extends ToolPriority {
moduleDegree?: number; // Number of dependencies/connections
isHub?: boolean; // Whether this tool's module is identified as a hub
clusterId?: number; // Assigned cluster ID from Hub-Exclusion clustering
centralityScore?: number; // Centrality score [0-1] β higher = more architecturally important
}
function computeCentralityScores(tools: ToolPriority[], clusteringResult: HubExclusionResult): Map<string, number>;
function sortToolsByClusterAwarePriority(tools: { name: string }[], clusteringResult?: HubExclusionResult): typeof tools;
function generateClusterAwareFilterReport(tools: { name: string }[], limit: number, clusteringResult?: HubExclusionResult): string;
const CATEGORY_TO_MODULE: Record<string, string | readonly string[]> = {
fileSystem: ['fileSystemTools.ts', 'tools/fileSystemTools.ts'],
webResearch: ['webResearchTools.ts', 'tools/webResearchTools.ts'],
// ... 20 categories mapped to source files with dual-name support (bare + path-prefixed)
};
// _syncFromSessionMemory() β Called lazily when search_projects or get_project_info is invoked
async function _syncFromSessionMemory(): Promise<void> {
// Load context entries from working dir + plugin root
const entries = await loadContextEntries();
for (const entry of entries) {
if ('decision' in entry.data && typeof entry.data.decision === 'string') {
const decisionText = entry.data.decision as string;
// Extract project names from past decisions
// e.g., "switched to ai-toolbox at C:\Source Code\..."
const match = extractProjectNameFromDecision(decisionText);
if (match) {
await registerProject(match.name, match.path);
}
}
}
}
// Called from search_projects tool β ensures registry is up-to-date before querying
async function searchProjects(query: string): Promise<ProjectInfo[]> {
await _syncFromSessionMemory(); // Lazy sync β NEW
return enhancedSearchProjects(query, 10);
}
promptPreprocessor() β detectProjectKeywords(message)
β
βββ If match found β Inject confirmation prompt
β ("REGISTERED PROJECT DETECTED: ai-toolbox at C:\...")
β
βββ User replies "YES" β AI calls register_project(workingDir, confirmed=true)
β
βΌ
registryManager._syncFromSessionMemory() β Ensures future searches find it
// New cross-module relationships:
toolPriority.ts β hubExclusionClustering.js (centrality scoring integration)
contextTiers.ts β (standalone β used by ContextStorageManager for tier-provenance)
projectAutoDetect.ts β DEPRECATED: No longer called from index.ts at startup. Registration requires explicitConfirmation=true via register_project tool. See src/index.ts comment: "NO AUTO-REGISTRATION ON STARTUP"
confidenceTypes.ts β all tool modules (via createToolResult<T>() helper functions)
hubExclusionClustering.ts β analysis utility (analyzeAiToolboxDependencies() pre-populated graph)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Atomic Write Utility β
β (src/utils/atomicWrite.ts) β
β β
β ββββββββββββββββββββ ββββββββββββββββββββ β
β β atomicWrite() β β atomicWriteBinaryβ β
β β (text files) β β File() β β
β β β β (binary files) β β
β ββββββββββ¬ββββββββββ ββββββββββ¬ββββββββββ β
β β β β
β βΌ βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 1. Generate random temp filename β β
β β crypto.randomBytes(9) β 72-bit entropy β β
β β Format: {original}.{hex}.tmp β β
β ββββββββββββββββββββββ¬βββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 2. Write content to temp file β β
β β fs.promises.writeFile(tempPath, data) β β
β β (text: UTF-8 | binary: raw buffer) β β
β ββββββββββββββββββββββ¬βββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 3. Atomic rename (survives crashes) β β
β β fs.promises.rename(tempPath, original) β β
β β OS-level atomic operation β β
β ββββββββββββββββββββββ¬βββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 4. Cleanup on failure (if rename fails) β β
β β fs.promises.unlink(tempPath) β β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Original file remains intact even if process crashes
between steps 2 and 3 β temp file orphaned but safe.
import * as crypto from 'crypto';
import { writeFile, rename, unlink } from 'fs/promises';
export async function atomicWrite(filePath: string, content: string): Promise<void> {
// Generate randomized temp filename (72-bit entropy)
const tempFile = `${filePath}.${crypto.randomBytes(9).toString('hex')}.tmp`;
try {
await writeFile(tempFile, content, 'utf-8'); // Step 1: Write to temp
await rename(tempFile, filePath); // Step 2: Atomic rename
} catch (err) {
await unlink(tempFile).catch(() => {}); // Step 3: Cleanup on failure
throw err; // Re-throw original error
}
}
export async function atomicWriteBinaryFile(
filePath: string,
buffer: Buffer
): Promise<void> {
const tempFile = `${filePath}.${crypto.randomBytes(9).toString('hex')}.tmp`;
try {
await writeFile(tempFile, buffer); // Raw buffer write (no encoding)
await rename(tempFile, filePath); // Atomic rename
} catch (err) {
await unlink(tempFile).catch(() => {}); // Cleanup on failure
throw err; // Re-throw original error
}
}
// BEFORE atomic write attempt β create .bak backup
await fs.copyFile(originalPath, `${originalPath}.bak`);
try {
await atomicWrite(originalPath, newContent); // Attempt async atomic write
} catch (err) {
// Rollback: restore from .bak backup
await fs.copyFile(`${originalPath}.bak`, originalPath);
throw new Error(`Atomic write failed β restored from backup: ${err.message}`);
}
esm + cjs (dual-package compatibility)es2020 / node platform@lmstudio/sdk, puppeteer, sharp, tesseract.js, isomorphic-git, pdf-parse, mammoth, archiver, unzipper, node-notifier, pixelmatch, pngjs.d.ts files via dts: truebackgroundCommandManagersaveToFile()getMemoryFilePath()change_directorypersistenceEnabled === false, returns in-memory keys directly (test isolation). When enabled, reloads from disk before returning (handles working dir changes mid-session)._ready promise) to prevent race conditionsglobalprojectsessionripgrep (pithings/ripgrep-node 0.3.1, ESM-only WASM build of rg) is dynamically imported on first use only; a missing or broken package degrades to a typed fallback and never breaks plugin boot.--no-ignore --no-require-git --hidden (the walker scans dot-dirs when no include pattern is given), -i passed as a parameter (grep_files compiles every regex with 'i'), caller-supplied exclusion globs, and depth-budget parity via --max-depth=cap+1 (rg 15.x WASM includes one boundary level less; cap 0 emits no flag β pinned by unit test).--max-filesize: production skip records carry exact byte counts, and rg's size units/rounding differ..contenthistoryTextLength parameter is passed from promptPreprocessor.ts after native API iteration in Step 0.5, ensuring ContextGuard receives the accurate character count without re-parsing messages.detectApiServer() function initially only attempted port 1234. If LM Studio was running on a different port, or if the server wasn't ready during plugin initialization, connections failed immediately and threw errors that propagated up as [LM Studio API] β οΈ Could not connect..., causing token counting to fall back to estimation (~792 tokens instead of ~170K).fetchTokenCount() threw exceptions that cluttered production logs and disrupted tool execution pipelines..getText() method first (common in SDK v1.x), then checks for a .text property, falling back to JSON.stringify() only as an absolute last resort.RecodeConfig.ruleConfigs| lineOperations.ts |
| 1 |
| β Yes |
| Utility toggle |
| Markdown Preview | markdownPreviewTools.ts | 1 | β Yes | Utility toggle |
| Task Planning | taskPlanningTools.ts | 3 | β Yes | Enabled (default) |
| Total Registered | 131 unique tools (24 modules) |
src/tools/fs.promisesnpm run build # Compiles src/ β dist/ with sourcemaps
npm run typecheck # Validates types without emitting (tsc --noEmit)
npm run lint # ESLint static analysis
// index.ts
export function main(context: PluginContext) {
// 1. Register config schematics (UI toggles)
context.withConfigSchematics(configSchematics);
// 2. Register prompt preprocessor (Document RAG + ContextGuard)
context.withPromptPreprocessor(preprocess);
// 3. Register tools provider (all registered categories based on config)
context.withToolsProvider(toolsProvider);
// 4. Setup cleanup handlers
process.on('SIGTERM', cleanupBrowserSession);
process.on('SIGINT', cleanupBrowserSession);
}
toolsProvider() called by LM Studio SDK
β
βΌ
createToolsProvider(config, stateManager, bgCommandManager)
β
βββ StateManager(config) βββββββΊ Load state from disk
βββ BackgroundCommandManager βββΊ Initialize process tracker
βββ Declarative Registry Pattern (v1.8.2+):
β
βββ TOOL_REGISTRIES array (20 entries, closure-based)
β βββ Each entry captures dependencies at definition time
β βββ Single for...of loop iterates all entries
β βββ Config key gating + GOD MODE bypass
β
βββ registerFileSystemTools() βββΊ 23 tools (enabled by default, incl. pattern_scan)
βββ registerWebResearchTools() βββΊ 3 tools (enabled by default)
βββ registerGitTools() βββΊ 15 tools (disabled by default)
βββ registerBrowserTools() βββΊ 5 tools (disabled by default)
βββ registerDatabaseTools() βββΊ 1 tool (disabled by default)
βββ registerDocumentTools() βββΊ 1 tool (enabled by default)
βββ registerBackgroundCommandTools() ββΊ 3 tools (disabled by default)
βββ registerImageProcessingTools() ββΊ 4 tools (enabled by default)
βββ registerHttpClientTools() βββΊ 3 tools (disabled by default)
βββ registerRagTools() βββΊ 7 tools (enabled by default: rag_index_files/pdf/docx/xlsx, rag_query_vector, rag_clear_index, rag_web_content β since v1.9.2/v1.9.10)
βββ registerUiGenerationTools() βββΊ 3 tools (disabled by default)
βββ registerContextManagementTools() ββΊ 12 tools (enabled by default)
βββ registerTextProcessingTools() βββΊ 4 tools (enabled by default)
βββ registerRefactorCodeTools() βββΊ 2 tools (enabled by default)
βββ registerExecutionTools() βββΊ 5 tools (mixed defaults)
β
βΌ
Return Tool[] to SDK βββΊ **131 unique tools** registered across 24 modules (configurable per user)
Session Activity Occurs
β
βΌ
auto_summarize_context() called
β
βββ Analyze tool usage patterns
βββ Detect configuration changes
βββ Identify important decisions
βββ Generate summary
βΌ
ContextStorageManager.addEntry(entry)
β
βββ Load existing entries from .ai_toolbox_memory.msgpack (Working Dir β Plugin Root fallback)
βββ Apply default scope: 'global' (v1.9.1+) unless explicitly specified
βββ Set TTL for session-scoped entries: 24h (v1.9.1+)
βββ Increment frequency counter on existing entries with matching ID
βββ Append new entry to beginning of array
βββ Limit to 1000 entries (prevent unbounded growth)
βββ Save atomically (temp file + rename)
βΌ
Persistent Storage (.ai_toolbox_memory.msgpack)
β
βββ get_context_memory(limit, type?) β Retrieve recent entries
β βββ Prune expired session entries (TTL check: 24h threshold, v1.9.1+)
β βββ Apply heuristic scoring: Recency(70%) + Frequency(30%) sort
β βββ Return top N scored entries
β
βββ search_context(query, maxResults) β Text-based search
β βββ Prune expired session entries (v1.9.1+)
β βββ Filter by title/content/tags match
β βββ Apply heuristic scoring to results
β βββ Return top N scored matches
β
βββ context_summary() β Statistics & counts
βββ delete_context_entry(id) β Remove specific entry
Session Activity Occurs
β
βΌ
auto_summarize_context() called
β
βββ Analyze tool usage patterns
βββ Detect configuration changes
βββ Identify important decisions
βββ Generate summary
βΌ
ContextStorageManager.addEntry(entry)
β
βββ Load existing entries from .ai_toolbox_memory.msgpack β .session_context/.ai_toolbox_memory.msgpack
βββ Append new entry to beginning of array
βββ Limit to 1000 entries (prevent unbounded growth)
βββ Save atomically (temp file + rename)
βΌ
Persistent Storage (.ai_toolbox_memory.msgpack)
β
βββ get_context_memory() β Retrieve recent entries
βββ search_context(query) β Text-based search
βββ context_summary() β Statistics & counts
βββ delete_context_entry(id) β Remove specific entry
// Simplified registration pattern (actual implementation uses closure-based registry)
export async function toolsProvider(ctl: ToolsProviderController): Promise<Tool[]> {
const pluginConfig = ctl.getPluginConfig(configSchematics);
// Construct typed PluginConfig from .get() calls
const config: PluginConfig = { /* ... */ };
// Initialize managers (singleton pattern)
if (!stateManager) stateManager = new StateManager(config);
if (!backgroundCommandManager) backgroundCommandManager = new BackgroundCommandManager(config);
const tools: Tool[] = [];
// --- Declarative Registry Definition (v1.8.2+) ---
const TOOL_REGISTRIES: ToolRegistryEntry[] = [
{ key: 'fileSystem', register: () => registerFileSystemTools(config, stateManager) },
{ key: 'webSearch', register: () => registerWebResearchTools(config) },
// ... 18 more entries (20 total)
];
// --- Registry Loop (replaces ~80 lines of if/else blocks) ---
for (const entry of TOOL_REGISTRIES) {
if (config[entry.key] || isGodMode) {
tools.push(...entry.register());
}
}
return tools;
}
class StateManager {
private state: Map<string, StateEntry>;
private maxSize: number;
private persistenceEnabled: boolean;
private memoryFile!: string; // Resolved at runtime
set(key, value): void // In-memory + async disk write
get<T>(key): T | undefined // In-memory retrieval
delete(key): boolean // In-memory + async disk update
getAllKeys(): Promise<string[]> // Waits for initialization
clear(): void // Resets in-memory state
}
// save_session_summary writes:
await stateManager.set(`${summaryId}_data`, compressed); // Base64-encoded gzip stream < 10k chars
await stateManager.set(`${summaryId}_timestamp`, Date.now());
// get_session_summary reads with backward-compatible fallback:
const keys = await stateManager.getAllKeys(); // Waits for loadFromFile(), returns all keys
const compressedData = stateManager.get(summaryKey);
try {
const decompressed = zlib.gunzipSync(Buffer.from(compressedData, 'base64')).toString('utf-8');
sessionSummary = JSON.parse(decompressed); // New format (v1.5.15+)
} catch (parseErr) {
// Fallback for legacy uncompressed summaries (pre-v1.5.15)
if (typeof compressedData === 'string' && compressedData.startsWith('{')) {
try {
sessionSummary = JSON.parse(compressedData); // Legacy format
} catch (legacyErr) {
throw new Error(`Legacy summary parsing failed: ${String(legacyErr)}`);
}
} else {
throw parseErr; // Corrupted or unknown format
}
}
export type MemoryScope = 'global' | 'project' | 'session';
class ContextStorageManager {
private storagePath: string; // .ai_toolbox_memory.msgpack
load(): Promise<ContextEntry[]>
save(entries: ContextEntry[]): Promise<void>
addEntry(entry: ContextEntry): Promise<void> // Applies default scope + TTL
getRecentEntries(limit, type?): Promise<ContextEntry[]> // Heuristic scoring applied
searchEntries(query, maxResults): Promise<ContextEntry[]> // Heuristic scoring applied
deleteEntry(id): Promise<boolean>
clearAll(): Promise<void>
getSummary(): Promise<ContextSummary>
pruneExpiredSessionEntries(): Promise<number> // TTL pruning (v1.9.1+)
}
interface ContextEntry {
id: string;
timestamp: number;
date: string;
type: 'decision' | 'pattern' | 'configuration' | 'file_change' | 'error' | 'summary';
title: string;
content: string;
tags?: string[];
scope?: MemoryScope; // NEW (v1.9.1): Context isolation
frequency?: number; // NEW (v1.9.1): Access count for scoring
ttl_ms?: number; // NEW (v1.9.1): Expiration threshold
}
// Recency Decay: Exponential decay based on age (lambda = 1 day)
const recencyFactor = Math.exp(-ageMs / (24 * 60 * 60 * 1000));
// Frequency Saturation: Prevents infinite bias toward frequently accessed entries
const frequencyFactor = freq / (freq + 5);
// Weighted composite score
return (recencyFactor * 0.7) + (frequencyFactor * 0.3);
Input β Path Validation β Binary Detection β Command Sanitization β SQL Validation
(validatePath) (isBinaryFile) (sanitizeCommand) (validateSQLQuery)
let currentWorkingDir: string = BASE_DIR;
getWorkingDir(): string
setWorkingDir(newDir: string): boolean
resetWorkingDir(): void
resolvePath(userPath: string): string
getAllowedBases(): string[]
// former src/tools/gatewayTools.ts (REMOVED from codebase 24.08; shown for historical reference)
export async function getGatewayTools(
provider: ToolsProvider,
config: PluginConfig
): Promise<Tool[]> {
const exploreTools = tool({
name: 'explore_tools',
description: 'Discover available tools and their categories...',
parameters: { category: z.string().optional() },
implementation: async (params) => {
await provider.getAvailableTools(); // Ensure registry loaded
return { success: true, categories: [...] }; // Returns category names only
}
});
const executeGatewayTool = tool({
name: 'execute_gateway_tool',
description: 'Executes a specific tool by its name...',
parameters: {
toolName: z.string(),
arguments: z.record(z.unknown())
},
implementation: async (params) => {
return await provider.executeTool(params.toolName, params.arguments); // Delegates to registry
}
});
return [exploreTools, executeGatewayTool];
}
User Message β AI calls explore_tools(category="fileSystem")
β Returns: { success: true, categories: ["read_file", "write_file", ...] }
β AI decides to use read_file
β AI calls execute_gateway_tool(toolName="read_file", arguments={file_name: "example.txt"})
β Gateway delegates to provider.executeTool("read_file", args)
β Tool executes with full validation, security checks, error handling
User Path Input
β
βββ Empty check βββββββββββββββββΊ Reject
β
βββ UNC path check (\\\) βββββββΊ Reject
β
βββ Relative path?
β β
β βββ Yes: Resolve against basePath
β β β
β β βββ Within base? ββββΊ Allow
β β βββ Outside base? βββΊ Reject
β β
β βββ No (absolute):
β β
β βββ In allowed bases? βββΊ Allow
β βββ Outside allowed? ββββΊ Reject
Command String
β
βΌ
Layer 1: Dangerous Pattern Blocking
β
βββ Null byte injection ββββββββββΊ Reject
βββ IFS tampering ββββββββββββββββΊ Reject
βββ Dangerous patterns (rm -rf, sudo, etc.) ββΊ Reject
βββ Too many pipes (>2) ββββββββββΊ Reject
βββ Multiple semicolons (>1) ββββββΊ Reject
βββ Command substitution ($(), ``) ββΊ Reject
βββ Environment modification ββββββΊ Reject
β
βΌ
Layer 2: Tool-Category Enforcement
β
βββ classifyCommand() β Set<string>
β β
β βββ git * / api.github.com β 'gitOperations'
β βββ duckduckgo / google / bing β 'webSearch'
β βββ puppeteer / playwright / chromium β 'browserAutomation'
β βββ sqlite3 / mysql / psql β 'databaseQueries'
β βββ curl / wget / http β 'httpClient'
β βββ nohup / disown / & β 'backgroundCommands'
β β
β βΌ
β Check against config toggles
β β
β βββ Category disabled + !godMode ββΊ Reject
β βββ Category enabled or godMode βββΊ Allow
β
βΌ
Allow Execution
JavaScript Code
β
βββ require() detection ββββββββββΊ Reject
βββ eval() detection βββββββββββββΊ Reject
βββ fs/child_process access ββββββΊ Reject
βββ Function constructor βββββββββΊ Reject
βββ Dynamic import() βββββββββββββΊ Reject
βββ __proto__ access βββββββββββββΊ Reject
// Stops calculating if minimum possible score drops below threshold
function levenshteinSimilarity(a: string, b: string, minScore: number): number | null {
// Quick rejection for very different lengths
if (lenDiff / maxLen > (1 - minScore)) return null;
// Two-row optimization (saves memory vs full matrix)
// Early exit when row minimum exceeds threshold
}
// Concurrency-controlled batch processing
async function findFilesAsync(dirPath, pattern, maxDepth, concurrencyLimit = 4) {
// Process directories in batches
for (const batch of batches) {
await Promise.all(batch.map(dir => searchDir(dir, depth + 1)));
}
}
regex-mode directory scan:
phase 1 searchCandidates(ripgrepEngine.ts) β awaited BEFORE scan start (cost outside the 15 s deadline window)
exit 0 β allow-set of rg-named candidate files (relativized against targetDir; RC-D fix)
exit 1 β no matches (clean negative; set stays null β full-JS walk runs anyway, byte-for-byte fallback)
exit 2 / import failure / WASI error β { status:'fallback-required', reason } (e.g. Rust-dialect parse
errors: lookarounds/backreferences) β same full-JS fallback, all hang guards intact
phase 2 existing walkDirectory/processFile pipeline on the candidate set only: per-file stat + size gate FIRST
(exact-byte skip records), line-cap probe read for non-named files (RC-E fix: byte-identical
skipped_files contract), then unchanged processWithRegex shaping
User Message
β
βΌ
promptPreprocessor()
β
βββ Check temporalAwareness config
β β
β βββ Enabled?
β β
β βββ Yes: Get cached datetime (5min TTL)
β β β
β β βββ Format: Standard ([Zeit: ...]) or HEUTE IST Mode
β β β
β β βββ Append timestamp to message end
β β
β βββ No: Skip
β
βΌ
Final Prompt sent to LLM (with timestamp suffix)
User Message Arrives β Step 0: Attachments + Checkpoint Suffix
β
βΌ
Step 0.5: ContextGuard Token Counting & Auto-Tracker Threshold Check
β
βββ Count tokens via History Text Length Γ 0.25 (+10% buffer)
βββ autoTracker.checkAndGeneratePrompt() β threshold warning if β₯75% used
βββ Inject checkpointSuffix into all return paths (unified injection)
β
βΌ
Step 0.6: Auto-Tracker Checkpoint Reply Handling + Message Analysis
β
βββ Detect "YES"/"NO" (or German "JA"/"NEIN") reply to pending checkpoint prompt β flush/save memory
βββ Analyze message for tracking triggers (decisions, completions, errors)
βββ Buffer actions for later flush at next threshold checkpoint
β
βΌ
Step 0.7: Project Keyword Detection β NEW (v1.9.8+)
β
βββ Read project_registry.json from disk
βββ Extract candidate words from user message (filter stop-words)
βββ Match against registered projects with fuzzy normalization (hyphenβunderscore)
βββ If matched β Inject "β οΈ REGISTERED PROJECT DETECTED" confirm-first banner
β (Does NOT change CWD this turn; one-shot switch only after an explicit YES/JA reply in a later message)
β
βΌ
Step 1: Directory Path Detection β Unchanged
β
βββ Detect Windows/Unix/Relative paths in message
βββ If found β Inject "WORKING DIRECTORY DETECTED" confirmation prompt
β
βΌ
Step 2: Document RAG (if enabled)
β
βββ Yes: Load embedding model
β β
β βββ Process files β chunks
β β
β βββ Semantic retrieval
β β
β βββ Filter by affinity threshold
β β
β βββ Inject relevant chunks into prompt
β
βββ No: Pass through unchanged
browser_open_page(url)
β
βΌ
BrowserSessionManager.getBrowser()
β
βββ Browser exists & connected? ββββΊ Reuse
β
βββ No: Launch new Puppeteer instance
β
βββ Retry with exponential backoff (max 2)
β
βββ Reset inactivity timer (5 min)
β
βΌ
Navigate to URL
β
βββ Wait for selector (optional)
β
βββ Take screenshot (optional)
β
βββ Extract text content
Session Activity Detected
β
βΌ
auto_summarize_context(sessionEvents, configChanges)
β
βββ Analyze tool usage patterns (>3 uses = frequent pattern)
βββ Track configuration changes
βββ Identify important decisions
βββ Generate session summary
βΌ
ContextStorageManager.addEntry(entry)
β
βββ Load existing entries from .ai_toolbox_memory.msgpack
βββ Prepend new entry to array
βββ Enforce 1000-entry limit
βββ Atomic save (temp file + rename)
βΌ
Persistent Storage (.ai_toolbox_memory.msgpack)
β
βββ get_context_memory(limit, type?) β Retrieve entries
βββ search_context(query, maxResults) β Text-based search
βββ context_summary() β Statistics & counts
βββ delete_context_entry(id) / clearContextMemory(confirm) β Management
User calls get_session_summary() or get_memory()
β
βΌ
1οΈβ£ Check Working Directory File:
{current_working_dir}/.session_context/.ai_toolbox_memory.msgpack
β
βββ Found? Decode msgpack β Return β
(Local-first hit)
β
βββ Missing/Empty/Corrupt? Continue...
β
βΌ
2οΈβ£ Check Plugin Root File:
{plugin_root}/.session_context/.ai_toolbox_memory.msgpack
β
βββ Found? Decode msgpack β Return β οΈ (Fallback hit)
β
βββ Missing/Empty/Corrupt? Continue...
β
βΌ
3οΈβ£ Check In-Memory State (RAM):
stateManager.get('session_summary_latest') or memory_* keys
β
βββ Found? Return β οΈ (Last resort)
βββ Not found? Return β error
User Message Arrives β Step 0.5: ContextGuard Token Counting
β
βββ autoTracker.checkAndGeneratePrompt(tokenCount, maxTokens):
β ββ Calculate usagePercentage = (effectiveTokens / maxTokens) * 100
β ββ Compare against threshold (default: 75%)
β ββ If >= threshold β Generate warning + FSM: IDLE β THRESHOLD_REACHED
β
βΌ
Warning injected into user message prompt:
β οΈ SESSION WARNING: You have reached {usage}% of your token limit...
User replies "YES" (German "JA" normalized) β Step 0.6: Checkpoint Reply Detection
β
βββ autoTracker.hasPendingWarning()? YES β
βββ replyMatch === 'YES'? YES β
βββ autoTracker.processUserReply('YES') β FSM: THRESHOLD_REACHED β CONFIRMED
βββ autoTracker.checkAndSaveTokenThreshold():
ββ flushActionsToMemory() β Save buffered decisions/completions/errors
ββ autoSaveSessionMemory() β Create checkpoint entry with token stats
User replies "NO" (German "NEIN" normalized) β Step 0.6: Checkpoint Reply Detection
β
βββ autoTracker.hasPendingWarning()? YES β
βββ replyMatch === 'NO'? YES β
βββ autoTracker.processUserReply('NO') β FSM: THRESHOLD_REACHED β DECLINED β IDLE (reset)
tool result β tokenStatsManager.recordToolResult()
βββ measures payload size β running mid-loop delta for this turn
βββ logs [AutoTracker] [DELTA] β¦ | chat used β N tok
where N = turnBaselineTokens (TokenCheck baseline at turn start) + midLoopEstTokens
next preprocess()/threshold evaluation
βββ compares against historyCount + running deltas (FIX #20),
so 75% / 90% triggers fire inside long multi-tool turns, not only between messages
User Message Arrives
β
βΌ
promptPreprocessor()
β
βββ Check contextGuardEnabled config
β β
β βββ Enabled?
β β
β βββ Yes: Count tokens in history
β β β
β β βββ Below 90% threshold? βββΊ Skip compression
β β β
β β βββ Above 90% threshold?
β β β
β β βΌ
β β compressHistory(messages)
β β β
β β βββ Identify messages to compress (all except last 10)
β β βββ Send to summary model
β β β βββ Use contextGuardSummaryModel or current chat model
β β β
β β βββ Generate summary with preserved file paths/names
β β β
β β βββ Calculate tokens saved
β β β
β β βββ Inject visual indicator:
β β β
β β βββ π§ Emoji header
β β βββ Messages compressed count
β β βββ Tokens before β after (e.g., "~85k β ~42k")
β β βββ Percentage saved (e.g., "Saved ~43,000 tokens (~51%)")
β β βββ Timestamp
β β βββ Visual separator lines
β β
β βββ No: Skip ContextGuard processing
β
βΌ
Final Prompt sent to LLM (with or without compression indicator)
π§ **ContextGuard Compression Active**
βββββββββββββββββββββββββββββββββββββββββββββββ
β’ Compressed 15 message(s) into summary
β’ Tokens before: ~85k β after: ~42k
β’ **Saved ~43,000 tokens (~51%)**
β’ Timestamp: 19:15:32
βββββββββββββββββββββββββββββββββββββββββββββββ
### CONTEXT SUMMARY (from 15 messages)
[Summary content here...]
User Message Arrives (ContextGuard Enabled)
β
βΌ
promptPreprocessor() β Native History API Iteration
β
βββ history.getLength() β Get message count
βββ For each message i from 0 to length-1:
β βββ msg = history.at(i) β Retrieve message by index
β βββ msg.getText() β Extract text content via getter method
β βββ msg.getToolCallRequests() β Serialize tool calls if present
β βββ msg.getToolCallResults() β Serialize tool results if present
β
βΌ
contextGuard.countTokens(messages, imageCount, modelId, systemPrompt, historyTextLength)
β
βββ PRIMARY METHOD: History Text Length Γ 0.25 ratio (v1.8.8+) β effective ~0.275 with +10% buffer
β βββ If historyTextLength provided from native API iteration:
β β β
β β βββ primaryTokenCount = Math.ceil(historyTextLength * 0.25)
β β βββ Add image tokens if applicable (+500 per image)
β β βββ Return totalTokens β β
Matches LM Studio sidebar exactly
β β
β βββ Verified at ~130K tokens for 544,578 chars β <0.5% deviation from sidebar
β
βββ VERIFIED at ~130K tokens for 544,578 chars β <0.3% deviation from sidebar (improved from <0.5%)
βββ FALLBACK: SDK-native countTokens() Γ calibration (if history unavailable)
β βββ Format messages into prompt string
β βββ Call model.countTokens(promptString) via LM Studio SDK
β βββ Apply TOKEN_SCALING_FACTOR = 65 for overhead compensation β β οΈ Legacy fallback
β
βΌ
Threshold Check: totalTokens >= tokenLimit * 0.9?
β
βββ Yes: compressHistory(messages) β Uses History Text Length Γ 0.24 for compressedPreview too
βββ No: Skip compression
index.ts
βββ toolsProvider.ts
β βββ config.ts
β βββ stateManager.ts
β βββ backgroundCommands.ts
β βββ tools/*.ts (15 registered modules)
β βββ security.ts (shared)
β βββ workingDir.ts (shared)
β βββ performanceUtils.ts (shared)
βββ config.ts
βββ promptPreprocessor.ts
βββ config.ts
ConfigSchema (Zod)
βββ Tool Gating (13 booleans)
βββ Execution Tools (4 booleans)
βββ Search Settings (3 fields)
βββ Browser Settings (2 fields)
βββ Git Settings (2 fields)
βββ Document RAG (3 fields)
βββ Security Settings (4 fields)
βββ State Management (2 fields)
βββ i18n (1 field)
βββ Notifications (1 field)
βββ Temporal Awareness (2 fields: temporalAwareness, dateFormatStyle)
βββ ContextGuard (6 fields): v1.4.2
βββ contextGuardEnabled (boolean) β Master toggle
βββ contextGuardTokenLimit (number 1K-200K) β Compression threshold
βββ contextGuardSmartReading (boolean) β Keyword-based file reading
βββ contextGuardSummaryModel (string) β Dedicated summary model name
βββ contextGuardTerminalFilterEnabled (boolean) β Terminal output filtering
βββ contextGuardTerminalFilterLength (number 100-20K) β Max terminal chars
src/
βββ index.ts # Plugin entry point
βββ toolsProvider.ts # Tool registration (conditional config gating)
βββ config.ts # Zod schema + UI schematics
βββ security.ts # Path/SQL/command validators
βββ stateManager.ts # Persistent state management
βββ workingDir.ts # Working directory manager
βββ performanceUtils.ts # Caching, async search, Levenshtein
βββ promptPreprocessor.ts # Document RAG + ContextGuard integration
βββ backgroundCommands.ts # Background process manager
βββ fuzzySearch.ts # Fuzzy file search implementation
βββ locales/ # i18n translation files
β βββ en.ts
β βββ de.ts
β βββ zh-CN.ts
β βββ zh-TW.ts
βββ tools/ # Tool category modules (30 source files)
β βββ fileSystemTools.ts # File system operations (23 tools β REGISTERED, incl. pattern_scan)
β βββ patternScan.ts # pattern_scan search engine (clean-room module, ReDoS-gated; tool registered in fileSystemTools.ts)
β βββ webResearchTools.ts # Web research & search (3 tools β REGISTERED; rag_web_content served by vectorRagTools.ts since v1.9.10)
β βββ browserAutomationTools.ts # Browser automation (5 tools β REGISTERED)
β βββ gitGithubTools.ts # Git local ops + GitHub API (15 tools β REGISTERED)
β βββ databaseTools.ts # Database queries (1 tool β REGISTERED)
β βββ documentTools.ts # Document parsing (PDF/DOCX) (1 tool β REGISTERED)
β βββ backgroundCommandTools.ts # Background process management (3 tools β REGISTERED)
β βββ executionTools.ts # Code execution JS/Python/Terminal (5 tools β REGISTERED)
β βββ utilityTools.ts # Utility tools (~25 tools β REGISTERED under 'utility' toggle)
β βββ imageProcessingTools.ts # Image processing & OCR (4 tools β REGISTERED)
β βββ httpClientTools.ts # HTTP client operations (3 tools β REGISTERED)
β βββ vectorRagTools.ts # Vector RAG semantic search (7 tools β REGISTERED: rag_index_files, rag_index_pdf, rag_index_docx, rag_index_xlsx, rag_query_vector, rag_clear_index, rag_web_content)
β βββ textProcessingTools.ts # Text transformation (4 tools β REGISTERED)
β βββ uiGenerationTools.ts # UI component generation (3 tools β REGISTERED)
β βββ contextManagementTools.ts # Context management & tracking (12 tools β REGISTERED)
β βββ refactorCodeTools.ts # AST-based code refactoring (2 tools β REGISTERED)
β βββ dataVisualizationTools.ts # Chart generation (1 tool β REGISTERED under 'utility' toggle)
β βββ backupTools.ts # Backup & restore operations (4 tools β REGISTERED under 'utility' toggle)
β βββ cleanupBackupsTool.ts # Cleanup backups utility (1 tool β REGISTERED under 'utility' toggle)
β βββ lineOperations.ts # Line-level text operations (1 tool β REGISTERED under 'utility' toggle)
β βββ taskPlanningTools.ts # Task planning & execution tracking (3 tools β REGISTERED)
β βββ markdownPreviewTools.ts # Markdown preview generation (1 tool β REGISTERED under 'utility' toggle)
β βββ fileModTracker.ts # File modification tracker (REGISTERED)
β βββ # networkToolsRegistry.ts β REMOVED 24.08 (was an orphan file with zero imports; deletion was tracked backlog since v1.9.3 and completed in the rag_web_content fix suite)
β βββ toolPriority.ts # Cluster-aware tool priority ranking (REGISTERED)
β βββ # backupUtils.ts / executionRegistry.ts / toolProtocolWarnings.ts / utilityRegistry.ts β REMOVED 01.09.2026 (Tier-1 dead code: zero referencers; see CHANGELOG_v2.md entry ~18:30)
β βββ restoreFromBak.ts # Backup restoration utility (REGISTERED)
β βββ attachmentManager.ts # Attachment handling & management (REGISTERED)
β βββ browserActions.ts # Browser action execution & validation (REGISTERED)
β βββ findLMStudioHome.ts # LM Studio home directory detection & fallback (REGISTERED)
β βββ lmStudioApi.ts # LM Studio REST API integration layer (REGISTERED)
β βββ tokenStatsManager.ts # Token statistics tracking & management (REGISTERED)
βββ types/ # Type definitions
βββ dom-augment.d.ts # DOM type augmentations for browser automation
βββ node-notifier.d.ts # Node.js notifier type declarations
βββ types.d.ts # Core shared type definitions
tests/ # Jest test suite (25 suites)
βββ security.test.ts # Core security validation tests
βββ security.edge-cases.test.ts # Security boundary & edge case testing
βββ config.test.ts # Zod schema + UI schematics validation
βββ stateManager.test.ts # Persistence, path resolution, atomic writes
βββ fileSystemTools.test.ts # File system operation tests (23 tools, incl. pattern_scan)
βββ webResearchTools.test.ts # Multi-engine search & fetch tests
βββ browserAutomationTools.test.ts # Puppeteer session management tests
βββ gitGithubTools.test.ts # Git local ops + GitHub API tests
βββ databaseTools.test.ts # SQLite query validation tests
βββ executionTools.test.ts # JS/Python/Terminal sandboxed execution tests
βββ utilityTools.test.ts # Utility tools (backup, chart, line ops) tests
βββ backgroundCommands.test.ts # Background process management tests
βββ toolsProvider.test.ts # Declarative registry pattern integration tests
βββ performanceUtils.test.ts # Caching, async search, Levenshtein tests
βββ fuzzySearch.test.ts # Fuzzy file search similarity scoring tests
βββ workingDir.test.ts # Working directory manager path resolution tests
βββ findLMStudioHome.test.ts # LM Studio home detection & fallback tests
βββ i18n.test.ts # Translation file loading & formatting tests
βββ autoTracker.test.ts # Token threshold checkpointing & session memory tests (v1.6.6+)
βββ browserActions.test.ts # Browser action execution & validation tests
βββ fileSearch.test.ts # Recursive file search with exclusion patterns tests
βββ grep_files.test.ts # Regex/Literal matching, ReDoS protection, performance tests
βββ refactorCodeTools.test.ts # AST-based refactoring & dry-run diff tests (v1.5.30+)
βββ hubExclusionClustering.test.ts # Hub-exclusion clustering algorithm verification (83 tests) β NEW v1.9.8
βββ projectAutoDetect.test.ts # Project auto-detection & registration workflow tests β NEW v1.9.8
src/tools/recodeTool/
βββ rules/
β βββ unusedImports.ts β Tier 1: Implemented β
(extracted from refactorCodeTools.ts) β the only live rule as of 01.09.2026
β βββ # deadCodeDetection / modulePathNormalization / typeInference / asyncModernizer β REMOVED 01.09.2026 (Tier-1 dead code: never wired into any tool operation; see CHANGELOG_v2.md entry ~18:30)
βββ recodeEngine.ts β AST transformation orchestrator with dry-run diff support (LCS-based)
βββ recodeTypes.ts β Shared interfaces & schemas (RuleContext, RuleResult, RecodeRule)
export type Confidence = 'EXTRACTED' | 'INFERRED' | 'AMBIGUOUS';
interface ToolResultMetadata {
confidence: Confidence; // EXTRACTED (deterministic), INFERRED (semantic), AMBIGUOUS (uncertain)
provenance?: string; // e.g., "file:src/utils.ts L42", "rag_query_vector"
note?: string; // Additional context for confidence assessment
}
// Helper functions
function determineConfidence(operationType, success, fallbackUsed): Confidence;
function createToolResult<T>(data: T, confidence: Confidence, options?): { success: true; data: T & ToolResultMetadata };
function createErrorResult(message: string, provenance?): { success: false; error: string; data: ToolResultMetadata };
Build Dependency Graph β Calculate Degrees β Identify Hubs (80th percentile) β Create Non-Hub Subgraph β Louvain Community Detection β Majority-Vote Hub Reattachment β Cluster Density & Modularity Calculation
function buildDependencyGraph(sourceDirs: string[]): Map<string, Set<string>>;
function addEdge(adjacency, source, target): void;
function calculateDegrees(adjacency): Map<string, number>;
function identifyHubs(degrees, hubThresholdPercentile = 80): Set<string>;
function louvainCommunityDetection(adjacency): Map<string, number>;
function reattachHubsByMajorityVote(hubs, adjacency, nonHubCommunities): Record<string, number>;
function calculateClusterDensity(members, adjacency): number;
function calculateModularity(edges, nodeDegrees, clusterAssignments): number;
function performHubExclusionClustering(adjacency, hubThresholdPercentile = 80): HubExclusionResult;
function analyzeAiToolboxDependencies(): HubExclusionResult;
interface HubExclusionResult {
nodes: ModuleNode[]; // All modules with degrees (sorted by degree descending)
edges: Edge[]; // All connections in the graph
hubs: string[]; // Identified hub module IDs
nonHubs: string[]; // Non-hub modules for clustering
clusters: ClusterInfo[]; // Community clusters with density metrics
hubAssignments: Record<string, number>; // Hub β cluster ID mapping via majority-vote
hubThresholdPercentile: number; // Threshold used (default: 80th percentile)
modularity?: number; // Overall clustering quality [0-1]
}
interface ClusterInfo {
clusterId: number; // Sequential cluster identifier (0-indexed)
members: string[]; // Module IDs in this cluster
size: number; // Number of members
density?: number; // Internal edge density [0-1]
}
interface ProjectDetectionResult {
path: string; // Absolute path to detected project
isValid: boolean; // Whether this looks like a valid project (β₯0.3 confidence)
name?: string; // Detected project name from package.json or fallback
sourceDirs?: string[]; // Source directories within the project
confidence: number; // Detection confidence score [0-1]
}
// Confidence signals:
// - package.json exists: +0.4 (strongest signal)
// - src/ or lib/ directory exists: +0.3
// - .git directory exists: +0.1
// - tsconfig.json or jest.config.* exists: +0.2
function normalizeProjectName(name: string): string; // "ai-toolbox" β "ai_toolbox", "@lmstudio/ai-toolbox" β "lmstudio_ai_toolbox"
function generateNameVariants(name: string): string[]; // "aitoolbox" β ["aitoolbox", "ai-tool-box"]
async function searchWithAutoRegister(query, cwd, maxResults = 10): Promise<Array<{ name: string; path: string }>> {
let results = await enhancedSearchProjects(query, maxResults);
if (results.length === 0) {
const autoDetected = autoDetectAndRegister(cwd, query, true /* explicitConfirmation */);
if (autoDetected.registered) {
results = await enhancedSearchProjects(query, maxResults);
}
}
return results;
}
function initializeProjectDetection(cwd: string): void; // β οΈ DEPRECATED (v1.9.8+): No longer called from index.ts at startup. Registration requires explicitConfirmation=true via register_project tool. See src/index.ts comment: "NO AUTO-REGISTRATION ON STARTUP"
// Step 0.7 in promptPreprocessor.ts (v1.9.8+) β NEW
async function detectProjectKeywords(message: string): Promise<string | null> {
// 1. Read project_registry.json from disk
const registry = await readProjectRegistry();
// 2. Extract candidate words from user message
const words = extractCandidateWords(message); // Filter stop-words, lowercase
// 3. Fuzzy-match against registered projects (hyphenβunderscore normalization)
for (const word of words) {
for (const project of registry.projects) {
if (normalizeProjectName(word) === normalizeProjectName(project.name)) {
return `REGISTERED PROJECT DETECTED: ${project.name} at ${project.path}`;
}
}
}
return null; // No match β fall through to directory detection (Step 1)
}
// _syncFromSessionMemory() in registry manager β NEW (v1.9.8+)
async function _syncFromSessionMemory(): Promise<void> {
// Read .ai_toolbox_memory.msgpack from working dir and plugin root
const entries = await loadContextEntries();
for (const entry of entries) {
if ('decision' in entry.data && typeof entry.data.decision === 'string') {
const decision = entry.data.decision as string;
// Match project names from past decisions (e.g., "switched to ai-toolbox")
const match = extractProjectNameFromDecision(decision);
if (match) {
await registerProject(match.name, match.path);
}
}
}
}
export type ContextOrigin = 'ast' | 'semantic';
interface ContextNode {
id: string;
_origin: ContextOrigin; // "ast" (raw file/AST) or "semantic" (derived insight)
label?: string; // Human-readable label
source_file?: string; // Original file path (for ast origin)
data?: unknown; // Payload/data
timestamp?: number; // Optional timestamp for ordering
}
function replaceTier(oldNodes: ContextNode[], newNodes: ContextNode[]): ContextNode[] {
const oldAst = oldNodes.filter(n => n._origin === 'ast');
const oldSem = oldNodes.filter(n => n._origin === 'semantic');
const newAst = newNodes.filter(n => n._origin === 'ast');
const newSem = newNodes.filter(n => n._origin === 'semantic');
return [
...oldAst.filter(a => !newAst.some(n => n.id === a.id)), // Old AST not replaced
...newAst, // New AST
...oldSem.filter(s => !newSem.some(n => n.id === s.id)), // Old Sem not replaced
...newSem // New Sem
];
}
function createAstNode(id: string, data: unknown, sourceFile?: string): ContextNode;
function createSemanticNode(id: string, data: unknown, label?: string): ContextNode;
export type PriorityTier = 'critical' | 'high' | 'standard' | 'optional' | 'background';
const PRIORITY_TIER_VALUES: Record<PriorityTier, number> = {
critical: 1, // File system tools (23 tools β core workflow)
high: 2, // Web research, execution, git operations (30+ tools β essential workflows)
standard: 3, // Browser automation, image processing, RAG, HTTP client (25+ tools β useful but not essential)
optional: 4, // Context management tools (12 tools β specialized or low-usage)
background: 5 // Backup, cleanup, chart generation, markdown preview (8+ tools β utility/maintenance)
};
interface ClusterAwarePriority extends ToolPriority {
moduleDegree?: number; // Number of dependencies/connections
isHub?: boolean; // Whether this tool's module is identified as a hub
clusterId?: number; // Assigned cluster ID from Hub-Exclusion clustering
centralityScore?: number; // Centrality score [0-1] β higher = more architecturally important
}
function computeCentralityScores(tools: ToolPriority[], clusteringResult: HubExclusionResult): Map<string, number>;
function sortToolsByClusterAwarePriority(tools: { name: string }[], clusteringResult?: HubExclusionResult): typeof tools;
function generateClusterAwareFilterReport(tools: { name: string }[], limit: number, clusteringResult?: HubExclusionResult): string;
const CATEGORY_TO_MODULE: Record<string, string | readonly string[]> = {
fileSystem: ['fileSystemTools.ts', 'tools/fileSystemTools.ts'],
webResearch: ['webResearchTools.ts', 'tools/webResearchTools.ts'],
// ... 20 categories mapped to source files with dual-name support (bare + path-prefixed)
};
// _syncFromSessionMemory() β Called lazily when search_projects or get_project_info is invoked
async function _syncFromSessionMemory(): Promise<void> {
// Load context entries from working dir + plugin root
const entries = await loadContextEntries();
for (const entry of entries) {
if ('decision' in entry.data && typeof entry.data.decision === 'string') {
const decisionText = entry.data.decision as string;
// Extract project names from past decisions
// e.g., "switched to ai-toolbox at C:\Source Code\..."
const match = extractProjectNameFromDecision(decisionText);
if (match) {
await registerProject(match.name, match.path);
}
}
}
}
// Called from search_projects tool β ensures registry is up-to-date before querying
async function searchProjects(query: string): Promise<ProjectInfo[]> {
await _syncFromSessionMemory(); // Lazy sync β NEW
return enhancedSearchProjects(query, 10);
}
promptPreprocessor() β detectProjectKeywords(message)
β
βββ If match found β Inject confirmation prompt
β ("REGISTERED PROJECT DETECTED: ai-toolbox at C:\...")
β
βββ User replies "YES" β AI calls register_project(workingDir, confirmed=true)
β
βΌ
registryManager._syncFromSessionMemory() β Ensures future searches find it
// New cross-module relationships:
toolPriority.ts β hubExclusionClustering.js (centrality scoring integration)
contextTiers.ts β (standalone β used by ContextStorageManager for tier-provenance)
projectAutoDetect.ts β DEPRECATED: No longer called from index.ts at startup. Registration requires explicitConfirmation=true via register_project tool. See src/index.ts comment: "NO AUTO-REGISTRATION ON STARTUP"
confidenceTypes.ts β all tool modules (via createToolResult<T>() helper functions)
hubExclusionClustering.ts β analysis utility (analyzeAiToolboxDependencies() pre-populated graph)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Atomic Write Utility β
β (src/utils/atomicWrite.ts) β
β β
β ββββββββββββββββββββ ββββββββββββββββββββ β
β β atomicWrite() β β atomicWriteBinaryβ β
β β (text files) β β File() β β
β β β β (binary files) β β
β ββββββββββ¬ββββββββββ ββββββββββ¬ββββββββββ β
β β β β
β βΌ βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 1. Generate random temp filename β β
β β crypto.randomBytes(9) β 72-bit entropy β β
β β Format: {original}.{hex}.tmp β β
β ββββββββββββββββββββββ¬βββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 2. Write content to temp file β β
β β fs.promises.writeFile(tempPath, data) β β
β β (text: UTF-8 | binary: raw buffer) β β
β ββββββββββββββββββββββ¬βββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 3. Atomic rename (survives crashes) β β
β β fs.promises.rename(tempPath, original) β β
β β OS-level atomic operation β β
β ββββββββββββββββββββββ¬βββββββββββββββββββββββββ β
β β β
β βΌ β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
β β 4. Cleanup on failure (if rename fails) β β
β β fs.promises.unlink(tempPath) β β
β βββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Original file remains intact even if process crashes
between steps 2 and 3 β temp file orphaned but safe.
import * as crypto from 'crypto';
import { writeFile, rename, unlink } from 'fs/promises';
export async function atomicWrite(filePath: string, content: string): Promise<void> {
// Generate randomized temp filename (72-bit entropy)
const tempFile = `${filePath}.${crypto.randomBytes(9).toString('hex')}.tmp`;
try {
await writeFile(tempFile, content, 'utf-8'); // Step 1: Write to temp
await rename(tempFile, filePath); // Step 2: Atomic rename
} catch (err) {
await unlink(tempFile).catch(() => {}); // Step 3: Cleanup on failure
throw err; // Re-throw original error
}
}
export async function atomicWriteBinaryFile(
filePath: string,
buffer: Buffer
): Promise<void> {
const tempFile = `${filePath}.${crypto.randomBytes(9).toString('hex')}.tmp`;
try {
await writeFile(tempFile, buffer); // Raw buffer write (no encoding)
await rename(tempFile, filePath); // Atomic rename
} catch (err) {
await unlink(tempFile).catch(() => {}); // Cleanup on failure
throw err; // Re-throw original error
}
}
// BEFORE atomic write attempt β create .bak backup
await fs.copyFile(originalPath, `${originalPath}.bak`);
try {
await atomicWrite(originalPath, newContent); // Attempt async atomic write
} catch (err) {
// Rollback: restore from .bak backup
await fs.copyFile(`${originalPath}.bak`, originalPath);
throw new Error(`Atomic write failed β restored from backup: ${err.message}`);
}