AI analytics#1339
Open
aadesh18 wants to merge 106 commits into
Open
Conversation
- Added new internal API endpoint for documentation tools, allowing actions such as listing available docs, searching, and fetching specific documentation by ID. - Updated environment configuration to support optional internal secret for enhanced security. - Refactored existing search functionality to utilize the new docs tools API instead of the previous MCP server. - Improved error handling and response parsing for documentation-related requests. - Expanded documentation to clarify the relationship between the new tools and existing API functionalities. This update streamlines the documentation access process and enhances the overall developer experience.
- Introduced error capturing for failed HTTP requests in the docs tools API, improving debugging capabilities. - Updated the API response for unsupported methods to include an 'Allow' header, clarifying the expected request type. These changes enhance the robustness of the documentation tools integration and improve developer experience.
- Updated the key name in the capabilities section of the API documentation to follow a consistent naming convention, improving clarity and maintainability.
The .gitmodules was updated in d22593d to point at apps/backend/src/private/implementation, but the gitlink entry (mode 160000) was never added to the tree. This caused `git clone --recurse-submodules` to silently skip the private submodule. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…on for docs tools - Added `STACK_DOCS_INTERNAL_BASE_URL` to backend `.env` and `.env.development` files for AI tool bundle configuration. - Removed references to `STACK_INTERNAL_DOCS_TOOLS_SECRET` from backend and docs environment files and validation logic from the docs tools API route. - Introduced a new `.env` file for the docs app with essential configuration variables.
… into llm-mcp-flow
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Collaborator
Author
|
@greptile review |
nams1570
approved these changes
May 14, 2026
…minal callbacks - Added a new test file for `handleStreamMode` that verifies logging behavior for various terminal callback scenarios. - Introduced a guard in `handleStreamMode` to ensure that logging occurs at most once per request lifecycle, addressing the issue of double-logging when both `onError` and `onAbort` are triggered. - Tests include cases for single callbacks and multiple callbacks firing in rapid succession, ensuring correct logging behavior under all conditions.
…rift test The server schema added `qa_entries.requestId` (4th column) and `add_manual_qa.requestId` (6th arg) on May 11 but the auto-generated client bindings were last regenerated on May 5 and never picked up the new field. Because BSATN is positional, every column after position 3 in `my_visible_qa_entries` would deserialize one slot off (question bytes read as answer, etc.) once the new schema is published — corrupting the Knowledge Base view in the internal-tool. Add `requestId` to `my_visible_qa_entries_table.ts`, `types.ts:QaEntries`, and `add_manual_qa_reducer.ts` in the same positions the codegen would produce, plus a structural test that parses both the server schema and the generated bindings and asserts field/arg orders match. The test fails on the prior state with the exact missing-`requestId` diff and passes after the patch, catching any future drift in CI. Co-authored-by: Cursor <cursoragent@cursor.com>
Collaborator
|
@greptile-ai please review Addressed the remaining 2 issues from your latest summary in 1a8e38e:
Also added |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
4 issues found across 100 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/backend/src/app/api/latest/integrations/ai-proxy/[[...path]]/route.ts">
<violation number="1" location="apps/backend/src/app/api/latest/integrations/ai-proxy/[[...path]]/route.ts:55">
P2: The `passthrough()` fallback will produce an empty response if `observeAndLog` throws after consuming the response body. Once `response.arrayBuffer()` is called inside `observeAndLog`, the original `response.body` stream is exhausted, so the catch-block's `new Response(response.body, ...)` sends nothing to the client.</violation>
</file>
Partial review: This PR has more than 50 files, so cubic reviewed the highest-priority files first. During the trial, paid plans get a higher file limit.
You can try an ultrareview to bypass the file limit, comment @cubic-dev-ai ultrareview. Learn more.
Fix all with cubic | Re-trigger cubic
| responseHeaders, | ||
| }); | ||
| } catch (e) { | ||
| captureError("ai-proxy-log-pipeline", e); |
There was a problem hiding this comment.
P2: The passthrough() fallback will produce an empty response if observeAndLog throws after consuming the response body. Once response.arrayBuffer() is called inside observeAndLog, the original response.body stream is exhausted, so the catch-block's new Response(response.body, ...) sends nothing to the client.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/backend/src/app/api/latest/integrations/ai-proxy/[[...path]]/route.ts, line 55:
<comment>The `passthrough()` fallback will produce an empty response if `observeAndLog` throws after consuming the response body. Once `response.arrayBuffer()` is called inside `observeAndLog`, the original `response.body` stream is exhausted, so the catch-block's `new Response(response.body, ...)` sends nothing to the client.</comment>
<file context>
@@ -1,97 +1,60 @@
+ responseHeaders,
});
+ } catch (e) {
+ captureError("ai-proxy-log-pipeline", e);
+ return passthrough();
}
</file context>
- Introduced unit tests for `observeAndLog` in `ai-proxy-handlers.test.ts` to ensure correct behavior when logging functions throw errors. - Added tests for `callSql` in `spacetimedb-client.test.ts` to verify 401 enrollment retry logic and error handling. - Created tests for `buildProxyLogRow` in `ai-proxy-logger.test.ts` to validate tool-name extraction from parsed logs. - Implemented validation tests for tool names in `index.test.ts` to ensure consistency with defined tool names. These additions enhance test coverage and reliability of the AI-related functionalities.
mantrakp04
approved these changes
May 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR ships AI analytics on top of the existing internal MCP Review tool: a new Unified AI Endpoint Analytics tab over the AI proxy/query path, plus the supporting backend telemetry (
ai-proxy-handlers,ai-query-handlers, SpacetimeDB log sync, OpenRouter usage attribution, MCP-call / Q&A reducers). It also reworks several MCP Review components and tightens the in-product AI dashboard builder (vibe-coding chat + sandbox host) with tests around the new handlers and SpacetimeDB client.Base: dev → Head: ai-analytics· 104 files changed · +7,874 / -1,312What this PR actually adds vs. what was already there
The MCP Review tool itself (Call Logs / Knowledge Base / Analytics tabs) shipped in #1321 ("LLM MCP Flow") — that's the
devbaseline shown in the before-shots. This PR layers on top of that:dev(from #1321)MCP Review ToolshellAnalytics.tsx— MCP-only QA-score / flag-types dashboardUsage.tsx+UsageDetail.tsx— full Unified AI Endpoint Analytics surface (new file)CallLogList.tsx,CallLogDetail.tsx,ConversationReplay.tsx,KnowledgeBase.tsx,AddManualQa.tsxmcp_call_logSpacetimeDB table (public: true)mcp_call_logreshaped (sharding, indices, optional fields) + brand-newai_query_logtable + reducers (log_mcp_call_reducer,update_mcp_qa_review_reducer)ai-proxy-handlers.ts,ai-query-handlers.ts,ai-proxy-logger.ts,mcp-call-logger.ts(+ tests), QA pipeline (qa-reviewer,reviewer-auth,verified-qa), models/prompts/toolsdashboards/[id]/page-client.tsx,vibe-coding/*,assistant-ui/thread.tsx(Grok model swap from #1476 + sandbox host fixes)Screenshots are captured against a live local dev env (
NEXT_PUBLIC_STACK_PORT_PREFIX=82) with SpacetimeDB bindings published per branch — dev'smcp_call_log-only schema for the before-pass, then this PR's schema (mcp_call_log+ai_query_log) for the after-pass — so the table-renders below are real, not stubs.MCP Review home (was: "Call Logs")
Tab rename + new empty-state messaging. Same data on both sides (zero logged MCP calls in this seed), but the dev shell is just a filter + table, whereas this PR adds the KPI strip, calls-over-time chart, QA score distribution, top flag types, response-time card, and tool-usage card on top — all visible even with zero calls.
Unified AI Endpoint Analytics — populated
The headline new surface —
Usage.tsx+UsageDetail.tsxdid not exist ondev. The before-pass shows dev's old "Analytics" tab, which was scoped to MCP QA review stats. The after-pass shows this PR's tab populated with 11 live AI calls fired through the dashboard's vibe-coding chat during this session.What's wired on the after side:
command-center-ask-ai,create-dashboard,docs-ask-ai, email-assistant family,email-wysiwyg-editor,rewrite-template-source,run-query,wysiwyg-edit), and a model chip (x-ai/grok-build-0.1).ai_query_logvia SpacetimeDB subscriptions.update-dashboard,patch-dashboard,sql-query), latency histogram.CALLS (11)with sortable columns: time, system prompt, model, mode, in/out tokens, cache read/write, $ cost, duration, status.Knowledge Base — populated
Q&A management surface. Backed by
mcp_call_log.qa_*and the newupdate_mcp_qa_review_reducer. Below: seeded with 2 published entries + 1 draft via "+ Add Q&A". The structural diff is the schema rework (sharding, indexes, optional fields), not so much new chrome here.Each row carries status (Draft / Published), source (
manual), timestamp, author, and per-row Publish / Unpublish / Edit / Delete actions. The "+ Add Q&A" modal:Dashboard view — vibe-coding chat + sandbox host
apps/dashboard/.../dashboards/[dashboardId]/page-client.tsx, thevibe-coding/chat pieces, and the dashboard sandbox host all change here. The chat now drives a live IDE-style preview while a layout is being assembled, with chat history + Generate / Stop controls.dashboard.tsxdev baseline (empty state on dev looks similar — the meaningful diff is the sandbox host wiring + the Grok model swap from #1476)
Notes for reviewers
error. That's a model-side issue (Grok build-0.1 default in dev-forwarding mode rejects this PR's tool-call schema), not a logging bug — the proxy still attributes, times, and records every call, which is exactly the property the new dashboard is meant to expose. With a working tool-call response you'd see non-zero Output Tokens + Total Cost.origin/devwithpnpm --filter @stackframe/internal-tool spacetime:publish:localso dev'smcp_call_logschema is live (otherwise the page hangs on "Connecting to SpacetimeDB…"); then back toai-analytics, re-publish (this DESTROYS the dev database —--delete-data=on-conflict), and re-fire calls. The dev-side data shown is purely the chrome / empty state, since dev's schema is what gets wiped each cycle.Test plan
pnpm typecheckcleanpnpm lintcleanai-proxy-handlers,ai-query-handlers,spacetimedb-client,spacetimedb-bindings-sync,tools/index,ai-proxy-logger)