[feat] Add HTTP Config API to PluginManager for synchronous connection config updates

[Abhi011999]

Problem

I was exploring a usecase where I am programmatically creating the .spc and related config files in a multi-account & multi-tenant scenario (in steampipe service mode).
In a high-throughput and long running scenarios the files are created rapidly and the creds are valid in a short time window (e.g. for k8s its 15mins max). With this specific usecase I am facing a race-condition where the steampipe doesn't pickup the creds, the flow somewhat looks likes this:

1. .spc file is created
2. [Steampipe's go file watcher is in its 4sec watch timeout window]
3. A reuqest comes which requires new and updated creds above but it's not currently picked up and initialized in the plugin (same tenant, same account, concurrent runs)
4. The plugin query throws 403/404 error

This introduced a race window where queries could run with stale/missing credentials.

Solution

Add an opt-in HTTP endpoint (for service mode) on PluginManager that updates one connection config synchronously and returns only after plugin propagation completes.

Before:

• write/update .spc
• wait for file watcher polling/debounce
• propagate to running plugin via async handler

After:

POST /v1/connection/config
     |
     |- validate request
     |- lock per connection (serialize same-connection concurrent updates)
     |- lock plugin manager state
     |- merge config into in-memory map
     |- handleConnectionConfigChanges() (sync)
     |    |- Diff() for added/changed
     |    |- UpdateConnectionConfigs gRPC to running plugin(s)
     |- unlock plugin manager state
     |- if new connection: CREATE SCHEMA + IMPORT FOREIGN SCHEMA
     |- unlock connection lock
     v
200 OK (config is active)

Key behavior

• Opt-in: server starts only when STEAMPIPE_CONFIG_API_PORT is set and non-zero.
• Local-only: binds to 127.0.0.1:<port>.
• Synchronous guarantee: HTTP 200 means in-memory plugin config update has been applied.
• Per-connection serialization: concurrent updates for the same connection are serialized.
• No .spc writes in this path: new-connection schema creation uses SQL import only.

Remaining scope

• Aggregator fidelity without any config reload path: for API-only connections that have no persisted config metadata, scan path can still fall back to single-connection behavior. Full zero-disk aggregator fidelity requires plugin-manager metadata RPC (or equivalent in-memory metadata channel) for connection type/children.
• Method-not-allowed response shape: 405 currently uses http.Error; can be normalized to JSON if strict response-shape consistency is required.

[Abhi011999]

These changes are majorly to eliminate all the disk I/O, effectively eliminating the race conditions happening due to file watcher debounces.
There are some downstream change that are also effectively needed in steampipe-postgres-ftw.
This will be the main issue to track those changes as well (and the related PRs).

[Abhi011999]

@graza-io @pskrbasu do let me know if this makes sense, i have a PR for this, it doesn't include docs yet

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.