What is Lakemine
Lakemine finds where your Databricks spend is being wasted, ranks every finding by dollar impact, and tells you how to fix it.
It installs into your own Databricks account and stays there. It reads the usage and billing information Databricks already keeps about your workspaces, works out where money is going that does not need to, and writes the findings into result tables in a catalog you choose. A dashboard in your workspace reads those tables.
One scheduled job does all of it, by default once a day at 06:00 UTC over a trailing 30-day window. Nobody has to run anything by hand.
The analysis is rule based
Every finding is produced by an explicit rule applied to your usage data, and every savings figure is arithmetic you can check. Seventeen detection engines run on each scan, each one owning a single failure mode.
There is no model in the loop. Nothing is inferred, predicted, or scored by a system you cannot inspect. When Lakemine says a cluster is costing you $2,140 a month more than it needs to, the Engines section shows you the query that found it and the arithmetic that priced it.
What it covers
One install covers every workspace attached to your account’s metastore, not just the one you install into.
Cost and billing coverage is account wide, because Databricks keeps billing data at the account level. Some of the finer telemetry is stored per region, so workspaces outside your install region get cost-only coverage. The installer tells you which ones before you deploy, so this is never a surprise you discover afterwards.
Databricks on AWS is supported and live-validated. Azure is supported, with full install validation pending. GCP is expected to work but is not yet validated. See Requirements.
What you get out of it
The findings land in two places, and they answer different questions.
The cost dashboard, in your workspace
A Lakeview dashboard named “Databricks Cost Optimization”, deployed into the hub workspace you chose. This is where you read the results. It has two pages:
- Account Overview. Rollups across the account, cost and savings by workspace, the top opportunities ranked by dollar impact, and cost broken down by tag.
- Workspace Detail. Pick one workspace and drill into it.
Access is controlled by one group membership list that you own. Everyone in the viewers group chosen at install can open the dashboard. Nobody else can. See Access model in depth.
The Lakemine portal
app.lakemine.ai tells you whether Lakemine itself is healthy and licensed. Your installs and their last check-in, your plan and subscription, and the aggregated totals each scan reported.
The portal is not where you read your cost findings. Your findings are in the Lakeview dashboard inside your own workspace, and they never leave it. The portal exists for licensing, install health, and metering.
What leaves your environment
Your usage data and findings do not. The scan reads the Databricks system tables in place, and the results stay in your own catalog.
Lakemine receives two kinds of data. Install metadata: your workspace name, ID and URL, the results schema name and SQL warehouse ID, and your scan schedule, used to register and manage your installs. Aggregate usage figures: totals and counts such as cost, savings and findings per detection, used for metering and your portal. We never receive query text, table contents, or Databricks user identities.
Install metadata
The installer sends this when it registers an install, and again at the end of every deploy. It is what the portal’s Installs page shows.
| Field | What it is |
|---|---|
install_id | A SHA-256 hash of your hub workspace URL. Identifies this install. |
workspace_url, workspace_name, workspace_id | The hub workspace you chose. |
result_schema | The results schema, as catalog.schema. |
sql_warehouse | The ID of the SQL warehouse the scan uses. |
service_principal_name | The name of the identity the scan runs as. |
dashboard_viewers | The name of the group that can open the dashboard. |
scan_window, scan_schedule | The lookback window and schedule you chose. |
os, os_version, arch, app_version | The machine and installer version that ran the install. |
| Links | Links to the dashboard, job, results schema, SQL warehouse and first scan run in your workspace. |
What the scan job sends
Four calls go from the scan job to the Lakemine portal, and nowhere else. Every connection is started from your side.
| Call | When | What it carries |
|---|---|---|
| Subscription check | Before every scan. | The install’s subscription key. Nothing about you or your Databricks estate, not your email, not your workspace, not a number from a scan. The portal answers with whether the subscription is active, the install’s status, your plan and its expiry. If the portal cannot be reached, the last successful answer stands for 7 days, so an outage on the Lakemine side never interrupts your scans. |
| Token refresh | Before each usage report. | The metering credential from the lakemine secret scope. The portal returns a replacement, which the scan stores back in the scope. |
| Usage report | After each scan. | The aggregate figures below. |
| Failure event | When a scan stops before producing results. | install_id, run_id, engine_version, scanned_at, and one of three coarse categories: auth_failed, detection_error or write_failed. No error text and no stack trace. |
The usage report
One report per scan, used to bill you for the product and to fill in your portal. It is
schema version "3". Fields that Lakemine could not compute for a run are left empty
rather than sent as zero.
| Field | What it is |
|---|---|
schema_version, install_id, run_id, engine_version | Contract version, the install, one ID per run (a UUID), and the engine release. |
scanned_at, period_start, period_end | When the scan ran and the window it covered. |
workspaces_covered | How many workspaces the scan analyzed. |
total_cost_analyzed_usd, total_dbus | Spend and DBUs over the window. |
dbus_by_day[] | Per-day DBUs and cost. |
cluster_count, warehouse_count, job_count, pipeline_count, sql_query_count | Counts of resources and SQL statements. |
findings_count, total_estimated_savings_usd | How many findings, and their estimated savings. |
realized_savings_usd, gross_savings_usd, optimized_cost_usd, savings_pct | The savings roll-up. |
monthly_actual_run_rate_usd, monthly_optimized_run_rate_usd, monthly_savings_run_rate_usd | Monthly run rates. |
job_runs_total, job_runs_failed, avg_job_duration_s, job_duration_trend_pct | Job activity over the window. |
sql_user_count | How many distinct people ran SQL. A count only, with no names. |
findings[] | For each detection and severity: how many findings, and their monthly and window savings. |
cost_by_sku[] | Databricks SKU names with their DBUs and cost. SKU names are Databricks product names, not names of your resources. |
cost_by_source_type[] | Spend by category (job, dashboard, alert, notebook, genie_space, other) with execution counts. The individual job, dashboard or notebook is not identified. |
estate_stats | Counts such as clusters with autoscaling, Photon or auto-termination, warehouses by type, jobs on all-purpose compute, continuous pipelines, distinct query count, total query work in seconds, distinct tag keys, and tagged cost. |
detections_run | The IDs of the detections that ran, so degraded coverage is visible. |
What is not in it: no raw usage rows, no query text, no table names, no cluster or job names, no Databricks user names or other identities, no resource IDs.
If no metering credential is configured, metering is skipped. It is best effort and never fails a scan.
What the installer stores on your computer
The installer is a desktop app with no background service. While it is installed, it keeps a small amount of data on your machine:
- Your Lakemine sign-in. An encrypted refresh token in the app’s data folder, protected by the operating system keychain (Keychain on macOS, DPAPI on Windows, libsecret on Linux). It lets a relaunch skip the password. If the system has no real keychain, nothing is saved and you sign in on each launch.
- Your Databricks sign-in. The Databricks CLI caches a short-lived OAuth session in a local profile. The installer does not control how the CLI protects that cache. Databricks credentials are never sent to Lakemine.
- Cost figures from your results tables. If you use the installer’s Dashboard view, the figures it queries from your workspace are saved as plain JSON files in the app’s data folder, per account and workspace. They stay on your machine.
- A working copy of the bundle. The rendered dashboard and bundle configuration.
The scan’s metering credential is never written to your machine. It is stored only in the
lakemine secret scope in your workspace.
Signing out deletes the stored refresh token and that account’s cached cost figures. It does not remove the Databricks CLI profiles. Only uninstalling does that. Uninstalling removes the CLI profiles the installer created, the working copy of the bundle, and that workspace’s cached cost figures.
What Lakemine never has
No standing access. You grant access at install, in your own environment, under your own admin identity. Lakemine holds no credentials to your account at any point. Uninstalling revokes everything.
No source code is shipped. The analysis ships as a compiled, sourceless wheel. No detection source is synced into your environment.
No write access to your data. The scan reads the system.* schemas and writes your
data only to the results schema you chose. Its other write-type grants cover only
Lakemine’s own secret scope and install folder. The full grant list is in
Security overview.
Next
- Access model in depth covers the permission model and why it is built the way it is.
- Requirements lists what you need before you start.
- Quickstart takes you from download to first findings.