How it works

A plain-language description of what Lakemine does inside your Databricks account, who needs which permissions, and why the access model is built the way it is.

For the step-by-step install, see Quickstart. For the exact permission statements, see the Connecting Databricks section.

The shape of it

Everything lands under one hub workspace you pick at install:

• A scheduled job, lakemine-scan, which runs the compiled analysis wheel.
• A dashboard, “Databricks Cost Optimization”, with two pages.
• The compiled wheel itself, under /Workspace/Applications/lakemine/artifacts/.
• A results schema in the catalog you chose, default schema name lakemine, created by the scan on its first run.

The results schema holds seven small tables that the dashboard reads: run_summary, opportunities, cost_by_workspace, workspaces, cost_by_day, cost_by_user, and cost_by_tag. Every one of them has install_id as its first column. They are rebuilt on each run. The dashboard queries these tables, never system.*, so opening it is cheap.

Who can deploy it

Installing is a one-time administrative act. It needs three permissions, which one person often holds together.

If the person running the installer is not a metastore admin, everything else still completes and the installer prints the exact commands for a metastore admin to run. Until those are applied the scheduled scan cannot read your usage data, so this is the one step that genuinely blocks results.

Who can see the dashboard

Everyone in the viewers group chosen at install, and nobody else.

During install you pick a group. An existing one such as your BI team, your FinOps group, or your workspace admins, or a dedicated lakemine-viewers group the installer creates for the purpose. That group’s membership is the entire access control for the cost data.

• Add someone to the group and they see the dashboard.
• Remove them and they do not.
• No database permissions, no SQL, no ticket to a data platform team.

Viewers need no access of their own to the result tables, to the system usage tables, or to the catalog. The next section explains how that works.

Two identities, clearly separated

Lakemine splits who installs it from who runs it.

You deploy it. The install is done under the admin’s own identity. That is what makes it auditable, and it is why the admin permissions above are needed.

A dedicated service principal runs it. Every scheduled scan from then on runs as a machine identity created at install, named lakemine-scan, never as a person.

An earlier design ran the scan as whoever installed it. That tied a permanent, daily business process to one employee’s account. Their permissions changing, their leaving the company, or a colleague simply re-running the installer would silently break the scan or hand it different access. Splitting the two removes that dependency entirely.

The service principal owns the results and answers the dashboard’s questions

Because the scheduled job runs as the service principal:

1. It reads the Databricks system usage tables, using the access granted at install.
2. It writes and owns the result tables, so it can always read back what it wrote.
3. It publishes the dashboard under its own credentials. Every chart you open runs its queries as the service principal, not as you.

That third point is what makes the access model simple. A viewer opening the dashboard is not querying your data warehouse under their own name. The appliance’s identity answers on their behalf. So a viewer needs no permission on anything: not the result tables, not the catalog, not the system usage tables.

The only question Databricks asks about a viewer is whether they are in the group.

Why re-deploying is safe

Because the appliance’s identity is fixed and independent of people, an upgrade or re-install by a different administrator changes nothing about how the appliance runs. The same service principal keeps the same access, keeps owning the same tables, and keeps answering the dashboard’s queries. Nothing re-points at the new deployer, and no viewer loses access.

This matters more than it sounds. It means the appliance survives staff changes, delegation, and routine upgrades without anyone re-granting anything.

What this buys you

• A single control point for access. One group membership list, owned by you, with no data-platform involvement to add or remove a viewer.
• No standing dependency on an individual. No key-person risk in a daily process.
• Least privilege for people. Viewers see cost findings without being granted access to underlying usage data they do not need.
• Auditability. The install is attributable to a named admin. Everything after it is attributable to one clearly named machine identity.

When it runs

• Daily at 06:00 UTC by default. You can choose every 12 hours, weekly on Mondays, or paused for manual runs only, at install time.
• Each run scans a trailing 30-day window by default, configurable between 7 and 90 days, across every workspace the run identity can see, in one pass.
• Each run overwrites the results snapshot. The dashboard always shows the latest scan.

Daily rather than hourly keeps the compute you pay for modest. The dashboard reads the small results tables rather than system.*, so reading it costs almost nothing.

Credentials, and where they live

The install writes two values into a lakemine secret scope in your workspace: an opaque subscription key, readable only by the scan identity, and the metering credential.

Neither is a bundle variable or a job parameter. Those are stored in plaintext in the job spec, and anything an admin can edit there is not a credential.

What leaves your environment

Your usage data does not. Two outbound calls go to the Lakemine portal and nowhere else: a subscription check that carries no data about your estate, and one aggregated metering payload after each scan that contains counts and totals only.

The full field list is in What is Lakemine.

Restricted-egress and air-gapped workspaces

The appliance needs outbound access to the Lakemine portal. If your workspace blocks it:

• Metering simply skips. It is best effort and never fails a scan.
• The subscription check runs on its last successful answer for 7 days. Past that the scan stops and the job reports why, and your dashboard keeps its last snapshot.

A fully air-gapped install therefore needs an allowlist entry for the portal host. Talk to us before deploying into one.