Section D · Production / Cloud

Dashboards & BI

Self-serve principles, metric layers, dashboard hygiene, and the "vanity vs decision" distinction every stakeholder feels but never names.

Why BI gets undervalued (and shouldn't be)

BI is the layer where data science meets the rest of the company. A great model is invisible if its output never reaches the decision-maker. The most-effective senior DS treat BI as a serious craft — not "spinning up a dashboard" but designing artifacts that compound trust and drive behavior.

Bad BI signals:

Dashboards nobody opens (check usage logs — most BI tools have them).
Stakeholders building their own spreadsheets to reconcile your numbers.
Two dashboards reporting the same metric with different values.
"Quick questions" you keep answering manually because the dashboard doesn't quite cover it.

Dashboards that get used

A used dashboard has four properties:

One audience. Don't make a dashboard for "everyone." A board deck, a product team weekly, and a sales pipeline view are three different artifacts.
One decision it informs. Be able to answer "what would a viewer do differently because of this dashboard?" If the answer is "be more informed," it's a vanity dashboard — kill it.
Trust. Numbers match what the consumer can verify elsewhere. Definitions are visible. The dashboard explains itself.
Low maintenance cost. Built on stable upstream models, breaks rarely, doesn't require manual refresh.

Tool tradeoffs

Tool	Strengths	Weaknesses
Looker	Metric layer (LookML), explore-style self-serve, enterprise governance	Expensive, LookML lock-in, slow to iterate on
Mode	Analyst-friendly, fast iteration, SQL + Python notebooks	No real metric layer, governance is weak
Hex	Notebooks + dashboards in one, modern UX, app-like interactivity	Newer, smaller talent pool, governance still maturing
Metabase	Cheap or free, good enough for early-stage	Outgrown around Series B
Tableau	Sophisticated viz, enterprise fixture	Disconnected from warehouse modeling, expensive
Sigma / ThoughtSpot	Spreadsheet-style UX for execs, NL queries	Less analyst-friendly

At Series-B–C stage AI startup, Mode or Hex is usually the right pick — fast iteration matters more than enterprise governance. Looker becomes worth its cost around Series C when you have multiple analyst teams that need a shared metric layer.

Metric layer

The single most under-discussed BI investment. A metric layer defines metrics once, with version control, tests, and lineage — and serves them to every BI tool, app, and notebook.

Without it

"Conversion rate" is defined three different ways in three dashboards.
The sales team's CRM dashboard disagrees with the product team's BI dashboard, and reconciling takes a week.
A junior analyst ships a new dashboard with a subtly different definition, and the number propagates into a board deck.

Options

LookML (Looker-native).
dbt semantic layer (works across many tools).
Cube (open-source, headless).
MetricFlow (the engine underneath dbt's semantic layer).

The senior move

Treat metric definitions as code: version-controlled, code-reviewed, tested. The metric layer is where this lives. Selling this investment to leadership is part of the senior-DS pitch — the ROI shows up as fewer reconciliation meetings and faster onboarding for new analysts.

Dashboard hygiene

The discipline that separates production-quality BI from notebook-style ad-hoc:

Timestamp on every chart: "data through <date>". Catches stale dashboards.
Definition tooltips: hover any metric, see how it's computed.
Date filters with sensible defaults: trailing 90 days is usually right.
No more than ~7 charts per dashboard: more and viewers don't read any of them.
Comparison context on every metric: "MAU = 1.2M (+5% WoW, +12% YoY)". A number without context is noise.
Anomaly callouts: highlight metrics outside historical ranges so viewers don't have to spot them.
Owner and refresh cadence visible: who maintains this, when does data refresh?

Vanity vs decision metrics

A vanity metric is one a leader likes to see but nobody changes behavior based on. A decision metric is one that, when it moves, triggers action. Senior DS spend their time making decision metrics visible — and politely retiring vanity ones.

Examples

Vanity	Decision metric (same area)
Total signups all-time	Weekly new activations (signed up AND completed setup)
Page views	Time-to-first-key-action
"Total users"	Weekly active users by cohort
Total revenue all-time	Net revenue retention by cohort
NPS	30-day repeat-purchase rate

Anti-patterns

"Everything in one dashboard." Different audiences want different things. Split.
Charts without context. A 5% MoM lift looks impressive — until you see that the metric has 8% week-over-week noise.
Dashboards that compute on the fly with complex SQL. Slow, breaks easily. Move logic into dbt models.
Pie charts with too many slices. Use a bar chart.
Y-axes that don't start at zero (when they should). Visually exaggerates moves.
Color encoding that doesn't reuse a consistent palette. Confuses viewers across dashboards.
"Final" dashboards that nobody maintains. Document an owner. Sunset dashboards quarterly.

Interview probes

Show probe 1: "How do you decide what goes on a dashboard?"

Start with the audience and the decision they're making. A weekly product review dashboard for a single team has ~5 charts: north-star metric, leading indicators, top 2–3 experiments, and a guardrails section. If a chart doesn't change a decision, it doesn't belong. The "kill criterion" is: would removing this chart prompt any viewer to ask where it went?

Show probe 2: "Two dashboards report MAU and they disagree. How do you fix it?"

Long-term: build a metric layer so MAU is defined once. Short-term: trace both definitions, identify the divergence (is one filtering bots, the other not? Different date truncation? Different definition of 'active'?), pick the right one, kill the other, and write up the decision so it doesn't happen again. Communicate the discrepancy to stakeholders proactively — they probably noticed.

Show probe 3: "What's your process for retiring a dashboard?"

Quarterly sweep: pull view counts (BI tools log this). Anything not opened in 60 days is a candidate. Reach out to the named owner and the last viewer. If neither defends it, mark for sunset with 30 days notice. Sunset means archived, not deleted — easier to restore than rebuild if someone needed it.

Show probe 4: "Should every metric have a confidence interval on a dashboard?"

No. Most production dashboards report point estimates because the audience won't process the intervals. Use intervals on experiment dashboards and forecasting dashboards where the uncertainty drives decisions. On steady-state product dashboards, instead show comparison context: WoW, MoM, YoY — that's a more digestible form of "is this normal."

✓ Check yourself

Answer from memory before opening each one.

A dashboard has 22 charts for four audiences, and the usage log shows almost nobody opens it. What do you change?

Show answer
Split it by audience, give each dashboard one decision it informs, and cut each to about seven charts. A chart that would not make a viewer ask where it went should be removed. See dashboards that get used and dashboard hygiene.
Two dashboards both show “conversion rate,” at 17% and 19%. Name three definition differences that could cause this, and say where the lasting fix lives.

Show answer
Different numerators or denominators (all visitors vs. signed-in users), different time windows or attribution rules, and different inclusion filters such as bots or test accounts. The lasting fix is one version-controlled definition in a metric layer that every dashboard reads. See metric layer.
A 40-person startup has two analysts and a CEO who wants one board view. Which BI tool fits, and when does that answer change?

Show answer
A low-cost, quick-to-start tool such as Metabase, or Mode or Hex if the analysts want SQL and notebooks. Looker becomes worth its cost when several analyst teams need a shared governed metric layer, around Series C. See tool trade-offs.
Leadership likes a chart of total signups, all time. What do you propose in its place, and how do you make the swap acceptable?

Show answer
Weekly new activations, meaning signups that completed setup, because it can fall and so it can trigger action. Run both side by side for a quarter and show the cases where the all-time total hid a weak week. See vanity vs decision metrics.
A chart shows a 5% month-over-month lift, but the metric swings 8% week to week. What is missing, and how does the dashboard supply it?

Show answer
Context: whether 5% is outside normal variation. Show the trailing range or comparison to week-over-week and year-over-year, mark anomalies, and state the data-through date. See anti-patterns and dashboard hygiene.

Takeaway

A dashboard that gets used has one audience, one decision, visible definitions and stable upstream models.
Two dashboards that disagree on one metric are a definition problem, and the fix is a version-controlled, tested metric layer.
Pick tools by stage: Metabase early, Mode or Hex for fast analyst iteration, Looker once several analyst teams need shared governed metrics.
Context makes a number readable: a data-through date, a comparison, a definition and an owner on every chart.
Replace vanity metrics with ones that can trigger action, and retire dashboards nobody opens.
Heavy logic belongs in dbt models; the BI tool should read, not compute.

Next, the vocabulary and economics of AI products, which shape the metrics you will be asked about. → LLM & AI-SaaS Domain Context