Readit Newsoliver101 commented on Show HN: Lightdash – An open source Looker alternative github.com/lightdash/ligh... · Posted by u/oliver101
Neat. What if we're using dbt Cloud?
We're building support for dbt cloud users right now. Using dbt locally has allowed us to piggy-back on dbt's query runner and we're having to build that ourselves to support dbt cloud.
This should roll out in the next 2 weeks.
oliver101 commented on Show HN: Lightdash – An open source Looker alternative github.com/lightdash/ligh... · Posted by u/oliver101
Well done! I was looking for an open source LookML a while back and found Rakam[0]. It seems they added the dbt layer after the fact while you started with that concept.
Lightdash looks slick, good luck!
By the way, what happened with Hubble?
Thanks! With Hubble, we realised that while some companies wanted a separate tool to monitor data quality (e.g. data governance teams in financial integrations) most modern data teams want to test inside their existing code base (e.g. dbt). Also the majority of the company tends to interact with the data through their BI tool and we think that's where adding data quality makes most sense.
For example: flagging a dashboard as out of date, or showing that a report depends on data with failing tests.
There's rich metadata in the transform layer that just isn't getting pulled through to existing reporting/BI/viz tools.
We still have a lot of love for Hubble and data quality monitoring. By connecting dbt and Lightdash, we finally get some of those data monitoring features we always wanted.
Thanks for sharing Rakam, they always stood out for their choice of using dbt as their transform layer, it's really cool.
oliver101 commented on Show HN: Lightdash – An open source Looker alternative github.com/lightdash/ligh... · Posted by u/oliver101
Hey HN,
We're really excited to release the first public version of Lightdash!
Lightdash is an open source alternative to looker that lets analysts define data transformations and metrics in one place. Lightdash gives analysts a BI platform built on the open-source tools they already love (dbt).
Github: https://github.com/lightdash/lightdash
Demo: https://www.loom.com/share/f3725e98ce4840bda3f719da647f58b0
Install: https://docs.lightdash.com/get-started/setup-the-demo-projec...
We believe that the future of the modern data stack lies in having a single source of truth for all your metrics.
Tools like dbt have made it possible for analysts to manage their transformations using SQL. But existing BI tools still hide away lots of extra business logic, meaning that metrics get scattered across the company (you know those 5 different calculations of revenue XD) and data context gets lost between tools.
With Lightdash, your BI tool is fully integrated with your dbt project. This means:
- You define your metrics right beside the rest of your data transformations, in dbt.
- Developing metrics becomes lightning fast: change some SQL or a metric and immediately see your data viz update
- All your dbt metadata (column and table descriptions, lineage, freshness, test results) is kept in sync with lightdash so you don't have to try to maintain it in multiple places.
Lightdash is still in the early days and we've got lots of work to do. Today, Lightdash supports most popular databases and warehouses but is only tested with PostgreSQL and BigQuery - so, if you try it with another database, it'd be great to hear about your experience using it!
We'd love any feedback or to hear about how you're solving BI at your company today :)
Thanks!
The lightdash team
oliver101 commented on Show HN: Generate LookML views from DBT models automatically github.com/hubble-data/db... · Posted by u/oliver101
Hey all!
Wrote a simple CLI tool that converts dbt models into looker view files. Once you've built your dbt project, run dbt2looker and copy the files over to looker.
Features:
- Auto-generates a Looker view per dbt model
- Supports dbt model and column-level descriptions
- Automatically maps raw column types to looker types
- Creates dimension groups for datetime/timestamp/date types
- Currently supports: BigQuery, Snowflake, Redshift (postgres to come)
I recently ran a little shootout between Superset, Metabase, and Lightdash — all open source with hosted options. All have nontrivial weaknesses but I ended up picking Lightdash. Superset is the best of them at data visualization but I honestly found it almost useless for self-serve BI by business users if you have existing star schema. This issue on how to do joins in Superset (with stalebot making a mess XD) is everything difficult about Superset for BI in a nutshell. https://github.com/apache/superset/issues/8645
Metabase is pretty great and it's definitely the right choice for a startup looking to get low cost BI set up. It still has a very table centric view, but feels built for _BI_ rather than visualization alone.
Lightdash has significant warts (YAML, pivoting being done in the frontend, no symmetric aggregates) but the Looker inspiration is obvious and it makes it easy to present _groups of tables_ to business users ready to rock. I liked Looker before Google acquired it. My business users are comfortable with star and snowflake schemas (not that they know those words) and it was easy to drop Lightdash on top of our existing data warehouse.
(one of the maintainers of Lightdash) You touched on some of our most interesting problems here! Would be especially interested to hear about what you liked / didn't like about symmetric aggregates in Looker and how you find dev with YAML. If you have an idea of how you'd like these to look in Lightdash, the team would be really open to making that a reality.
For pivoting in the backend, this is coming! Issue here: https://github.com/lightdash/lightdash/issues/2907