[{"data":1,"prerenderedAt":669},["ShallowReactive",2],{"categories-init":3,"stack-gcp-elt-pipeline":4},true,{"stack_id":5,"slug":6,"name":7,"tagline":8,"long_description":9,"key_features":10,"use_cases":17,"pros":23,"cons":28,"cover_image_url":32,"scores":33,"options":42,"additions":220,"option_groups":337,"multi_select_option_types":338,"tools_by_category":339,"related_stacks":558,"faqs":635,"pricing":651,"system_requirements":32,"experience_level":624,"project_type":600,"stack_type_slug":565,"stack_type_icon_url":566,"published_date":151,"last_updated_date":32,"seo_meta":665},109,"gcp-elt-pipeline","GCP ELT Pipeline","Fivetran to BigQuery, dbt transforms, Dagster orchestrates, Metabase visualizes on GCP.","The GCP ELT Pipeline uses Google Cloud's ecosystem for a **managed, scalable analytics stack**. Fivetran handles data ingestion with 300+ managed connectors that keep data continuously synchronized into BigQuery. dbt transforms raw BigQuery data into clean models with SQL, tests, and documentation. Dagster orchestrates the pipeline as typed assets with built-in lineage visualization. Metabase provides the BI dashboard layer on top of the dbt-modeled BigQuery tables.\n\nBigQuery's serverless architecture eliminates cluster management: it scales automatically and charges per query, making it cost-effective for variable analytical workloads. Fivetran's fully managed connectors mean zero ingestion maintenance once set up. Dagster's asset-based model tracks **data lineage from source to dashboard**, making debugging and impact analysis straightforward.\n\nThis is a **premium, low-maintenance data stack** for organizations that prioritize reliability and scalability over total cost control.",[11,12,13,14,15,16],"Fivetran fully managed connectors with automatic schema migration to BigQuery","BigQuery serverless SQL data warehouse with no cluster provisioning","dbt SQL transformations with testing and column-level documentation","Dagster asset-based orchestration with visual lineage and data catalog","Metabase BI dashboards connected to dbt-modeled BigQuery views","GCP IAM and VPC for secure data access across the pipeline",[18,19,20,21,22],"GCP-centric organizations building a production analytics pipeline with minimal maintenance","Teams that want fully managed ingestion to eliminate connector maintenance overhead","Data teams that prioritize lineage visibility and pipeline reliability over cost","Organizations with variable query patterns where BigQuery's per-query pricing is efficient","Analytics programs that need audit trails and data catalog capabilities via Dagster",[24,25,26,27],"Fivetran fully managed connectors require almost zero ingestion maintenance","BigQuery serverless eliminates cluster sizing and management decisions","Dagster asset lineage provides clear visibility into data dependencies","Metabase OSS keeps visualization costs low on top of a premium pipeline",[29,30,31],"Fivetran is one of the most expensive ingestion tools per connector-month","BigQuery per-query costs can become significant for exploratory heavy users","Four distinct tools add integration surface and learning overhead",null,{"popularity":34,"learning_curve":36,"flexibility":37,"performance":38,"portability":40},{"score":35,"reasoning":32},4,{"score":35,"reasoning":32},{"score":35,"reasoning":32},{"score":39,"reasoning":32},5,{"score":41,"reasoning":32},2,{"database":43,"orm":47,"authentication":51,"analytics":55,"coding_agent":59,"llm":63,"language":67,"frontend_framework":71,"cms":75,"hosting":79,"reverse_proxy":83,"self_hosted_paas":87,"orchestrator":91},{"tools":44,"descriptions":45,"aliases":46,"see_all":32},[],{},{},{"tools":48,"descriptions":49,"aliases":50,"see_all":32},[],{},{},{"tools":52,"descriptions":53,"aliases":54,"see_all":32},[],{},{},{"tools":56,"descriptions":57,"aliases":58,"see_all":32},[],{},{},{"tools":60,"descriptions":61,"aliases":62,"see_all":32},[],{},{},{"tools":64,"descriptions":65,"aliases":66,"see_all":32},[],{},{},{"tools":68,"descriptions":69,"aliases":70,"see_all":32},[],{},{},{"tools":72,"descriptions":73,"aliases":74,"see_all":32},[],{},{},{"tools":76,"descriptions":77,"aliases":78,"see_all":32},[],{},{},{"tools":80,"descriptions":81,"aliases":82,"see_all":32},[],{},{},{"tools":84,"descriptions":85,"aliases":86,"see_all":32},[],{},{},{"tools":88,"descriptions":89,"aliases":90,"see_all":32},[],{},{},{"tools":92,"descriptions":215,"aliases":219,"see_all":32},[93,152,186],{"tool_id":94,"name":95,"slug":96,"tooltip_description":97,"logo_url":98,"logo_bg":99,"pricing_model":100,"learning_curve_score":35,"popularity_score":41,"hosting_assignment_type":104,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":106,"subcategory":110,"categories":114,"subcategories":117,"flexibility_score":39,"performance_score":35,"portability_score":35,"is_featured":119,"tags":120,"score_reasonings":144,"published_date":150,"last_updated_date":151},76,"Dagster","dagster","Open-source data orchestration platform that provides a unified interface for building, testing, and monitoring data assets with a software-defined approach.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fdagster.svg","white",{"slug":101,"display_name":102,"description":103},"freemium","Freemium","A free tier is available; additional features, usage limits, or managed hosting require a paid plan.","self_hostable","open",{"category_id":107,"name":108,"slug":109},13,"Data Engineering & ETL","data-engineering-etl",{"subcategory_id":111,"name":112,"slug":113},28,"Orchestration","orchestration",[115],{"category_id":107,"name":108,"slug":109,"is_primary":3,"display_order":116},0,[118],{"subcategory_id":111,"name":112,"slug":113,"category_id":107,"is_primary":3,"display_order":116},false,[121,126,131,136,140],{"tag_id":122,"name":123,"slug":124,"tag_type":125},1,"Python","python","technology",{"tag_id":127,"name":128,"slug":129,"tag_type":130},11,"Open Source","open-source","feature",{"tag_id":132,"name":133,"slug":134,"tag_type":135},27,"Data Engineering","data-engineering","use_case",{"tag_id":137,"name":138,"slug":139,"tag_type":135},33,"Workflow Automation","workflow-automation",{"tag_id":141,"name":142,"slug":143,"tag_type":135},37,"Data Pipelines","data-pipelines",{"learning_curve":145,"performance":146,"portability":147,"flexibility":148,"popularity":149},"Assets, resources, IO managers, and sensors are powerful but require significant dedicated study.","Asset materialization is efficient; orchestrator overhead is minimal.","Open source and Python-based; asset concepts transfer to Prefect and other modern orchestrators.","Assets, sensors, IO managers, and resources compose freely; fully open source.","Growing but remains behind Airflow in mindshare; popular in data-forward engineering teams.","2026-05-29","2026-09-27",{"tool_id":153,"name":154,"slug":155,"tooltip_description":156,"logo_url":157,"logo_bg":158,"pricing_model":159,"learning_curve_score":35,"popularity_score":162,"hosting_assignment_type":163,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":164,"subcategory":165,"categories":166,"subcategories":168,"flexibility_score":162,"performance_score":35,"portability_score":35,"is_featured":119,"tags":170,"score_reasonings":180,"published_date":150,"last_updated_date":151},9,"Apache Airflow","apache-airflow","Apache Airflow is an open-source workflow orchestration platform that lets you define, schedule, and monitor data pipelines as Python code using Directed Acyclic Graphs (DAGs). It is the de facto standard for orchestrating data engineering workflows at scale.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fapache-airflow.svg","dark",{"slug":160,"display_name":128,"description":161},"open_source","Source code is publicly available and free to use, modify, and distribute. No paid plans from the project itself.",3,"deployable",{"category_id":107,"name":108,"slug":109},{"subcategory_id":111,"name":112,"slug":113},[167],{"category_id":107,"name":108,"slug":109,"is_primary":3,"display_order":116},[169],{"subcategory_id":111,"name":112,"slug":113,"category_id":107,"is_primary":3,"display_order":116},[171,172,173,177,178,179],{"tag_id":122,"name":123,"slug":124,"tag_type":125},{"tag_id":127,"name":128,"slug":129,"tag_type":130},{"tag_id":174,"name":175,"slug":176,"tag_type":130},12,"Self-hostable","self-hostable",{"tag_id":132,"name":133,"slug":134,"tag_type":135},{"tag_id":137,"name":138,"slug":139,"tag_type":135},{"tag_id":141,"name":142,"slug":143,"tag_type":135},{"learning_curve":181,"flexibility":182,"performance":183,"portability":184,"popularity":185},"DAG concepts, operators, and executor configurations take weeks to fully master.","DAG-based orchestration is powerful but the scheduler model constrains architecture choices.","Scheduler overhead is minimal; task throughput scales with the executor configuration.","Open source; DAG patterns transfer to Prefect and Dagster with moderate adjustment.","De facto standard for data workflow orchestration; used across data engineering teams globally.",{"tool_id":187,"name":188,"slug":189,"tooltip_description":190,"logo_url":191,"logo_bg":99,"pricing_model":192,"learning_curve_score":41,"popularity_score":162,"hosting_assignment_type":104,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":193,"subcategory":194,"categories":195,"subcategories":197,"flexibility_score":35,"performance_score":162,"portability_score":35,"is_featured":119,"tags":199,"score_reasonings":209,"published_date":150,"last_updated_date":151},132,"Prefect","prefect","Python-native workflow orchestration framework for building resilient data pipelines. Lighter and more developer-friendly than Airflow, with a managed cloud option and a fully free self-hosted edition.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fprefect.svg",{"slug":101,"display_name":102,"description":103},{"category_id":107,"name":108,"slug":109},{"subcategory_id":111,"name":112,"slug":113},[196],{"category_id":107,"name":108,"slug":109,"is_primary":3,"display_order":116},[198],{"subcategory_id":111,"name":112,"slug":113,"category_id":107,"is_primary":3,"display_order":116},[200,203,204],{"tag_id":107,"name":201,"slug":202,"tag_type":130},"Free Tier","free-tier",{"tag_id":174,"name":175,"slug":176,"tag_type":130},{"tag_id":205,"name":206,"slug":207,"tag_type":208},40,"Web","web","platform",{"learning_curve":210,"flexibility":211,"performance":212,"popularity":213,"portability":214},"Decorate existing Python functions with @flow and @task — no new DSL to learn; local testing works identically to production execution.","Supports any Python logic with bring-your-own-compute via work pools — runs flows on Kubernetes, cloud functions, or local processes without vendor lock-in.","Orchestration overhead is minimal; end-to-end flow latency is dominated by the tasks themselves, not the Prefect scheduler or server.","Growing adoption among Python data teams but significantly smaller than Airflow; roughly 17k GitHub stars as of 2026 with frequent release cadence.","Apache 2.0 licensed engine; Python flow patterns transfer to Dagster with moderate effort; all underlying task logic is plain Python with no proprietary abstractions.",{"dagster":216,"apache-airflow":217,"prefect":218},"The default: asset-based orchestration where each BigQuery table and dbt model is a tracked asset with lineage, backed by first-party dagster-gcp and dagster-dbt integrations.","Swap in Airflow for the most battle-tested option on GCP: Google's own managed offering (Cloud Composer) runs it natively, and its BigQuery and dbt operators are mature.","Swap in Prefect for lighter-weight orchestration: the prefect-gcp collection provides BigQuery tasks, and dynamic flows suit pipelines whose steps vary by branch or config.",{},{"ci_cd":221,"containerization":299},{"tools":222,"descriptions":291,"aliases":294,"preface":295,"see_all":296},[223,264],{"tool_id":224,"name":225,"slug":226,"tooltip_description":227,"logo_url":228,"logo_bg":99,"pricing_model":229,"learning_curve_score":162,"popularity_score":39,"hosting_assignment_type":32,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":230,"subcategory":233,"categories":237,"subcategories":239,"flexibility_score":39,"performance_score":35,"portability_score":35,"is_featured":119,"tags":241,"score_reasonings":258,"published_date":150,"last_updated_date":151},164,"GitHub Actions","github-actions","GitHub's integrated CI\u002FCD platform that automates build, test, and deployment workflows using YAML-based configurations. Runs on GitHub-hosted or self-hosted runners with a rich marketplace of pre-built integrations.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fgithub-actions.svg",{"slug":101,"display_name":102,"description":103},{"category_id":174,"name":231,"slug":232},"DevOps & CI\u002FCD","devops-cicd",{"subcategory_id":234,"name":235,"slug":236},34,"CI\u002FCD Pipelines","cicd-pipelines",[238],{"category_id":174,"name":231,"slug":232,"is_primary":3,"display_order":116},[240],{"subcategory_id":234,"name":235,"slug":236,"category_id":174,"is_primary":3,"display_order":116},[242,243,244,245,249,253],{"tag_id":107,"name":201,"slug":202,"tag_type":130},{"tag_id":174,"name":175,"slug":176,"tag_type":130},{"tag_id":205,"name":206,"slug":207,"tag_type":208},{"tag_id":246,"name":247,"slug":248,"tag_type":135},36,"CI\u002FCD","ci-cd",{"tag_id":250,"name":251,"slug":252,"tag_type":130},24,"Docker Compatible","docker-compatible",{"tag_id":254,"name":255,"slug":256,"tag_type":257},45,"Declarative","declarative","paradigm",{"flexibility":259,"learning_curve":260,"performance":261,"popularity":262,"portability":263},"Self-hosted runner support across any OS or cloud, custom labels, matrix builds, Kubernetes scaling via Actions Runner Controller, and a marketplace of thousands of community-built actions give teams virtually unlimited configuration options for any workflow or environment.","Basic single-job workflows are accessible to any developer comfortable with YAML, and GitHub's documentation lowers the barrier further. However, advanced patterns — composite actions, reusable workflows, OIDC-based cloud auth, and conditional matrix strategies — involve non-obvious syntax and a complex permission model that requires meaningful time to master.","The platform handles massive scale reliably, but GitHub-hosted runner performance can vary between runs, making consistent benchmarking difficult. Teams running on self-hosted or larger hosted runners achieve stable, high throughput; the managed offering trades predictable latency for zero infrastructure overhead.","GitHub Actions is the dominant CI\u002FCD platform with tens of millions of repositories using it, thousands of marketplace actions, and widespread enterprise adoption. It is cited as the most-used CI\u002FCD solution in multiple developer surveys.","Self-hosted runners can run on any cloud or on-premises infrastructure, and the YAML workflow model is readable and auditable. The main constraint is tight coupling to GitHub events and APIs — moving workflows to another CI\u002FCD platform requires meaningful rewriting rather than a simple lift-and-shift.",{"tool_id":265,"name":266,"slug":267,"tooltip_description":268,"logo_url":269,"logo_bg":99,"pricing_model":270,"learning_curve_score":162,"popularity_score":35,"hosting_assignment_type":32,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":271,"subcategory":272,"categories":273,"subcategories":275,"flexibility_score":39,"performance_score":162,"portability_score":162,"is_featured":119,"tags":277,"score_reasonings":285,"published_date":150,"last_updated_date":151},165,"GitLab CI\u002FCD","gitlab-cicd","GitLab's built-in CI\u002FCD system configured through .gitlab-ci.yml files stored in your repository. Supports both GitLab-hosted and self-hosted runners for flexible pipeline execution across diverse environments.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fgitlab.svg",{"slug":101,"display_name":102,"description":103},{"category_id":174,"name":231,"slug":232},{"subcategory_id":234,"name":235,"slug":236},[274],{"category_id":174,"name":231,"slug":232,"is_primary":3,"display_order":116},[276],{"subcategory_id":234,"name":235,"slug":236,"category_id":174,"is_primary":3,"display_order":116},[278,279,280,281,282,283,284],{"tag_id":107,"name":201,"slug":202,"tag_type":130},{"tag_id":174,"name":175,"slug":176,"tag_type":130},{"tag_id":205,"name":206,"slug":207,"tag_type":208},{"tag_id":246,"name":247,"slug":248,"tag_type":135},{"tag_id":127,"name":128,"slug":129,"tag_type":130},{"tag_id":250,"name":251,"slug":252,"tag_type":130},{"tag_id":254,"name":255,"slug":256,"tag_type":257},{"learning_curve":286,"flexibility":287,"performance":288,"popularity":289,"portability":290},"The core .gitlab-ci.yml syntax is approachable for developers with YAML and Linux fundamentals, and GitLab's documentation is thorough. However, the platform's breadth — runner executors, pipeline inheritance, CI\u002FCD Catalog, merge trains, and security scanning configuration — creates a long tail of advanced concepts that teams discover gradually over months.","Multiple runner executor types (Docker, Kubernetes, Shell, machine), unlimited self-hosted runner capacity, parent-child pipelines, reusable CI\u002FCD Components, and the ability to self-host the entire platform give teams complete control over their pipeline environment and infrastructure.","Pipeline performance depends heavily on runner configuration and caching strategy. Shared GitLab-hosted runners can experience queue delays during peak periods, and Docker+machine executors add startup overhead. Teams running optimized self-hosted Kubernetes runners with effective caching achieve significantly faster pipelines, but the default shared experience is average for the CI\u002FCD category.","GitLab CI\u002FCD is used by over 100,000 organizations and is particularly strong among enterprises and security-conscious DevOps teams. It consistently ranks among the top CI\u002FCD platforms in developer surveys, though GitHub Actions has overtaken it in raw adoption volume — especially among open-source and smaller team segments.","GitLab CE is fully open source and self-hostable on any infrastructure, which is a strong portability advantage. However, .gitlab-ci.yml pipelines are tightly coupled to GitLab's API and runner ecosystem — migrating pipeline definitions to another CI\u002FCD system (GitHub Actions, CircleCI, etc.) requires substantial rewriting rather than a simple port.",{"github-actions":292,"gitlab-cicd":293},"Runs dbt test and validates Dagster job definitions on every push, catching a broken model or asset before it reaches the BigQuery tables Metabase reads from.","The same dbt-and-Dagster validation step, for teams running this pipeline's code from a GitLab repo instead.",{},"Add CI\u002FCD when you want a dedicated pipeline for running tests, linting, or multi-stage builds before a deploy goes out. Many hosting platforms already redeploy automatically on every push on their own — a CI\u002FCD tool adds the most value on top of that by gating the deploy on a passing test suite, and matters even more when the hosting choice does not auto-deploy at all, such as a self-hosted server.",{"kind":297,"slug":236,"name":235,"href":298},"subcategory","\u002Ftools\u002Fcategories\u002Fdevops-cicd\u002Fcicd-pipelines",{"tools":300,"descriptions":331,"aliases":333,"preface":334,"see_all":335},[301],{"tool_id":302,"name":303,"slug":304,"tooltip_description":305,"logo_url":306,"logo_bg":158,"pricing_model":307,"learning_curve_score":35,"popularity_score":39,"hosting_assignment_type":32,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":308,"subcategory":309,"categories":312,"subcategories":314,"flexibility_score":39,"performance_score":35,"portability_score":39,"is_featured":3,"tags":316,"score_reasonings":325,"published_date":150,"last_updated_date":151},72,"Docker","docker","Container platform for packaging applications and their dependencies into portable images that run the same on a laptop, in CI, and in production, with Docker Desktop, Compose, and Docker Hub.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fdocker.svg",{"slug":101,"display_name":102,"description":103},{"category_id":174,"name":231,"slug":232},{"subcategory_id":137,"name":310,"slug":311},"Containerization","containerization",[313],{"category_id":174,"name":231,"slug":232,"is_primary":3,"display_order":116},[315],{"subcategory_id":137,"name":310,"slug":311,"category_id":174,"is_primary":3,"display_order":116},[317,318,319,320,321],{"tag_id":127,"name":128,"slug":129,"tag_type":130},{"tag_id":174,"name":175,"slug":176,"tag_type":130},{"tag_id":250,"name":251,"slug":252,"tag_type":130},{"tag_id":246,"name":247,"slug":248,"tag_type":135},{"tag_id":322,"name":323,"slug":324,"tag_type":208},43,"Cross-platform","cross-platform",{"learning_curve":326,"performance":327,"portability":328,"flexibility":329,"popularity":330},"Container images, networking, volumes, and multi-stage builds all need deliberate learning.","Container overhead is minimal; near-native performance for most workloads.","Open standard; containers built with Docker run on any container-compatible platform.","Any runtime, any architecture; multi-stage builds and Compose profiles support complex systems.","The standard for containerization; present in virtually every modern software project.",{"docker":332},"Package the dbt project and any custom Dagster job code identically across a laptop, CI, and wherever the scheduled run happens.",{},"Add containerization when you want the app packaged the same way across local development, staging, and production, or need to deploy somewhere that isn't a managed serverless platform.",{"kind":297,"slug":311,"name":310,"href":336},"\u002Ftools\u002Fcategories\u002Fdevops-cicd\u002Fcontainerization",{},[],{"Programming Languages":340,"Databases":376,"Hosting & Cloud":419,"Data Engineering & ETL":455,"BI & Analytics":523},[341],{"tool_id":35,"name":123,"slug":124,"tooltip_description":342,"logo_url":343,"logo_bg":158,"pricing_model":344,"learning_curve_score":41,"popularity_score":39,"hosting_assignment_type":32,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":345,"subcategory":32,"categories":348,"subcategories":350,"flexibility_score":39,"performance_score":162,"portability_score":39,"is_featured":3,"tags":351,"score_reasonings":370,"published_date":150,"last_updated_date":151},"Python is a high-level, interpreted, dynamically typed programming language emphasising readability and simplicity. It dominates data science, machine learning, and general-purpose scripting.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fpython.svg",{"slug":160,"display_name":128,"description":161},{"category_id":162,"name":346,"slug":347},"Programming Languages","programming-languages",[349],{"category_id":162,"name":346,"slug":347,"is_primary":3,"display_order":116},[],[352,353,354,358,362,366],{"tag_id":122,"name":123,"slug":124,"tag_type":125},{"tag_id":127,"name":128,"slug":129,"tag_type":130},{"tag_id":355,"name":356,"slug":357,"tag_type":135},25,"Machine Learning","machine-learning",{"tag_id":359,"name":360,"slug":361,"tag_type":135},39,"Data Science","data-science",{"tag_id":363,"name":364,"slug":365,"tag_type":257},48,"Functional","functional",{"tag_id":367,"name":368,"slug":369,"tag_type":257},49,"Object-oriented","object-oriented",{"learning_curve":371,"flexibility":372,"performance":373,"popularity":374,"portability":375},"Clean, readable syntax with vast learning resources; beginner-friendly from day one.","No constraints; equally suited to scripting, data science, web servers, and systems programming.","Interpreted and GIL-limited; efficient for I\u002FO-bound work but slow for CPU-intensive tasks.","The most widely used programming language globally; dominant in data science, AI, and automation.","Universal language; skills transfer across every domain and environment.",[377],{"tool_id":378,"name":379,"slug":380,"tooltip_description":381,"logo_url":382,"logo_bg":158,"pricing_model":383,"learning_curve_score":162,"popularity_score":162,"hosting_assignment_type":163,"hosting_provider_restriction":105,"hosting_target_restriction":387,"hosting_compatible_tool_ids":388,"parent_tool_id":32,"category":390,"subcategory":393,"categories":397,"subcategories":399,"flexibility_score":162,"performance_score":35,"portability_score":41,"is_featured":119,"tags":401,"score_reasonings":413,"published_date":150,"last_updated_date":151},55,"BigQuery","bigquery","BigQuery is Google Cloud's fully managed, serverless data warehouse built for large-scale analytics. It uses a columnar storage format and separates compute from storage, charging per TB of data queried with no infrastructure to manage.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fbigquery.png",{"slug":384,"display_name":385,"description":386},"usage_based","Usage-Based","Pricing scales with consumption: API calls, data volume, compute time, or similar metered units.","allowlist",[389],51,{"category_id":35,"name":391,"slug":392},"Databases","databases",{"subcategory_id":394,"name":395,"slug":396},26,"OLAP Databases","olap-databases",[398],{"category_id":35,"name":391,"slug":392,"is_primary":3,"display_order":116},[400],{"subcategory_id":394,"name":395,"slug":396,"category_id":35,"is_primary":3,"display_order":116},[402,405,409,412],{"tag_id":35,"name":403,"slug":404,"tag_type":125},"SQL","sql",{"tag_id":406,"name":407,"slug":408,"tag_type":130},14,"Serverless","serverless",{"tag_id":394,"name":410,"slug":411,"tag_type":135},"Data Visualization","data-visualization",{"tag_id":132,"name":133,"slug":134,"tag_type":135},{"learning_curve":414,"performance":415,"portability":416,"flexibility":417,"popularity":418},"SQL interface is familiar; partitioning, clustering, and cost control take experience.","Distributed columnar storage processes petabyte-scale queries in seconds.","Google-specific SQL dialect and billing model; migrating to another warehouse is expensive.","SQL-first with limited output options; constrained to BigQuery's data model and job patterns.","Standard for analytics in Google Cloud environments; respected in data engineering.",[420],{"tool_id":389,"name":421,"slug":422,"tooltip_description":423,"logo_url":424,"logo_bg":158,"pricing_model":425,"learning_curve_score":162,"popularity_score":162,"hosting_assignment_type":32,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":426,"subcategory":429,"categories":432,"subcategories":434,"flexibility_score":39,"performance_score":39,"portability_score":162,"is_featured":119,"tags":436,"score_reasonings":449,"published_date":150,"last_updated_date":151},"Google Cloud Platform","gcp","Google's cloud platform with strong data, analytics, AI, and Kubernetes capabilities. The third-largest cloud provider, after AWS and Azure.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fgcp.svg",{"slug":384,"display_name":385,"description":386},{"category_id":39,"name":427,"slug":428},"Hosting & Cloud","hosting-cloud",{"subcategory_id":205,"name":430,"slug":431},"Cloud Providers","cloud-providers",[433],{"category_id":39,"name":427,"slug":428,"is_primary":3,"display_order":116},[435],{"subcategory_id":205,"name":430,"slug":431,"category_id":39,"is_primary":3,"display_order":116},[437,438,439,443,447,448],{"tag_id":107,"name":201,"slug":202,"tag_type":130},{"tag_id":406,"name":407,"slug":408,"tag_type":130},{"tag_id":440,"name":441,"slug":442,"tag_type":130},20,"Auto-scaling","auto-scaling",{"tag_id":444,"name":445,"slug":446,"tag_type":130},21,"Multi-region","multi-region",{"tag_id":250,"name":251,"slug":252,"tag_type":130},{"tag_id":355,"name":356,"slug":357,"tag_type":135},{"learning_curve":450,"flexibility":451,"performance":452,"portability":453,"popularity":454},"Slightly more approachable than AWS; BigQuery and Cloud Run have excellent documentation.","Deep service catalog; Cloud Run, GKE, and BigQuery combine freely for any architecture.","BigQuery's columnar engine handles petabyte queries in seconds; Cloud Run scales fast.","Similar to AWS; proprietary managed services create meaningful switching costs.","Strong in data and AI workloads; second or third cloud by market share in most segments.",[456,482,508],{"tool_id":457,"name":458,"slug":459,"tooltip_description":460,"logo_url":461,"logo_bg":158,"pricing_model":462,"learning_curve_score":41,"popularity_score":41,"hosting_assignment_type":463,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":464,"subcategory":465,"categories":469,"subcategories":471,"flexibility_score":162,"performance_score":35,"portability_score":41,"is_featured":119,"tags":473,"score_reasonings":476,"published_date":150,"last_updated_date":151},75,"Fivetran","fivetran","Automated data integration platform that syncs data from SaaS applications, databases, and APIs into your data warehouse with zero configuration.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Ffivetran.png",{"slug":384,"display_name":385,"description":386},"managed_only",{"category_id":107,"name":108,"slug":109},{"subcategory_id":466,"name":467,"slug":468},29,"Extract & Load","extract-and-load",[470],{"category_id":107,"name":108,"slug":109,"is_primary":3,"display_order":116},[472],{"subcategory_id":466,"name":467,"slug":468,"category_id":107,"is_primary":3,"display_order":116},[474,475],{"tag_id":132,"name":133,"slug":134,"tag_type":135},{"tag_id":141,"name":142,"slug":143,"tag_type":135},{"learning_curve":477,"performance":478,"portability":479,"flexibility":480,"popularity":481},"Managed UI; data connectors configure in minutes with minimal technical knowledge.","Managed sync pipelines with efficient incremental loading; fast for typical data volumes.","Managed SaaS pipelines are Fivetran-specific; no self-hosting; migration requires rebuilding connectors.","Wide connector catalog but fixed architecture; custom connectors require the Fivetran SDK.","Respected in data engineering but expensive; niche compared to open-source alternatives.",{"tool_id":483,"name":484,"slug":484,"tooltip_description":485,"logo_url":486,"logo_bg":158,"pricing_model":487,"learning_curve_score":162,"popularity_score":162,"hosting_assignment_type":32,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":488,"subcategory":489,"categories":493,"subcategories":495,"flexibility_score":35,"performance_score":162,"portability_score":35,"is_featured":119,"tags":497,"score_reasonings":502,"published_date":150,"last_updated_date":151},74,"dbt","Transformation tool that enables data teams to transform data in their warehouse using SQL and software engineering best practices like version control, testing, and modularity.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fdbt.png",{"slug":101,"display_name":102,"description":103},{"category_id":107,"name":108,"slug":109},{"subcategory_id":490,"name":491,"slug":492},30,"Transformation","transformation",[494],{"category_id":107,"name":108,"slug":109,"is_primary":3,"display_order":116},[496],{"subcategory_id":490,"name":491,"slug":492,"category_id":107,"is_primary":3,"display_order":116},[498,499,500,501],{"tag_id":35,"name":403,"slug":404,"tag_type":125},{"tag_id":127,"name":128,"slug":129,"tag_type":130},{"tag_id":132,"name":133,"slug":134,"tag_type":135},{"tag_id":141,"name":142,"slug":143,"tag_type":135},{"learning_curve":503,"performance":504,"portability":505,"flexibility":506,"popularity":507},"SQL-first approach is familiar, but project structure, refs, macros, and tests take time.","Transformation speed depends on the underlying warehouse; dbt itself adds minimal overhead.","SQL-based transformations are relatively portable; moving to SQLMesh is feasible.","Macros, custom tests, and the package ecosystem make SQL transformations highly composable.","De facto standard for data transformation in the modern data stack.",{"tool_id":94,"name":95,"slug":96,"tooltip_description":97,"logo_url":98,"logo_bg":99,"pricing_model":509,"learning_curve_score":35,"popularity_score":41,"hosting_assignment_type":104,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":510,"subcategory":511,"categories":512,"subcategories":514,"flexibility_score":39,"performance_score":35,"portability_score":35,"is_featured":119,"tags":516,"score_reasonings":522,"published_date":150,"last_updated_date":151},{"slug":101,"display_name":102,"description":103},{"category_id":107,"name":108,"slug":109},{"subcategory_id":111,"name":112,"slug":113},[513],{"category_id":107,"name":108,"slug":109,"is_primary":3,"display_order":116},[515],{"subcategory_id":111,"name":112,"slug":113,"category_id":107,"is_primary":3,"display_order":116},[517,518,519,520,521],{"tag_id":122,"name":123,"slug":124,"tag_type":125},{"tag_id":127,"name":128,"slug":129,"tag_type":130},{"tag_id":132,"name":133,"slug":134,"tag_type":135},{"tag_id":137,"name":138,"slug":139,"tag_type":135},{"tag_id":141,"name":142,"slug":143,"tag_type":135},{"learning_curve":145,"performance":146,"portability":147,"flexibility":148,"popularity":149},[524],{"tool_id":525,"name":526,"slug":527,"tooltip_description":528,"logo_url":529,"logo_bg":158,"pricing_model":530,"learning_curve_score":122,"popularity_score":41,"hosting_assignment_type":104,"hosting_provider_restriction":105,"hosting_target_restriction":105,"hosting_compatible_tool_ids":32,"parent_tool_id":32,"category":531,"subcategory":535,"categories":538,"subcategories":540,"flexibility_score":162,"performance_score":162,"portability_score":162,"is_featured":119,"tags":542,"score_reasonings":552,"published_date":150,"last_updated_date":151},89,"Metabase","metabase","Metabase is an open-source BI tool that lets anyone in your company ask questions and explore data through an intuitive no-SQL interface, with optional embedded analytics for products.","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fmetabase.svg",{"slug":101,"display_name":102,"description":103},{"category_id":532,"name":533,"slug":534},16,"BI & Analytics","bi-analytics",{"subcategory_id":141,"name":536,"slug":537},"BI Tools","bi-tools",[539],{"category_id":532,"name":533,"slug":534,"is_primary":3,"display_order":116},[541],{"subcategory_id":141,"name":536,"slug":537,"category_id":532,"is_primary":3,"display_order":116},[543,544,545,546,547,548],{"tag_id":35,"name":403,"slug":404,"tag_type":125},{"tag_id":127,"name":128,"slug":129,"tag_type":130},{"tag_id":174,"name":175,"slug":176,"tag_type":130},{"tag_id":107,"name":201,"slug":202,"tag_type":130},{"tag_id":394,"name":410,"slug":411,"tag_type":135},{"tag_id":549,"name":550,"slug":551,"tag_type":135},38,"Dashboards","dashboards",{"learning_curve":553,"portability":554,"flexibility":555,"performance":556,"popularity":557},"Business users can run queries without SQL; self-service BI from day one.","Open source; SQL-based questions transfer; dashboard migration requires manual recreation.","Question builder and dashboards are flexible within Metabase's model; SQL mode unlocks more.","Adequate for business BI; complex queries on large datasets benefit from result caching.","Popular for internal BI dashboards; competes in a crowded field with established players.",[559,580,595,619],{"stack_id":153,"slug":560,"name":561,"tagline":562,"experience_level":563,"project_type":564,"stack_type_slug":565,"stack_type_icon_url":566,"score_popularity":162,"score_learning_curve":39,"catalog_display_order":32,"published_date":32,"last_updated_date":32,"core_tool_previews":567},"mlops-pipeline","MLOps Pipeline","End-to-end ML pipelines from training to production monitoring.","advanced","ml_project","project","https:\u002F\u002Fassets.tekyous.dev\u002Ficons\u002Fstack-types\u002Fproject.svg",[568,572,573,578,579],{"tool_id":406,"slug":569,"name":570,"logo_url":571,"logo_bg":158},"fastapi","FastAPI","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Ffastapi.svg",{"tool_id":35,"slug":124,"name":123,"logo_url":343,"logo_bg":158},{"tool_id":574,"slug":575,"name":576,"logo_url":577,"logo_bg":158},57,"snowflake","Snowflake","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fsnowflake.svg",{"tool_id":153,"slug":155,"name":154,"logo_url":157,"logo_bg":158},{"tool_id":483,"slug":484,"name":484,"logo_url":486,"logo_bg":158},{"stack_id":581,"slug":582,"name":583,"tagline":584,"experience_level":563,"project_type":585,"stack_type_slug":565,"stack_type_icon_url":566,"score_popularity":35,"score_learning_curve":35,"catalog_display_order":32,"published_date":32,"last_updated_date":32,"core_tool_previews":586},111,"databricks-lakehouse-pipeline","Databricks Lakehouse Pipeline","Databricks unified lakehouse for large-scale data engineering, ML, and SQL analytics.","data_pipeline",[587,588,593,594],{"tool_id":35,"slug":124,"name":123,"logo_url":343,"logo_bg":158},{"tool_id":589,"slug":590,"name":591,"logo_url":592,"logo_bg":158},58,"databricks","Databricks","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fdatabricks.svg",{"tool_id":153,"slug":155,"name":154,"logo_url":157,"logo_bg":158},{"tool_id":483,"slug":484,"name":484,"logo_url":486,"logo_bg":158},{"stack_id":596,"slug":597,"name":598,"tagline":599,"experience_level":563,"project_type":600,"stack_type_slug":565,"stack_type_icon_url":566,"score_popularity":162,"score_learning_curve":39,"catalog_display_order":32,"published_date":32,"last_updated_date":32,"core_tool_previews":601},110,"streaming-analytics-pipeline","Streaming Analytics Pipeline","Real-time streaming analytics with Kafka, dbt, ClickHouse, and Grafana dashboards.","dashboard",[602,603,608,613,614],{"tool_id":35,"slug":124,"name":123,"logo_url":343,"logo_bg":158},{"tool_id":604,"slug":605,"name":606,"logo_url":607,"logo_bg":158},135,"clickhouse","ClickHouse","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fclickhouse.svg",{"tool_id":609,"slug":610,"name":611,"logo_url":612,"logo_bg":99},133,"apache-kafka","Apache Kafka","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fapache-kafka.svg",{"tool_id":483,"slug":484,"name":484,"logo_url":486,"logo_bg":158},{"tool_id":615,"slug":616,"name":617,"logo_url":618,"logo_bg":158},91,"grafana","Grafana","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fgrafana.svg",{"stack_id":620,"slug":621,"name":622,"tagline":623,"experience_level":624,"project_type":585,"stack_type_slug":565,"stack_type_icon_url":566,"score_popularity":35,"score_learning_curve":35,"catalog_display_order":32,"published_date":32,"last_updated_date":32,"core_tool_previews":625},108,"modern-elt-stack","Modern ELT Stack","Airbyte extracts into Snowflake, dbt transforms, Airflow orchestrates: the modern ELT standard.","intermediate",[626,627,628,633,634],{"tool_id":35,"slug":124,"name":123,"logo_url":343,"logo_bg":158},{"tool_id":574,"slug":575,"name":576,"logo_url":577,"logo_bg":158},{"tool_id":629,"slug":630,"name":631,"logo_url":632,"logo_bg":158},131,"airbyte","Airbyte","https:\u002F\u002Fassets.tekyous.dev\u002Flogos\u002Ftools\u002Fairbyte.svg",{"tool_id":483,"slug":484,"name":484,"logo_url":486,"logo_bg":158},{"tool_id":153,"slug":155,"name":154,"logo_url":157,"logo_bg":158},[636,639,642,645,648],{"question":637,"answer":638},"Fivetran or Airbyte for ingestion?","Fivetran is fully managed with essentially no setup, billed by monthly active rows synced. Airbyte is free to self-host and has more connectors overall, at the cost of running and maintaining it yourself.",{"question":640,"answer":641},"Why Dagster instead of Airflow for orchestration?","Dagster's asset-based model tracks the data each step produces, not just the tasks that ran, which makes debugging a broken pipeline and understanding data lineage more direct than Airflow's task-centric DAGs.",{"question":643,"answer":644},"How do I keep BigQuery query costs predictable?","On on-demand pricing you pay for bytes scanned, so the goal is to scan less. Partition large tables by date and cluster them on the columns people filter by, and turn on the require-partition-filter setting so a query without a date range fails instead of reading the whole table. Build big dbt models as incremental so each run processes only new data. On the Metabase side, dashboards query BigQuery every time they load, so point them at small aggregated dbt models rather than raw tables and enable Metabase's query caching. A per-user or per-project cap on bytes billed stops one runaway query from becoming the month's biggest line item.",{"question":646,"answer":647},"Where do Dagster and Metabase run in a serverless setup?","Not in BigQuery or Fivetran, so they need a home. Fivetran and BigQuery are fully managed, but open-source Dagster and Metabase are services you host. On GCP the common choices are Cloud Run or a small Compute Engine VM for Metabase, and a VM or a GKE cluster for Dagster, each with a Cloud SQL PostgreSQL database for its own metadata (Metabase's built-in H2 database isn't meant for production). The alternative is to buy those two layers managed as well: Dagster+ and Metabase Cloud are hosted versions, at a subscription that replaces the hosting work.",{"question":649,"answer":650},"How is this different from the Airbyte + dbt + Snowflake + Tableau stack?","Both are warehouse-plus-dbt pipelines; the choices around dbt differ. This stack stays on Google Cloud with BigQuery, uses Fivetran for fully managed ingestion, orchestrates with Dagster, and keeps BI cheap with open-source Metabase. The Snowflake stack is cloud-neutral, uses Airbyte (free to self-host), and pays for Tableau seats to get deeper enterprise dashboards. Pick this one when the company already runs on GCP and wants ingestion that needs no attention; pick the Snowflake stack when the warehouse must span clouds or when a large business audience needs Tableau's dashboard depth.",{"summary":652,"starting_cost_label":653,"has_free_tier":3,"line_items":654},"dbt Core, Dagster, and Metabase's open-source edition are all free to self-host. Fivetran and BigQuery are the real cost drivers: Fivetran bills by monthly active rows past its free tier, and BigQuery bills by query bytes scanned and storage, so total cost tracks data volume and how often the pipeline runs rather than a flat fee.","Free to start, usage-based at scale",[655,658,661],{"label":458,"cost":656,"note":657},"Free-usage-based","The free tier covers light usage; paid tiers bill by monthly active rows synced, which scales with data volume.",{"label":379,"cost":659,"note":660},"Usage-based","Billed by query bytes scanned and storage; GCP's Always Free tier covers a modest amount of both each month.",{"label":662,"cost":663,"note":664},"dbt Core, Dagster, Metabase","Free (open source)","All three are free to self-host with no usage limits.",{"title":666,"description":667,"og_image":32,"canonical":668},"GCP ELT Pipeline: Tools, Pricing & How to Deploy | Tekyous","Fivetran to BigQuery, dbt transforms, Dagster orchestrates, Metabase visualizes on GCP. Compare GCP ELT Pipeline tools, pricing & how to deploy on Tekyous.","https:\u002F\u002Ftekyous.dev\u002Fstacks\u002Fgcp-elt-pipeline",1790518839447]