Senior Data Engineer, Solutions Architecture

About this role

<h2 class="lumesse-section-header lumesse-section-header-0 jdesc-section-header"><span style="color: rgb(0, 32, 96);">What The Role Is</span></h2> <p>We are seeking a talented Senior Data Engineer to design, build, and maintain our data infrastructure supporting mission-critical energy operations. You'll work at the intersection of renewable energy and data, developing pipelines that process everything from real-time asset performance data to complex trading and risk analytics. This hybrid role offers the opportunity to make a direct impact on clean energy operations while working with a cutting-edge data stack including Snowflake, Dagster, dbt, Modal, and GitLab.</p> <h2 class="lumesse-section-header lumesse-section-header-1 jdesc-section-header"><span style="color: rgb(0, 32, 96);">What You'll Be Doing</span></h2> <h3>Analytics Infrastructure & Data Warehouse Management</h3> <ul> <li>Design, deploy, and maintain scalable data infrastructure to support enterprise analytics and reporting needs</li> <li>Manage Snowflake instances, including performance tuning, security configuration, and capacity planning for growing data volumes</li> <li>Optimize query performance and resource utilization to control costs and improve processing speed</li> </ul> <h3>Data Pipeline Development & Orchestration</h3> <ul> <li>Build and orchestrate complex ETL/ELT workflows using Dagster to ensure reliable, automated data processing for asset management and energy trading</li> <li>Develop robust data pipelines that handle high-volume, time-sensitive energy market data and asset generation and performance metrics</li> <li>Implement workflow automation and dependency management for critical business operations</li> </ul> <h3>Data Transformation & Analytics Support</h3> <ul> <li>Develop and maintain dbt models to transform raw data into business-ready analytical datasets and dimensional models</li> <li>Create efficient SQL-based transformations for complex energy market calculations and asset performance metrics</li> <li>Support advanced analytics initiatives through proper data preparation and feature engineering</li> </ul> <h3>Data Quality & Governance</h3> <ul> <li>Implement comprehensive data validation, testing, and monitoring frameworks to ensure accuracy and consistency across all energy and financial data assets</li> <li>Establish data lineage tracking and privacy controls to meet regulatory compliance requirements in the energy sector</li> <li>Develop alerting and monitoring systems for data pipelines, including error handling, SLA monitoring, and incident response</li> </ul> <h3>CI/CD & DevOps</h3> <ul> <li>Lead continuous integration and deployment initiatives for Dagster and dbt pipelines, and Streamlit/Gradio application deployments to Linux servers</li> <li>Implement automated testing and deployment automation for data pipelines and analytics applications</li> <li>Manage version control and infrastructure as code practices</li> </ul> <h3>Cross-functional Collaboration</h3> <ul> <li>Partner with Analytics Engineers, Data Scientists, and business stakeholders to understand requirements and deliver solutions</li> <li>Work closely with asset management and trading groups to ensure real-time data availability for market operations and risk calculations</li> <li>Collaborate with credit risk teams to develop data models supporting financial analysis and regulatory reporting</li> <li>Translate business requirements into technical solutions and communicate data insights to stakeholders</li> </ul> <h3>Documentation & Security</h3> <ul> <li>Create and maintain technical documentation, data dictionaries, and onboarding materials for data assets</li> <li>Implement role-based access controls, data encryption, and security best practices across the data stack</li> <li>Monitor and optimize cloud infrastructure costs, implement resource allocation strategies, and provide cost forecasting</li> </ul> <h2 class="lumesse-section-header lumesse-section-header-2 jdesc-section-header"><span style="color: rgb(0, 32, 96);">What You'll Bring</span></h2> <ul> <li>Experience: 4+ years of hands-on data engineering experience in production environments</li> <li>Education: Bachelor's degree in Computer Science, Engineering, or a related field</li> <li>Data Orchestration: Proficiency in Dagster for pipeline scheduling, dependency management, and workflow automation; Airflow experience a plus</li> <li>Cloud Data Warehousing: Advanced-level Snowflake administration, including virtual warehouses, clustering, security, and cost optimization</li> <li>Data Transformation: Proficiency in dbt for data modeling, testing, documentation, and version control of analytical transformations</li> <li>Programming: Strong Python and SQL skills for data processing and automation</li> <li>CI/CD: 3+ years of experience with continuous integration and continuous deployment practices and tools; proficiency in GitLab CI/CD required (GitHub Actions experience a plus)</li> <li>Database Technologies: Advanced SQL skills, database design principles, and experience with multiple database platforms</li> <li>Cloud Platforms: Proficiency in AWS/Azure/GCP data services, storage solutions (S3, Azure Blob, GCS), and infrastructure…