Open role
Solution Architect Databricks OR Spark
HNM Solutions B.V.
About the role
Languages*
English
We are looking for a highly hands-on Solution Architect with deep expertise in Apache Spark and Databricks. This is not a purely advisory or whiteboarding role—the candidate must be capable of writing production code, optimising complex batch and streaming workloads, and delivering scalable platform solutions.
What you'll do
* Design and optimise Spark batch and Structured Streaming workloads.
* Build Delta Live Tables pipelines, Databricks Workflows and Unity Catalog governance models.
* Implement Terraform, Terragrunt and Databricks Asset Bundles with GitHub Actions CI/CD.
* Develop reusable, metadata-driven PySpark frameworks.
* Design fine-grained permissions, row-level security and data models.
* Support Databricks Apps and full-stack components using Node.js and React.
* Integrate Databricks with on-premises and custom data platforms.
* Translate loosely defined business requirements into technical solutions and product priorities.
* Advise global engineering teams on performance, cost reduction and platform adoption.
Mandatory skills:
* Advanced Apache Spark performance tuning: shuffles, partitioning, joins, memory management and AQE.
* Production Spark Structured Streaming: watermarking, triggers, state management, exactly-once processing and checkpoint recovery.
* Strong Spark SQL and query-plan optimisation.
* Deep hands-on experience with Delta Live Tables, Databricks Workflows and Unity Catalog.
* Production-level Python/PySpark.
* Terraform and Databricks Asset Bundles.
* Strong coding and debugging capabilities.
* Ability to work autonomously, define priorities and deliver without detailed requirements or structured sprints.
Strongly preferred:
* Databricks Apps.
* Fine-grained access control and row-level security.
* GitHub Actions and end-to-end CI/CD.
* Lakebase, Delta tables and Azure Data Factory/Airflow.
* Monitoring using OTLP, VictoriaMetrics or Grafana.
* Financial markets, proprietary trading, market-making or quantitative-finance experience