Organization: aKube Inc
Location: United States
Category: IT Services Location: Onsite/ Hybrid/ Remote: Remote Duration:1 Month Rate Range: Upto $89/hr on C2C or $82/hr on W2 Work Authorization: GC, USC, All valid EADs except H1B, OPT, CPT Must Have: Advanced Databricks engineering experience Delta Lake Data ingestion from new and diverse sources Medallion Architecture: Bronze, Silver, Gold Databricks Genie Spaces Genie Space debugging, query tuning, and performance optimization Advanced SQL and Spark/PySpark Responsibilities: Build and support Databricks pipelines for new data source ingestion. Design and develop Bronze, Silver, and Gold data layers. Optimize Delta Lake tables, queries, and platform performance. Debug Genie Spaces and improve natural language query response quality. Tune underlying SQL and data models used by Genie Spaces. Support short-term delivery needs across the Databricks platform. Qualifications: Strong hands-on Databricks and Delta Lake engineering background. Proven experience building Medallion Architecture in production. Experience integrating multiple data sources into centralized data platforms. Direct experience with Databricks Genie Spaces is required. Strong SQL, Spark, and data performance tuning experience. Nice to Have: Unity Catalog Databricks workflows Data governance and observability Lakehouse migration experience Details Originally posted on Himalayas
To excel in this role, you will need to possess a strong understanding of Databricks and its ecosystem, including:
* Advanced knowledge of Databricks’ features, such as Databricks Genie Spaces, Medallion Architecture, and Delta Lake.
* Experience with data ingestion from diverse sources, including cloud and on-premises data sources.
* Strong SQL and Spark/PySpark skills, with the ability to design and develop optimized data layers.
* Experience with data governance, observability, and data quality, including data validation, data cleansing, and data transformation.
* Familiarity with data engineering tools and technologies, such as Apache Spark, Apache Hadoop, and NoSQL databases.
* Experience with data architecture design, including data modeling, data warehousing, and data integration.
In addition to the technical skills, the ideal candidate will also possess excellent communication and collaboration skills, with the ability to work effectively in a fast-paced, dynamic environment.
Some of the key skills and technologies that the ideal candidate should have include:
* Databricks Genie Spaces: experience with debugging, query tuning, and performance optimization is essential.
* Medallion Architecture: a strong understanding of Bronze, Silver, and Gold data layers is required.
* Delta Lake: experience with data ingestion, data governance, and data observability is necessary.
* SQL and Spark/PySpark: strong skills in designing and developing optimized data layers are essential.
* Data governance and observability: experience with data validation, data cleansing, and data transformation is required.
* Data engineering tools and technologies: familiarity with Apache Spark, Apache Hadoop, and NoSQL databases is necessary.
Furthermore, the ideal candidate should have experience with:
* Data architecture design: including data modeling, data warehousing, and data integration.
* Data governance and observability: including data validation, data cleansing, and data transformation.
* Lakehouse migration: experience with migrating data from traditional data warehouses to lakehouse architectures is a plus.
* Databricks workflows: experience with designing and developing optimized Databricks workflows is required.
Get Latest Updates: Join our WhatsApp Group