Data Engineer - PySpark & Databricks, Morris Plains, New Jersey, Job at Brandon Consulting Associates, Inc., Morris Plains, NJ

TlJiZnhwVytCUU9YWWF3MVdvNGlkbC9UVFE9PQ==
  • Brandon Consulting Associates, Inc.
  • Morris Plains, NJ

Job Description

(W2 - ONLY - No Sponsorship)

Job Description

We are seeking a Data Engineer with strong experience in PySpark and Databricks to design, build, and optimize scalable data pipelines and analytics solutions. The role involves working with large-scale datasets, cloud-based platforms, and modern data engineering practices to support advanced analytics, reporting, and data-driven decision-making.

Responsibilities:

  • Design, develop, and maintain ETL/ELT pipelines using PySpark on Databricks
  • Build and optimize batch and streaming data pipelines

  • Implement Delta Lake solutions (Delta tables, time travel, ACID transactions)

  • Collaborate with data scientists, analysts, and architects to deliver analytics-ready datasets

  • Optimize Spark jobs for performance, scalability, and cost - Integrate data from multiple sources (RDBMS, APIs, files, cloud storage

  • Implement data quality checks, validation, and monitoring

  • Manage Databricks notebooks, jobs, clusters, and workflows

  • Follow data governance, security, and compliance standards

  • Participate in code reviews and contribute to best practices

Qualifications

Technical Skills

  • PySpark: Hands-on experience with Data Frames, RDDs, joins, transformations, and actions.
  • Databricks: Job optimization, cluster configuration, repartitioning, and Shuffle mechanics
  • AWS: S3 buckets, IAM, CloudWatch, and integration with Databricks. - SQL: Strong query skills for analytics and ETL.
  • Performance tuning: Partitioning, caching, broadcast joins, and skew handling
  • Delta Lake, Medallion Architecture, Spark Streaming, Spark ML, and CI/CD pipelines. Data & Platform Knowledge
  • ETL/ELT design patterns - Handling large-scale structured and semi-structured data
  • Performance tuning (partitioning, caching, broadcast joins) - Understanding of data warehousing concepts

Skill Category

Years of Experience you have

PySpark & Distributed Data Processing

Databricks Platform & Spark Optimization

Delta Lake & Medallion Architecture

AWS Data Ecosystem & Integration

Data Quality, Governance & SQL

Brandon Consulting Associates, Inc. is an EQUAL OPPORTUNITY EMPLOYER and has been in business for 29years.

Job Tags

Similar Jobs

Summit Foot and Ankle

Podiatrist, DPM Job at Summit Foot and Ankle

 ...person gets the care they need and deserve. Our goal is to help every patient walk out of our clinic pain-free! QUALIFICATIONS FOR A PODIATRIST DPM Doctor of Podiatric Medicine Licensure in the State of Georgia 1+ years of experience in podiatry, preferred... 

Humina Resource

Network Engineer Job at Humina Resource

 ...Network Engineer Japanese Bilingual Full-time | Entry-Level Welcome! | Japanese / English Bilingual -------------------------------- About the Role A Japanese IT company providing a wide range of technology servicesincluding system development, software... 

Amazon Data Services, Inc.

Data Center Infrastructure Engineer Intern Job at Amazon Data Services, Inc.

 ...invention and progressive thinking. As an Intern, you will engage with an experienced...  ...to provide full life-cycle support to AWS Data Centers from design inception through site...  ...and maintenance. You will be the go to engineering resource for your region when technical... 

Copper Mountain College

Men's and Women's Cross-Country / Long-Distance Track Head Coach, Part-Time Faculty Job at Copper Mountain College

 ...Men's and Women's Cross-Country / Long-Distance Track Head Coach, Part-Time Faculty Salary: $58.05 - $78.08 Hourly Job Type: Part-Time Department: Academic Services Closing: Location: Joshua Tree, CA Job Number: 2025-12-17-PTHCCC Position... 

Morgan, Lewis & Bockius LLP

Junior Full Stack Software Developer Job at Morgan, Lewis & Bockius LLP

 ...of commerce, law, and government across North America, Asia, Europe, and the Middle East, is seeking to hire a motivated Junior Full-Stack Developer to support the development of modern, cloud-based web applications. This role is ideal for someone early in their career...