Data Engineer - PySpark & Databricks, Morris Plains, New Jersey, Job at Brandon Consulting Associates, Inc., Morris Plains, NJ

TlJiZnhwVytCUU9YWWF3MVdvNGlkbC9UVFE9PQ==
  • Brandon Consulting Associates, Inc.
  • Morris Plains, NJ

Job Description

(W2 - ONLY - No Sponsorship)

Job Description

We are seeking a Data Engineer with strong experience in PySpark and Databricks to design, build, and optimize scalable data pipelines and analytics solutions. The role involves working with large-scale datasets, cloud-based platforms, and modern data engineering practices to support advanced analytics, reporting, and data-driven decision-making.

Responsibilities:

  • Design, develop, and maintain ETL/ELT pipelines using PySpark on Databricks
  • Build and optimize batch and streaming data pipelines

  • Implement Delta Lake solutions (Delta tables, time travel, ACID transactions)

  • Collaborate with data scientists, analysts, and architects to deliver analytics-ready datasets

  • Optimize Spark jobs for performance, scalability, and cost - Integrate data from multiple sources (RDBMS, APIs, files, cloud storage

  • Implement data quality checks, validation, and monitoring

  • Manage Databricks notebooks, jobs, clusters, and workflows

  • Follow data governance, security, and compliance standards

  • Participate in code reviews and contribute to best practices

Qualifications

Technical Skills

  • PySpark: Hands-on experience with Data Frames, RDDs, joins, transformations, and actions.
  • Databricks: Job optimization, cluster configuration, repartitioning, and Shuffle mechanics
  • AWS: S3 buckets, IAM, CloudWatch, and integration with Databricks. - SQL: Strong query skills for analytics and ETL.
  • Performance tuning: Partitioning, caching, broadcast joins, and skew handling
  • Delta Lake, Medallion Architecture, Spark Streaming, Spark ML, and CI/CD pipelines. Data & Platform Knowledge
  • ETL/ELT design patterns - Handling large-scale structured and semi-structured data
  • Performance tuning (partitioning, caching, broadcast joins) - Understanding of data warehousing concepts

Skill Category

Years of Experience you have

PySpark & Distributed Data Processing

Databricks Platform & Spark Optimization

Delta Lake & Medallion Architecture

AWS Data Ecosystem & Integration

Data Quality, Governance & SQL

Brandon Consulting Associates, Inc. is an EQUAL OPPORTUNITY EMPLOYER and has been in business for 29years.

Job Tags

Similar Jobs

Hometown Nannies Plus LLC

NANNY / FAMILY ASSISTANT | Full-Time (40+ hours) | Live-Out | 2-month-old baby. | Flexible Schedule; Occasional Weekends /Overnights & Domestic & International Travel Required | $35/hr., Benefits: Paid Vacation, Health Insurance/Stipend, Holidays Off | START 2-15-26 Job at Hometown Nannies Plus LLC

 ...NANNY / FAMILY ASSISTANT | WESTPORT, CT Full-Time | Live-Out | $35/hr Overnight Coverage: $...  ..., Health Insurance or Stipend, Holidays Off A warm, professional family in Westport...  .../week) Flexible schedule; occasional weekends or overnights Domestic and international... 

CornerStone Staffing

Medical Coder Job at CornerStone Staffing

 ...coding experience specializing in Neurosurgery or OB/GYN . Certification: Active coding certification required (CPC, CCS, RHIT, RHIA, or equivalent). OB/GYN applicants: COBGC certification through AAPC preferred. Technical Skills: Proficiency with... 

Humina Resource

Network Engineer Job at Humina Resource

 ...Network Engineer Japanese Bilingual Full-time | Entry-Level Welcome! | Japanese / English Bilingual -------------------------------- About the Role A Japanese IT company providing a wide range of technology servicesincluding system development, software... 

Ganna Walska Lotusland

Visitor Center and Garden Shop Manager Job at Ganna Walska Lotusland

Visitor Center and Garden Shop Manager Location Santa Barbara, CA : Title: Visitor Center and Garden Shop Manager Classification: Full-Time 40-hour work week nonexempt Department: Visitor Services Reports to: Director of Visitor Services and Education Programs... 

The TJX Companies, Inc.

70120 - Merchandise Associate Job at The TJX Companies, Inc.

 ...prioritizes your development. Whether youre working in our four global Home Offices, Distribution Centers or Retail StoresTJ Maxx, Marshalls, Homegoods, Homesense, Sierra, Winners, and TK Maxx, youll find abundant opportunities to learn, thrive, and make an impact. Come...