Senior Data Engineer

San Francisco Office

BigCommerce named a “Best and Brightest” place to work in San Francisco is looking for a Senior Data Engineer. This is an exciting opportunity to build a real time big data pipeline and SaaS analytics platform using latest cloud technologies for the #1 e-commerce SaaS platform. Our perfect candidate will have a background in Computer Science or a related technical field and possess extensive experience in building distributed scalable data pipelines and working with large data stores. You will be working with cross-functional teams in highly visible role. You are scrappy, focused on results, a self-starter and have demonstrated success in using engineering and analytics to drive the understanding, growth and success of a product.

What you’ll do:

  • Build scalable and reliable near real time data pipeline on cloud (AWS and GCP) that collects, transforms, loads and curates data from various internal and external data sources
  • Build a scalable distributed data store that will be central source of truth
  • Own data quality for the pipelines you build and make them auditable
  • Build self service tools that helps our data consumers to extract, analyze and visualize data faster
  • Evaluate new technologies and build prototypes for continuous improvements in Data Engineering
  • Partner with Infrastructure and Engineering teams to ensure instrumentation, logging and monitoring is in place
  • Implement Machine learning algorithms

Who you are:

  • Extensive experience in using big data technologies such as Spark, Kafka, Hadoop, HBase and Hive or their equivalents
  • Experience with AWS and/or GCP
  • 5+ years of experience with Java, Scala and Python
  • 5+ years of experience with SQL (MySQL, Redshift, etc)
  • 3+ years of experience in building and monitoring near real time scalable ETL pipelines
  • Experience with shell scripting
  • Excellent written and verbal communication skills
  • BS or MS in Computer Science or related technical field

Nice to have:

  • Experience with Machine learning algorithms will be a huge plus.
  • Experience with tools such as AirFlow for scheduling and monitoring of data workflow will be a plus
  • Experience with managed data processing services of cloud platforms such as GCP and/or AWS will be a plus

Experience working with a team distributed over time zones would be a plus

Share job posting