Spark Developer

Columbus, OH

Opportunity Details

Full TimeApplication Development

Columbus, OH

We’re Hiring an Apache Spark Developer

This role is responsible for building and maintaining the architecture and systems for automating predictive models and the components used to train, predict, and track the accuracy of them. This includes gathering, structuring and analyzing data in high volumes and velocity.

Primary Responsibilities:

  • Collect, store and manage large data sets from multiple source systems and third parties.

  • Design, build, and continuously improve the architecture and systems to support predictive models in a distributed environment.

  • Create processes to monitor the prediction systems to ensure they are performant and produce accurate predictions.

  • Collaborate with data science team to support any data needs and understand model structures to be automated by the system.

  • Work closely with data science and analytics teams to identify correct data segmentation, best algorithm to apply, and which features to use in the model.

  • Anticipate future demands of initiatives related to people, technology, and business when designing solutions.

  • Collaborate with Business Intelligence and other IT teams when designing processes to persist model predictions, calling models from other systems via API, and providing visuals for reporting data insights to the business.


    Required Skills:

  • At least 1 years of hands on experience in Spark (fundamentals, RDD and dataframe APIs)

  • At least 2 years of experience with Python ML stack (scikit-learn, pandas, numpy) Experience with SparkML (or any ML on a big data platform)

  • At least 2 years of experience in Hadoop and knowledge of HDFS design patterns

  • Strong understanding of machine learning theory (supervised and unsupervised algorithms, regularization, cross-validation, hyper-parameter grid search, predictive performance evaluation, optimization concepts)

  • Advanced SQL Skills (multiple table queries, subqueries, window functions)

  • Ability to quickly adjust to a growing organization’s needs, new markets, and opportunities in a collaborative Agile environment Team oriented with focus on collaboration, knowledge sharing, and relationship building High degree of responsiveness, reliability, and integrity Decisiveness, good judgment, analytical, and problem-solving skills


    Preferred Skills:

  • C# programming Experience with neural networks and streaming data Experience with Azure data stack Background in statistics

Think this opportunity is a good fit for you?
Apply Now

“My career found purpose, autonomy, and mastery at Improving. Improving’s foundation of Trust has changed everything in my career and personal life. There’s a humble pride I have partnering with fellow Improvers. We are a team of professionals driven by a persistent habit of Excellence – consistently involved in helping others Improve with purpose.”

― Ronnie Cooke, Principle Agile Consultant