You will help in building a Data platform to facilitate the analysis of structured and unstructured data. The platform will initially be used to provide daily analysis of our firm wide risk, with further use cases following. You will architect a custom data pipeline solution for a big data platform. Additionally, you will work on moving portions of our systems to a cloud-based environment, as we will be leveraging AWS extensively.
Gather and process raw data at scale (including writing systems and scripts, calling APIs, write SQL queries, etc.).
Build solutions to process unstructured data into a form suitable for analysis and ingest into the platform.
Support business decisions with ad hoc analysis as needed.
Work with other members of the team to ensure data is ingested, stored and managed properly to support our analytical needs
Re-Architecting existing systems with an emphasis on the Cloud.
Helping define automation and management of cloud infrastructure using AWS native tools, open source tools, and third party products.
Automating the build and management of cloud infrastructure using AWS native tools, open source tools, and third party products.
Aggregate, correlate, and report on both logs and metrics, use them for performance tuning and troubleshooting, and to trigger automated actions based on events or thresholds.
4+ years of experience building scalable and reliable data pipelines based on Big Data processing technologies like Hadoop, Cassandra, MapReduce, Spark, etc.
4+ years of combined hands-on experience in Java, Python,Scala, R and the ability to write reusable and efficient code to automate analyses and data processes.
4+ years of experience designing and supporting AWS environments.
Experience processing large amounts of structured and unstructured data.
Highly analytical with good problem solving skills and creativity; able to work independently in a fast-paced environment
Extensive background in building highly automated DevOps infrastructure.
Strong knowledge of and experience with statistics; potentially other advanced math as well.
Broad understanding of financial services instruments