Data Engineer
Insight Global
Job Description
As a member of the US Data Strategy team, you’ll work in a fast-paced environment where you will be responsible for the detailed technical design and development of applications using various technology platforms. The role performs tasks of technical depth and breadth, utilizing a solid understanding of business dynamics to conduct impact analysis and provide feedback on problems with recommended solutions.
The Data Engineer determines methods and approaches to projects, transforming business requirements specifications into programming instructions, designing, coding and testing programs. In addition, the role plays a key role in the development and implementation of database management solutions, supporting the company’s backup plans. The role uses judgement and autonomy on day to day tasks, exercising considerable latitude to conduct data source analysis and negotiate with users regarding requirements.
• Configure and develop custom ETL Solutions to ingest data into Azure SQL Data Warehouse, code data quality and transformation logic for data movement within data warehouse, develop code to publish data from data warehouse to data mart for consumption by applications or BI tools.
• Design and develop SQL Server data objects including tables, schemas, views, functions and stored procedures.
• Designing and implementing data ingestion pipelines from multiple sources using Azure DataBricks Apache Spark and/or Azure DataBricks, developing scalable and re-usable frameworks for ingesting of data sets, integrating the end to end data pipeline - to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times, working with event based / streaming technologies to ingest and process data is required.
• Work with other members of the project team to support delivery of additional project components (API interfaces, Search), evaluate the performance and applicability of multiple tools against customer requirements, document and maintain data lineage, validate application code against unit and security tests, along with entitlement models.
• Work with technical experts at the platform and enterprise levels to define custom and integrated software solutions, work in an agile SDLC and contribute from discovery to delivery of a feature and interface with business partners to understand requirements and present solutions within the capabilities of the enterprise.
• Collaborates directly with stakeholders, end-users, Product Owners, Business Analysts, and functional partners – Data, Information Security, Identity Management, Compliance, Legal, and Risk and develops proofs of concept (PoC) as needed to evaluate product and technical strategies, while owning technical and functional specifications as needed.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Skills and Requirements
• 5+ years of experience as a data engineer with a strong knowledge of approaches, tools, techniques and roles and responsibilities in providing technical or business guidance to business and technical partners.
• Hands on experience designing and delivering solutions using the Azure Data Analytics platform (Cortana Intelligence Platform) including Azure Storage, Azure SQL Data Warehouse, Azure Data Lake, Azure Cosmos DB, Azure Stream Analytics
• Proficiency in ETL, Azure SQL and REST APIs
• Expertise in designing and deploying data applications on Azure cloud solutions
• Hands on experience in performance tuning and optimizing code running in DataBricks, Talend or similar ETL tools
• Proficient in programming languages like PySpark and Python and solid understanding of SQL, T-SQL and/or PL/SQL
• Strong analytical and problem-solving skills particularly those that apply to a big data environment
Exposure to Integration and BI Tools such as Tableau, Cognos or Power BI with significant automation experience Microsoft Azure Big Data Architecture certification
Confirmar seu email: Enviar Email
Todos os Empregos de Insight Global