**Introduction** The Backend Enablement team drives Instana’s backend and data platform evolution — designing, optimizing, and future-proofing our multi-cloud (including hybrid deployments) and multi-tenant observability data stack. As a Senior Data Engineer / Data Scientist, you will lead research, planning, and implementation efforts to modernize our data infrastructure, improve performance and cost efficiency, and align our systems with industry-leading data engineering and compliance practices. What We Offer · Opportunity to shape the next generation of observability data infrastructure within a global IBM product. · Collaborative science-driven, focused on quality, automation, and long-term maintainability. · Work alongside experienced architects and engineers on projects spanning schema evolution, encryption, and CI/CD optimization. · Continuous learning and professional certification support. **Your role and responsibilities** Key Responsibilities · To build scalable, schema-driven, and compliant data pipelines powering real-time analytics, search, and long-term storage across Kafka, ClickHouse, PostgreSQL, and OpenSearch ecosystems. · ClickHouse Optimization & Scaling — Research, plan, and implement ClickHouse cluster optimizations for multi-tenant workloads. Design rollup and retention strategies to balance performance, cost, and compliance. · Big Data Tooling & Integrations — Introduce and evaluate Big Data components such as S3/object storage, serverless databases, Kafka, Kafka Connect, and Kafka Schema Registry, adopting modern data-warehouse practices. Design data-in-motion and data-at-rest pipelines with strong encryption, efficient compression and schema-driven governance. · Elasticsearch → OpenSearch Migration — Lead research, planning, and phased migration of existing Elasticsearch clusters and APIs to OpenSearch. Ensure backward compatibility, observability, and search performance parity. · Datastore End-of-Life (EOL) Management — Research, plan, and implement decommissioning strategies for Cassandra and other legacy datastores. Define migration paths, data export/import automation, and service continuity procedures. · Schema & Data Format Engineering — Define and implement schema-driven data formats such as Parquet, Avro, and Cryo, or columnar compression for analytical workloads. Support versioned schema evolution across ClickHouse, PostgreSQL, and OpenSearch. · Compliance, Security & Data Governance — Embed compliance requirements (SOC2, FedRAMP, GDPR) into data retention, protection, and encryption layers. Contribute to audit-ready documentation, validation tooling, and policy enforcement automation. **Required technical and professional expertise** · Proven expertise with analytical databases and data pipeline design (ClickHouse, PostgreSQL, BigQuery, Redshift, etc.). · Strong programming background in Python, SQL, and Bash (plus optional Java or JavaScript experience). · Experience with Kafka, Kafka Connect, and Schema Registry, plus search systems (Elasticsearch/OpenSearch) and NoSQL (Cassandra, DynamoDB). · Solid understanding of schema-driven design, data compression, and high-throughput ETL/ELT pipelines. · Familiarity with data protection, compliance, and security frameworks (SOC2, FedRAMP, GDPR, FIPS). · Practical experience in public cloud ecosystems (AWS, GCP, IBM Cloud) and certifications are a strong plus. **Preferred technical and professional experience** · Experience with containerized test environments (Docker, Testcontainers) and CI/CD systems (CodeBuild, Jenkins). · Contributions to data platform migrations or modernization projects. · Knowledge of infrastructure-as-code, serverless services, and observability tooling. · Certification or formal training in Data Engineering, Cloud Architecture, or Information Security. IBM is committed to creating a diverse environment and is proud to be an equal-opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, caste, genetics, pregnancy, disability, neurodivergence, age, veteran status, or other characteristics. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.