Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And download my free book about data streaming use cases, including customer stories across all industries.

The Data Streaming Use Case Show: Episode #1 – Manufacturing and Automotive

These Siemens success stories are part of The Data Streaming Use Case Show, a new industry webinar series hosted by me.

In the first episode, we focus on the manufacturing and automotive industries. It features:

• Experts from Siemens Digital Industries and Siemens Healthineers
• The Founder of ‘IoT Use Case‘, a content and community platform focused on real-world industrial IoT applications
• Deep insights into how industrial companies combine OT, IT, cloud, and data streaming with the shift left architecture.

The series explores real-world solutions across industries, showing how leaders turn data into action through open architectures and real-time platforms.

Siemens Digital Industries: Company and Vision

Siemens Digital Industries is the technology and software arm of Siemens AG, focused on advancing industrial automation and digitalization. It empowers manufacturers and machine builders to become more agile, efficient, and resilient through intelligent software and integrated systems.

Its business model bridges the physical and digital worlds—combining operational technology (OT) with modern information technology (IT). From programmable logic controllers to industrial IoT, Siemens delivers end-to-end solutions across industries.

Today, the company is transforming itself into a software- and cloud-driven organization, focusing strongly on edge computing, real-time analytics, and data streaming as key enablers of modern manufacturing.

With edge and cloud working in harmony, Siemens helps industrial enterprises break up monoliths and develop toward modular, flexible architectures. These software-driven approaches make plants and factories more adaptive, intelligent, and autonomous.

Data Streaming at Industrial Companies

In industrial settings, data is continuously generated by machines, production systems, robots, and logistics processes. But traditional batch-oriented IT systems are not designed to handle this in real time.

To make smarter, faster decisions, companies need to process data as it is generated. That’s where data streaming comes in.

Apache Kafka and Apache Flink enable event-driven architectures. These allow industrial data to flow in real time, from edge to cloud, across hybrid environments.

Check out my other blogs about use cases and architecture for manufacturing and Industrial IoT powered by data streaming.

Edge and Hybrid Cloud as a Standard

Modern industrial use cases are increasingly hybrid by design. Machines and controllers produce data at the edge. Decisions must be made close to the source. However, cloud platforms offer powerful compute and AI capabilities.

Siemens leverages edge devices to capture and preprocess data on-site. Data streaming with Confluent provides Siemens a real-time backbone for integrating this data with cloud-based systems, including Snowflake, SAP, Salesforce, and others.

This hybrid architecture supports low latency, high availability, and full control over data processing and analytics workflows.

The Shift Left Architecture for Industrial IoT

In many industrial architectures, Kafka has traditionally been used to ingest data into analytics platforms like Snowflake or Databricks. Processing, transformation, and enrichment happened late in the data pipeline.

But Siemens is shifting that model.

The Shift Left Architecture moves processing closer to the source, directly into the streaming layer. Instead of waiting to transform data in a data warehouse, Siemens now applies stream processing in real time, using Confluent Cloud and Kafka topics.

This shift enables faster decision-making, better data quality, and broader reuse of high-quality data across both analytical and operational systems.

For a deeper look at how Shift Left is transforming industrial architectures, read the full article about the Shift Left Architecture with Data Streaming.

Siemens Data Streaming Success Story: Modular Intralogistics Platform

A key example of this new architecture is Siemens’ Modular Intralogistics Platform, used in manufacturing plants for material handling and supply chain optimization. I explored the shift left architecture in our data streaming use case show with Stefan Baer, Senior Key Expert – Data Streaming at Siemens IT.

Traditionally, intralogistic systems were tightly coupled, with rigid integrations between

• Enterprise Resource Planning (ERP): Order management, master data
• Manufacturing Operations Management (MOM): Production scheduling, quality, maintenance
• Warehouse Execution System (EWM): Inventory, picking, warehouse automation
• Execution Management System (eMS): Transport control, automated guided vehicle (AGV) orchestration, conveyor logic

The new approach breaks this down into package business capabilities—each one modular, orchestrated, and connected through Confluent Cloud.

Key benefits:

• Real-time orchestration of logistics operations
• Automated material delivery—no manual reordering required
• ERP and MOM systems integrated flexibly via Kafka
• High adaptability through modular components
• GenAI used for package station load optimization

Stream processing with Apache Flink transforms events in motion. For example, when a production order changes or material shortages occur, the system reacts instantly—adjusting delivery routes, triggering alerts, or rebalancing station loads using AI.

Architecture: Data Products + Shift Left

At the heart of the solution is a combination of data products and stream processing:

• Kafka Topics serve as real-time interfaces and persistency layer between business domains.
• Confluent Cloud hosts the event streaming infrastructure as a fully-managed service with low latency, elasticity, and critical SLAs.
• Stream processing with serverless Flink logic enriches and transforms data in motion.
• Snowflake receives curated, ready-to-use data for analytics.
• Other operational and analytical downstream consumers—such as GenAI modules or shop floor dashboards—access the same consistent data in real time.

This reuse of data products ensures consistent semantics, reduces duplication, and simplifies governance.

By processing data earlier in the pipeline, Siemens improves both data quality and system responsiveness. This model replaces brittle, point-to-point integrations with a more sustainable, scalable platform architecture.

Business Value of Data Streaming and Shift Left at Siemens Digital Industries

The combination of real-time data streaming, modular data products, and Shift Left design principles unlocks significant value:

• Faster response to dynamic events in production and logistics
• Improved operational resilience and agility
• Higher quality data for both analytics and AI
• Reuse across multiple consumers (analytics, operations, automation)
• Lower integration costs and easier scaling

This approach is not just technically superior—it supports measurable business outcomes like shorter lead times, lower stock levels, and increased manufacturing throughput.

Siemens Healthineers: Shift Left with IoT, Data Streaming, AI/ML, Confluent and Snowflake in Manufacturing and Healthcare

In a recent blog post, I explored how Siemens Healthineers uses Apache Kafka and Flink to transform both manufacturing and healthcare with a wide range of data streaming use cases. From predictive maintenance to real-time logistics, their approach is a textbook example of how to modernize complex environments with an event-driven architecture and data streaming—even if they don’t explicitly label it “shift left.”

Their architecture enables proactive decision-making by pushing real-time insights and automation earlier in the process. Examples include telemetry streaming from medical devices, machine integration with SAP and KUKA robots, and logistics event streaming from SAP for faster packaging and delivery. Each use case shows how real-time data—combined with cloud-native platforms like Confluent and Snowflake—improves efficiency, reliability, and responsiveness.

Just like the intralogistics example from Siemens Digital Industries, Healthineers applies shift-left thinking by enabling teams to act on data sooner, reduce latency, and prevent costly delays. This approach enhances not only operational workflows but also outcomes that matter, like patient care and regulatory compliance.

This is shift left in action: embedding intelligence and quality controls early, where they have the greatest impact.

Rethinking Industrial Data Architectures with Data Streaming and Shift Left Architecture

Siemens Digital Industries is demonstrating what’s possible when you rethink the data architecture beyond just analytics in a data lake.

With data streaming leveraging Confluent Cloud, data products for modular software, and a Shift Left approach, Siemens is transforming traditional factories into intelligent, event-driven operations. A data streaming platform based on Apache Kafka is no longer just an ingestion layer. It is a central nervous system for real-time processing and decision-making.

This is not about chasing trends. It’s about building resilient, scalable, and future-proof industrial systems. And it’s just the beginning.

To learn more, watch the on-demand industry use case show with Siemens Digital Industries and Siemens Healthineers or connect with us to explore what data streaming can do for your organization.

Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter. And download my free book about data streaming use cases.

The post Shift Left Architecture at Siemens: Real-Time Innovation in Manufacturing and Logistics with Data Streaming appeared first on Kai Waehner.

Just as mainframe offloading modernized enterprise IT, offloading and replacing legacy OT middleware is the next wave of digital transformation. Companies are shifting from vendor-locked, heavyweight OT middleware to real-time, event-driven architectures using Apache Kafka and Apache Flink—enabling cost efficiency, agility, and seamless edge-to-cloud integration.

This blog explores why and how organizations are replacing traditional OT middleware with data streaming, the benefits of this shift, and architectural patterns for hybrid and edge deployments.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch. And make sure to download my free book about data streaming use cases, including architectures and customer stories for hybrid IT/OT integration scenarios.

Why Replace Legacy OT Middleware?

Industrial environments have long relied on OT middleware like OSIsoft PI, proprietary SCADA systems, and industry-specific data buses. These solutions were designed for polling-based communication, siloed data storage, and batch integration. But today’s real-time, AI-driven, and cloud-native use cases demand more.

Challenges: Proprietary, Monolithic, Expensive

• High Costs – Licensing, maintenance, and scaling expenses grow exponentially.
• Proprietary & Rigid – Vendor lock-in restricts flexibility and data sharing.
• Batch & Polling-Based – Limited ability to process and act on real-time events.
• Complex Integration – Difficult to connect with cloud and modern IT systems.
• Limited Scalability – Not built for the massive data volumes of IoT and edge computing.

Just as PLCs are transitioning to virtual PLCs, eliminating hardware constraints and enabling software-defined industrial control, OT middleware is undergoing a similar shift. Moving from monolithic, proprietary middleware to event-driven, streaming architectures with Kafka and Flink allows organizations to scale dynamically, integrate seamlessly with IT, and process industrial data in real time—without vendor lock-in or infrastructure bottlenecks.

The Data Streaming Approach: Kafka & Flink as the Foundation for Modern OT Middleware

Data streaming is NOT a direct replacement for OT middleware, but it serves as the foundation for modernizing industrial data architectures. With Kafka and Flink, enterprises can offload or replace OT middleware to achieve real-time processing, edge-to-cloud integration, and open interoperability.

While Kafka and Flink provide real-time, scalable, and event-driven capabilities, last-mile integration with PLCs, sensors, and industrial equipment still requires OT-specific SDKs, open interfaces, or lightweight middleware. This includes support for MQTT, OPC UA or open-source solutions like Apache PLC4X to ensure seamless connectivity with OT systems.

Apache Kafka: The Backbone of Real-Time OT Data Streaming

Kafka acts as the central nervous system for industrial data to ensure low-latency, scalable, and fault-tolerant event streaming between OT and IT systems.

• Aggregates and normalizes OT data from sensors, PLCs, SCADA, and edge devices.
• Bridges OT and IT by integrating with ERP, MES, cloud analytics, and AI/ML platforms.
• Operates seamlessly in hybrid, multi-cloud, and edge environments, ensuring real-time data flow.
• Works with open OT standards like MQTT and OPC UA, reducing reliance on proprietary middleware solutions.

And just to be clear: Apache Kafka and similar technologies support “IT real-time” (meaning milliseconds of latency and sometimes latency spikes). This is NOT about hard real-time in the OT world for embedded systems or safety critical applications.

Apache Flink: The Real-Time Processing Engine for OT Data

Flink powers real-time analytics, complex event processing, and anomaly detection for streaming industrial data.

• Processes sensor data in real-time for predictive maintenance, energy optimization, and fault detection.
• Enables stateful stream processing, supporting event correlation, time-series aggregation, and trend analysis.
• Runs on edge devices, data centers, or cloud platforms, providing scalability and deployment flexibility.

By leveraging Kafka and Flink, enterprises can process OT and IT data only once, ensuring a real-time, unified data architecture that eliminates redundant processing across separate systems. This approach enhances operational efficiency, reduces costs, and accelerates digital transformation while still integrating seamlessly with existing industrial protocols and interfaces.

Unifying Operational (OT) and Analytical (IT) Workloads

As industries modernize, a shift-left architecture approach ensures that operational data is not just consumed for real-time operational OT workloads but is also made available for transactional and analytical IT use cases—without unnecessary duplication or transformation overhead.

The Shift-Left Architecture: Bringing Advanced Analytics Closer to Industrial IoT

In traditional architectures, OT data is first collected, processed, and stored in proprietary or siloed middleware systems before being moved later to IT systems for analysis. This delayed, multi-step process leads to inefficiencies, including:

• High latency between data collection and actionable insights.
• Redundant data storage and transformations, increasing complexity and cost.
• Disjointed AI/ML pipelines, where models are trained on outdated, pre-processed data rather than real-time information.

A shift-left approach eliminates these inefficiencies by bringing analytics, AI/ML, and data science closer to the raw, real-time data streams from the OT environments.

Instead of waiting for batch pipelines to extract and move data for analysis, a modern architecture integrates real-time streaming with open table formats to ensure immediate usability across both operational and analytical workloads.

Open Table Format with Apache Iceberg / Delta Lake for Unified Workloads and Single Storage Layer

By integrating open table formats like Apache Iceberg and Delta Lake, organizations can:

• Unify operational and analytical workloads to enable both real-time data streaming and batch analytics in a single architecture.
• Eliminate data silos, ensuring that OT and IT teams access the same high-quality, time-series data without duplication.
• Ensure schema evolution and ACID transactions to enable robust and flexible long-term data storage and retrieval.
• Enable real-time and historical analytics, allowing engineers, business users, and AI/ML models to query both fresh and historical data efficiently.
• Reduce the need for complex ETL pipelines, as data is written once and made available for multiple workloadssimultaneously. And no need to use the anti-pattern of Reverse ETL.

The Result: An Open, Cloud-Native, Future-Proof Data Historian for Industrial IoT

This open, hybrid OT/IT architecture allows organizations to maintain real-time industrial automation and monitoring with Kafka and Flink, while ensuring structured, queryable, and analytics-ready data with Iceberg or Delta Lake. The shift-left approach ensures that data streams remain useful beyond their initial OT function, powering AI-driven automation, predictive maintenance, and business intelligence in near real-time rather than relying on outdated and inconsistent batch processes.

By adopting this unified, streaming-first architecture to build an open and cloud-native data historian, organizations can:

• Process data once and make it available for both real-time decisions and long-term analytics.
• Reduce costs and complexity by eliminating unnecessary data duplication and movement.
• Improve AI/ML effectiveness by feeding models with real-time, high-fidelity OT data.
• Ensure compliance and historical traceability without compromising real-time performance.

This approach future-proofs industrial data infrastructures, allowing enterprises to seamlessly integrate IT and OT, while supporting cloud, edge, and hybrid environments for maximum scalability and resilience.

Key Benefits of Offloading OT Middleware to Data Streaming

• Lower Costs – Reduce licensing fees and maintenance overhead.
• Real-Time Insights – No more waiting for batch updates; analyze events as they happen.
• One Unified Data Pipeline – Process data once and make it available for both OT and IT use cases.
• Edge and Hybrid Cloud Flexibility – Run analytics at the edge, on-premise, or in the cloud.
• Open Standards & Interoperability – Support MQTT, OPC UA, REST/HTTP, Kafka, and Flink, avoiding vendor lock-in.
• Scalability & Reliability – Handle massive sensor and machine data streams continuously without performance degradation.

A Step-by-Step Approach: Offloading vs. Replacing OT Middleware with Data Streaming

Companies transitioning from legacy OT middleware have several strategies by leveraging data streaming as an integration and migration platform:

1. Hybrid Data Processing
2. Lift-and-Shift
3. Full OT Middleware Replacement

1. Hybrid Data Streaming: Process Once for OT and IT

Why?

Traditional OT architectures often duplicate data processing across multiple siloed systems, leading to higher costs, slower insights, and operational inefficiencies. Many enterprises still process data inside expensive legacy OT middleware, only to extract and reprocess it again for IT, analytics, and cloud applications.

A hybrid approach using Kafka and Flink enables organizations to offload processing from legacy middleware while ensuring real-time, scalable, and cost-efficient data streaming across OT, IT, cloud, and edge environments.

How?

Connect to the existing OT middleware via:

• A Kafka Connector (if available).
• HTTP APIs, OPC UA, or MQTT for data extraction.
• Custom integrations for proprietary OT protocols.
• Lightweight edge processing to pre-filter data before ingestion.

Use Kafka for real-time ingestion, ensuring all OT data is available in a scalable, event-driven pipeline.

Process data once with Flink to:

• Apply real-time transformations, aggregations, and filtering at scale.
• Perform predictive analytics and anomaly detection before storing or forwarding data.
• Enrich OT data with IT context (e.g., adding metadata from ERP or MES).

Distribute processed data to the right destinations, such as:

• Time-series databases for historical analysis and monitoring.
• Enterprise IT systems (ERP, MES, CMMS, BI tools) for decision-making.
• Cloud analytics and AI platforms for advanced insights.
• Edge and on-prem applications that need real-time operational intelligence.

Result?

• Eliminate redundant processing across OT and IT, reducing costs.
• Real-time data availability for analytics, automation, and AI-driven decision-making.
• Unified, event-driven architecture that integrates seamlessly with on-premise, edge, hybrid, and cloud environments.
• Flexibility to migrate OT workloads over time, without disrupting current operations.

By offloading costly data processing from legacy OT middleware, enterprises can modernize their industrial data infrastructure while maintaining interoperability, efficiency, and scalability.

2. Lift-and-Shift: Reduce Costs While Keeping Existing OT Integrations

Why?

Many enterprises rely on legacy OT middleware like OSIsoft PI, proprietary SCADA systems, or industry-specific data hubs for storing and processing industrial data. However, these solutions come with high licensing costs, limited scalability, and an inflexible architecture.

A lift-and-shift approach provides an immediate cost reduction by offloading data ingestion and storage to Apache Kafka while keeping existing integrations intact. This allows organizations to modernize their infrastructure without disrupting current operations.

How?

Use the Stranger Fig Design Pattern as a gradual modernization approach where new systems incrementally replace legacy components, reducing risk and ensuring a seamless transition:

“The most important reason to consider a strangler fig application over a cut-over rewrite is reduced risk.” Martin Fowler

Replace expensive OT middleware for ingestion and storage:

• Deploy Kafka as a scalable, real-time event backbone to collect and distribute data.
• Offload sensor, PLC, and SCADA data from OSIsoft PI, legacy brokers, or proprietary middleware.
• Maintain the connectivity with existing OT applications to prevent workflow disruption.

Streamline OT data processing:

• Store and distribute data in Kafka instead of proprietary, high-cost middleware storage.
• Leverage schema-based data governance to ensure compatibility across IT and OT systems.
• Reduce data duplication by ingesting once and distributing to all required systems.

Maintain existing IT and analytics integrations:

• Keep connections to ERP, MES, and BI platforms via Kafka connectors.
• Continue using existing dashboards and reports while transitioning to modern analytics platforms.
• Avoid vendor lock-in and enable future migration to cloud or hybrid solutions.

Result?

• Immediate cost savings by reducing reliance on expensive middleware storage and licensing fees.
• No disruption to existing workflows, ensuring continued operational efficiency.
• Scalable, future-ready architecture with the flexibility to expand to edge, cloud, or hybrid environments over time.
• Real-time data streaming capabilities, paving the way for predictive analytics, AI-driven automation, and IoT-driven optimizations.

A lift-and-shift approach serves as a stepping stone toward full OT modernization, allowing enterprises to gradually transition to a fully event-driven, real-time architecture.

3. Full OT Middleware Replacement: Cloud-Native, Scalable, and Future-Proof

Why?

Legacy OT middleware systems were designed for on-premise, batch-based, and proprietary environments, making them expensive, inflexible, and difficult to scale. As industries embrace cloud-native architectures, edge computing, and real-time analytics, replacing traditional OT middleware with event-driven streaming platforms enables greater flexibility, cost efficiency, and real-time operational intelligence.

A full OT middleware replacement eliminates vendor lock-in, outdated integration methods, and high-maintenance costs while enabling scalable, event-driven data processing that works across edge, on-premise, and cloud environments.

How?

Use Kafka and Flink as the Core Data Streaming Platform

• Kafka replaces legacy data brokers and middleware storage by handling high-throughput event ingestion and real-time data distribution.
• Flink provides advanced real-time analytics, anomaly detection, and predictive maintenance capabilities.
• Process OT and IT data in real-time, eliminating batch-based limitations.

Replace Proprietary Connectors with Lightweight, Open Standards

• Deploy MQTT or OPC UA gateways to enable seamless communication with sensors, PLCs, SCADA, and industrial controllers.
• Eliminate complex, costly middleware like OSIsoft PI with low-latency, open-source integration.
• Leverage Apache PLC4X for industrial protocol connectivity, avoiding proprietary vendor constraints.

Adopt a Cloud-Native, Hybrid, or On-Premise Storage Strategy

• Store time-series data in scalable, purpose-built databases like InfluxDB or TimescaleDB.
• Enable real-time query capabilities for monitoring, analytics, and AI-driven automation.
• Ensure data availability across on-premise infrastructure, hybrid cloud, and multi-cloud deployments.

Modernize IT and Business Integrations

• Enable seamless OT-to-IT integration with ERP, MES, BI, and AI/ML platforms.
• Stream data directly into cloud-based analytics services, digital twins, and AI models.
• Build real-time dashboards and event-driven applications for operators, engineers, and business stakeholders.

Result?

• Fully event-driven and cloud-native OT architecture that eliminates legacy bottlenecks.
• Real-time data streaming and processing across all industrial environments.
• Scalability for high-throughput workloads, supporting edge, hybrid, and multi-cloud use cases.
• Lower operational costs and reduced maintenance overhead by replacing proprietary, heavyweight OT middleware.
• Future-ready, open, and extensible architecture built on Kafka, Flink, and industry-standard protocols.

By fully replacing OT middleware, organizations gain real-time visibility, predictive analytics, and scalable industrial automation, unlocking new business value while ensuring seamless IT/OT integration.

Helin is an excellent example for a cloud-native IT/OT data solution powered by Kafka and Flink to focus on real-time data integration and analytics, particularly in the context of industrial and operational environments. Its industry focus on maritime and energy sector, but this is relevant across all IIoT industries.

Why This Matters: The Future of OT is Real-Time & Open for Data Sharing

The next generation of OT architectures is being built on open standards, real-time streaming, and hybrid cloud.

• Most new industrial sensors, machines, and control systems are now designed with Kafka, MQTT, and OPC UA compatibility.
• Modern IT architectures demand event-driven data pipelines for AI, analytics, and automation.
• Edge and hybrid computing require scalable, fault-tolerant, real-time processing.

Use Kafka Cluster Linking for seamless bi-directional data replication and command&control, ensuring low-latency, high-availability data synchronization across on-premise, edge, and cloud environments.

Enable multi-region and hybrid edge to cloud architectures with real-time data mirroring to allow organizations to maintain data consistency across global deployments while ensuring business continuity and failover capabilities.

It’s Time to Move Beyond Legacy OT Middleware to Open Standards like MQTT, OPC-UA, Kafka

The days of expensive, proprietary, and rigid OT middleware are numbered (at least for new deployments). Industrial enterprises need real-time, scalable, and open architectures to meet the growing demands of automation, predictive maintenance, and industrial IoT. By embracing open IoT and data streaming technologies, companies can seamlessly bridge the gap between Operational Technology (OT) and IT, ensuring efficient, event-driven communication across industrial systems.

MQTT, OPC-UA and Apache Kafka are a match in heaven for industrial IoT:

• MQTT enables lightweight, publish-subscribe messaging for industrial sensors and edge devices.
• OPC-UA provides secure, interoperable communication between industrial control systems and modern applications.
• Kafka acts as the high-performance event backbone, allowing data from OT systems to be streamed, processed, and analyzed in real time.

Whether lifting and shifting, optimizing hybrid processing, or fully replacing legacy middleware, data streaming is the foundation for the next generation of OT and IT integration. With Kafka at the core, enterprises can decouple systems, enhance scalability, and unlock real-time analytics across the entire industrial landscape.

Stay ahead of the curve! Subscribe to my newsletter for insights into data streaming and connect with me on LinkedIn to continue the conversation. And make sure to download my free book about data streaming use cases and industry success stories.

The post Modernizing OT Middleware: The Shift to Open Industrial IoT Architectures with Data Streaming appeared first on Kai Waehner.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch.

Siemens Healthineers: Shaping the Future of Healthcare Technology

Who They Are

Siemens AG, a global powerhouse in industrial manufacturing, energy, and technology, has been a leader in innovation for over 170 years. Known for its groundbreaking contributions across sectors, Siemens combines engineering expertise with digitalization to shape industries worldwide. Within this ecosystem, Siemens Healthineers stands out as a pivotal player in healthcare technology.

With over 71,000 employees operating in 70+ countries, Siemens Healthineers supports critical clinical decisions in healthcare. Over 90% of leading hospitals worldwide collaborate with them, and their technologies influence over 70% of critical clinical decisions.

Their Vision

Siemens Healthineers focuses on innovation through data and AI, aiming to streamline healthcare delivery. With more than 24,000 technical intellectual property rights, including 15,000 granted patents, their technological foundation enables precision medicine, enhanced diagnostics, and patient-centric solutions.

Siemens Healthineers and Data Streaming for Healthcare and Manufacturing

Siemens is a large conglomerate. I already covered a few data streaming use cases at other Siemens divisions. For instance, the integration project from SAP ERP on-premise to Salesforce CRM in the cloud.

At the Data in Motion Tour 2024 in Frankfurt, Arash Attarzadeh (“Apache Kafka Jedi“) from Siemens Heathineers presented various very interesting success stories that leverage data streaming using Apache Kafka, Flink, Confluent, and its entire ecosystem.

Why Data Streaming with Apache Kafka and Flink?

Healthcare and manufacturing processes generate massive volumes of real-time data. Whether it’s monitoring devices on production floors, analyzing telemetry data from hospitals, or optimizing logistics, Siemens Healthineers recognizes that data streaming enables:

• Real-time insights: Immediate and continuously action on events as they happen.
• Improved decision-making: Faster and more accurate responses.
• Cost efficiency: Reduced downtime and optimized operations.

Healthineers Data Cloud

The Siemens Healthineers Data Cloud serves as the backbone of their data strategy. Built on a robust technology stack, it facilitates real-time data ingestion, transformation, and analytics using tools like Confluent Cloud (including Apache Kafka and Flink) and Snowflake.

This combination of leading SaaS solutions enables seamless integration of streaming data with batch processes and diverse analytics platforms.

Technology Stack: Healthineers Data Cloud

Key Components

• Confluent Cloud (Apache Kafka): For real-time data ingestion, data integration and stream processing.
• Snowflake: A centralized warehouse for analytics and reporting.
• Matillion: Batch ETL processes for structured and semi-structured data.
• IoT Data Integration: Sensors and PLCs collect data from manufacturing floors, often via MQTT.

Many other solutions are critical for some use cases. Siemens Healthineers also uses Databricks, dbt, OPC-UA, and many other systems for the end-to-end data pipelines.

Diverse Data Ingestion

• Real-Time Streaming: IoT data (sensors, PLCs) is ingested within minutes.
• Batch Processing: Structured and semi-structured data from SAP systems.
• Change Data Capture (CDC): Data changes in SAP sources are captured and available in under 30 minutes.

Not every data integration process is or can be real-time. Data consistency is still one of the most underrated capabilities of data streaming. Apache Kafka supports real-time, batch and request-response APIs communicating with each other in a consistent way.

Use Cases for Data Streaming at Siemens Healthineers

Siemens Healthineers described six different use cases that leverage data streaming together with various other IoT, software and cloud services:

1. Machine monitoring and predictive maintenance
2. Data integration layer for analytics
3. Machine and robot integration
4. Telemetry data processing for improved diagnostics
5. Real-time logistics with SAP events for better supply chain efficiency
6. Track and Trace Orders for improved customer satisfaction and ensured compliance

Let’s take a look at them in the following subsections.

1. Machine Monitoring and Predictive Maintenance in Manufacturing

Goal: To ensure the smooth operation of production devices through predictive maintenance.

Using data streaming, real-time IoT data from drill machines is ingested into Kafka topics, where it’s analyzed to predict maintenance needs. By using a TensorFlow machine learning model for infererence with Apache Kafka, Siemens Healthineers can:

• Reduce machine downtime.
• Optimize maintenance schedules.
• Increase productivity in manufacturing CT scanners.

Business Value: Predictive maintenance reduces operational costs and prevents production halts, ensuring timely delivery of critical medical equipment.

2. IQ-Data Intelligence from IoT and SAP to Cloud

Goal: Develop an end-to-end data integration layer for analytics.

Data from various lifecycle phases (e.g., SAP systems, IoT interfaces via MQTT using Mosquitto, external sources) is streamed into a consistent model using stream processing with ksqlDB. The resulting data backend supports the development of MLOps architectures and enables advanced analytics.

Business Value: Streamlined data integration accelerates the development of AI applications, helping data scientists and analysts make quicker, more informed decisions.

3. Machine Integration with SAP and KUKA Robots

Goal: Integrate machine data for analytics and real-time insights.

Data from SAP systems (such as SAP ME and SAP PCO) and machines like KUKA robots is streamed into Snowflake for analytics. MQTT brokers and Apache Kafka manage real-time data ingestion and facilitate predictive analytics.

Business Value: Enhanced machine integration improves production quality and supports the shift toward smart manufacturing processes.

4. Digital Healthcare Service Operations using Data Streaming

Goal: Stream telemetry data from Siemens Healthineers products for analytics.

Telemetry data from hospital devices is streamed via WebSockets to Kafka and combined with ksqlDB for continuous stream processing. Insights are fed back to clients for improved diagnostics.

Business Value: By leveraging real-time device data, Siemens Healthineers enhances the reliability of its medical equipment and improves patient outcomes.

5. Real-Time Logistics with SAP Events and Confluent Cloud

Goal: Stream SAP logistics event data for real-time packaging and shipping updates.

Using Confluent Cloud, Siemens Healthineers reduces delays in packaging and shipping by enabling real-time insights into logistics processes.

Business Value: Improved packaging planning reduces delivery times and enhances supply chain efficiency, ensuring faster deployment of medical devices.

6. Track and Trace Orders with Apache Kafka and Snowflake

Goal: Real-time order tracking using streaming data.

Data from Siemens Healthineers orders is streamed into Snowflake using Kafka for real-time monitoring. This enables detailed tracking of orders throughout the supply chain.

Business Value: Enhanced order visibility improves customer satisfaction and ensures compliance with regulatory requirements.

Real-Time Data as a Catalyst for Healthcare and Manufacturing Innovation at Siemens Healthineers

Siemens Healthineers’ innovative use of data streaming exemplifies how real-time insights can drive efficiency, reliability, and innovation in healthcare and manufacturing. By leveraging tools like Confluent (including Apache Kafka and Flink), MQTT and Snowflake and transitiing some workloads to the cloud, they’ve built a robust infrastructure to handle diverse data streams, improve decision-making, and deliver tangible business outcomes.

From predictive maintenance to enhanced supply chain visibility, the adoption of data streaming unlocks value at every stage of the production and service lifecycle. For Siemens Healthineers, these advancements translate into better patient care, streamlined operations, and a competitive edge in the dynamic healthcare industry.

To learn more about the relationship between these key technologies and their applications in different use cases, explore the articles below:

• Data Streaming with Apache Kafka and MQTT – A Blog Series about Use Cases
• Apache Kafka and Snowflake – Friends, NOT Enemies
• Machine Learning / AI and Data Streaming in the Healthcare Sector

Do you have similar use cases and architectures like Siemens Healthineers to leverage data streaming with Apache Kafka and Flink in the healthcare and manufacturing sector? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post How Siemens Healthineers Leverages Data Streaming with Apache Kafka and Flink in Manufacturing and Healthcare appeared first on Kai Waehner.

Cardinal Health is at the forefront of leveraging real-time data streaming to transform healthcare and manufacturing operations. With Apache Kafka and Confluent Cloud, Cardinal Health has modernized its legacy systems, enhanced real-time analytics, and improved efficiency across its Pharma and Med divisions. This blog explores Cardinal Health’s journey, exploring how its event-driven architecture powers use cases like supply chain optimization, and medical device and equipment management. By integrating data streaming with platforms like Apigee, Dell Boomi and SAP, Cardinal Health sets a benchmark for IT modernization and innovation in the healthcare and pharma sectors.

Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter and follow me on LinkedIn or X (former Twitter) to stay in touch.

Data Streaming in Healthcare and Pharma

Many healthcare companies leverage Apache Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap.

I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

• Data Streaming Use Cases and Architectures for Healthcare (including Slide Deck)
• Legacy Modernization and Hybrid Cloud (Optum / UnitedHealth Group, Centene, Bayer)
• Streaming ETL (Bayer, Babylon Health)
• Real-time Analytics (Cerner, Celmatix, CDC/Centers for Disease Control and Prevention)
• Machine Learning and Data Science (Recursion, Humana)
• Open API and Omnichannel (Care.com, Invitae)

Apache Kafka and Flink for Stream Processing and Analytics

Apache Kafka and Apache Flink enable continuous stream processing and analytics in healthcare and pharma, providing the ability to ingest, process, and analyze massive volumes of data with low latency.

These technologies empower use cases such as real-time patient monitoring, personalized medicine, supply chain optimization, and regulatory compliance, ensuring timely insights and operational efficiency.

The State of Data Streaming in Healthcare and Pharma

Recently, I did a webinar that explores the healthcare industry’s trends and architectures for data streaming. The primary focus is the data streaming architectures and case studies.

Check out the on-demand recording:

Cardinal Health’s Data Streaming Adoption

Cardinal Health is an American multinational health care services company, and the 14th highest revenue generating company in the United States with 48,000 employees. It plays a critical role in healthcare by distributing pharmaceuticals, manufacturing medical products, and providing data solutions to healthcare facilities.

With operations in over 30 countries and a presence in 90% of U.S. hospitals, Cardinal Health’s ability to innovate is directly tied to its extensive use of data. Their journey into event-driven architecture (EDA) with data streaming using Apache Kafka and Confluent Cloud has been a transformative step in achieving real-time operational excellence and enabling scalable digital transformation.

At the data streaming conference Current 2024 in Austin, Texas, Cardinal Health presented how they “devised their enablement strategy and proceeded to build an ecosystem of information and learning support to entice leaders, architects, and developers to get on the EDA-with-Confluent-Cloud-Kafka bandwagon“.

Data streaming is a journey. Cardinal Health is an excellent success story about adopting a fully managed SaaS data streaming platform in the cloud, learning from technical and organizations challenges and best practices.

The Challenge: Modernizing Data Management

Cardinal Health faced significant challenges with outdated middleware technologies and a data-at-rest approach reliant on batch processing. These limitations led to inefficiencies, such as delayed updates on order statuses and manual interventions in customer service.

Recognizing the need for change, the company sought a modern solution to transform its data management and address the growing demand for real-time insights.

The Strategy: Adopting Event-Driven Architecture

In 2022, Cardinal Health implemented a strategy to shift from traditional data processing methods to an event-driven architecture powered by Kafka via the fully-managed Confluent Cloud SaaS solution.

The goal was not just to deploy a new tool, but to change how data was managed, shared, and leveraged across the organization. This involved:

• Establishing an event-driven architecture (EDA) team to act as consultants for business units, guiding them on how best to use Kafka and data streaming.
• Partnering with Confluent Professional Services to create a scalable and secure platform tailored to Cardinal Health’s needs.
• Integrating change management strategies to ensure teams adopted EDA successfully and moved beyond familiar legacy systems.

Building Organizational Capability

Key to Cardinal Health’s success was an intentional focus on learning and performance enablement:

1. Comprehensive Training Programs: Cardinal Health created an ecosystem of training resources, including live events, asynchronous content, and structured learning paths. These were designed to cater to diverse roles and skill levels across the organization, from business users to developers.
2. Agile Learning Development: By adopting agile methodologies, the training team rapidly delivered concise, modular content that could be easily updated and scaled.
3. Cross-Team Engagement: Events like tech talks and interactive sessions with internal and external experts fostered a culture of continuous learning and collaboration.

Results and Business Impact

In just 17 months, Cardinal Health expanded its use of Kafka from a single topic to over 58 applications across its pharmaceutical and medical product divisions. This rapid growth reflects the organization’s commitment to scaling its digital capabilities.

Key outcomes include:

• Improved Efficiency: Real-time data streaming has streamlined operations, reducing manual interventions and ensuring timely updates for both internal teams and customers.
• Scalability: The cloud-native and elastic platform allows Cardinal Health to handle peak loads seamlessly, such as during high-demand periods.
• Cost Savings: The adoption of EDA has optimized resource use, resulting in significant cost reductions.
• Enhanced Customer Experience: Real-time transparency into order statuses and operational data has improved customer satisfaction and reduced support calls.

Lessons Learned and Best Practices

Cardinal Health’s journey provides valuable insights for organizations embarking on similar transformations:

• Define a Clear Strategy: Aligning technical initiatives with business goals ensures meaningful impact and sustained investment.
• Partner with Experts: Collaborating with Confluent and other professionals accelerated Cardinal Health’s deployment and knowledge-building efforts.
• Invest in Training: A robust training and enablement ecosystem empowers teams to adopt new technologies effectively and confidently.
• Engage Stakeholders: Frequent communication and recognition programs foster a culture of adoption and innovation.

Cardinal Health’s Pharma and Med Use Cases for Data Streaming

A wide range of use cases across the Pharma and Med divisions was implemented by Cardinal Health leveraging data sharing across the event-driven architecture (EDA). Real-time data to improve operational efficiency, enhances customer experiences, and drive business agility.

The data streaming platform CHEDA (Cardinal Health Event-Driven Architecture) integrates and shares data across business units for unified insights and operations:

Cardinal Health’s event-driven architecture is central to enabling data sharing and real-time responsiveness, boosting operational efficiency, and providing a seamless customer experience while keeping the systems truly decoupled from each other. By integrating EDA across its Pharma, Med and other divisions, the organization has modernized its data infrastructure, paving the way for future innovation and growth.

The next two subsections explore the use cases and integrated platforms for Cardinal Health’s Pharma and Med divisions. Cardinal Health has seamlessly integrated Confluent with other data integration and application integration platforms, respectively API gateways, such as Apigee, Dell Boomi, Axway, and Qlik Attunity. Keep in mind that data streaming with Kafka and Flink is complementary to other integration platforms. Data streaming is its own software category and not the same as an ETL tool, Enterprise Service Bus (ESB) or Integration Platform as a Service (iPaaS).

Pharma: Distribution of Pharmaceuticals Supporting Retail Pharmacies, Hospitals and Specialty Care Providers

Cardinal Health’s Pharma division focuses on the distribution of pharmaceuticals, supporting retail pharmacies, hospitals, and specialty care providers with a robust supply chain network. This division also provides solutions like specialty drug management, contract handling, and real-time analytics to optimize operations and improve patient outcomes.

Use cases for data streaming with Cardinal Health Event-Driven Architecture in the pharma division include:

• Generics and Flu Vaccination Management
• Therapy and Specialty Platforms
• Contract and Chargeback Management
• E-commerce and Ordering
• Data Analytics and Reporting
• Warehouse and Logistics

Integrated platforms include JARVIS/Vantus, IBM, Rimsys RIMS, Veeva Vault QMS, Nuctrac, Palantir, PRISM, ISOTRAK, and many more.

Med: Manufacturing and Distribution of Medical Devices, Supplies and Laboratory Products for Healthcare Facilities

The Med division of Cardinal Health specializes in the manufacturing and distribution of medical devices, surgical supplies, and laboratory products for healthcare facilities. It also offers integrated solutions for inventory management, compliance, and logistics to enhance efficiency and ensure timely delivery of critical medical products.

Use cases for data streaming with Cardinal Health Event-Driven Architecture in the med division include:

• Medical Device and Equipment Management
• Regulatory and Compliance Solutions
• Ordering and Customer Systems
• Supply Chain Monitoring
• Back-office Operations

Integrated platforms include Edgepark Data Warehouse, SAP ECC, SAP MDG (Master Data Governance), SAP S/4Hana, Kodiak Canada ERP, Edgepark, Vastera Tradephere, and many more.

Cardinal Health: A Data-Driven Transformation in Healthcare and Pharma

Cardinal Health shows the transformative power of data streaming and event-driven architecture (EDA) in healthcare and pharma. By modernizing legacy systems and deploying real-time data streaming across its Pharma and Med divisions, Cardinal Health has enhanced efficiency, improved supply chain management, enabled seamless data sharing across business units and applications, and delivered better customer experiences.

With use cases ranging from Generics and Flu Vaccination Management to Therapy and Specialty Platforms, and optimizing supply chain operations, Cardinal Health’s EDA platform has become a foundation for data sharing and innovation. By connecting to other API gateways and data integration platforms such as Axway or Apigee and industry-specific applications such as Edgepark, PRISM or Nuctrac, the company has built a robust ecosystem that supports real-time responsiveness and operational efficiency across business units. Cardinal Health’s journey showcases how embracing real-time data streaming can drive significant value across critical healthcare and pharma operations.

How do you leverage data streaming with Apache Kafka and Flink in the healthcare and pharma sector? How does your enterprise architecture look like? What data platforms and applications do you interconnect? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Data Streaming in Healthcare and Pharma: Use Cases and Insights from Cardinal Health appeared first on Kai Waehner.

Data Streaming and Industrial IoT / Industry 4.0

Data streaming with Apache Kafka plays a critical role in Industrial IoT by enabling real-time data ingestion, processing, and analysis from various industrial devices and sensors. Kafka’s high throughput and scalability ensure that it can reliably handle and integrate massive streams of data from IoT devices into analytics platforms for valuable insights. This real-time capability enhances predictive maintenance, operational efficiency, and overall automation in industrial settings.

Here is an exemplary hybrid industrial IoT architecture with data streaming at the edge in the factory and 5G supply chain environments synchronizing in real-time with business applications and analytics / AI platforms in the cloud:

Brose – A Global Automotive Supplier

Brose is a global automotive supplier headquartered in beautiful Franconia, Bavaria, Germany. The company has a global presence with 70 locations, 25 countries, 5 continents, and about 30,000 employees.

Brose specializes in mechatronic systems for vehicle doors, seats, and electric motors. They develop and manufacture innovative products that enhance vehicle comfort, safety, and efficiency, serving major car manufacturers worldwide.

Brose’s Hybrid Architecture for Industry 4.0 with Eclipse Kura, OPC UA, Kafka, SAP and MuleSoft

Brose is an excellent example of combining data streaming using Confluent with other technologies like open source Eclipse Kura and OPC-UA for the OT and edge site, and IT infrastructure and cloud software like SAP, Splunk, SQL Server, AWS Kinesis and MuleSoft:

Here is how it works according to Sven Matuschzik, Director of IT-Platforms and Databases at Brose:

Regional Kafka on-premise clusters are embedded within the IIoT and production platform, facilitating seamless data flow from the shop floor to the business world in combination with other integration tools. This hybrid IoT streaming architecture connects machines to the IT infrastructure, mastering various technologies, and ensuring zero trust security with micro-segmentation. It manages latencies between sites and central IT, enables two-way communication between machines and the IT world, and maintains high data quality from the shop floor.

For more insights from Brose (and Siemens) about IoT and data streaming with Apache Kafka, listen to the following interactive discussion.

Interactive Discussion with Siemens and Brose about Data Streaming and IoT

Brose and Siemens discussed with me

• the practical strategies employed by Brose and Siemens to integrate data streaming in IoT for real-time data utilization.
• the challenges faced by both companies in embracing data streaming, and reveal how they overcame barriers to maximize their potential with a hybrid cloud infrastructure.
• how these enterprise architectures will be expanded, including real-time data sharing with customers, partners, and suppliers, and the potential impact of artificial intelligence (AI), including GenAI, on data streaming efforts, providing valuable insights to drive business outcomes and operational efficiency.
• the significance of event-driven architectures and data streaming for enhanced manufacturing processes to improve overall equipment effectiveness (OEE) and seamlessly integrate with existing IT systems like SAP ERP and Salesforce CRM to optimize their operations.

Here is the video recording with Stefan Baer from Siemens and Sven Matuschzik from Brose:

Data Streaming with Apache Kafka to Unify Industrial IoT Workloads from Edge to Cloud with Apache Kafka

Many manufacturers leverage data streaming powered by Apache Kafka to unify the OT/IT world from edge sites like factories to the data center or public cloud for analytics and business applications.

I wrote a lot about data streaming with Apache Kafka and Flink in Industry 4.0, Industrial IoT and OT/IT modernization. Here are a few of my favourite articles:

• The State of Data Streaming in Manufacturing
• How Michelin improves Aftermarket Sales and Customer Service with Data Streaming
• Apache Kafka for Synchronization from Smart Factory to Cloud at BMW
• Modernizing SCADA Systems and OT/IT Integration with Data Streaming at 50Hertz
• OPC UA, MQTT, and Apache Kafka – The Trinity of Data Streaming in IoT
• Apache Kafka as Data Historian – an IIoT / Industry 4.0 Real Time Data Lake

How does your IoT architecture look like? Do you already use data streaming? What are the use cases and enterprise architecture? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka in Manufacturing at Automotive Supplier Brose for Industrial IoT Use Cases appeared first on Kai Waehner.

The State of Data Streaming for Manufacturing in 2023

The evolution of industrial IoT, manufacturing 4.0, and digitalized B2B and customer relations require modern, open, and scalable information sharing. Data streaming allows integrating and correlating data in real-time at any scale. Trends like software-defined manufacturing and data streaming help modernize and innovate the entire engineering and sales lifecycle.

I have recently presented an overview of trending enterprise architectures in the manufacturing industry and data streaming customer stories from BMW, Mercedes, Michelin, and Siemens. A complete slide deck and on-demand video recording are included:

This blog post explores one of the enterprise architectures and case studies in more detail: Context-specific aftersales and service management in real-time with data streaming.

What is Aftermarket Sales and Service? And how does Data Streaming help?

The aftermarket is the secondary market of the manufacturing industry, concerned with the production, distribution, retailing, and installation of all parts, chemicals, equipment, and accessories after the product’s sale by the original equipment manufacturer (OEM) to the consumer. The term ‘aftermarket’ is mainly used in the automotive industry but as relevant in other manufacturing industries.

“Aftermarket sales and service are vital to manufacturers’ strategies” according to McKinsey.

Enterprises leverage data streaming for collecting data from cars, dealerships, customers, and many other backend systems to make automated context-specific decision-making in real-time when it is relevant (predictive maintenance) or valuable (cross-/upselling).

Challenges with Aftermarket Customer Communication

Manufacturers face many challenges when seeking to implement digital tools for aftermarket services. McKinsey defined research points to five central priorities – all grounded in data – for improving aftermarket services: People, operations, offers, a network of external partners, and digital tools.

While these priorities are related, digitalization is relevant across all business processes in aftermarket services:

Disclaimer: The McKinsey research focuses on aerospace and defense, but the challenges look very similar in other industries, in my experience from customer conversations.

Data Streaming to make Context-specific Decisions in Real-Time

“The newest aftermarket frontier features the robust use of modern technological developments such as advanced sensors, big data, and artificial intelligence.” says McKinsey.

Data streaming helps transform the global supply chain, including aftermarket business processes, with real-time data integration and correlation to make context-specific decisions.

McKinsey mentioned various digital tools that are valuable for aftermarket services:

Interestingly, this coincides with what I see from applications built with data streaming. One key reason is that data streaming with Apache Kafka enablement data consistency across real-time and non-real-time applications.

Omnichannel retail and aftersales are very challenging for most enterprises. That’s why many enterprise architectures rely on data streaming for their context-specific customer 360 infrastructure and real-time applications.

Michelin: Context-specific Aftermarket Sales and Customer Service

Michelin is a French multinational tire manufacturing company for almost every type of vehicle. The company sells a broad spectrum of tires. They manufacture products for automobiles, motorcycles, bicycles, aircraft, space shuttles, and heavy equipment.

Michelin’s many inventions include the removable tire, the ‘pneurail’ (a tire for rubber-tired metros), and the radial tire.

Michelin presented at Kafka Summit how they moved from monolithic orchestrator to data streaming with microservices. This project was all about replacing a huge and complex Business Process Management tool (Oracle BPM), an orchestrator of their internal logistic flows.

And when Michelin says huge, they really mean it: over 24 processes, 150 millions of tyres moved representing 10 billions € of Michelin turnover. So why replacing such a critical component in their Information System? Mainly because “it was built like a monolithic ERP and became difficult to maintain, not to say a potential single point of failure”. Michelin replaced it with a choreography of micro-services around our Kafka cluster.

From spaghetti integration to decoupled microservices

Michelin faced the same challenges as most manufacturers: Slow data processing, conflicting information, and complex supply chains. Hence, Michelin moved from a spaghetti integration architecture and batch processing to decoupled microservices and real-time event-driven architecture.

They optimized unreliable and outdated reporting on inventory, especially for raw and semi-finished materials by connecting various systems across the supply chain, including DRP, TMS, ERP, WMS, and more. Apache Kafka provides the data fabric for data integration and to ensure truly decoupled and independent microservices.

From human processes to predictive mobility services

However, the supply chain does not end with manufacturing the best tires. Michelin aims to provide the best services and customer experience via data-driven analytics. As part of this initiative, Michelin migrated from orchestration and a single point of failure with a legacy BPM engine to a flexible choreography and true decoupling with an event-driven architecture leveraging Apache Kafka:

Michelin implemented mobility solutions to provide mobility assistance and fleet services to its diverse customer base. For instance, predictive insights notify customers to replace tires or show the best routes to optimize fuel. The new business process choreography enables proactive marketing and aftersales. Context-specific customer service is possible as the event-driven architecture gives access to the right data at the right time (e.g. when the customer calls the service hotline).

The technical infrastructure is based on cloud-native technologies such as Kubernetes (elastic infrastructure), Apache Kafka (data streaming with components like Kafka Connect and Kafka Streams), and Zeebe (a modern BPM and workflow engine).

From self-managed operations to fully managed cloud

Michelin’s commercial supply chain spans 170 countries. Michelin relies on a real-time inventory system to efficiently manage the flow of products and materials within their massive network.

A strategic decision was the move to a fully managed data streaming service to focus on business logic and innovation in manufacturing, after-sales, and service management. The migration of self-managed Kafka to Confluent Cloud cut operations costs by 35%.

Many companies replace existing legacy BPM engines with workflow orchestration powered by Apache Kafka.

Lightboard Video: How Data Streaming improves Aftermarket Sales and Customer Service

Here is a five-minute lightboard video that describes how data streaming helps with modernizing non-scalable and inflexible data infrastructure for improving the end-to-end supply chain, including aftermarket sales and customer service:

If you liked this video, make sure to follow the Confluent YouTube channel for many more lightboard videos across all industries.

Apache Kafka for automated business processed and improved aftermarket

The Michelin case study explored how a manufacturer improved the end-to-end supply chain from production to aftermarket sales and customer service. For more case studies, check out the free “The State of Data Streaming in Manufacturing” on-demand recording or read the related blog post.

Critical aftermarket sales and customer services challenges are missing information, rising costs, customer churn, and decreasing revenue. Real-time monitoring and context-specific decision-making improve the customer journey and retention. Learn more by reading how data streaming enables building a control tower for real-time supply chain operations.

How do you leverage data streaming in your aftermarket use cases for sales and service management? Did you already build a real-time infrastructure across your supply chain? Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post How Michelin improves Aftermarket Sales and Customer Service with Data Streaming appeared first on Kai Waehner.

General trends in the manufacturing industry

Researchers, analysts, startups, and last but not least, labs and the first real-world rollouts of traditional players show a few upcoming trends in the manufacturing industry:

• Software-defined manufacturing (see the “SDM4FZI” research project of the German Federal Ministry of Economics and Climate Protection, BMWK)
• Industry 4.0 and closed-loop manufacturing (see a great video recording from the research company “IoT Analytics”
• Automation and robotics (see a recent McKinsey report from January 2023)

Let’s explore the goals and impact of these trends.

Software-defined manufacturing

Software-defined manufacturing for the vehicle and supplier industry (SDM4FZI) is a German research project exploring the vision and future of significantly digitalized and automated manufacturing processes across the supply chain.

The four fundamental principles are on the wish list of every factory manager and shop floor team:

• Versatile: Highly versatile production systems with simultaneous control of complexity through virtualization
• Continuous: Continuity of the data chain from planning to operation (life cycle) and from the sensor to the cloud
• Automated: Fully automated software development process from requirements analysis and testing to distribution and execution in the digital factory
• Optimized: Risk-free optimization of applications in the virtual factory before deployment in the real world

The primary goal is a lot size of 1 and flexible, adaptive manufacturing leveraging digitalization with digital twins along the entire engineering and production process.

Industry 4.0 and closed-loop manufacturing

Closed-loop manufacturing means the synchronization of product lifecycle activities with product performance. The research company IoT Analytics describes the principle in a nice diagram:

Information is connected and shared bi-directionally across the product journey from design and engineering across the entire production line up to aftersales. The following video explores the details:

Unlocking the industrial potential of robotics and automation

Industrial companies are set to spend heavily on robotics and automation. Industrial-company executives expect benefits in output quality, efficiency, and uptime. McKinsey published a great analysis of the investment and benefits of robotics and automation:

Robotics and automation implicitly include digitalizing the entire manufacturing process: the shop floor level production line and the integration with MES, ERP, and the rest of the IT world, like analytics, business intelligence, customer 360, etc.

Data streaming in manufacturing

Adopting trends like software-defined manufacturing and robotic automation is only possible if manufacturers can provide and correlate information at the right time in the proper context. Real-time, which means using the information in milliseconds, seconds, or minutes, is almost always better than processing data later (whatever later means):

Data streaming combines the power of real-time messaging at any scale with storage for true decoupling, data integration, and data correlation capabilities. Apache Kafka is the de facto standard for data streaming.

“Apache Kafka in Manufacturing and Industry 4.0” is a good article for starting with an industry-specific point of view on data streaming.

Just-in-time manufacturing with data streaming

An adaptive manufacturing strategy starts with real-time data:

• Just-in-time (JIT) vs. make to forecast
• Fixed vs. variable price contracts
• Build vs. buy plant capacity
• Staffed vs. lights-out third shift
• Linking vs. not linking prices for materials and finished goods

Data streaming shines for such a use case as it can take real-time information of robots or sensors and correlate it with data of MES and ERP systems. For instance, almost every customer I talk to integrates Kafka with the SAP ecosystem.

This is just one example. Data streaming with the Apache Kafka ecosystem and cloud services are used throughout the manufacturing process, supply chain, and after-sales. Search my blog for various articles related to this topic: Search Kai’s blog.

Top benefits of event stream processing for manufacturing

Ventana Research defined the top data and application areas (excluding analytics) for event stream processing. They explored the top benefits:

Architecture Trends

The manufacturing applies various trends for enterprise architectures for cost, flexibility, security, and latency reasons. The three major topics I see these days at customers are:

• IT deployments everywhere (edge, hybrid, multi-cloud)
• Real-time analytics at the shop floor level
• Data consistency across real-time and batch systems

Everywhere: Data streaming across edge and cloud

The following diagram shows an architecture from the shipping industry. IT workloads run where it makes most sense, like fully managed in the cloud, traditional software in the data center, IT workloads at the edge, or embedded into a machine or IoT device:

Data synchronization between the different things, sites, and regions is critical. Data streaming enables a reliable and scalable real-time replication between different technologies, APIs, and communication paradigms.

Analytics: Machine learning applied to the shop floor and IoT workloads

Data streaming enables many use cases from OT and shop floor level to customer-facing aftersales and service management. One of the most critical scenarios is low-latency analytics for condition monitoring and predictive maintenance.

Straightforward stream processing in stateless applications is already powerful. The following diagram shows condition monitoring for temperature spikes in real time:

More advanced stateful streaming analytics allows continuous aggregation and correlation of events from one or more machines, for instance, to create a constant sliding window over the last 60 minutes to detect anomalies:

Such a scenario can implement business logic or even deploy analytic models that the data science team trained in the cloud with data streaming and machine learning in the data lake.

Data consistency: Integration and correlation between real-time and batch systems

More and more people understand that a real differentiator of data streaming is its capability to work with real time, batch, and request-response APIs at the same time to ensure data consistency across different technologies and communication paradigms:

Not every system is or will be real time. The storage capability of the commit log appends data and stores it for as long as needed (hours, days, years). Besides real-time consumption, batch applications and web APIs can reprocess historical data in guaranteed order for reporting/analytics or interactive queries.

New customer stories for data streaming in manufacturing

So much innovation is happening in the manufacturing industry. Automation and digitalization change how car makers, energy providers, and other enterprises leverage data to increase overall equipment effectiveness (OEE) and improve the customer experience.

Most enterprises use a cloud-first (but still hybrid) approach to improve time-to-market, increase flexibility, and focus on business logic instead of operating IT infrastructure.

Here are a few customer stories from worldwide manufacturers across industries:

• BMW: From smart factories to the cloud
• Michelin: From shop floor to customer service
• 50Hertz: From monolithic SCADA to cloud-native IoT
• Siemens: From SAP ERP on-premise to Salesforce CRM in the cloud
• Mercedes: From manual business processes to seamless customer 360

Find more details about these case studies in the below slide deck.

Resources to learn more

This blog post is just the starting point. Learn more in the following on-demand webinar recording, the related slide deck, and further resources, including pretty cool lightboard videos about use cases.

On-demand video recording

The video recording explores the manufacturing industry’s trends and architectures for data streaming. The primary focus is the data streaming case studies from BMW, Michelin, 50Hertz, Siemens, and Mercedes:

Slides

If you prefer learning from slides, check out the deck used for the above recording:

Case studies and lightboard videos for data streaming in the manufacturing industry

The state of data streaming for manufacturing in 2023 is fascinating. New use cases and case studies come up every month. This includes better data governance across the entire organization, collecting data from the IoT and OT worlds in real-time, customer service and aftersales, data sharing and B2B partnerships with suppliers and sales partners for new business models, and so many more scenarios.

We recorded lightboard videos showing the value of data streaming simply and effectively. These five-minute videos explore the business value of data streaming, related architectures, and customer stories:

• Manufacturing – An Overview about Data Streaming
• Connecting Smart Factories and the Cloud in Real-Time
• Modernizing SCADA and OT/IT in the Energy Sector
• From On-Premise SAP to Salesforce CRM in the Cloud
• Predictive Maintenance & Condition Monitoring

And this is just the beginning. Every month, we will talk about the status of data streaming in a different industry. Manufacturing was the first. Next is financial services, then retail, and so on…

Let’s connect on LinkedIn and discuss it! Join the data streaming community and stay informed about new blog posts by subscribing to my newsletter.

The post The State of Data Streaming for Manufacturing appeared first on Kai Waehner.

What is a SCADA system?

Supervisory control and data acquisition (SCADA) is a control system architecture comprising computers, networked data communications, and graphical user interfaces for high-level supervision of machines and processes. It also covers sensors and other devices, such as programmable logic controllers, which interface with process plants or machinery.

While many people refer to specific commercial products, SCADA is a concept or architecture. It can include various components, functions, and products (from different vendors) on different levels:

Wikipedia has a detailed article explaining the terms, history, components, and functions of SCADA. The evolution describes four generations of SCADA systems:

1. First generation: Monolithic
2. Second generation: Distributed
3. Third generation: Networked
4. Fourth generation: Web-based

The evolution did not stop here. The following explores the 5. generation: Cloud-native and open SCADA systems.

How does Apache Kafka help in Industrial IoT?

Industrial IoT (IIoT) and Industry 4.0 create a few new challenges across industries:

• The need for a much bigger scale
• The demand for real-time information
• Hybrid architectures with mission-critical workloads at the edge and analytics in elastic public cloud infrastructure.
• A flexible Open API culture and data sharing across OT/IT environments, and between partners (e.g., supplier, OEM, and mobility service).

Apache Kafka is unique in its characteristics for IoT infrastructures, being very scalable (for transactional and analytical requirements and SLAs), reliable, and open. Hence, many new Industrial IoT projects adopt Apache Kafka for various use cases, including data hub between OT and IT, global integration of smart factories for analytics, predictive maintenance, customer 360, and many other scenarios.

Cloud-native data historian powered by Apache Kafka (operating at the edge or in the cloud)

Data Historian is a well-known concept in Industrial IoT. It helps to ensure and improve the Overall Equipment Effectiveness (OEE). The term often overlaps with SCADA. Some people even use it as a synonym.

Apache Kafka can be used as a component of a Data Historian to improve the OEE and reduce/eliminate the most common causes of equipment-based productivity loss in manufacturing (aka Six Big Losses):

Continuous real-time data ingestion, processing, and monitoring 24/7 at scale is a crucial requirement for thriving Industry 4.0 initiatives. Data Streaming with Apache Kafka and its ecosystem brings enormous value to implementing these modern IoT architectures.

Let’s explore a concrete example of a cloud-native SCADA system.

50hertz: A case study for a cloud-native SCADA system built with Apache Kafka

50hertz is a transmission system operator for electricity in Germany. The company secures electricity supply to 18 million people in northern and eastern Germany.

The infrastructure must operate 24 hours, seven days a week. Various shift teams and a mission-critical SCADA infrastructure supervise and control the OT systems.

50hertz presented their OT/IT and SCADA modernization leveraging data streaming with Apache Kafka at the Confluent Data in Motion tour 2021. The on-demand video recording is available (the speech is in German, unfortunately).

The Journey of 50hertz in a big picture

Look at this fantastic picture of 50hertz’s digital transformation journey from monolithic and proprietary legacy technology to a modern cloud-native integration platform powered by Kafka to modernize their IoT ecosystem, such as SCADA systems:

Notice the details in the above picture:

• The legacy infrastructure on the left side glues and patches together different components. It almost breaks together. No changes are possible to existing components.
• The new infrastructure on the ride side is based on flexible, standardized containers. It is easy to scale, add, or remove applications. The communication happens via standard sizes and schemas.
• The bridge in the middle shows the journey. This is a brownfield approach where the old and new world has to communicate with each other for many years. Over time, the company can shut down more and more of the legacy infrastructure.

A great example of innovation in the energy sector! Let’s explore the details of building a cloud-native SCADA system with Apache Kafka:

Challenges of the monolithic legacy IoT infrastructure

The old IT/OT infrastructure and SCADA system are monolithic, proprietary, not scalable, and miss open APIs based on standard interfaces:

A very common infrastructure setup. Most existing OT/IT infrastructures have exactly the same challenges. This is how factories and production lines were built in the past decades.

The consequence is inflexibility regarding software updates, hardware changes, security fixes, and no option for scalability or innovation. Applications run in disconnected mode and are air-gapped from the internet because the old Windows servers are not even supported and no longer get security patches.

Digital transformation in the industrial space requires modernization. Legacy infrastructure still needs to be integrated into most scenarios. Not every company starts from scratch like Tesla, building brand new factories that are built with automation and digitalization from scratch.

Cloud-native SCADA with Kafka to enable innovation (and legacy integration)

50hertz next-generation Modular Control Center System (MCCS) leverages a central, scalable, event-based integration platform based on Confluent:

The first four containers include the Supervisory & Control (SCADA), Load Frequency Control (LFC), and Time Series Management & Forecasting applications. Each container can have multiple services/functions that follow the event-based microservices pattern.

50hertz provides central governance for security, protocols, and data schemas (CIM compliant) between platform containers/ modules. The cloud-native 24/7 SCADA system is developed in the cloud and deployed in safety-critical edge environments.

More on data streaming and Industrial IoT

If you want to learn more about real-world case studies, use cases, and technical architectures for data streaming with Apache Kafka in IIoT scenarios, check out these articles:

• Use cases and architectures for Apache Kafka for Industrial IoT and Manufacturing 4.0
• Apache Kafka as Data Historian – an IIoT / Industry 4.0 Real Time Data Lake
• OPC UA, MQTT, and Apache Kafka – The Trinity of Data Streaming in IoT
• Use cases and architecture for Kafka and MQTT
• Apache Kafka Landscape for Industrial IoT in the Automotive Industry
• Data streaming in air-gapped and zero-trust environments
• Real-Time Logistics, Shipping, and Transportation with Apache Kafka
• A Real-Time Supply Chain Control Tower powered by Kafka
• A postmodern ERP system built with Kafka

If this is insufficient, please let me know what else you need to know…

Cloud-native architectures and Open API are the future of Industrial IoT

50hertz is a tremendous real-world case study about the modernization of the OT/IT world. A modern SCADA architecture requires real-time data processing at any scale, true decoupling between data producers and consumers (no matter what API these apps use), and open interfaces to integrate with any other application like MES, ERP, cloud services, and so on.

From the IT side, this is nothing new. The last decade brought up scalable open source technologies like Kafka, Spark, Flink, Iceberg, and many more, plus related fully managed, elastic cloud services like Confluent Cloud, Databricks, Snowflake, and so on.

However, the OT side has to change. Instead of using monolithic legacy systems, unsupported and unstable Windows servers, and proprietary protocols, next-generation SCADA systems need to use the same cloud-native IT systems, adopt modern OT hardware/software combinations, and integrate the old and new world to enable digitalization and innovation in industry verticals like manufacturing, automotive, military, energy, and so on.

What role plays data streaming in your Industrial IoT environments and OT/IT modernization? Do you run everything around Kafka in the cloud or operate hybrid edge scenarios? What tasks does Kafka take over – is it “just” the data hub, or are IoT use cases built with it, too? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post A cloud-native SCADA System for Industrial IoT built with Apache Kafka appeared first on Kai Waehner.

Blog Series – Kafka in Healthcare

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

• Overview – Data Streaming Use Cases and Architectures for Healthcare (including Slide Deck)
• THIS POST: Legacy Modernization and Hybrid Cloud (Optum / UnitedHealth Group, Centene, Bayer)
• Streaming ETL (Bayer, Babylon Health)
• Real-time Analytics (Cerner, Celmatix, CDC/Centers for Disease Control and Prevention)
• Machine Learning and Data Science (Recursion, Humana)
• Open API and Omnichannel (Care.com, Invitae)

Stay tuned for a dedicated blog post for each of these topics as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Subscribe to my newsletter to get an email after each publication (no spam or ads).

Legacy Modernization and Hybrid Multi-Cloud with Kafka

Application modernization benefits from the Apache Kafka ecosystem for hybrid integration scenarios.

Most enterprises require a reliable and scalable integration between legacy systems such as IBM Mainframe, Oracle, SAP ERP, and modern cloud-native applications like Snowflake, MongoDB Atlas, or AWS Lambda.

I already explored “architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments” some time ago:

TL;DR: Various alternatives exist to deploy Apache Kafka across data centers, regions, and continents. There is no single best architecture. It always depends on characteristics such as RPO / RTO, SLAs, latency, throughput, etc.

Some deployments focus on on-prem to cloud integration. Others link Kafka clusters on multiple cloud providers. Technologies such as Apache Kafka’s  MirrorMaker 2, Confluent Replicator, Confluent Multi-Region-Clusters, and Confluent Cluster Linking help build such an infrastructure.

Let’s look at a few real-world deployments in the healthcare sector.

Optum (United Health Group) – Cloud-native Kafka-as-a-Service

Optum is an American pharmacy benefit manager and health care provider. It is a subsidiary of UnitedHealth Group. The Apache Kafka infrastructure is provided as an internal service, centrally managed, and used by over 200 internal application teams.

Optum built a repeatable, scalable, cost-efficient way to standardize data. They leverage the whole Kafka ecosystem:

• Data ingestion from multiple resources (Kafka Connect)
• Data enrichment (Table Joins & Streaming API)
• Aggregation and metrics calculation (Kafka Streams API)
• Sinking data to database (Kafka Connect)
• Near real-time APIs to serve the data

Optum’s Kafka Summit talk explored the journey and maturity curve for their data streaming evolution:

As you can see, the journey started with a self-managed Kafka cluster on-premises. Over time, they migrated to a cloud-native Kubernetes environment and built an internal Kafka-as-a-Service offering. Right now, Optum works on multi-cloud enterprise architecture to deploy across multiple cloud service providers.

Centene – Data Integration for M&A across Infrastructures

Centene is the largest Medicaid and Medicare Managed Care Provider in the US. The healthcare insurer acts as an intermediary for government-sponsored and privately insured healthcare programs. Centene’s mission is to “help people live healthier lives and to help make the health system work better for everyone”.

The critical challenge of Centene is interesting: Growth! Many mergers and acquisitions happened in the last decade: Envolve, HealthNet, Fidelis, and Wellcare.

Data integration and processing at scale in real-time between various systems, infrastructures, and cloud environments is a considerable challenge. Kafka provides Centene with valuable capabilities, as they explained in an online talk:

• Highly scalable
• High autonomy/decoupling
• High availability & data resiliency
• Real-time data transfer
• Complex stream processing

The event-driven integration architecture leverages Apache Kafka with MongoDB:

Bayer – Hybrid Multi-Cloud Data Streaming

Bayer AG is a German multinational pharmaceutical and life sciences company and one of the largest pharmaceutical companies in the world. They leverage Kafka in various use cases and business domains. The following scenario is from Monsanto.

Bayer adopted a cloud-first strategy and started a multi-year transition to the cloud to provide real-time data flows across hybrid and multi-cloud infrastructures.

The Kafka-based cross-data center DataHub facilitated migration and the shift to real-time stream processing. It offers strong enterprise adoption and supports a myriad of use cases. The Apache Kafka ecosystem is the “middleware” to build a bi-directional streaming replication and integration architecture between on-premises data centers and multiple cloud providers:

The Kafka journey of Bayer started on AWS. Afterward, some project teams worked on GCP. In parallel, DevOps and cloud-native technologies modernized the underlying infrastructure. Today, Bayer operates a multi-cloud infrastructure with mature, reliable, and scalable stream processing use cases:

Learn about Bayer’s journey and how they built their hybrid and multi-cloud Enterprise DataHub with Apache Kafka and its ecosystem: Bayer’s Kafka Summit talk.

Data Streaming with Kafka across Hybrid and Multi-cloud Infrastructures

Think about IoT sensor analytics, cybersecurity, patient communication, insurance, research, and many other domains. Real-time data beats slow data in the healthcare supply chain almost everywhere.

This blog post explored the value of data streaming with Apache Kafka to modernize IT infrastructure and build hybrid multi-cloud architectures. Real-world deployments from Optum, Centene, and Bayer showed how enterprises deploy Kafka successfully for different use cases in the enterprise architecture.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Legacy Modernization and Hybrid Multi-Cloud with Kafka in Healthcare appeared first on Kai Waehner.

Here is the entire blog series:

• Data Streaming Use Cases and Architectures for Healthcare (including Slide Deck)
• THIS POST: Legacy Modernization and Hybrid Cloud (Optum / UnitedHealth Group, Centene, Bayer)
• Streaming ETL (Bayer, Babylon Health)
• Real-time Analytics (Cerner, Celmatix, CDC/Centers for Disease Control and Prevention)
• Machine Learning and Data Science (Recursion, Humana)
• Open API and Omnichannel (Care.com, Invitae)

Healthcare – A broad spectrum of very different domains

Health care is the maintenance or improvement of health via the prevention, diagnosis, treatment, amelioration, or cure of disease, illness, injury, and other physical and mental impairments.

Health professionals and allied health fields deliver health care. Medicine, dentistry, pharmacy, midwifery, nursing, optometry, audiology, psychology, occupational therapy, physical therapy, athletic training, and other health professions are all part of health care. It includes work done in providing primary care, secondary care, tertiary care, and public health. Access to health care varies across countries, communities, and individuals, influenced by social and economic conditions, as well as health policies.

The Healthcare Industry

The healthcare industry (also called the medical industry or health economy) is one of the world’s largest industries. It aggregates and integrates sectors within the economic system that provide goods and services to treat patients with curative, preventive, rehabilitative, and palliative care.

This industry includes the generation and commercialization of goods and services lending themselves to maintaining and re-establishing health. The modern healthcare industry has three essential branches: Services, products, and finance, and may be divided into many sectors and categories.

The blog series explores the technical architectures and use cases relevant across the healthcare supply chain. The slide deck shows various real-world deployments from different domains.

Real-time data beats slow data in the healthcare sector

Processing information in the proper context at the right time is crucial for most use cases across the healthcare value chain. Real-time data processing with the Kafka ecosystem reduces risks, improves efficiency, and decreases cost in many domains. Here are some examples:

A real-time Kappa architecture beats batch processing with Lambda architectures and adds business value in almost all use cases.

Apache Kafka to process data in motion across the healthcare supply chain

The beauty of the Kafka ecosystem is the capability to provide a truly decoupled infrastructure for workloads at any scale, including transactional and analytical use cases.

Data flows in and out of various systems. Some are real-time. Others are batch or web service APIs. Some are modern and cloud-native microservices. Others are monolithic proprietary on-premise applications. Some use open standards like Health Level Seven (HL7) with FHIR, others use open data formats like JSON, and some use proprietary data formats.

Here is an example of public health data automation. It leverages Apache Kafka to connect claims and clinical data from proprietary legacy systems with modern cloud-native microservices:

Real-World Deployments of Apache Kafka in the Healthcare Industry

Many healthcare companies leverage Kafka today. Use cases exist in every domain across the healthcare value chain. Most companies deploy data streaming in different business domains. Use cases often overlap. I tried to categorize a few real-world deployments into different technical scenarios and added a few real-world examples:

• Legacy Modernization and Hybrid Cloud (Optum / UnitedHealth Group, Centene, Bayer)
• Streaming ETL (Bayer, Babylon Health)
• Real-time Analytics (Cerner, Celmatix, CDC/Centers for Disease Control and Prevention)
• Machine Learning and Data Science (Recursion, Humana)
• Open API and Omnichannel (Care.com, Invitae)

Stay tuned for a dedicated blog post for each of these topics as part of this blog series. I will link the blogs here as soon as they are available (in the next few weeks). Subscribe to my newsletter to get an email after each publication (no spam or ads).

Slide Deck – Apache Kafka in Healthcare

Here is a slide deck that covers an introduction, use cases, and architectures for data streaming with Apache Kafka in the healthcare sector:

The other blogs of the series take a deeper look into the use cases and architectures.

Data Streaming as Game Changer in Healthcare

Think about IoT sensor analytics, cybersecurity, patient communication, insurance, research, and many other domains. Real-time data beats slow data in the healthcare supply chain almost everywhere.

The above slide deck is an excellent overview of the added value of data streaming to modernize IT infrastructure and build innovative new applications in the cloud or hybrid scenarios. Several real-world deployments show how game-changing the technology is for this very traditional and often still paper-driven sector.

How do you leverage data streaming with Apache Kafka in the healthcare industry? What architecture does your platform use? Which products do you combine with data streaming? Let’s connect on LinkedIn and discuss it! Stay informed about new blog posts by subscribing to my newsletter.

The post Apache Kafka in the Healthcare Industry appeared first on Kai Waehner.