{"id":262,"date":"2020-09-24T19:51:33","date_gmt":"2020-09-24T19:51:33","guid":{"rendered":"https:\/\/machine-learning.webcloning.com\/2020\/09\/24\/ingest-data-from-apache-kafka\/"},"modified":"2020-09-24T19:51:33","modified_gmt":"2020-09-24T19:51:33","slug":"ingest-data-from-apache-kafka","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2020\/09\/24\/ingest-data-from-apache-kafka\/","title":{"rendered":"Ingest data from Apache Kafka"},"content":{"rendered":"
\n<\/p>\n

This is part of the Learning path: Get started with IBM Streams<\/a><\/strong>.<\/p>\n

Summary<\/h2>\n

In this developer code pattern, we walk you through the basics of creating a streaming application powered by Apache Kafka, one of the most popular open source distributed event-streaming platforms used for creating real-time data pipeline and streaming apps. The application will be built using IBM Streams on IBM Cloud Pak\u00ae for Data.<\/p>\n

Description<\/h2>\n

In this pattern, we walk you through the basics of creating a streaming application powered by Apache Kafka. Our app will be built using IBM Streams on IBM Cloud Pak for Data. IBM Streams provides a built-in IDE (Streams Flows) that allows you to visually create a streaming app. The IBM Cloud Pak for Data platform provides additional support, such as integration with multiple data sources, built-in analytics, Jupyter Notebooks, and machine learning.<\/p>\n

For our Apache Kafka service, we will be using IBM Event Streams on IBM Cloud, which is a high-throughput message bus built on the Kafka platform. In the following examples, we will show it as both a source and a target of clickstream data \u2014 data captured from user clicks as they browsed online shopping websites.<\/p>\n

Flow<\/h2>\n

\"flow\"<\/p>\n

    \n
  1. User creates streaming app in IBM Streams.<\/li>\n
  2. Streaming app uses Kafka service via IBM Event Streams to send\/recieve messages.<\/li>\n
  3. Jupyter notebook is generated from IBM Streams app.<\/li>\n
  4. User executes streaming app in Jupyter notebook.<\/li>\n
  5. Jupyter notebook accesses Kafka service via IBM Event Streams to send\/receive messages.<\/li>\n<\/ol>\n

    Instructions<\/h2>\n

    Ready to get started? The README<\/a> explains the steps to:<\/p>\n

      \n
    1. Clone the repo<\/li>\n
    2. Provison Event Streams on IBM Cloud<\/li>\n
    3. Create sample Kafka console Python app<\/li>\n
    4. Add IBM Streams service to Cloud Pak for Data<\/li>\n
    5. Create a new project in Cloud Pak for Data<\/li>\n
    6. Create a Streams Flow in Cloud Pak for Data<\/li>\n
    7. Create a Streams Flow with Kafka as source<\/li>\n
    8. Use Streams Flow option to generate a notebook<\/li>\n
    9. Run the generated Streams Flow notebook<\/li>\n<\/ol>\n

      This pattern is part of the Learning path: Get started with IBM Streams<\/a>. To continue the series and learn more about IBM Streams, check out a code pattern titled Build a streaming app using a Python API<\/a>. <\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"

      https:\/\/developer.ibm.com\/patterns\/add-event-streams-and-a-db-in-python-to-clickstream\/<\/p>\n","protected":false},"author":0,"featured_media":263,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/262"}],"collection":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/comments?post=262"}],"version-history":[{"count":0,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/posts\/262\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media\/263"}],"wp:attachment":[{"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/media?parent=262"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/categories?post=262"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salarydistribution.com\/machine-learning\/wp-json\/wp\/v2\/tags?post=262"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}