Apache Kafka 1.0 cookbook : over 100 practical recipes on using distributed enterprise messaging to handle real-time data / Raul Estrada.

Format:

Book

Author/Creator:

Estrada, Raul, author.

Language:

English

Subjects (All):

Apache (Computer file : Apache Group).

Electronic data processing.

Physical Description:

1 online resource (222 pages) : illustrations (some color)

Edition:

1st edition

Place of Publication:

Birmingham, England : Packt, 2017.

System Details:

text file

Biography/History:

Estrada Raul: Raul Estrada has been a programmer since 1996 and a Java developer since 2001. He loves all topics related to computer science. With more than 15 years of experience in high-availability and enterprise software, he has been designing and implementing architectures since 2003. His specialization is in systems integration, and he mainly participates in projects related to the financial sector. He has been an enterprise architect for BEA Systems and Oracle Inc. , but he also enjoys web, mobile, and game programming. Raul is a supporter of free software and enjoys experimenting with new technologies, frameworks, languages, and methods. Raul is the author of other Packt Publishing titles, such as Fast Data Processing Systems with SMACK and Apache Kafka Cookbook.

Summary:

Simplify real-time data processing by leveraging the power of Apache Kafka 1.0 About This Book Use Kafka 1.0 features such as Confluent platforms and Kafka streams to build efficient streaming data applications to handle and process your data Integrate Kafka with other Big Data tools such as Apache Hadoop, Apache Spark, and more Hands-on recipes to help you design, operate, maintain, and secure your Apache Kafka cluster with ease Who This Book Is For This book is for developers and Kafka administrators who are looking for quick, practical solutions to problems encountered while operating, managing or monitoring Apache Kafka. If you are a developer, some knowledge of Scala or Java will help, while for administrators, some working knowledge of Kafka will be useful. What You Will Learn Install and configure Apache Kafka 1.0 to get optimal performance Create and configure Kafka Producers and Consumers Operate your Kafka clusters efficiently by implementing the mirroring technique Work with the new Confluent platform and Kafka streams, and achieve high availability with Kafka Monitor Kafka using tools such as Graphite and Ganglia Integrate Kafka with third-party tools such as Elasticsearch, Logstash, Apache Hadoop, Apache Spark, and more In Detail Apache Kafka provides a unified, high-throughput, low-latency platform to handle real-time data feeds. This book will show you how to use Kafka efficiently, and contains practical solutions to the common problems that developers and administrators usually face while working with it. This practical guide contains easy-to-follow recipes to help you set up, configure, and use Apache Kafka in the best possible manner. You will use Apache Kafka Consumers and Producers to build effective real-time streaming applications. The book covers the recently released Kafka version 1.0, the Confluent Platform and Kafka Streams. The programming aspect covered in the book will teach you how to perform important tasks such as message validation, enrichment and composition.Recipes focusing on optimizing the performance of your Kafka cluster, and integrate Kafka with a variety of third-party tools such as Apache Hadoop, Apache Spark, and Elasticsearch will help ease your day to day collaboration with Kafka greatly. Finally, we cover tasks related to monitoring and securing your Apache Kafka cluster using tools such as Ganglia and Graphite. If you're looking to become the go-to person in your organization when it comes to work...

Contents:

Cover

Copyright

Credits

About the Author

About the Reviewers

www.PacktPub.com

Customer Feedback

Dedication

Table of Contents

Preface

Chapter 1: Configuring Kafka

Introduction

Installing Kafka

Getting ready

How to do it...

Installing Java in Linux

Installing Scala in Linux

Installing Kafka in Linux

There's more...

See also

Running Kafka

Configuring Kafka brokers

How it works...

Configuring Kafka topics

There's more…

Creating a message console producer

Creating a message console consumer

Configuring the broker settings

How it works…

Configuring threads and performance

Configuring the log settings

Configuring the replica settings

Configuring the ZooKeeper settings

How to do it…

Configuring other miscellaneous parameters

Chapter 2: Kafka Clusters

Configuring a single-node single-broker cluster - SNSB

Starting ZooKeeper

Starting the broker

SNSB - creating a topic, producer, and consumer

How to do it.

Creating a topic

Starting the producer

Starting the consumer

Configuring a single-node multiple-broker cluster - SNMB

SNMB - creating a topic, producer, and consumer

Creating a topic

Starting a producer

Starting a consumer

Configuring a multiple-node multiple-broker cluster - MNMB

Chapter 3: Message Validation

Modeling the events

Setting up the project

Reading from Kafka

Writing to Kafka

Running ProcessingApp

Coding the validator

Running the validator

Chapter 4: Message Enrichment

Geolocation extractor

Geolocation enricher

Currency price extractor

Currency price enricher

See also.

Running the currency price enricher

Open weather extractor

Location temperature enricher

Running the location temperature enricher

Chapter 5: The Confluent Platform

Installing the Confluent Platform

Using Kafka operations

Monitoring with the Confluent Control Center

Using the Schema Registry

Using the Kafka REST Proxy

Using Kafka Connect

Chapter 6: Kafka Streams

Running the streaming application

Chapter 7: Managing Kafka

Managing consumer groups

Dumping log segments

Importing ZooKeeper offsets

Using the GetOffsetShell

Using the JMX tool.

Getting ready

Using the MirrorMaker tool

Replaying log producer

Using state change log merger

Chapter 8: Operating Kafka

Adding or removing topics

Modifying message topics

Implementing a graceful shutdown

Balancing leadership

Expanding clusters

Increasing the replication factor

Decommissioning brokers

Checking the consumer position

Chapter 9: Monitoring and Security

Monitoring server statistics

Monitoring producer statistics

Monitoring consumer statistics

Connecting with the help of Graphite

Monitoring with the help of Ganglia

Implementing authentication using SSL

Implementing authentication using SASL/Kerberos

Chapter 10: Third-Party Tool Integration

Moving data between Kafka nodes with Flume

Writing to an HDFS cluster with Gobblin

Moving data from Kafka to Elastic with Logstash

Connecting Spark streams and Kafka

Ingesting data from Kafka to Storm

Pushing data from Kafka to Elastic

Inserting data from Kafka to SolrCloud

Building a Kafka producer with Akka

Building a Kafka consumer with Akka

Storing data in Cassandra

Running Kafka on Mesos

Reading Kafka with Apache Beam

Writing to Kafka from Apache Beam

Index.

Notes:

Includes index.

Description based on online resource; title from PDF title page (ebrary, viewed February 5, 2018).

OCLC:

1021185680