Audience
Anyone seeking a solution to simplify data integration for their streaming and batch data ecosystems
About Apache Gobblin
A distributed data integration framework that simplifies common aspects of Big Data integration such as data ingestion, replication, organization, and lifecycle management for both streaming and batch data ecosystems. Runs as a standalone application on a single box. Also supports embedded mode. Runs as an mapreduce application on multiple Hadoop versions. Also supports Azkaban for launching mapreduce jobs. Runs as a standalone cluster with primary and worker nodes. This mode supports high availability and can run on bare metals as well. Runs as an elastic cluster on public cloud. This mode supports high availability. Gobblin as it exists today is a framework that can be used to build different data integration applications like ingest, replication, etc. Each of these applications is typically configured as a separate job and executed through a scheduler like Azkaban.
Other Popular Alternatives & Related Software
Tencent Cloud Elastic MapReduce
EMR enables you to scale the managed Hadoop clusters manually or automatically according to your business curves or monitoring metrics. EMR's storage-computation separation even allows you to terminate a cluster to maximize resource efficiency. EMR supports hot failover for CBS-based nodes. It features a primary/secondary disaster recovery mechanism where the secondary node starts within seconds when the primary node fails, ensuring the high availability of big data services. The metadata of its components such as Hive supports remote disaster recovery. Computation-storage separation ensures high data persistence for COS data storage. EMR is equipped with a comprehensive monitoring system that helps you quickly identify and locate cluster exceptions to ensure stable cluster operations. VPCs provide a convenient network isolation method that facilitates your network policy planning for managed Hadoop clusters.
Learn more
Apache Spark
Apache Spark™ is a unified analytics engine for large-scale data processing. Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine. Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python, R, and SQL shells. Spark powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application. Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud. It can access diverse data sources. You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN, on Mesos, or on Kubernetes. Access data in HDFS, Alluxio, Apache Cassandra, Apache HBase, Apache Hive, and hundreds of other data sources.
Learn more
MLlib
Apache Spark's MLlib is a scalable machine learning library that integrates seamlessly with Spark's APIs, supporting Java, Scala, Python, and R. It offers a comprehensive suite of algorithms and utilities, including classification, regression, clustering, collaborative filtering, and tools for constructing machine learning pipelines. MLlib's high-quality algorithms leverage Spark's iterative computation capabilities, delivering performance up to 100 times faster than traditional MapReduce implementations. It is designed to operate across diverse environments, running on Hadoop, Apache Mesos, Kubernetes, standalone clusters, or in the cloud, and accessing various data sources such as HDFS, HBase, and local files. This flexibility makes MLlib a robust solution for scalable and efficient machine learning tasks within the Apache Spark ecosystem.
Learn more
Oracle Big Data Service
Oracle Big Data Service makes it easy for customers to deploy Hadoop clusters of all sizes, with VM shapes ranging from 1 OCPU to a dedicated bare metal environment. Customers choose between high-performance NVmE storage or cost-effective block storage, and can grow or shrink their clusters. Quickly create Hadoop-based data lakes to extend or complement customer data warehouses, and ensure that all data is both accessible and managed cost-effectively. Query, visualize and transform data so data scientists can build machine learning models using the included notebook with its R, Python and SQL support. Move customer-managed Hadoop clusters to a fully-managed cloud-based service, reducing management costs and improving resource utilization.
Learn more
Integrations
Company Information
Apache Software Foundation
United States
gobblin.apache.org
Videos and Screen Captures
Other Useful Business Software
The full-stack observability platform that protects your dataLayer, tags and conversion data
Code-Cube.io detects issues instantly, alerts you in real time and helps you resolve them fast.
No manual QA. No unreliable data. Just data you can trust and act on.
Product Details
Platforms Supported
Windows
Linux
Training
Documentation
Support
Online
