Data Engineering

A centralized repository that stores structured, semi-structured, and unstructured data at any scale, allowing for flexible analysis.

A series of processes and tools used to move, transform, and store data from source systems to destinations like data warehouses.

A centralized database optimized for analytical queries and reporting, typically storing historical data from multiple sources.

A process used in data integration where data is extracted from sources, transformed into a usable format, and loaded into a target system.

A modern variation of ETL where raw data is first loaded into the target system and then transformed in place.

The management of data availability, usability, integrity, and security in enterprise systems, based on internal standards and policies.

The structure of a database or dataset, including table definitions, columns, data types, and relationships.

A data processing method that collects and processes data in large blocks or batches at scheduled intervals.

The real-time processing of data as it is generated, allowing for immediate analysis and action.

Extremely large and complex data sets that require advanced tools and techniques to capture, store, manage, and analyze.

The process of creating a conceptual representation of data structures and relationships to support business processes.

A measure of the condition of data based on factors such as accuracy, completeness, reliability, and timeliness.

An open-source platform used to programmatically author, schedule, and monitor data workflows.

A distributed streaming platform used for building real-time data pipelines and streaming apps, capable of handling high-throughput data feeds.

An open-source distributed computing system used for big data processing, featuring in-memory computation for increased speed.

Want to explore more? Stay tuned for new terms and updates!

Last updated on 23 April, 2025