AWS Big Data

OCTOBER 11, 2023

Organizations with legacy, on-premises, near-real-time analytics solutions typically rely on self-managed relational databases as their data store for analytics workloads. Near-real-time streaming analytics captures the value of operational data and metrics to provide new insights to create business opportunities.

Management 107

Management Metadata Analytics Dashboards 107

AWS Big Data

APRIL 27, 2023

Apache Iceberg is an open table format for data lakes that manages large collections of files as tables. It supports modern analytical data lake operations such as create table as select (CTAS), upsert and merge, and time travel queries. However, this requires knowledge of a table’s current snapshots.

Data Lake 74

Data Lake Snapshot Optimization Data Transformation 74

AWS Big Data

JULY 20, 2023

One key component that plays a central role in modern data architectures is the data lake, which allows organizations to store and analyze large amounts of data in a cost-effective manner and run advanced analytics and machine learning (ML) at scale. To overcome these issues, Orca decided to build a data lake.

Data Lake 88

Data Lake Analytics Snapshot Optimization 88

AWS Big Data

MARCH 3, 2023

In this post, we share how the AWS Data Lab helped Tricentis to improve their software as a service (SaaS) Tricentis Analytics platform with insights powered by Amazon Redshift. While aggregating, summarizing, and aligning to a common information model, all transformations must not affect the integrity of data from its source.

Software 52

Software Data Lake Testing Cost-Benefit 52

AWS Big Data

MARCH 3, 2023

You can then apply transformations and store data in Delta format for managing inserts, updates, and deletes. Amazon EMR Serverless is a serverless option in Amazon EMR that makes it easy for data analysts and engineers to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers.

Data Lake 84

Data Lake Dashboards Metrics Metadata 84

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 106

Data Lake Management Metrics Data Warehouse 106

AWS Big Data

JANUARY 27, 2023

The AWS Glue Data Catalog provides a uniform repository where disparate systems can store and find metadata to keep track of data in data silos. Apache Flink is a widely used data processing engine for scalable streaming ETL, analytics, and event-driven applications. Transformed data can be stored in Amazon S3.

Data Lake 62

Data Lake Metadata Business Analysis Data-driven 62

DataKitchen

JANUARY 23, 2023

To capture a more complete picture of the data’s journey, it is important to have a DataOps Observability system in place. Data lineage is static and often lags by weeks or months. Data lineage is often considered static because it is typically based on snapshots of data and metadata taken at a specific time.

Testing 130

Testing Data Governance Data Quality Data-driven 130

Jet Global

NOVEMBER 7, 2023

The Challenges of Extracting Enterprise Data Currently, various use cases require data extraction from your OCA ERP, including data warehousing, data harmonization, feeding downstream systems for analytical or operational purposes, leveraging data mining, predictive analysis, and AI-driven or augmented BI disciplines.

Enterprise 52

Enterprise Data Warehouse Operational Reporting Reporting 52