AWS Big Data

OCTOBER 11, 2023

Traditionally, such a legacy call center analytics platform would be built on a relational database that stores data from streaming sources. Data transformations through stored procedures and use of materialized views to curate datasets and generate insights is a known pattern with relational databases.

Management 104

Management Metadata Analytics Dashboards 104

AWS Big Data

MARCH 16, 2023

A source of unpredictable workloads is dbt Cloud , which SafetyCulture uses to manage data transformations in the form of models. Refer to Managing Amazon Redshift Serverless using the console for setup steps. We create a datashare called prod_datashare to allow the serverless instance access to data in the provisioned cluster.

Data Warehouse 52

Data Warehouse Testing Snapshot Modeling 52

DataKitchen

JANUARY 23, 2023

It is important to have additional tools and processes in place to understand the impact of data errors and to minimize their effect on the data pipeline and downstream systems. These operations can include data movement, validation, cleaning, transformation, aggregation, analysis, and more.

Testing 130

Testing Data Governance Data Quality Data-driven 130

O'Reilly on Data

OCTOBER 19, 2021

but to reference concrete tooling used today in order to ground what could otherwise be a somewhat abstract exercise. Adapted from the book Effective Data Science Infrastructure. To manage the dynamism, we can resort to taking snapshots that represent immutable points in time: of models, of data, of code, and of internal state.

IT 346

IT Testing Experimentation Software 346