AWS Big Data

MARCH 23, 2023

Large organizations often have lines of businesses (LoBs) that operate with autonomy in managing their business data. It makes sharing data across LoBs non-trivial. These organizations have adopted a federated model, with each LoB having the autonomy to make decisions on their data.

Interactive 66

Interactive Snapshot Data Lake Software 66

AWS Big Data

SEPTEMBER 13, 2023

A modern data architecture is an evolutionary architecture pattern designed to integrate a data lake, data warehouse, and purpose-built stores with a unified governance model. Of those tables, some are larger (such as in terms of record volume) than others, and some are updated more frequently than others.

Data Lake 90

Data Lake Data Processing Metadata Snapshot 90

Domino Data Lab

JANUARY 11, 2021

Arming data science teams with the access and capabilities needed to establish a two-way flow of information is one critical challenge many organizations face when it comes to unlocking value from their modeling efforts. Domino Data Lab and Snowflake: Better Together. Writing data from Domino into Snowflake.

Recreation/Entertainment 103

Recreation/Entertainment Data Science Data Warehouse Modeling 103

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale.

Data Lake 113

Data Lake Metadata Optimization Statistics 113

AWS Big Data

JULY 3, 2023

Terminology Let’s first discuss some of the terminology used in this post: Research data lake on Amazon S3 – A data lake is a large, centralized repository that allows you to manage all your structured and unstructured data at any scale. This is where the tagging feature in Apache Iceberg comes in handy.

Snapshot 82

Snapshot Data Lake Testing Strategy 82

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 85

Data Lake Analytics Snapshot Optimization 85

AWS Big Data

JULY 19, 2023

You can structure your data, measure business processes, and get valuable insights quickly can be done by using a dimensional model. Amazon Redshift provides built-in features to accelerate the process of modeling, orchestrating, and reporting from a dimensional model. Declare the grain of your data.

Modeling 67

Modeling Sales Data Warehouse Snapshot 67

AWS Big Data

FEBRUARY 1, 2023

With data volumes exhibiting a double-digit percentage growth rate year on year and the COVID pandemic disrupting global logistics in 2021, it became more critical to scale and generate near-real-time data. The response times for these data sources are critical to our key stakeholders.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

AWS Big Data

JULY 28, 2023

Tagging Consider tagging your Amazon Redshift resources to quickly identify which clusters and snapshots contain the PII data, the owners, the data retention policy, and so on. Redshift resources, such as namespaces, workgroups, snapshots, and clusters can be tagged. Tags provide metadata about resources at a glance.

Snapshot 80

Snapshot Metadata Measurement Data Warehouse 80

O'Reilly on Data

OCTOBER 19, 2021

Let’s start by considering the job of a non-ML software engineer: writing traditional software deals with well-defined, narrowly-scoped inputs, which the engineer can exhaustively and cleanly model in the code. Not only is data larger, but models—deep learning models in particular—are much larger than before.

IT 346

IT Testing Experimentation Software 346

AWS Big Data

AUGUST 2, 2023

As the only user-provided input is the amount spent on an activity, EIO LCA methods aren’t useful for modeling improved efficiency. The data architecture diagram below shows an example of how you could use AWS services to calculate and visualize an organization’s estimated carbon footprint.

Data Lake 69

Data Lake Measurement Visualization Data Architecture 69

AWS Big Data

MARCH 18, 2024

With these new attributes, you are able to build a segmentation model to identify distinct groups of customers that you can target with personalized messaging. This data is available to subscribe to on AWS Data Exchange—and with data sharing, you don’t need to pay to store a copy of it in your account in order to query it.

Data Warehouse 81

Data Warehouse Visualization Snapshot Data-driven 81

AWS Big Data

MARCH 20, 2023

Personalized recommendations – User behavior based on clickstream events can be captured up to the last second before enriching it for personalization and sending it to the model to predict the recommendations. The data from the S3 data lake is used for batch processing and analytics through Amazon EMR and Amazon Redshift.

Analytics 52

Analytics Slice and Dice Data Processing Data Lake 52

Cloudera

MARCH 23, 2022

For example, a Jupyter notebook in CML, can use Spark or Python framework to directly access an Iceberg table to build a forecast model, while new data is ingested via NiFi flows, and a SQL analyst monitors revenue targets using Data Visualization. Simplify data management . 2: Open formats. Reproducibility for ML Ops.

Metadata 88

Metadata Data Architecture Machine Learning Cost-Benefit 88

Corinium

APRIL 25, 2019

Then when there is a breach, it comes as a shock, “wow, I didn’t even know that application had access to so much sensitive data”. Step One in any data security program should first be to discover and classify datasets that are sensitive, and know where that data is, and understand who really needs it to do their jobs.

Insurance 150

Insurance Risk IoT Cost-Benefit 150

AWS Big Data

MARCH 3, 2023

Every event in the data source can be relevant, and our customers don’t tolerate data loss, poor data quality, or discrepancies between the source and Tricentis Analytics. While aggregating, summarizing, and aligning to a common information model, all transformations must not affect the integrity of data from its source.

Software 52

Software Data Lake Testing Cost-Benefit 52

AWS Big Data

NOVEMBER 29, 2023

We have seen a strong customer demand to expand its scope to cloud-based data lakes because data lakes are increasingly the enterprise solution for large-scale data initiatives due to their power and capabilities. The team uses dbt-glue to build a transformed gold model optimized for business intelligence (BI).

Data Lake 102

Data Lake Management Metrics Data Warehouse 102

AWS Big Data

JUNE 21, 2023

In traditional databases, we would model such applications using a normalized data model (entity-relation diagram). In this post, we walk through the process of exporting data from a DynamoDB table to Amazon Redshift. We discuss data model design for both NoSQL databases and SQL data warehouses.

Data Warehouse 78

Data Warehouse Data Lake OLAP Cost-Benefit 78

AWS Big Data

JULY 28, 2023

Amazon Redshift now makes it easier for you to run queries in AWS data lakes by automatically mounting the AWS Glue Data Catalog. You no longer have to create an external schema in Amazon Redshift to use the data lake tables cataloged in the Data Catalog.

Data Lake 65

Data Lake Data Governance Data Warehouse Modeling 65

IBM Big Data Hub

SEPTEMBER 25, 2023

Uber’s DNA as an analytics company At its core, Uber’s business model is deceptively simple: connect a customer at point A to their destination at point B. Uber understood that digital superiority required the capture of all their transactional data, not just a sampling. It lands as raw data in HDFS.

OLAP 92

OLAP Data Lake Data-driven Snapshot 92