article thumbnail

Use AWS Glue ETL to perform merge, partition evolution, and schema evolution on Apache Iceberg

AWS Big Data

As enterprises collect increasing amounts of data from various sources, the structure and organization of that data often need to change over time to meet evolving analytical needs. Schema evolution enables adding, deleting, renaming, or modifying columns without needing to rewrite existing data.

Snapshot 108
article thumbnail

Themes and Conferences per Pacoid, Episode 8

Domino Data Lab

So we had three tiers providing a separation of concerns: presentation, logic, data. Note that data warehouse (DW) and business intelligence (BI) practices both emerged circa 1990. Given those two, plus SQL gaining eminence as a database strategy, a decidedly relational picture coalesced throughout the decade. It’s a mess.

article thumbnail

Themes and Conferences per Pacoid, Episode 12

Domino Data Lab

Consider the following timeline: 2001 – Physics grad students are getting hired in quantity by hedge funds to work on Wall St. to join data science teams, e.g., to support advertising, social networks, gaming, and so on—I hired more than a few. 2018 – Global reckoning about data governance, aka “Oops! The best words.