Cloud leader AWS shifts its database focus to DataZone and Zero-ETL

Cloud leader AWS’s influence on IT trends takes many forms, but none has been more impactful than its standard database services.

At most annual Re:Invention conferences, AWS has rolled out a shiny new database that cements the company’s presence among cloud-based databases. These are sometimes open source and mostly Purpose-built.

But this year was different. At AWS re:Invent 2022, the company turned its sights to making its existing array of cloud data tools more palatable to enterprise IT. That means data integration and data management are now the focus.

To that end the company released Amazon DataZone data management services to catalog and manage data stored in the AWS cloud and on-premises. Also, DataZone can support third-party resources via APIs, AWS said, citing partners like Databricks, Snowflake and Tableau in this context.

The timing is right. Organizations are finding that the number of different sources of data they need to connect is growing dramatically. Management and governance of dispersed data holdings becomes more difficult.

Integrating data feeds

Now as never before, cost efficiency is driving IT to the cloud, AWS CEO Adam Selipsky told re:Invent attendees. Today for AWS, cost-effective data engines in a columnar MPP data warehouse are starting to raise the economics of data analytics with the 2012 introduction of Aurora and Redshift, AWS’ version of open source PostgreSQL.

RDS in the database procession brought by Aurora and Redshift. NeptuneDynamoDB, DocumentDB, ElasticCache, TimestreamAnd this Quantum Ledger DBSome of the stirring controversy came as start-ups wrestled with cloud giant AWS’s aggressive approach to open-source licensing.

Celebski didn’t get around to reviewing the new database – despite updates to several existing machines. Instead, he promoted the idea of ​​combining the existing portfolio more effectively.

“Having all these tools to store and analyze data reveals the next challenge that people face … you can combine information across these different methods of data exploration to see the whole picture and really get insights,” he said.

Give them ETL

In his re:invention address, Celebski took aim at the integration challenges surrounding Extract Transform Load (ETL), a long-standing underpinning of high technology that innovators have recently been rethinking.

He announced new integrations between Amazon Aurora and Amazon Redshift services, as well as eliminating the need for ETL between Spark and Redshift.

Celebski’s intention here is clear. Maybe it’s time to dial in “Zero ETL” because of low code/no code. It is a phase of data processing that involves repetitive custom scripting that is necessary and typically illuminated when digital transformation is the ultimate goal of the organization.

The sluggish work of ETL data preparation can stand in the way of progress. To show IT’s frustration with the process, Celebski read an excerpt from a customer letter that described ETL as a “thankless, unsustainable black hole.” New Aurora and Redshift capabilities help customers move toward a zero-ETL future on AWS, he said.

Table echoes

Although perhaps hidden Machine learning And other announcements, the focus on big data management issues at re:Invent 2022 suggests a new maturity in AWS’s approach to IT’s data needs.

There is also the implication that Adam Celebski is charting a new course for the AWS cloud. Given his years at the helm Business Intelligence Provider IndexThis is not entirely unexpected.

Under his watch, Tableau distinguished itself for innovation in visual data presentation and established itself as an expert in ease of use and drag-and-drop integration support for structured and unstructured data sets.

AWS’s DataZone and Zero-ETL fit neatly into a similar picture of cloud data evolution. Future moves will be closely watched to see if AWS moves more in data building.