Darryl R.’s Post

Curious about how to build a multi-cloud resource data lake for SecOps, ITOps, and FinOps functions and how to normalize the data to perform analytics against it? Check out this Amazon Web Services (AWS) bog post that details how to avoid operational challenges with applications distributed across a cloud estate that spans not only multiple Cloud Service Providers (CSPs) but also regions and accounts on AWS (or Subscriptions and Projects on other providers).

Cloud service providers such as AWS and GCP offer hundreds of services, and sometimes it can be a little confusing to figure out what solution does what and how data engineers and data scientists might use them. In this article, I wanted to discuss some of the services that are useful to know as a data engineer as well as provide a combination of real-world examples where I have used said services. I’ll also only be focusing on AWS for now. So let’s dive into the cloud services you should know as a data engineer. https://lnkd.in/gqKfD_Rc

The Amazon Athena Neptune connector is an AWS Lambda function that connects Athena to Neptune to query the graph. This post provides a guide to integrate the connector and query Neptune using Gremlin and SPARQL. #aws #awscloud #cloud #amazonathena #amazonneptune #awsglue #intermediate200 #technicalhowto

Amazon DataZone now integrates with AWS Lake Formation hybrid access mode to simplify secure and governed data sharing in the AWS Glue Data Catalog. This helps customers use Amazon DataZone for data access control across on-premises and cloud data lakes. #aws #awscloud #cloud #amazondatazone #announcements #awsglue #awslakeformation

🎨 Learn how Amazon DataZone uses popular AWS services you may already have in your environment, including Amazon Redshift, Amazon Athena, AWS Glue & AWS Lake Formation, as well as on-premises & third-party sources. 📄 https://go.aws/3US1tuU

Optimize Querying Your Data in Amazon S3. 📊 Amazon S3 is a popular cloud storage service. But accessing data can slow down queries.👎 Here are tips to speed things up!💨 First, enable S3 Inventory to track your data. This avoids full S3 scans for queries.🔍 You can also use S3 analytics for insights.📈 Next, optimize your data layout in S3.👌 Partition by date, ID or other fields to limit scanning. Also compress files to speed transfers.🔋 You can use prefixes to separate data sets too. This isolates queries to relevant folders only.🗂️ Finally, use S3 formats like Parquet for faster analysis. The columnar format avoids excess data reads. 🚦And use S3 Select to filter results, avoiding full file transfers.✂️ There's so much more! Read on to optimize S3 performance today.👇 https://lnkd.in/ePvc4pSB #aws #amazon #dataanalytics #datascience #datalake #amazons3

How to Setup TTL for Records and Practical Use Cases https://lnkd.in/ddEDJZwJ Amazon Web Services (AWS) #aws #awslambda #businesscompassllc

"🚀 Mastering S3 Data Migration: A Comprehensive Guide to Copying Files Between Buckets with AWS Lambda 🌐 Looking to streamline your data workflows on AWS? Learn how to effortlessly copy files between S3 buckets using AWS Lambda with our step-by-step guide! Whether you're migrating data, syncing backups, or optimizing storage, this tutorial covers everything from setup to execution, ensuring you harness the full power of serverless computing for efficient data management. Dive into the details and elevate your cloud storage strategy today! #AWS #CloudComputing #DataManagement #Serverless" https://lnkd.in/geZCqkdj

Migrating data to the cloud can be complicated, but with Protegrity and Amazon Web Services (AWS), it can be both efficient and effective. In this post by Alexandre Charlet, Principal Solution Engineer at Protegrity, and Venkatesh A. of AWS, they describe value of a joint solution. I learned a lot. I hope you do too. https://lnkd.in/eeQ-TFdJ

Data governance is a key enabler for adopting a data-driven culture to drive innovation. Amazon DataZone is a managed data service that makes it easier to catalog, discover, share, and govern data across AWS, on premises, and third-party sources. #aws #awscloud #cloud #advanced300 #amazondatazone #technicalhowto