Nelson J Pérez

💡 The Redshift team at AWS recently published a study analyzing half a billion queries on 32 million tables over a 3-month period. The findings? 67% of queries scanned less than 1 GB of data, and 99.8% of scanned less than 10TB. This reveals a reality check—most data workloads are far smaller than we often assume. So why overcomplicate things with massive infrastructure? Embrace the simple joys of small data

234

11 Comments

See how simple it is to create chatbots with Microsoft Azure On Your Data and vector embeddings from an existing Elasticsearch index: https://lnkd.in/gBHZbkjq #ElasticSearchLabs #RAG #VectorDatabase

2

As data increased in volume, velocity, and variety, so, in turn, did the need for tools that could help process and manage those larger data sets coming at us at ever faster speeds. As a result, frameworks such as Apache Spark and Apache Flink became popular due to their abilities to handle big data processing in a fast, efficient, and scalable manner. But we often find that sometimes it can be difficult to understand which use cases are best suited for Spark as well as for Flink (or even which might be suited for both). In this article, we’ll discuss some of those unique benefits for both Spark and Flink and help you understand the difference between the two, and go over real use cases, including ones where the engineers were trying to decide between Spark vs. Flink. https://lnkd.in/g_XHFkQ6

60

1 Comment

Run this advanced state-of-the-art model with language understanding, superior reasoning, and text generation. Build with this NIM today.

4

Here is a good explanation of Fabric Data Warehouse by James!

4

Startups oftentimes face a huge existential challenge right out of the gate: Choosing a data infrastructure that supports their products as they scale, but that also provides the flexibility and performance they need to build, iterate, and grow. Redis for Startups is our new program to help early-stage companies get access to the Redis solutions and resources they need to build better apps. Every rocket ship needs help getting off the ground. Come see how Redis can be your booster: https://redis.io/startups/ Read more here: https://lnkd.in/g-gaP6Yx

146

5 Comments

Are you wondering how Dragonfly compares to Redis? Our new video provides a comprehensive breakdown of the key differences, including performance, features, and scalability. Whether you’re choosing a new DB solution or just curious, this video is a must-watch. Engage with us in the comments and share your thoughts! Watch here: https://hubs.la/Q02Kczst0 #Dragonfly #Redis #Database #TechComparison #DeveloperCommunity

3

"I think all of this is indicative of this broader #DataOps trend, of you want everything unified in one platform so that you have full control and visibility over all workloads. You need access to strong orchestration, lots of compute. If you’re training ML models, you need strong observability to make sure that you understand how everything is working together. And that’s very much how we view building our products and influencing the market over the next couple of years towards this full DataOps platform, where you don’t have to go buy six different tools. You can just come to one.” - Julian LaNeve, CTO, Astronomer Julian spoke with BigDATAwire about the significance of Astro Observe, our vision for Astro moving forward beyond data orchestration, and how Ford Motor Company is leaning on Astro for ML/AI data operations of its Advanced Driver Assistance Systems (ADAS) car initiatives. https://bit.ly/3Nzslv7

21

Excited to share our Redshift integration here today 😎 DATA ENGINEERS ON AWS: With Orchestra, you get: 👉 End-to-end data lineage for Assets in Redshift 👉 Alerting for when dbt tasks or tests on Redshift assets fail 👉 Monitor Data Quality in Redshift over time 👉 Stitch Redshift together with the rest of your #datastack Read more here 🖇 https://lnkd.in/dRwM-VzT #redshift #aws #dataengineering

11

4 Comments

OpenAI announced the release of new features in their fine-tuning API. The features will give model developers more control over the fine-tuning process and better insight into their model performance. #OpenAI #Developers #API

2

Elasticsearch open inference API adds support for OpenAI chat completions. Learn more in this blog post: https://gag.gl/hEDnXw #ElasticSearchLabs #OpenAI

1

Today at Noon EST, Join Anant Corporation and Dremio Senior Evangelist, Alex Merced ,for a deep dive into the innovative world of Apache Iceberg powered Reflections in his talk, "Dremio's Apache Iceberg Powered Reflections: Query Acceleration Deep Dive." Discover how to simplify your data infrastructure and accelerate your analytics with Dremio's innovative approach to query optimization. #Apache #Iceberg #Query #Acceleration

2

We recently started a new LinkedIn Live series for everything Apache Hudi and Open Lakehouses 🎉 In the very 1st episode, Dipankar went over the fundamentals of Apache Hudi and showed hands-on examples of running transactions with compute engines like Apache Spark. If you missed the live, here's the recording: https://lnkd.in/dm-24JzC Check it out and share with the Hudi community! #dataengineering #softwareengineering #lakehouse

15

Open Source models are on the rise and getting better every day. Snowflake has recently released five models in various sizes on Hugging Face, which is great news for anyone looking to leverage AI through RAG or a semantic search service. If you're interested in leveraging the latest developments in AI, this is definitely worth checking out. #AI #opensource #Snowflake #HuggingFace #RAG #semanticsearch

3

Implement a Q&A experience using a RAG technique with Elasticsearch as a vector database by following the steps in this blog: https://gag.gl/V2hwuA #ElasticSearchLabs #RAG #GenAI

2

Lately, I've received a lot more inquiries from organizations interested in building out a Semantic Layer. It makes a ton of sense: with a Data Vault organizations can create highly flexible, highly normalized data models that evolve at the speed of business. With a Semantic Layer, organizations can create governed metrics, denormalized consistently across use-cases. 👉 And so, on Tuesday 13 August, join Tim Kirschke from Scalefree and I as we demonstrate how to build an end-to-end Semantic Layer on top of a Data Vault with dbt Labs and TurboVault4dbt! Registration details in the link below 👇 Interested in learning more about Semantic Layer and how it may be able to help your organization? Take a peek at the blog post below! https://lnkd.in/dgk2B7We #data #dbt #datavault #semanticlayer

66

3 Comments

Amazon Managed Service for Apache Flink has just introduced two powerful new APIs: ListApplicationOperations and DescribeApplicationOperation. These APIs are game-changers for monitoring and managing your Flink applications. Key benefits:
 • Enhanced visibility into application operations
 • Real-time status updates on operation progress
 • Detailed insights on success, failure, and rollbacks
 • Improved ability to take timely follow-up actions This update further solidifies Amazon Managed Service for Apache Flink as a top choice for real-time data processing and analytics. It seamlessly integrates with other AWS services like Amazon MSK, Kinesis Data Streams, OpenSearch Service, DynamoDB streams, and S3, making it a versatile solution for various streaming data scenarios. If you’re working on projects involving real-time data transformation and analysis, these new APIs are definitely worth exploring. They could significantly streamline your operational workflows and improve application management. Check out the link below for more details and start leveraging these APIs to supercharge your Flink applications today! #AWS #ApacheFlink #DataEngineering #StreamProcessing #CloudComputing https://lnkd.in/gViDbtdU

53