Andrew Fogarty’s Post

Databricks offers compute resources to process your data. With its managed clusters, you can seamlessly scale your workloads. Clusters consist of a set of one or more virtual machines (VM) instances (nodes) that work together to process data. A typical case, in the picture below, is when a cluster has a driver node and one or more worker nodes. There are two main types of clusters: 1. All-purpose clusters are used for interactive analysis, data engineering, and data science. It’s a balance between flexibility and performance, and multiple users can share these clusters and work in collaboration. 2. Job clusters are designed specifically for running automated jobs. These clusters are ephemeral, spinning up when a job starts and terminating when it ends to optimize resource usage and overall cost. Additionally, Databricks offers other cluster types like Single-Node, Multi-Node, Auto-Scaling, High Concurrency, and GPU-Enabled clusters, each providing for different processing needs. #Databricks #DataAnalytics #DataEngineering #CloudComputing

Here is why the technical interview is funny.... If you interview for an Azure Data Engineer role, just remember that the documentation for all of Azure Synapse is 9,049 pages long. You literally won't know everything. This is just one service in Azure by the way 🤣 🤣 🤣 #data #dataengineer #tech #azure #microsoft #interview #technicalinterview #2024

When you are running a PySpark job, understanding how to estimate the required resources is critical. Here are 5 key factors to guide your decision-making process: 1.Data Size: The volume of data directly impacts the resources you’ll need. 2.Job Duration: Is it a quick job, or will it run for an extended period? 3.Resource Type: Determine whether your job demands more memory or CPU power. 4.Caching Requirements: Will caching be needed, and how does it affect your resource plan? 5.Completion Time: How fast do you need the job to finish? Pro Tip: More resources don’t always mean better performance. For example, if you have 80 partitions and allocate 160 CPU cores, only 80 will be utilized—leaving the rest idle. #DataEngineering #PySpark #BigData #AWS #CloudComputing #CareerTips

10 Good Systems That Completely Transformed the Tech World.    1. Dynamo - Amazon’s Highly Available Key-Value Store  2. Google File System: Insights into a highly scalable File System  3. Scaling Memcached at Facebook: A look at the complexities of Caching  4. Cassandra: A look at the design and architecture of a distributed NoSQL database  5. Kafka: Internals of the distributed messaging platform  6. Amazon Aurora: To learn how Amazon provides high-availability and performance  7. Spanner: Design and architecture of Google’s globally distributed database  8. MapReduce: A detailed look at how MapReduce enables parallel processing of massive volumes of data  9. Shard Manager: Understanding the generic shard management framework  10- ScyllaDB: A distributed database that Follows PAC(Partition Tolerance, Availability, Consistency)   #systemdesign #coding #distubutedSystems #technology 

Impressed with the KX community”s breadth and depth of ecosystem. Here are just some of the partners we collaborate closely with to solve the largest data and high-performance computing problems on behalf of our mutual customers. ⚙ 𝐒𝐞𝐫𝐯𝐞𝐫𝐬 & 𝐒𝐭𝐨𝐫𝐚𝐠𝐞: #KX is the go-to database for the most latency-sensitive applications. But what servers and storage should you run it on? We’ve partnered with several of the leading hardware vendors to optimize the performance of KDB on their platforms. Audited benchmarks and reference configurations are available for each of the following: ◾ #Dell Technologies: PowerEdge servers + PowerScale & PowerFlex storage ◾ Lenovo: ThinkSystem ◾ Pure Storage: Flashblade ◾ DDN Storage: AI400X2 ◾ WEKA: WekaFS and WekaIO Matrix 🤝 𝐂𝐏𝐔𝐬 𝐚𝐧𝐝 𝐆𝐏𝐔𝐬: our engineering partnerships with the chip giants are rapidly expanding beyond CPU and GPU optimization to include integrations with their broader AI hardware and software layers. ◾ #NVIDIA: L40S, H100, Grace Hopper, RAPIDS, NeMo, NIM ◾ #Intel Corporation: Xeon, Sapphire Rapids, Optane ☁ 𝐇𝐲𝐩𝐞𝐫𝐬𝐜𝐚𝐥𝐞𝐫𝐬: naturally, KX runs on the platform of your choice. But did you also know that we’ve embedded KDB deeper in the cloud stack for higher performance and more native offerings? ◾ #Microsoft: KDB Insights on Azure - a fully managed Azure-native service ◾ #Amazon Web Services (AWS): Amazon Finspace with Managed KDB Insights – a fully managed AWS-native service. 🏚 𝐃𝐚𝐭𝐚 𝐖𝐚𝐫𝐞𝐡𝐨𝐮𝐬𝐞 𝐚𝐧𝐝 𝐋𝐚𝐤𝐞𝐡𝐨𝐮𝐬𝐞: want to leverage the scalability of Snowflake or Databricks but need to accelerate those AI and analytics workloads? ◾ #Databricks: run PyKX directly from the PySpark console.   ◾ #Snowflake: run PyKX directly in Snowpark. 💡 𝐕𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧: several years ago we made the decision to add ANSI SQL support in KDB. This has allowed customers to integrate their favourite front end tools and the response has been phenomenal. Open-source integrations are available for several of the top products including: ◾ #Tableau ◾ Power BI ◾ #Grafana Labs 📊 𝐃𝐚𝐭𝐚: One of the hardest parts of managing high frequency datasets is doing the integration between data feed and database. We’ve partnered with the following vendors to take all of the pain out of this part of the job. ◾ #ICE ◾ #Refinitiv ◾ #Bloomberg ◾ #BMLL ◾ Options Technology ◾ INQDATA 🤖 𝐀𝐈: we are undoubtedly in the early innings of a paradigm shift with respect to Generative AI. We’ve integrated with the following AI platforms and frameworks so that you can easily bring today’s state-of-the-art models to your data. ◾ #LlamaIndex ◾ #LangChain ◾ #Hugging Face 🛠️ 𝐒𝐞𝐫𝐯𝐢𝐜𝐞𝐬: Our services partners deliver enormous value to customers with their combination of technical and domain expertise day in and day out. ◾ #First Derivative ◾ #Data Intellect ◾ Caspian One ◾ #ExeQutionAnalytics ◾ Habla computing #community #ecosystem #innovation

#hiring #aws #dataengineer #mountainview #california #onsite #hybrid #[email protected] Backfill requirement : AWS Data Engineer : Mountain view CA (Hyrbid) Must have : #AWS , #Terraform , #Spark , #python Expertise with any of the following Object Oriented Languages (OOD): Java/J2EE, C#, VB.NET, Python, or sometimes C++. Java and Python preferred. Expertise with AWS (IAM, VPC), Spark, and Terraform are preferred. Expertise with Databricks is a strong bonus. Skills Multi-cloud data exploration Terraform infrastructure-as-code for managing AWS infrastructure and deep integration between enterprise tools (Starburst, Privacera, and Databricks) and Intuit services (LDAP, data decryption) Testing user flows for data analysis, processing, and visualization with Python Spark notebooks and SQL running on distributed compute to join data between AWS S3 and GCP BigQuery Developing data pipelines in Python Spark or SQL to push structured enterprise tool telemetry to our data lake Fine-grained access control for data exploration Terraform infrastructure-as-code for managing AWS infrastructure and deep integration between enterprise tools (Databricks and Privacera) Evaluating Databricks capabilities to sync Hive, Glue, and Unity Catalogs Evaluating Privacera capabilities or building new capabilities (AWS Lambda with Python) to sync Intuit access policies with Unity Catalog Testing user flows for data analysis, processing, and visualization with Python Spark notebooks on distributed compute or Databricks’ serverless SQL runtime

Why Azure Provides Two Access Keys Azure's dual access key system offers significant security benefits: ➡ Enhanced Security: By having two keys, you can rotate one periodically without affecting access. This reduces the risk of unauthorized access in case of a compromise. ➡ Business Continuity: If one key is accidentally leaked or compromised, you can quickly regenerate it, minimizing service disruption. ➡ Centralized Management: Azure Key Vault provides a secure and centralized location to store and manage access keys, reducing the risk of accidental exposure. #AzureDataFactory #DataPipelines #CloudComputing #DataEngineering #Azure #BlobStorage #DataLakeStorage #BigData #Analytics #DataAnalystJobs #DataEngineerJobs #BigDataAnalytics #DataDriven #TechJobs #ITJobs #Hiring #Python #SQL #DataVisualization #PowerBI #Tableau #ETL #DataScience #DataAnalytics #DataOps #DataPlatform #DataGovernance #DataQuality #DataMigration #DataIntegration Follow Priyanshu Shrivastav for more such content.

Azure Data Factory is hands down the best batch data job orchestrator on the market. As a data engineer, you can use this tool to copy data from one source system into another (including all of your SFTP jobs), to schedule and run Databricks jobs, and to schedule jobs using regular, open-source Python via Azure Machine Learning Service among many other things. It's trivial to schedule jobs using a timer, tumbling window, or event (i.e. a file appearing or disappearing), trivial to set up voice, SMS, or e-mail notifications for when your jobs fail (or completes if you're into that), and trivial to make your pipelines ultra dynamic via the use of parameters, variables, and loops, etc. It also makes ML engineers' jobs easy due to its ability to schedule, run and monitor batch machine learning jobs. It never fails, can be set up to run jobs on both cloud compute and on-prem compute, and I'm really not sure why it doesn't get more good press. It deserves it. Amongst all of the Azure tools, I find it the most useful and most reliable. #dataengineering #datascience #azure #microsoft #datafactory

Leverage Azure Databricks to streamline your data preparation process. You can efficiently query files from various data sources, including Azure Blob Storage, using Spark SQL. Whether handling large datasets or integrating multiple data streams, Azure Databricks offers a scalable solution for seamless data processing and analysis. #LeadingEdgeSolutions #LeadingEdge #Azure #BigData #AzureDatabricks #BlobStorage #SparkSQL #ITServices #ITSolutions #Recruitment #RecruitmentSolutions #Opportunities #Technology #Hiring #ITStaffing #USA #LinkedIn