Jean-Christophe Castellana

Remember back in 2012, when PostgreSQL introduced JSON support ? https://lnkd.in/e_Qb8rpc Now in PostgreSQL 17 we have JSON_TABLE and new JSON functions !

3

Dynamic Tables are generally available ! Dynamic tables allow you to create data pipelines that are easy to build, operate, and evolve. With general availability, the following new enhancements are added: Sharing and collaboration: Dynamic tables can now be shared across regions and clouds using Snowflake’s sharing and collaboration features. This makes it easy to share clean, enriched, and transformed data products with consumers in your organization, partner organizations, or the broader data cloud community, ensuring they stay updated according to your specified cadence. Disaster recovery and replication: Dynamic tables now support high availability through Snowflake’s replication infrastructure. You can build your production pipelines with peace of mind knowing that you’re supported with Snowflake’s disaster recovery solutions. Observability: New functionality added for better observability via Snowsight and programmatic interfaces. In Snowsight, there are new account-level views, visibility into warehouse consumption, improved graph and refresh history, and the ability to suspend and resume refreshes. Observability functions now include new account usage views, extended retention of information schema functions and added support for consistent metadata across Snowflake observability interfaces. Data Cloud integrations: Added support for clustering, transient dynamic tables, and governance policies (on sources of dynamic tables and dynamic tables themselves), allowing you to benefit from the best features of the Snowflake Data Cloud. Scalability: You can now create four times more dynamic tables in your account, and ten times more dynamic table sources feeding into another dynamic table. There are no longer any limits on the depth of a directed acyclic graph (DAG) that you can create. Query evolution and SELECT * support: Dynamic tables now automatically evolve to absorb new columns from base tables and incrementally repair without needing to rebuild the dynamic table when new columns are added. New documentation: We’ve added new articles to our documentation covering development best practices, performance optimization guides, troubleshooting pipeline issues, and other improvements. https://lnkd.in/epkbrVTJ Our partner Indexima offers automatic dynamic table creation&maintenance based on query patterns ! Learn more in the interview (in French) https://lnkd.in/eFKDSsxj

33

𝐇𝐢 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐚𝐛𝐫𝐢𝐜 𝐟𝐚𝐧𝐬 𝐚𝐧𝐝 𝐜𝐮𝐫𝐢𝐨𝐮𝐬 My solutions to perhaps improve this fantastic SAAS transversal Data platform Visit please my last article (study) ==> https://lnkd.in/eNzKARhQ Example : topic #10 and DBT "best" "mandatory ?" support on Fabric <== Over T SQL Warehouses mainly #azure #microsoft #datagovernance #microsoftfabric #bi #data #dataanalytics #datafactory #dataengineering #dbt

1

When I read this first, I was like, Tools updated for MySQL 10.1 ? That has not been released yet... https://lnkd.in/e9EDWHWq #mysql

1

𝐃𝐞𝐚𝐫𝐬 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐀𝐁𝐑𝐈𝐂 𝐚𝐧𝐝 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐅𝐚𝐧𝐬 🧭 May I share with you 1) on architecture side, an advise around => 𝐢𝐧𝐭𝐞𝐫𝐧𝐚𝐥 𝐞𝐱𝐢𝐬𝐭𝐢𝐧𝐠 𝐒𝐐𝐋 "𝐁𝐈" 𝐝𝐚𝐭𝐚 (somewhere on my company DWH or Lakehouse) consumption on Power BI semantic models ✔️Avoid a 𝐝𝐚𝐭𝐚𝐟𝐥𝐨𝐰 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 your SQL DWH/LH domain data product and your Semantic models (also data product ? ...another discussion 😁) 2) and a "big" requirement for Microsoft ==> 𝐓𝐡𝐞 "𝐦𝐚𝐠𝐢𝐜" & "𝐠𝐥𝐨𝐛𝐚𝐥" 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐝𝐚𝐭𝐚 𝐬𝐨𝐮𝐫𝐜𝐞 𝐇𝐔𝐁 (the famous universal data broker) around 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 to 𝐅𝐀𝐁𝐑𝐈𝐂 𝐃𝐚𝐭𝐚 (ERP , SAP , Folders , SQL , API access token , Bigquery DWH , Snowflake DWH , Databricks etc ..) usages ==> a 𝐇𝐔𝐁 access also on pbi desktop ✔️ Useful : On all "ingestion type" artifacts , a "one click"<== ready to use usage ✔️Don't forget pbi desktop please MS <== currently we give Bigquery service account secrets to PBI Desktop builders persons - not safe for bigquery ✔️Isolation : External data sources are already setuped (and shared with me) somewhere by domain owners ✔️ FABRIC consumers are data factory dfgen2 & Data pipeline , PBI semantic model & data flow , notebook , mirroring setup , event stream , T SQL stored proc , python script , shortcut etc.. ✔️ Datamesh governance compliant : By domain management ✔️ Security : Avoid to transmit credentials to several persons ✔️ Cloud or on prem data sources (offer a 360° vision) ✔️ Credentials fill in directly or Azure key vault system access or (managed the SAAS access to external BI system ? ) This day INTEGRATION word 'll reach a strong step on Fabric #data #dataanalytics #datagovernance #powerbi #microsoftfabric #bi #azure #microsoft #DataTransformation #DataCulture #BusinessExperts #DataAcculturation #DataAwareness #DataMesh #DataCatalog

5

2 commentaires

𝐃𝐞𝐚𝐫𝐬 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐀𝐁𝐑𝐈𝐂 𝐚𝐧𝐝 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 𝐅𝐚𝐧𝐬 🧭 May I share with you 1) on architecture side, an advise around => 𝐢𝐧𝐭𝐞𝐫𝐧𝐚𝐥 𝐞𝐱𝐢𝐬𝐭𝐢𝐧𝐠 𝐒𝐐𝐋 "𝐁𝐈" 𝐝𝐚𝐭𝐚 (somewhere on my company DWH or Lakehouse) consumption on Power BI semantic models ✔️Avoid a 𝐝𝐚𝐭𝐚𝐟𝐥𝐨𝐰 𝐛𝐞𝐭𝐰𝐞𝐞𝐧 your SQL DWH/LH domain data product and your Semantic models (also data product ? ...another discussion 😁) 2) and a "big" requirement for Microsoft ==> 𝐓𝐡𝐞 "𝐦𝐚𝐠𝐢𝐜" & "𝐠𝐥𝐨𝐛𝐚𝐥" 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 𝐝𝐚𝐭𝐚 𝐬𝐨𝐮𝐫𝐜𝐞 𝐇𝐔𝐁 (the famous universal data broker) around 𝐄𝐱𝐭𝐞𝐫𝐧𝐚𝐥 to 𝐅𝐀𝐁𝐑𝐈𝐂 𝐃𝐚𝐭𝐚 (ERP , SAP , Folders , SQL , API access token , Bigquery DWH , Snowflake DWH , Databricks etc ..) usages ==> a 𝐇𝐔𝐁 access also on pbi desktop ✔️ Useful : On all "ingestion type" artifacts , a "one click"<== ready to use usage ✔️Don't forget pbi desktop please MS <== currently we give Bigquery service account secrets to PBI Desktop builders persons - not safe for bigquery ✔️Isolation : External data sources are already setuped (and shared with me) somewhere by domain owners ✔️ FABRIC consumers are data factory dfgen2 & Data pipeline , PBI semantic model & data flow , notebook , mirroring setup , event stream , T SQL stored proc , python script , shortcut etc.. ✔️ Datamesh governance compliant : By domain management ✔️ Security : Avoid to transmit credentials to several persons ✔️ Cloud or on prem data sources (offer a 360° vision) ✔️ Credentials fill in directly or Azure key vault system access or (managed the SAAS access to external BI system ? ) This day INTEGRATION word 'll reach a strong step on Fabric #data #dataanalytics #datagovernance #powerbi #microsoftfabric #bi #azure #microsoft #DataTransformation #DataCulture #BusinessExperts #DataAcculturation #DataAwareness #DataMesh #DataCatalog

13

1 commentaire

𝗧𝗵𝗲 #𝗣𝗨𝗥𝗜𝗦 𝗦𝘁𝗼𝗿𝘆 🎉 𝗣𝗮𝗿𝘁 𝟭𝟬: Let's do it!       👉 𝗦𝗼𝘂𝗿𝗰𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝗰𝗼𝗻𝘁𝗲𝗻𝘁: https://lnkd.in/evr572PA   𝗪𝗮𝗻𝗻𝗮 𝘀𝘁𝗮𝗿𝘁 𝘀𝗺𝗮𝗹𝗹? 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗠𝗶𝗻𝗶𝗺𝘂𝗺 𝗩𝗶𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮 𝗦𝗽𝗮𝗰𝗲 (𝗠𝗩𝗗𝗦)! "A Minimum Viable Data Space (MVDS) is a combination of components that enable the creation of a data space with just enough features to be usable for secure and sovereign data exchange, as specified by the International Data Spaces Association (IDSA). The goal of an MVDS is to streamline the implementation process, making it easier and faster for experimenters to create a working data space with secure and sovereign data exchange. By starting with an MVDS, the development team can iterate quickly and respond to the requirements of the data space, making adjustments as necessary to meet the needs of users. (…) This allows them to start with a first working version (where secure and sovereign data exchange is granted), where the development team can iterate, identify and respond to the assumptions about the requirements of the data space.   𝗧𝗵𝗲 𝗠𝗩𝗗𝗦 𝗶𝘀 𝘁𝗵𝗲 𝘂𝗻𝗶𝗾𝘂𝗲 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝗱 𝗯𝘆 𝗜𝗗𝗦𝗔 𝗛𝗲𝗮𝗱 𝗢𝗳𝗳𝗶𝗰𝗲, 𝗮𝘀 𝗰𝘂𝗿𝗿𝗲𝗻𝘁 𝗯𝗲𝘀𝘁 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲."   𝗪𝗵𝗮𝘁 𝗮𝗿𝗲 𝘁𝗵𝗲 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗱 𝗳𝗼𝗿 𝗮 𝗠𝗩𝗗𝗦 (𝗮𝘀 𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗲𝗱 𝗯𝘆 𝗜𝗗𝗦𝗔)? "A minimum viable data space consists of: 1️⃣ 𝗧𝘄𝗼 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗼𝗿𝘀 (one acting as a data provider, and one as a data consumer) 2️⃣ An 𝗶𝗱𝗲𝗻𝘁𝗶𝘁𝘆 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝗿 (Dynamic Attribute Provisioning Service, Certificate Authority) 3️⃣ 𝗢𝗽𝘁𝗶𝗼𝗻𝗮𝗹 𝗮𝗻𝗱 𝗮𝗱𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀, such as a metadata broker, an app store, a clearing house, or a vocabulary provider, can be added to the MVDS to extend its functionality and enable more advanced features, such as searching for data sets.   𝗧𝗵𝗲 𝗠𝗩𝗗𝗦 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝘀 𝗮 𝘀𝘁𝗮𝗿𝘁𝗶𝗻𝗴 𝗽𝗼𝗶𝗻𝘁 𝗳𝗼𝗿 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗲𝗿𝘀 𝘁𝗼 𝗰𝗿𝗲𝗮𝘁𝗲 𝗮 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗱𝗮𝘁𝗮 𝘀𝗽𝗮𝗰𝗲 𝘁𝗵𝗮𝘁 𝗰𝗮𝗻 𝗯𝗲 𝗰𝘂𝘀𝘁𝗼𝗺𝗶𝘇𝗲𝗱 𝗮𝗻𝗱 𝗲𝘅𝗽𝗮𝗻𝗱𝗲𝗱 𝗮𝘀 𝗻𝗲𝗲𝗱𝗲𝗱 𝘁𝗼 𝗺𝗲𝗲𝘁 𝘀𝗽𝗲𝗰𝗶𝗳𝗶𝗰 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗺𝗲𝗻𝘁𝘀."   PURIS | Predictive Unit Realtime Information Service | International Data Spaces Association (IDSA) | iSHARE Foundation | Catena-X |   Sebastian Steinbuß | Gianfranco Cecconi | Christoph Mertens | Rajiv Rajani MSc | Michael Hahn   #dataexchange #datasharing #dataecosystem #dataspaces #data #supplychain #resilience #WeAreCatenaX #dustofflogistics #lovewhatido

22

𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐅𝐀𝐁𝐑𝐈𝐂 𝐮𝐬𝐞𝐫𝐬 Following my previous post around Compagny DATA Ingestion on Bronze/Raw layers https://lnkd.in/eK_Q9a9F Some ideas based on my DATA experiences to maybe improve the last SQL on your architecture plan 🚀𝐋𝐨𝐚𝐝 𝐞𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐭𝐥𝐲 𝐚𝐧𝐝 𝐪𝐮𝐢𝐜𝐤𝐥𝐲 𝐒𝐐𝐋 𝐋𝐇 (𝐠𝐨𝐥𝐝) 𝐨𝐫 𝐃𝐖𝐇 (𝐠𝐨𝐥𝐝) 𝐭𝐚𝐛𝐥𝐞𝐬 𝐏𝐫𝐨𝐯𝐢𝐝𝐞 𝐚 𝐧𝐢𝐜𝐞 𝐒𝐐𝐋 𝐝𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧𝐚𝐥 (𝐊𝐢𝐦𝐛𝐚𝐥𝐥) 𝐦𝐨𝐝𝐞𝐥 𝐭𝐨 𝐏𝐨𝐰𝐞𝐫 𝐁𝐈 (𝐚𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬 𝐜𝐮𝐛𝐞) 𝐚𝐬 𝐃𝐨𝐦𝐚𝐢𝐧 𝐃𝐀𝐓𝐀 𝐏𝐑𝐎𝐃𝐔𝐂𝐓 Big deal : Challenge on these stacks & last SQL step with strong DATA champion editor : 𝐃𝐁𝐓 , 𝐁𝐢𝐠𝐪𝐮𝐞𝐫𝐲 , 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞  We are on DATA ANALYSTS garden , they would like something very efficient and rich 𝐇𝐨𝐩𝐢𝐧𝐠 𝐜𝐨𝐦𝐢𝐧𝐠 𝐬𝐨𝐨𝐧 𝐬𝐨𝐦𝐞𝐭𝐡𝐢𝐧𝐠 𝐥𝐢𝐤𝐞 𝐃𝐁𝐓 𝐟𝐨𝐫 "𝐊𝐢𝐦𝐛𝐚𝐥𝐥" 𝐢𝐧𝐝𝐮𝐬𝐭𝐫𝐢𝐚𝐥 𝐭𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧𝐬 𝐚𝐧𝐝 𝐃𝐐𝐂 - 𝐃𝐚𝐭𝐚 𝐪𝐮𝐚𝐥𝐢𝐭𝐲 𝐜𝐨𝐧𝐭𝐫𝐨𝐥𝐬 -  - 𝐃𝐃𝐋 𝐃𝐞𝐩𝐥𝐨𝐲𝐦𝐞𝐧𝐭𝐬 -𝐃𝐌𝐋 𝐑𝐮𝐧𝐬 - 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐇𝐨𝐰 𝐭𝐨 𝐬𝐚𝐲 ? , 𝐈 𝐥𝐢𝐤𝐞 𝐓 𝐒𝐐𝐋 𝐒𝐭𝐨𝐫𝐞𝐝 𝐩𝐫𝐨𝐜 & 𝐧𝐨𝐭𝐞𝐛𝐨𝐨𝐤 𝐚𝐧𝐝 𝐈 𝐚𝐝𝐦𝐢𝐭 𝐭𝐡𝐚𝐭 𝐭𝐡𝐞𝐲 𝐝𝐨 𝐭𝐡𝐞 𝐣𝐨𝐛 𝐛𝐮𝐭 𝐢𝐭 𝐢𝐬 𝐯𝐞𝐫𝐲 𝐟𝐚𝐫 𝐟𝐫𝐨𝐦 𝐃𝐁𝐓 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐯𝐢𝐭𝐲 𝐚𝐧𝐝 𝐞𝐚𝐬𝐞 𝐨𝐟 T SQL functions on Fabric DWH needs to be very rich Not on the chart , sorry ==> If we want to hide some RAW columns & rows , we need perhaps , like on Bigquery to offer SQL Authorized views (to RAW DWH data) for data analysts (helping load gold tables) Goal : Reduce time to market on build sessions, Kimball transformation to be industrial but simple , improve maintenance , readability , performances 5) [Rows snapshot based primary key and Type2 columns changes] => provide automatically start / end row version datetime , surrogate key 6) [Merge based primary keys and Type 1 columns updates] <== Remark : Perform a real merge ( Fabric LH & TSQL DWH need to accept the famous merge) 5) & 6) will help to easily manage the slow change dimension table by merging on a same SQL view 😁 𝐌𝐀𝐍𝐃𝐀𝐓𝐎𝐑𝐘 𝐎𝐍 𝐀 𝐌𝐎𝐃𝐄𝐑𝐍 𝐃𝐀𝐓𝐀 𝐏𝐥𝐚𝐭𝐟𝐨𝐦 ==> 𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠 𝐧𝐞𝐞𝐝𝐬 𝐭𝐨 𝐛𝐞 𝐂𝐈𝐂𝐃 & 𝐝𝐚𝐭𝐚 𝐜𝐚𝐭𝐚𝐥𝐨𝐠 𝐜𝐨𝐦𝐩𝐥𝐢𝐚𝐧𝐭 𝐚𝐧𝐝 𝐰𝐞 𝐧𝐞𝐞𝐝 𝐭𝐨 𝐠𝐞𝐭 𝐚𝐧 𝐢𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐞𝐝 𝐬𝐭𝐫𝐨𝐧𝐠 𝐝𝐚𝐭𝐚 𝐪𝐮𝐚𝐥𝐢𝐭𝐲 𝐬𝐞𝐫𝐯𝐢𝐜𝐞 𝐨𝐧 𝐅𝐚𝐛𝐫𝐢𝐜 (𝐒𝐐𝐋 𝐚𝐧𝐝 𝐏𝐛𝐢 𝐝𝐚𝐭𝐚) 𝐚𝐫𝐨𝐮𝐧𝐝 𝐜𝐨𝐦𝐩𝐥𝐞𝐭𝐞𝐧𝐞𝐬𝐬 , 𝐫𝐮𝐥𝐞𝐬 , 𝐩𝐫𝐨𝐟𝐢𝐥𝐥𝐢𝐧𝐠 Coming soon : Ideas around improve TESTS ( non regression , best practices) on cicd , Power BI workload #data #dataanalytics #datagovernance #powerbi #microsoftfabric #bi #azure #microsoft #DataTransformation #DataCulture #BusinessExperts #DataAwareness #DataMesh #DataCatalog #dataenginners #microsoft #datafactory #dataengineering

4

2 commentaires

Scaling Data in Big Data Technologies : In the dynamic world of big data, scaling your data infrastructure is essential for maintaining performance and reliability. Here are some key strategies to help you scale effectively: #Horizontal Scaling: - Add More Nodes: Increase the number of nodes in your cluster to distribute the load. - Distributed Storage: Use systems like Hadoop HDFS or Amazon S3 for seamless scaling. #Partitioning and Sharding: - Data Partitioning: Split large datasets into manageable chunks. - Sharding: Distribute data across multiple databases for load balancing. #Optimization and Tuning: - Query Optimization: Use indexing and caching to speed up data retrieval. - Resource Management: Fine-tune CPU and memory allocation with tools like Kubernetes. #Data Pipeline Scaling: -Stream Processing: Utilize frameworks like Apache Kafka Streams for real-time data. - Batch Processing: Employ Apache Spark for efficient batch processing. #Cloud Solutions: - Scalable Cloud Services: Leverage AWS, GCP, or Azure for scalable storage and compute. - Serverless Architectures: Use AWS Lambda or Google Cloud Functions for automatic scaling. #Monitoring and Management - Continuous Monitoring: Implement tools like Prometheus to track performance. - Automated Management: Use Ansible or Terraform for infrastructure automation. #Data Governance and Security: - Governance Policies: Ensure data quality and compliance. - Security Measures: Implement encryption and access controls to protect data. By following these best practices, you can build a scalable, robust big data environment to meet your growing data needs. #BigData #DataEngineering

9

🚀 PySpark Data Engineer Interview Questions Set (Intermediate Level)🔥 🔍 𝐈𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧: 1️⃣ Can you provide an overview of your experience with PySpark and big data processing?   2️⃣ What motivated you to specialize in PySpark, and how have you applied it in your previous roles?  ⚙️ 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐁𝐚𝐬𝐢𝐜𝐬:  3️⃣ Explain the basic architecture of PySpark.   4️⃣ How does PySpark relate to Apache Spark?  💻 𝐃𝐚𝐭𝐚𝐅𝐫𝐚𝐦𝐞 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬:  5️⃣ Difference between DataFrame and RDD?   6️⃣ Explain transformations and actions in PySpark DataFrames.  🚀 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐢𝐧𝐠 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐉𝐨𝐛𝐬: 8️⃣ How do you optimize PySpark job performance?   9️⃣ Techniques for handling skewed data?  📦 𝐃𝐚𝐭𝐚 𝐒𝐞𝐫𝐢𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐂𝐨𝐦𝐩𝐫𝐞𝐬𝐬𝐢𝐨𝐧:  🔟 Explain data serialization in PySpark.   1️⃣1️⃣ Significance of choosing the right compression codec?  🧠 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐌𝐚𝐜𝐡𝐢𝐧𝐞 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠: 1️⃣8️⃣ Discuss your experience with PySpark’s MLlib.   1️⃣9️⃣ Examples of machine learning algorithms you’ve implemented? 🌐 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 𝐰𝐢𝐭𝐡 𝐎𝐭𝐡𝐞𝐫 𝐓𝐞𝐜𝐡𝐧𝐨𝐥𝐨𝐠𝐢𝐞𝐬:   2️⃣2️⃣ Have you integrated PySpark with other big data technologies?  🔧 𝐑𝐞𝐚𝐥-𝐰𝐨𝐫𝐥𝐝 𝐏𝐫𝐨𝐣𝐞𝐜𝐭 𝐒𝐜𝐞𝐧𝐚𝐫𝐢𝐨: 2️⃣5️⃣ Describe a challenging PySpark project you’ve worked on and how you overcame the challenges.  [Happy Learning 📚] 👉Follow Aishwarya Pani(She/Her) for more data engineering-related materials and information. Save and reshare ✅ 𝑹𝒆𝒑𝒐𝒔𝒕 𝒊𝒇 𝒚𝒐𝒖 𝒇𝒊𝒏𝒅 𝒊𝒕 𝒖𝒔𝒆𝒇𝒖𝒍 #PySpark #BigData #DataEngineering #MachineLearning #Spark #DataFrames #ClusterManagement #JobOptimization #TechInterview #DataScience

21

💠 Types of Databases every Data Scientist must know : 🔸 Relational Databases like MySQL & PostgreSQL - the traditional structured storerooms. 🔸 NoSQL Databases (e.g., MongoDB, Redis) - when flexibility & scalability are the game's names. 🔸 Graph Databases such as Neo4j - mapping relationships intricately as a spider weaves its web. 🔸 Distributed Databases (Apache Cassandra, Amazon DynamoDB) - spanning across multiple nodes for high availability. 🔸 In-memory databases like Redis - lightning-fast data access at the tip of your neurons. 🔸 Spatial Databases (PostGIS) - for when location matters as much as the data itself. 🔸 Time-Series Databases (InfluxDB) - capturing every moment in a data-ticking timeline. 🔸 Blockchain Databases (BigchainDB) - chaining blocks of data with security and transparency. 🔸 Object-Oriented Databases (db4o, ObjectDB) - when objects in programming find their storage mates. #databases #programming #coding #dataset #data #dataanalyst #dataanalysis #datascience #datascientist #db #dbms #dba #Blockchain #BigchainDB #ObjectDB #sql #nosql #oracle #db2

3

When does it actually make sense to convert your Relational Database to a Graph Database like Neo4j? 🤔 We’re used to traditional relational databases, but as data grows more complex and interconnected, sometimes an RDBMS isn’t enough. Here’s when you can consider moving to a Graph DB 👇: 1️⃣  If you feel that your relational database is struggling when dealing with many-to-many relationships or deeply nested data, then Graph DBs excel here by offering faster traversal between connected data points. 2️⃣ Updating the schema in an RDBMS can be a headache. Graph DBs provide schema flexibility allowing you to effortlessly add new relationships and data types without heavy restructuring. 3️⃣ If your data is naturally interconnected like social networks, fraud detection or supply chain networks, representing it in a graph structure mirrors reality, simplifying both storage and analysis. 4️⃣ Are complex queries like finding the shortest paths or identifying patterns causing performance bottlenecks? Graph DBs store relationships directly, allowing for faster, more efficient queries. But hold on... While Graph DBs offer many advantages, they come with their own set of performance challenges, especially in knowledge graph construction. Stay tuned to discover strategies for building efficient knowledge graphs! 👋 #graphdb #neo4j #dataengineering #datatransformation

66

1 commentaire

In this video I will explain about the Delta tables. Delta tables are a new type of table in Databricks that provide a powerful and efficient way to work with big data. They are optimized for fast, read-intensive, large-scale data processing and are ideal for use cases such as data lakes. Tutorial:- https://lnkd.in/gyKXzEpH #databricks #machinelearning #datascience #DatabricksIngestDatafromAzureSQL #AzureSQLDatabaseDatabricks, #DatabricksAzureSQL, #DataricksAzureDatabase,#Databricksreaddbtable,#DatabricksReadDatabaseTable, #SparkReadSQLTable,#SparkIngestDBTable,#SparkIngestDataBaseTable,#PysparkIngestDataBaseTable,#SparkLoadfromDBTable,#SparkReadfromDatabase,#DatabricksReadfromDatabase, #DatabricksJDBC,#SparkJdbc, #PysparkJDBC,#DatabricksAzureSQL,#SparkAzureSQLDB,#SparkAzureSQLDatabase #PySparkAzureSQLDatabase #DatabricksTutorial, #DatabricksMergeStatement, #AzureDatabricks,#Databricks,#Pyspark,#Spark,#AzureDatabricks,#AzureADF

3

🚀 DE Interview Experience 📢 Apache Spark ⌛ 1-Min-Read How to use write with partitionBy in Spark for efficient data storage and querying. 👉🏽 🧔🏽‍♂️Interviewer: Can you explain how you use the write method with partitionBy in Spark to optimize data storage? 👨‍🦰Candidate: Using the write method with partitionBy in Spark allows you to partition the output data based on the values of one or more columns. This improves query performance and makes data storage more efficient. 🧔🏽‍♂️Interviewer: Can you walk me through how you typically use partitionBy? 👨‍🦰Candidate: Let's say we have a DataFrame df containing sales data, and we want to partition the data by the region column before writing it to disk. df.write.partitionBy("region").parquet("output_path") In this example, the data is written to the output_path directory in Parquet format, partitioned by the region column. This means that the data will be organized into separate directories for each unique value in the region column. 🧔🏽‍♂️Interviewer: How does this improve performance? 👨‍🦰Candidate: Partitioning data by a specific column improves performance in several ways: Efficient Data Retrieval: When querying the data, Spark can skip reading entire partitions that do not match the query criteria, thus reducing the amount of data read from disk. Improved Compression: Partitioning can lead to better compression ratios since data within each partition tends to be more homogeneous. Parallel Processing: Each partition can be processed independently, enabling better parallelism and distributed processing. 🧔🏽‍♂️Interviewer: Can you provide a real-world example where you used partitionBy? 👨‍🦰Candidate: Absolutely. In a project where we needed to store large amounts of user activity data, we partitioned the data by date and user_id. user_activity_df.write.partitionBy("date", user_id").parquet("s3://bucket/user_activity") 🧔🏽‍♂️Interviewer: That’s an excellent example! Are there any other benefits to using partitionBy? 👨‍🦰Candidate: Yes, in addition to performance improvements, using partitionBy helps in data organization and management. It makes it easier to manage large datasets by breaking them down into more manageable chunks. This is particularly useful for archiving old data or performing maintenance tasks on specific partitions without affecting the entire dataset. #DataEngineering #ApacheSpark #DataPartitioning #BigData #DataStorage #DataOptimization #CareerGrowth #OneStepAnalytics

17

Which is better Scala spark or Pyspark for your Spark project ? The choice between Scala Spark and PySpark depends on your specific use case, expertise, and the requirements of your project: 1. Performance: • Scala Spark: Generally performs better since Spark is written in Scala, and the code written in Scala can directly interface with the Spark engine without any abstraction layer. This leads to faster execution and optimization. • PySpark: Although slightly slower than Scala due to the Python abstraction, PySpark is still performant and widely used for handling large datasets. The performance difference is often negligible for many use cases. 2. Ease of Use: • Scala Spark: Scala is more complex to learn and use, especially for those not familiar with functional programming. It has a steeper learning curve but offers more control over Spark operations. • PySpark: Python’s simplicity and ease of use make PySpark a preferred choice for data scientists and engineers who are familiar with Python. Python has a vast ecosystem for data science, which makes it easy to integrate PySpark with other tools. 3. Community and Ecosystem: • Scala Spark: Has a strong community, especially among big data engineers and performance-focused users. It’s commonly used for production-level code in enterprises. • PySpark: Python has a much larger community overall, particularly among data scientists and machine learning engineers. PySpark benefits from Python’s extensive libraries, making it easy to incorporate data science workflows. 4. Integration with Data Science Tools: • Scala Spark: Better for building and optimizing Spark-specific applications and jobs. • PySpark: Easier integration with popular data science libraries like pandas, NumPy, and scikit-learn, making it a great option if your work involves machine learning and exploratory data analysis. 5. Job Market: • Scala Spark: Tends to be more in demand in roles that focus on big data engineering and performance optimization. • PySpark: Often preferred in roles that combine data engineering and data science or machine learning tasks. Conclusion: • Choose Scala Spark if: • You’re focused on big data engineering, optimization, and need maximum performance. • You or your team are comfortable with Scala or JVM languages. • Choose PySpark if: • You prefer Python, need faster development cycles, and want to leverage Python’s rich ecosystem for data analysis, machine learning, or AI tasks. The decision should be based on the skills of your team, performance needs, and integration with other tools.

1

Explain the difference between repartition() and coalesce() in PySpark: My answer: Challenge accepted, let’s go! ➤ 𝗿𝗲𝗽𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻() 𝘁𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: The repartition() transformation is used to increase or decrease the number of partitions in a DataFrame or RDD. It performs a full shuffle of data across all nodes in the cluster, which can be costly but ensures even distribution of data. # Example DataFrame df = spark.createDataFrame([("Alice", 29), ("Bob", 34), ("Cathy", 28), ("David", 45)]) # Original number of partitions print(df.rdd.getNumPartitions()) # Output might be 4 (default) # Increase the number of partitions to 6 using repartition() repartitioned_df = df.repartition(6) # Check the number of partitions after repartitioning print(repartitioned_df.rdd.getNumPartitions()) # Output: 6 ➤ 𝗰𝗼𝗮𝗹𝗲𝘀𝗰𝗲() 𝘁𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻: The coalesce() transformation is used to reduce the number of partitions in a DataFrame or RDD. It avoids a full shuffle by moving data only between fewer partitions, making it more efficient than repartition() when reducing partitions. # Example DataFrame df = spark.createDataFrame([("Alice", 29), ("Bob", 34), ("Cathy", 28), ("David", 45)]) # Original number of partitions print(df.rdd.getNumPartitions()) # Output might be 4 (default) # Reduce the number of partitions to 2 using coalesce() coalesced_df = df.coalesce(2) # Check the number of partitions after coalescing print(coalesced_df.rdd.getNumPartitions()) # Output: 2 𝗞𝗲𝘆 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝗰𝗲: - repartition() involves a full shuffle and is used to both increase and decrease partitions. - coalesce() minimizes data movement by avoiding a full shuffle and is used primarily to decrease the number of partitions.

28

1 commentaire