Ziv K.’s Post

If you want to learn GPU programming and understand how GPUs work at the hardware level, this open source project is highly recommended! tiny-gpu is a minimalistic GPU implementation that has been optimized to allow you to learn the working principles of GPUs from scratch. With the growing trend of general-purpose GPUs (GPGPU) and ML accelerators (like Google's TPU), tiny-gpu focuses on emphasizing the general principles behind all these architectures, rather than the specific details of particular hardware. https://lnkd.in/gDJ5sAKE

Unlock the power of OpenMP with advanced device offload! Dive into strategies to enhance parallel computing efficiency by optimizing workloads across CPUs and GPUs. Explore practical tips to elevate your development and achieve better performance. Learn more: https://intel.ly/3ZjGraS #oneAPI #OpenMP

As the HPC community progresses further into the Exascale era, a key challenge is how to program our supercomputers. Even at the current scale of HPC, it is a major investment to write highly optimal distributed memory code that will efficiently leverage CPUs and GPUs. Domain Specific Languages (DSLs) are one solution to this programmability challenge. EPCC is part of the xDSL project which has been developing shared DSL compiler infrastructure. Read about this work in Nick Brown's article "A shared compiler stack for stencil-based domain specific languages" on our website: https://edin.ac/4cvdpsQ Gabriel Rodriguez-Canal

SCALE: A multi-vendor GPGPU platform. We are excited to announce the beta release of SCALE: a cross-platform solution to GPGPU programming with the potential to liberate the GPGPU market from its current single-vendor dominance. 🚀 We believe that it should be possible to write code once, and build/run it on any hardware platform. This has been a reality for CPU code for many years – now SCALE is the solution for GPUs. At Spectral Compute we believe that SCALE has the potential to transform the entire GPGPU market, removing supply chain concerns, high costs, shortening development times, eliminating the need to maintain multiple codebases, and so much more! We want everyone to have the freedom to choose their GPUs based on the hardware features that support them best, without having to move away from CUDA, which we all know and love. The SCALE platform:  • Extends CUDA – it’s fully compatible with CUDA, but adds great new     (optional) features. • Targets AMD GPUs – so CUDA code can be run on AMD GPUs, with no modifications required! • SCALE offers language extensions that make writing GPU code easier and more efficient. Writing SCALE feels more like writing modern C++ than C. • Avoid porting your existing codebase away from CUDA, which is required by alternative GPGPU programming solutions. Porting away from CUDA would be expensive, slow, and error-prone. • Provides implementations of the CUDA runtime and driver APIs for AMD GPUs • Is tested on an ever-growing set of open-source CUDA projects, including: NVIDIA Thrust, Blender Cycles, AMGX, llama-cpp, faiss, xgboost, GOMC, stdgpu, hashcat • Currently supports AMD gfx1030 (Navi 21, RDNA 2.0) and AMD gfx1100 (Navi 31, RDNA 3.0) GPUs. Our team has worked tirelessly for years to ensure that SCALE is the solution to wide CUDA compatibility that GPGPU programmers need. SCALE allows programs written using CUDA to be compiled for and executed on AMD GPU hardware, without rewriting them or compromising on performance. Help us liberate the GPGPU market today. AMD support is just the beginning. More vendors are on the way! 🥂 Download the free SCALE beta here: https://scale-lang.com/ 🥂 Best regards, Ruben van Dongen Business Development Manager Spectral Compute #GPGPU #NVIDIA #AMD #HPC #HIP #ZLUDA #Programming #performance #SCALE #Spectral #AI #Machinelearning #C #CPP #CUDA #AMGX #Goofit #Faiss #Hashcat #llamacpp #gomc #nvcc #GPU #compiler #blender

Software Engineering and GPUs - Getting Started with CUDA using Thrust: Several months back I posted about utilizing the thrust library in C++ as a high level interface for utilizing GPUs. Some pros of thrust: - Little to no boilerplate setup - very similar interfaces to the STL - pretty efficient for how much it abstracts away The main con is that it may not be suitable if you need to maximize performance. If you start to use thrust more heavily, it can be good to check on the calls it is making using nsight systems / compute to profile. That being said, you can get a nice performance boost simply by using it out of the box in a lot of cases. The amazing thing about thrust is how easily you can gain access to utilizing your GPU - which is what I want to showcase here. Definitions: Host = CPU Device = GPU Let's talk about the example code below: Line 15: Initialize a host vector with 32M elements. This is awesome because like a std::vector, this will clean up after itself, managing its memory. Line 19-21: I mask the higher order bits so I can assign uint8 vals to all 32M elements Line 28: Similarly, this will allocate device memory (and clean up after itself), but it will also perform the host to device copy. With CUDA, you have to manage all this yourself. Line 29-33: A for_each function that will apply the lambda expression to each element in the device array. The __device__ attribute specifies that this will run on GPU! Line 36: Also pretty cool, this handles the data copy from device back to host. That's it! You are now running a massively parallel for_each on your GPU. I find this to be pretty cool. If you like my content, feel free to follow or connect! #softwareengineering #GPU

Inferencing optimization just went next level with Microsoft releasing bitnet.cpp Bitnet.cpp is an inferencing framework for 1-bit LLMs. And it's almost 2x faster than Llama.cpp for equivalent models when running on CPUs. They've run some absolutely crazy evals like running a 100 Billion parameter model on just the CPU. This is beyond ridiculous. Even 6 months ago, this would have been a crazy achievement. Quoting the Matrix: This is a triumph equaled only by its monumental speeds. At 100 Billion parameters on the CPU, they are getting almost 5 tokens/ sec. It's not great but it's faster than humans can read, which means it's going to be good for many applications right off the bat. But not everything is clear right now about how good it is. I haven't run the benchmarks myself, but the ones listed on their GitHub repo don't mention much about how they compared it to llama.cpp. They have mentioned model sizes from 125M to 100B but haven't mentioned which quant it is. Also they have run benchmarks for 100 Billion parameter model and also said that their models are all existing models on HuggingFace. But where are these models? People are joking on reddit that we got an entire framework before good models to run this framework on. Overall though, strange times but crazy progress.

https://lnkd.in/dnF59CXh wow Microsoft Is commoditizing LLMs with 1-bit models running on CPUs. quality is not fully there but this appears to be a major step

Microsoft released the official inference framework. It doesn't require paying big money to run model at home now. The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. https://lnkd.in/gZmypn2b

Debugging CUDA applications using ARM64 and NVIDIA GPUs. Learn more >> #hpc #debugging #CUDA

Java #Llama2 fork extended with GPU support using Level Zero JNI library to run on Intel ARC and Integrated GPUs. The initial version from my colleague Michalis Papadimitriou included a port for #TornadoVM. We perform this study to analyze the performance impact of shared memory for LLMs applications such as Llama. The Level Zero JNI library (https://lnkd.in/g8jaZBVY) is an open source framework to program and manage GPUs from Java, and we implemented this library to dispatch SPIR-V kernels in TornadoVM. However, currently TornadoVM does not implement shared memory. 👉 We see that the new level-zero port achieves higher tok/s vs TornadoVM on Integrated GPUs. So, what's the catch? The Level-Zero JNI contains hand-written kernels with explicit runtime calls to manage and dispatch the code on GPUs, while in TornadoVM, well, it's parallelized and automatically accelerated from the Java sequential code. Besides, in near future, TornadoVM could implement shared memory buffers, and get extra performance on shared memory systems. Repo: https://lnkd.in/eT3n4AcN The system we took for evaluation: - 13th Gen Intel(R) Core(TM) i9-13900K - Intel ARC 770 - Intel HD Graphics UHD 770 - Level Zero: 65a95d9 - Level zero JNI: 614855c - TornadoVM: 1.0.8 and JDK 21 We showed all details at the oneAPI Dev Summit last week. https://lnkd.in/esGM-aGw