Navigating the AI landscape! 🤖🚀💼🌐 AI Leader, Investor, & Advisor | MassChallenge | Wharton VentureLab
Tel Aviv and IBM teams question the conventional benchmarking practices, which typically involve training models from scratch with random initialization as modeling long-range dependencies in sequences has led to notable architectural advancements, with state space models (SSMs) emerging as a significant alternative to Transformers. According to the team, this method may overestimate the differences between architectures. The researchers propose pretraining models using standard denoising objectives with downstream task data, a method they term selfpretraining (SPT). This approach significantly narrows the performance gap between Transformers and SSMs. For example, pretrained vanilla Transformers can match the performance of advanced SSMs like S4 on benchmarks such as the Long Range Arena (LRA). Specifically, SPT improved the best reported results of SSMs on the PathX-256 task by 20 points. Key findings from the study include: 1. Transformers vs. SSMs: Properly pretrained vanilla Transformers can achieve performance comparable to S4 on LRA tasks, challenging the notion that Transformers are less capable of modeling long-range dependencies. 2. Redundancy of Structured Parameterizations: Structured parameterizations in SSMs become mostly redundant with data-driven initialization through pretraining, suggesting that simpler models can match the performance of more complex architectures. 3. Effectiveness Across Data Scales: SPT is particularly beneficial when training data is scarce, with relative gains more pronounced with smaller datasets. 4. Adaptability of Convolution Kernels: Data-driven kernels learned via SPT adapt to specific task distributions, enhancing performance on long-sequence tasks. The study emphasizes the importance of incorporating a pretraining stage in model evaluation to ensure accurate performance estimation and simplify architecture design. This approach not only provides a fair comparison between different architectures but also highlights the efficiency of pretraining in leveraging task data. Arxiv: https://lnkd.in/enaH3mhu
