Erlend Rongen

https://lnkd.in/gxKxFeEA Hedging with Implied vs. Actual Volatility A detailed THREAD with an intuitive and mathematical explanation Follow Us on X, for more interesting Thread https://lnkd.in/ggsyM4sr

14

1 Comment

I've been following the development of the #tidymodels framework for machine learning in R since the beginning, and it never ceases to amaze me how the team at Posit PBC has added more and more functionality to {tidymodels}. (Screenshot from Hannah Frick's talk about "tidymodels for time-to-event data" at posit::conf(2024) yesterday, which I just rewatched.) #PositConf2024 #rstats #DataScience

88

🤔 Spurious Correlation 🤔 Correlation does not always imply Causation When people look for evidence to support their arguments, they may use spurious correlation to do so. “[Ratcliffe] informed the club’s ­approximately 1,000 ­employees that email traffic dropped by 20% when one of his companies ­experimented with work-from-home Fridays, which he cited as the reason for his diktat.” This insinuation implies that e-mail traffic means more work is getting done. So people sending less e-mails means they are doing less work. I would be highly doubtful about that, “e-mail” traffic is a sign that people are sending e-mails not necessarily being more effective. However, he wants to drive cultural change and used this as his reason. How would you measure productivity?

4

1 Comment

A new BIS paper by Gambacorta, Qiu, Shan and Rees finds that generative AI increases coder output by more than 50%, but that productivity gains are only statistically significant among entry-level/junior staff. In part, this reflects less frequent use of gen AI by senior programmers. This suggests that while generative AI can boost productivity, its adoption and usage patterns differ greatly based on experience levels. This highlights the importance of tailored strategies to maximize the benefits of AI across varying seniority levels in the workforce.

11

💫 Optimal Storage for #Trading Factors Calculated at Mid to High Frequencies 💻 Maintaining a large number of factors is a common challenge in #quantitative analysis. In many cases, the volume #factor data can far exceed that of high-frequency market data. For example, calculating factor #data for a year across 5,000 securities can yield substantial volumes: 10-minute intervals would generate 2.3TB yearly, 1-minute intervals 23TB yearly, and 3-second intervals 460TB yearly. This massive amount of data requires thoughtful storage solution planning. 📜 This tutorial compares different #DolphinDB storage solutions for factors calculated from high-frequency data using real-life scenarios to provide recommendations for efficient storage. ➡ https://lnkd.in/gFzd4NZK

5

Quantpedia Explains (Trading Strategies) Episode 95 is live - How to Improve Commodity Momentum - Boost commodity momentum performance by using a correlation filter to switch between momentum and reversal. - video: https://lnkd.in/dPGsV7gY - blog post: https://lnkd.in/dN_AQpg2

1

Multivariate Time Series (MTS) Forecasting: A Persistent Challenge in Receivables Management - Could the Dozerformer Be the Breakthrough? Accurate solutions for predicting bad debts and general debtor behavior are often proclaimed, but the reality is that it remains an unsolved problem. This challenge is not only a fascinating scientific pursuit but also the holy grail in the field of predictive analytics for receivables management. A truly effective method for accurately predicting bad debts or recovery rates would instantly have a massive impact on cost efficiency and debt recovery success. However, this issue proves to be extremely persistent. Predicting default rates and probabilities during the collection process deals with high-dimensional and strongly correlated data, and the temporal dimension must be considered at the level of events, debtor data, and payment behavior. As someone who has been dealing with this topic for many years, I have tried a multitude of methodologies myself, including classical attention or LSTM networks. While some of these methods have led to significant successes and substantial business impact, it has been partial at best. Therefore, it is all the more exciting that a new attention framework, the so-called Dozerformer, has been developed. This novel framework is designed to tackle the complex problem of Multivariate Time Series (MTS) forecasting. The proposed Dozer Attention mechanism, a key feature of the framework, is made up of three components: Local, Stride, and Vary. These components are designed to capture the essential attributes of MTS data, such as locality, seasonality, and global temporal dependencies, and adaptively utilize historical time steps based on the forecasting horizon. The Dozerformer was evaluated on nine benchmark datasets and demonstrated superior performance in both accuracy and efficiency compared to recent state-of-the-art methods. The study also includes an ablation study to validate the effectiveness of the Dozer Attention mechanism. The research concludes that the Dozerformer significantly outperforms current methods, making it a promising tool for solving complex and real problems in modeling and predicting multivariate time series. This development is particularly intriguing, and I am eager to see if this novel approach will truly facilitate a breakthrough for the transformer architecture in addressing complex, real-world challenges in creating models and forecasts for multivariate time series, particularly in the context of financial services and receivables management. #AI #genAI #generativeAI #artificialintelligence #PredictiveAnalytics #TimeSeries #Forecasting #TechNews #DebtCollection #creditrisk #CreditManagement #ClaimsManagement #quantitativefinance  #financialservices https://lnkd.in/eWRjpYWk

12

1 Comment

There is one big thing you probably don't realise about the diagram (which is excellent btw) from Dylan Anderson below. And that is that Timeliness and Completeness, probably two of the BIGGEST issues in Data Quality are in some ways not data quality issues at all. And that is because if there *is* no data (you could argue) there IS no data quality issue. Why is data not timely? Why does it matter? It happens because processes fail or data fails to get delivered on time. It matters because STAKEHOLDERS FIND OUT before you do --> and why is that? Because your architecture, and probably whatever is orchestrating things, isn't set-up well. The same is true of completeness. Apart from no-one realises. And then the system recovers. And then a stakeholders comes to you and says "hey where is last week's data by the way?" I think this really highlights the importance of ensuring you have robust architecture while also monitoring quality metrics both in a "blocking" sense (i.e. pipeline stops if something fails e.g. in dbt) vs. non-blocking (e.g. Data Quality testing tools or pipelines that are simply data quality tests). You can read some thoughts on this below And give Dylan a follow 😎 #dataengineering #dataquality #dbt

42

4 Comments

About input and output metrics, I had a random thought during some conversations in the last two weeks. I want to validate it more before I put it into writing. In general terms, BI (and, therefore, most data stacks) focus on output metrics. It is mainly revenue-based or COGS when in e-commerce. One exception is the addition of very high-level and early input metrics (visits in an e-commerce case). Marketing/Product analytics, in comparison, heavily focuses on input metrics like early funnels and retention developments but often lacks the output level. Again, one exception: high-level revenue (frequently more unreliable because it is tracked in the browser). What do you think? Kind of right or completely off? If this is the proper tendency, it explains to some degree why BI people don't get analytics and vice versa.

22

21 Comments

Something many CTOs and Heads of Data don’t realise is how shit hot you have to be to build everything in house The average salary for a data engineer is $130k in the US So yes of course you can build your own ELT - just for the sunk cost of at least $130k and all the lead time associated with building out and deploying your connectors. If you’re hiring scores of $200k+ data folks then yes you should obviously try to build stuff yourself. You have to consider whether this is a realistic strategy to pursue #totacostofownership #Elt #software #salary

115

51 Comments