Author Topic: AI, LLM, Macro forecasting, and Quants?  (Read 1493 times)

Financial.Velociraptor

  • Magnum Stache
  • ******
  • Posts: 2522
  • Age: 52
  • Location: Houston TX
  • Devour your prey raptors!
    • Living Universe Foundation
AI, LLM, Macro forecasting, and Quants?
« on: September 11, 2024, 09:04:23 AM »
For years there have been quant funds that have bots scouring news and various reporting statistics to influence quant investing.  There was once a hiccup where a company named Berkshire (but not BRK-A/B) had a press release with very favorable comps.  BRK-A, BRK-B briefly popped like 15% on the day as news bot driven trading pumped millions into the shares as an automated response. 

Where is that going?  And what triggers this for me is I looked at the betting futures on Kamala/Trump to see if the debate moved them (only a teeny bit).  What is the utility of these new technologies when comes to predicting macro trends?  Can they predict elections?  Can they predict say interest rates and/or inflation?  Can than spot the top in a bubble market? 

A lot of money is going into the hardware side of AI right now.  Surely, there are quant trading firms out there looking for the next advantage in the data centric arms race?  But I see no one reporting on their efforts.  Maybe it is TOP SECRET so the competition doesn't steal your ideas or best people?

ChpBstrd

  • Walrus Stache
  • *******
  • Posts: 8261
  • Location: A poor and backward Southern state known as minimum wage country
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #1 on: September 11, 2024, 12:51:41 PM »
The political betting markets may be too slow to react to Harris' performance of a lifetime because both the bots and the humans involved are consuming information from right-wing media saying that Trump clearly won. This is the analogue to the people trying to influence stock prices with fake press releases or pump-and-dump info, or the example you gave where the bot failed to understand the context of the information about "Berkshire". Humans make errors like that too, and EMH assumes their mistakes must even out across large numbers!

I think this illustrates a limitation to human and artificial intelligence. The outputs are only as good as the inputs. A better AI, or a smarter human, would be able to understand language information within a broader context of source reliability, the incentives sources have to provide misinformation, whether a piece of information is far-fetched, consistency with intellectual frameworks such as physics, medicine, or economics, and consistency with sensory information about the real world.

E.g. there were tons of doomer videos, podcasts, and interviews saying the US is already in recession in the 2nd quarter, but then GDP growth was announced at a massive +3%. My human critical thinking process notes the discrepancy, identifies the more reliable source of information, and discounts the value of the contradictory information's specific sources. Diving further, I draw conclusions about the tendency of internet users to seek out negative information, which drives ad clicks, which incentivizes content creators to produce the content that maximizes their revenue. A third level of cognition ties the bias of internet content consumers to prefer negative content with what I've previously learned about negativity bias, and this entire structure of thoughts leads me to broader conclusions about other ad-driven information sources and human behavior.

Thus, the observation of YouTuber doomers being wrong leads me to also discount the value of ad-supported Associated Press and Reuters articles, and inclines me to search for information sources which don't have the characteristic of being funded by internet traffic. A fourth level of cognition identifies subscription based sources, academic articles, and government statistic sites as potentially being more reliable than the entire class of ad-supported sources. A fifth level of cognition checks this impulse, and notes that these sources may have their own different systematic biases.

This string of thoughts through a cloud of information comes together as my own conclusions, behavioral tendencies, ideological frameworks, blind spots, and expectations. However different human minds would take a different path through all the inferential data, or perhaps stop at a lower level than where I stopped. Some human minds conclude the YouTubers are right and the government falsifies their data for political purposes. Are they wrong? Can we prove it?

This illustrates the challenge involved with deciding whether an AI is "working" or not. Human minds are a dime a dozen, so the hurdle for AI is to be significantly better at thinking than human experts*. Yet if human minds are all over the place in terms of their accuracy and validity, then what does a working AI look like? Perfect accuracy? 51%? Pass the Turing Test and hope for the best?

AI's are at a natural disadvantage because they lack sensory inputs from the real world. An AI can process a press release with positive language about a new car model, but cannot learn from a test drive or understand that the car is ugly. It can process a restaurant chain's financials but not understand that the food quality and service have slowly taken a dive. It can understand the specifications of a gadget, but not the functionality in a user's hand, or the aurora of status implied in the advertising campaign, or how any of this comes together as a human experience. The AI also does not naturally want. Maslow's Hierarchy is a piece of information in a framework, not a lived reality in a human meat-body with various urges, an unending stream of sensations, and a constantly changing environmental and cultural context for interpreting these motivations.

Thus it will take a lot of tricky programming for an AI to predict human behavior - and economics is at its root a behavioral science, not a branch of math. An AI could tell us that people like playing video games, but it will have a hard time determining which new games will be most popular.

If we threw a lot of AI power at the market, I think it would become less efficient at incorporating "all available information" and that would open up opportunities for quality testers, early adopters, fashion leaders, scientifically minded folks, contrarian thinkers, and narrative-thinkers who could spot situations where the AI was drinking its own kool-aid or where changing circumstances would affect humans in a certain way.

TL;DR: Critical thinking is a qualitative process and experience that may be hard for a neural network to emulate for reasons that may be inherent to being human versus being machine. Because inputs could lead to different outputs when run through different brains, or even the same brains at different times, there is no one correct and verifiable solution to inductively-informed narratives. This means we will not be able to tell whether an AI is thinking correctly by comparing it to human reference points, or even our own outputs. To understand economics and investing, the AI would need to emulate human psychology to some extent. E.g. How many people will buy an iPhone 16 when the specs have barely changed since the 15? None? All? 30%?

*In some applications, such as maybe reacting to earnings reports or exploiting momentary B/A spreads, an AI could be useful if it could draw faster conclusions, even if the accuracy was lower than a human whose conclusion would not arrive in time to exploit the gaps. But these are essentially the trading bots we already have, and have had for decades.

Financial.Velociraptor

  • Magnum Stache
  • ******
  • Posts: 2522
  • Age: 52
  • Location: Houston TX
  • Devour your prey raptors!
    • Living Universe Foundation
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #2 on: September 11, 2024, 01:11:59 PM »
@ChpBstrd

Thanks.  That was well reasoned.  I share many of your concerns about the limitations of the technology, especially the GIGO problem.  We have a Republic instead of a Democracy for a reason.  Imagine a USA with MANDATORY 100% voting.  The people who make the Kardashians a thing would decide (Kanye or the Rock, or maybe Madonna would become president). Or maybe 4chan would make Prezzy McPresident the POTUS...  So the 'killer app' is a source handicapping algorithm that can adapt and grow over time.   

I also remember DARPA had a forecasting market tool that got shut down because people lost their shit when it was discovered you could make a derivatives bet on terrorism (and for example enormously financially benefit from nuking Belgium.)  Such a tool can become sort of recursive where observations begin to alter the behavior of the underlying.

Michael in ABQ

  • Magnum Stache
  • ******
  • Posts: 2820
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #3 on: September 11, 2024, 01:27:45 PM »
For years there have been quant funds that have bots scouring news and various reporting statistics to influence quant investing.  There was once a hiccup where a company named Berkshire (but not BRK-A/B) had a press release with very favorable comps.  BRK-A, BRK-B briefly popped like 15% on the day as news bot driven trading pumped millions into the shares as an automated response. 

Where is that going?  And what triggers this for me is I looked at the betting futures on Kamala/Trump to see if the debate moved them (only a teeny bit).  What is the utility of these new technologies when comes to predicting macro trends?  Can they predict elections?  Can they predict say interest rates and/or inflation?  Can than spot the top in a bubble market? 

A lot of money is going into the hardware side of AI right now.  Surely, there are quant trading firms out there looking for the next advantage in the data centric arms race?  But I see no one reporting on their efforts.  Maybe it is TOP SECRET so the competition doesn't steal your ideas or best people?

Elections are very hard to predict as your model changes every 4 years and historic data quickly becomes irrelevant. The sample size is very small so it's hard to draw meaningful conclusions for any single election.

I'm sure there are companies out there trying to leverage LLMs and other "AI" for investing. However, everyone knows the models occasionally make stuff up so betting money on the results from one could be extremely risky. I expect that as they improve the market will become more efficient about quickly processing data and investing accordingly. But even with high frequency trading and all the information available the market is still inefficient and prone to react based on human biases.

Financial.Velociraptor

  • Magnum Stache
  • ******
  • Posts: 2522
  • Age: 52
  • Location: Houston TX
  • Devour your prey raptors!
    • Living Universe Foundation
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #4 on: September 11, 2024, 01:53:55 PM »
But even with high frequency trading and all the information available the market is still inefficient and prone to react based on human biases.

I remember back in grad school (B-school), I was made to 'prove' (using THE ONE HIGH AND HOLY MATHEMATICS) that markets are in fact, efficient.  There are good reasons why markets should be considered efficient over long periods of time especially in liquid markets. Over short periods of time?  Thinly traded markets?  Markets with substantially restricted access to information? 

However, you'd think the way data is accumulating at a geometric rate, most markets should be trending towards more efficient not less.  ChpBstrd spoke above quality of data (not all data is "information!")   Think I could sell a copy of Carl Sagan's "Bologna Detection Kit" to some big quant hedge funds???

SeattleCPA

  • Magnum Stache
  • ******
  • Posts: 2582
  • Age: 65
  • Location: Redmond, WA
    • Evergreen Small Business
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #5 on: September 11, 2024, 05:19:58 PM »
This isn't about the election or the debate.

But on the subject of LLMs, I am amazed at some of the stuff they CAN do. (Recently, I've been playing with ChatGPT4o to write JavaScript-based calculators for my blog posts. I also think you can use ChatGPT for something like determining a reasonable compensation amount for an S corporation. And writing a bunch of emails to job candidates or clients? That's something I did recently with ChatGPT. And gosh it is really productivity enhancing.)

But there's stuff it just can't do. Interpret tax law is one example. Some of you guys know, for example, that I've occasionally written ebooks for CPAs and tax attorneys about things like the Section 199A. (Section 199A lets a passthrough business NOT pay taxes on the last 20 percent of the income it earns.) And I tried to get ChatGPT to check the math used in my examples. There are dozens and dozens in the book. And I needed to update the formulas for the inflation that has occurred since the original edition. But it could not do it. Even with repeated, painstaking prompting, it could correctly apply the formulas.

Not sure this related to AI programs scrapping online content for actionable investment insights. Or picking up quantitative insights about investment options. But if an LLM can't calculate a number that appears on a tax return? Yikes.

MustacheAndaHalf

  • Walrus Stache
  • *******
  • Posts: 7633
  • Location: U.S. expat
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #6 on: September 12, 2024, 12:35:08 AM »
I argue that LLMs are not accurate enough, have certain biases by design, and lack critical information.

SeattleCPA's observation about poor math skills isn't isolated, as I've seen varying estimates of ChatGPT's math skills.  It does well on exams, but in those cases it has many prior tests with similar questions.  It can simply model prior tests, and have a good estimate for answering new tests.

The internet has racist material on it, which earlier versions of ChatGPT repeated.  To hide this material, ChatGPT was given a reverse bias, so that it preferred diversity in its results.  You can see that in some searches, but it is most memorable when similar technology is used to generate images.  An image of Nazi soldiers may include Asians... a picture of the founding fathers that includes Black people.  To fix one bias, they gave AI models the opposite bias, which again gives distortions they didn't quite get right.  If you ask ChatGPT who will win the election, and it is forced to ignore all racist views and add diversity... will it get an accurate result?

Finally, markets move when new information is released.  Company information is kept private, and secured.  The critical information ChatGPT needs isn't available to it.  Same with government reports that are released to everyone at the same time.  The key information to forecast markets isn't publicly available on the internet, most of the time (Covid-19 being an odd exception, with markets quickly brushing off the Federal Reserve's emergency meeting of March 8 2020).

SeattleCPA

  • Magnum Stache
  • ******
  • Posts: 2582
  • Age: 65
  • Location: Redmond, WA
    • Evergreen Small Business
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #7 on: September 12, 2024, 05:23:22 AM »
I argue that LLMs are not accurate enough, have certain biases by design, and lack critical information.


Totally agree.

FWIW I like Bloomberg columnist Matt Levine's observation (paraphrased here) that the LLMs are a like an adequate entry-level employee. They're useful. Some stuff, a lot of the time, they get right. But they aren't on the partner track at an investment banking, law of accounting firm.

All that said, I must add, for $20 a month, it's pretty convenient and dare I say it's almost addictive to have them tirelessly answer questions with high confidence in 2-3 seconds.
« Last Edit: September 16, 2024, 05:46:59 AM by SeattleCPA »

reeshau

  • Magnum Stache
  • ******
  • Posts: 3835
  • Location: Houston, TX Former locations: Detroit, Indianapolis, Dublin
  • FIRE'd Jan 2020
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #8 on: September 12, 2024, 08:02:15 AM »
But even with high frequency trading and all the information available the market is still inefficient and prone to react based on human biases.

I remember back in grad school (B-school), I was made to 'prove' (using THE ONE HIGH AND HOLY MATHEMATICS) that markets are in fact, efficient.  There are good reasons why markets should be considered efficient over long periods of time especially in liquid markets. Over short periods of time?  Thinly traded markets?  Markets with substantially restricted access to information? 

However, you'd think the way data is accumulating at a geometric rate, most markets should be trending towards more efficient not less.  ChpBstrd spoke above quality of data (not all data is "information!")   Think I could sell a copy of Carl Sagan's "Bologna Detection Kit" to some big quant hedge funds???

Economist Eugene Fama: ‘Efficient markets is a hypothesis. It’s not reality’

Send it to your B-school prof.

Financial.Velociraptor

  • Magnum Stache
  • ******
  • Posts: 2522
  • Age: 52
  • Location: Houston TX
  • Devour your prey raptors!
    • Living Universe Foundation
Re: AI, LLM, Macro forecasting, and Quants?
« Reply #9 on: September 12, 2024, 08:20:05 AM »

Economist Eugene Fama: ‘Efficient markets is a hypothesis. It’s not reality’

Send it to your B-school prof.

I had to de-educate myself on EMH.  Got no traction with the old profs (roughly Y2K period).  I don't think any of the old school EMH finance profs are left.  All the ones I've met at alumni events have a more nuanced view of what efficiency means.