Can GPT-4 qualify to be a CFA
This study aims at assessing the financial reasoning capabilities of LLMs. They have leveraged mock exam questions of the Chartered Financial Analyst (CFA) Program to conduct a comprehensive evaluation of ChatGPT and GPT-4 in financial analysis, considering Zero-Shot (ZS), Chain-of-Thought (CoT), and Few-Shot (FS) scenarios.No pre-training or fine-tuning was performed on the large language models.
The paper is cirumspect about it, since the actual grading criteria for CFA is kept a secret. Instead they have taken certain assumptions like:
- Level I: More than 60% in each topic and 70% overall;
- Level II: Achieve more than 50% in each topic and 60% overall.
Based on the estimated pass rates, ChatGPT will not be able to pass the CFA exam. But GPT-4 could have a decent chance of clearning the exam, if prompted by a FewShot (FS) or ChainOfThought (CoT) process.
The most prevalant errors were knowledge errors, where gaps in the knowledge base were magnified by the CoT process.
GPT has been used to train on specific domains like on a medical test, so its not a big jump from here to train a GPT model which can answer domain specific queries on finance.
Shows the deep impact that this new generation of models will have on the future of machine learning spplications
Arxiv:https://arxiv.org/abs/2310.08678
Disclaimer:
The research was funded by Faculty Research Awards of JP Morgan AI Research. This paper is for research purpose and does not constitute any investment research or investment advice, or a recommendation for the purchase or sale of any security, financial instrument, product or service.