A Comparative Study of Open-Source Large Language Models and Prompt Engineering Methods: Evaluating Zero-Shot, Few-Shot, and Chain-of-Thought Techniques

In this study, a comparison of two open-source large language models (LLMs), tiiuae/falcon-7b-instruct and GPT-2, was performed by comparing their performance using three prompt engineering techniques: Zero-Shot, Few-Shot, and Chain-of-Thought prompting. This research employed qualitative and quantitative methods of assessment through factual queries, summarization tasks, and reasoning-based questions. The results of this study show that Falcon-7b-instruct consistently produced higher clarity, conciseness, and accuracy than GPT-2 for all prompting styles. In the quantitative evaluation, it was found that, in general, Falcon-7b-instruct produced less lengthy and focused outputs. Falcon demonstrated the most significant improvement in reasoning ability, particularly with Chain-of-Thought prompting. Lastly, Zero-Shot prompting yielded the most effective results for simple factual questions. These findings will be useful for practitioners in selecting the most appropriate LLM for tasks requiring both high accuracy and computational efficiency, and they will contribute to the continued academic research on prompt engineering and LLM evaluation in the field of Natural Language Processing (NLP).