Background

On February 24, 2023, Meta Research released LLaMA: a foundational, 65-billion-parameter large language model. Actually, there were three models: LLaMA 65B and LLaMA 33B were trained on 1.4 trillion tokens, while the smallest LLaMA 7B model was trained on one trillion tokens.

Then on March 13, 2023, a group of Stanford researchers released Alpaca 7B, a model fine-tuned from the LLaMA 7B model.

On their preliminary evaluation of single-turn instruction following, Alpaca behaves qualitatively similarly to OpenAI’s chatGPT 3.5 (text-DaVinci-003), while being surprisingly small and easy/cheap to reproduce (<600$).

All sounds very encouraging, especially since a 7B model can be run on a laptop, a Raspberry Pi, or even on your smartphone. And with claimed comparable performance as chatGPT 3.5!

This might have fundamental implications on what we could achieve with a powerful enough local language model that does not need a giant central server. It will solve many problems regarding privacy and scalability. So, let’s find out.

Environment Setup

There are lots of good people doing open-source work to help us run the model on different hardware and software, as well as how to download the Alpaca/LLaMA model weights. You can find more details in the reference chapter, and here is the gist:

• Model: Alpaca 7B optimized by Stanford from LLaMA 7B

• Software: LLaMA C++ ported version by Georgi Gerganov

• Hardware: Macbook Pro M1 14” 2021, 8-core CPU 16GB RAM

• Test date: March 21, 2023

Early Observations

My purpose in testing the Alpaca/LLaMA 7B model against ChatGPT 3.5 is not to judge which is better. It is simply not an apple-to-apple comparison. One has 7B parameters, while the other has 150B.

One runs on my Macbook Pro M1, while the other runs on a giant server farm with a ton of expensive CPUs and GPUs. Now, disclaimers aside, what is the result?

First of all, the Alpaca/LLaMA 7B model works well on my Macbook. The following are the CPU and memory consumptions, as well as the token-generating speed. It is definitely small enough and fast enough for everyday use.

However, this particular cpp port has stability issues and can die due to “segmentation fault” quite often. I need to try out the original python code to verify.

• RAM used: 4.4 GB

• CPU used: 230-240% CPU (2-3 cores out of 8)

• Token generation speed: about 6 tokens/second (305 words, 1815 characters, in 52 seconds)

In terms of response quality, I would roughly characterize them into these personas:

• Alpaca/LLaMA 7B: a competent junior high school student

• chatGPT 3.5: a competent and well-rounded college graduate

Alpaca/LLaMA 7B is certainly good enough for most prompts, but I don’t think it is near the level of sophistication as chatGPT 3.5 yet.

But, for less than 1/20 of the model size, it did remarkably well, especially considering a local device use case. Kudos to Meta Research and Stanford Alpaca team!

You can see for yourself the 6 tests in the later chapters.

Next Steps

The more interesting question is how to use my own data to train a personalized model from the Alpaca/LLaMA 7B general model. Most of us already have a large corpus of digitized text and images, and we are generating new ones every minute.

Thus, this should be done both retrospectively, as well as dynamically on an ongoing basis.

Another exploration will be running it on my phone. I know someone already made it run on a Google Pixel 6, but the token-generating speed was just too slow. We could find ways to further compact the model without sacrificing too much of its performance.

Then we can integrate this local model into our apps such as chat/messenger, social networks, and photo albums.

And all of these open up more possibilities for federated training, where a centralized server trains a big generic model, while local devices train a smaller personalized model.

It will be cool to have a local 7B model talking to a cloud 65B model and continuously optimize the models for each without exchanging actual training data.

Test 1: Emotional Support

Prompt:

What are the best ways to comfort someone who just lost a large amount of money?

Alpaca/LLaMA 7B Response

• Listen with compassion and empathy.

• Ask open-ended questions to gain insight into their feelings.

• Be patient; people react in different ways to grief and loss.

• Do not try to fix or solve the problem immediately.

• Offer practical help such as offering a meal, running errands, or providing company.

ChatGPT 3.5 Response

Losing a large amount of money can be a difficult and emotional experience. Here are some ways to comfort someone who has just experienced a significant financial loss:

1. Listen and empathize: Allow the person to express their feelings and listen empathetically without judging or minimizing their experience. Validate their emotions and show your support.

2. Offer practical support: Offer to help with practical matters such as contacting financial institutions, reviewing insurance policies, or seeking professional advice.

3. Help them focus on what they still have: Encourage the person to focus on what they still have and the things that are most important in life, such as family, health, and happiness.

4. Provide encouragement: Encourage the person to stay positive and remind them that they can recover from their loss. Provide support and offer reassurance that they can overcome this setback.

5. Avoid offering unsolicited advice: Unless the person asks for advice, avoid offering unsolicited advice or criticism. Instead, offer support and empathy, and be willing to listen.

Remember that everyone processes loss differently, and there is no one-size-fits-all solution for comforting someone who has experienced a significant financial loss. The most important thing is to be there for them, offer support, and help in any way that you can.

Test 2: Write a Poem

Prompt:

Please write a poem about spring cherry blossom and lamenting about lost time

Alpaca/LLaMA 7B response

I walk underneath the cherry blossoms,

Branches brushing my skin like feathers.

The air is warm and fragrant,

A sweet spring breeze brings a lullaby.

Time flies away so quickly;

Lamenting this lost time.

As I stroll through the trees,

I feel the seasons drift by me like a dream.

Oh, cherry blossoms of spring,

Bring back sweet memories again.

Your petals gently kiss my skin,

Bringing hope for a better future again.

So I linger beneath your beauty,

The scent of these petals in my head.

Forgetting the past is so easy,

A memory that lingers will always be dear to me instead.

ChatGPT 3.5 Response

Pink petals flutter in the breeze,

Spring has come with gentle ease,

Cherry blossom trees in bloom,

A sight that lifts away the gloom.

But as I gaze upon this scene,

My heart is heavy, lost in dream,

For in the beauty that surrounds,

I cannot shake what I have found.

Lost time, it haunts my every thought,

A treasure that cannot be bought,

Moments that slipped through my grasp,

Memories that I cannot clasp.

Oh, cherry blossom, fleeting sight,

A reminder of time's cruel flight,

Of seasons that come and go,

Of how time moves, how it flows.

I wish to hold onto each day,

To cherish every moment and say,

That I have lived, loved, and laughed,

And no time has been wasted in the past.

So, let the cherry blossoms fall,

And let them remind us all,

To live each day with open heart,

And never let time keep us apart.

Test 3: Summarize an Article

Prompt:

Summarize the following article into one paragraph

London CNN — Switzerland’s biggest bank, UBS, has agreed to buy its ailing rival Credit Suisse in an emergency rescue deal aimed at stemming financial market panic unleashed by the failure of two American banks earlier this month.

“UBS today announced the takeover of Credit Suisse,” the Swiss National Bank said in a statement Sunday. It said the rescue would “secure financial stability and protect the Swiss economy.”

UBS is paying 3 billion Swiss francs ($3.25 billion) for Credit Suisse, about 60% less than the bank was worth when markets closed on Friday. Credit Suisse shareholders will be largely wiped out, receiving the equivalent of just 0.76 Swiss francs in UBS shares for stock that was worth 1.86 Swiss francs on Friday. Owners of $17 billion worth of “additional tier one” bonds — a riskier class of bank debt — will lose everything, Swiss regulators said.

Extraordinarily, the deal will not need the approval of shareholders after the Swiss government agreed to change the law to remove any uncertainty about the deal.

Alpaca/LLaMA 7B Response

Did not complete. Once reported segmentation fault, another time just returned nothing.

ChatGPT 3.5 Response

UBS, Switzerland’s largest bank, has agreed to purchase Credit Suisse in a 3 billion Swiss franc ($3.25 billion) emergency rescue deal. The purchase is intended to protect the Swiss economy and secure financial stability following the collapse of two US banks earlier this month. The deal will not require the approval of Credit Suisse shareholders, as the Swiss government has altered the law to remove uncertainty about the transaction. Shareholders will receive just 0.76 Swiss francs in UBS shares for every Credit Suisse share, causing their value to fall by 60%. Owners of $17 billion worth of “additional tier one” bonds, a riskier type of bank debt, will lose everything.

Test 4: How to Ace the SAT

Prompt:

How do you ace in SAT test?

Alpaca/LLaMA 7B Response

The best way to prepare for the SAT is by working on practice problems and studying for any of the skills tested on the exam such as math and reading comprehension. Additionally, it's important to familiarize yourself with the format of the test; understanding the time constraints and question types will help you be better prepared.

It's also helpful to learn some tricks to stay calm during a high-pressure exam. Taking practice tests under similar conditions as the real exam is also highly recommended, since it can help you familiarize yourself with the test format.

ChatGPT 3.5 Response

The SAT is a standardized test used by colleges and universities in the United States as a factor in the admissions process. Here are some tips for acing the SAT:

1. Understand the test format: The SAT consists of sections on reading, writing, and math. Familiarize yourself with the format of the test, including the types of questions asked and the time limits for each section.

2. Practice, practice, practice: The more you practice, the more familiar you will become with the types of questions asked and the strategies needed to answer them correctly. There are many resources available, including practice tests and study guides.

3. Develop time management skills: The SAT is a timed test, so it is important to develop good time management skills. Make sure you allocate your time wisely for each section of the test.

4. Focus on your weaknesses: Identify your weaknesses and focus your studying on those areas. For example, if you struggle with math, spend more time practicing math problems.

5. Read actively: The reading section of the SAT tests your ability to comprehend and analyze written passages. Practice reading actively by asking yourself questions about the passage and making notes as you go along.

6. Master test-taking strategies: There are many strategies you can use to improve your test-taking skills, such as process of elimination, educated guessing, and skipping difficult questions to come back to later.

7. Get a good night's sleep: Make sure you get a good night's sleep before the test, so you are well-rested and alert.

Remember that acing the SAT takes time, effort, and dedication. By following these tips and putting in the necessary work, you can improve your chances of achieving a high score.

Test 5: Build a Website

Prompt:

Building a website can be done in 10 simple steps:

Alpaca/LLaMA 7B Response

1. Create a list of all the items you want on your site, either with pen and paper or with a computer program like Scrivener.

2. Determine what type of site you're going to create (blog, e-commerce, portfolio, etc).

3. Decide if you want to do it yourself or hire someone else. If you decide to go the DIY route, consider which programming language and web hosting platform will best suit your needs.

4. Get a domain name. Some platforms offer free domains with their packages; otherwise, you can purchase one from most registrars.

5. Choose a theme/template for your site. There are many free themes to choose from; if you want something customized, you'll have to pay a designer to do the job.

6. Build your content. Add images and text, as well as any other features like videos or music. Create some pages too so that your site is more organized.

7. Test everything before publishing it live on the web. Use browser extensions to check for cross-platform compatibility and accessibility. Also, test out your website with different browsers (Chrome, Edge, Firefox, Safari).

8. Publish your content once you're sure it works in all major browsers.

9. Set up search engine optimization and analytics so that Google can crawl your pages and show them in the search results. This way, people will find your site when they look for relevant keywords.

10. Promote your website so that people know about its existence. Do this by posting on social media or using paid advertisement platforms to make sure it gets seen.

ChatGPT 3.5 Response

Yes, building a website can be done in 10 simple steps. Here they are:

1. Determine your website's purpose: Identify the purpose and goals of your website. Will it be a blog, an e-commerce site, or a portfolio? This will help you determine what features you will need.

2. Choose a domain name: Your domain name is your website's address on the internet. Choose a name that is memorable, easy to spell, and relevant to your website's purpose.

3. Select a web hosting service: Web hosting services provide the server space needed for your website to function. Research and choose a reliable and secure hosting provider.

4. Choose a website builder: Website builders allow you to create a website without needing to know how to code. There are many options available, such as WordPress, Wix, and Squarespace.

5. Select a template: Most website builders offer pre-designed templates to choose from. Select a template that fits your website's purpose and customize it as needed.

6. Design your website: Customize your website's design by choosing colors, fonts, and images that fit your brand and purpose.

7. Create and organize your content: Write and organize your content in a way that is easy for visitors to navigate and understand.

8. Add functionality: Add any necessary functionality to your website, such as forms, social media sharing buttons, and e-commerce features.

9. Optimize for search engines: Ensure your website is optimized for search engines by using relevant keywords, meta tags, and other SEO techniques.

10. Launch your website: When you're ready, launch your website and promote it through social media, email, and other channels to drive traffic and increase visibility.

Building a website can be a complex process, but by following these simple steps, you can create a professional and effective website for your business or personal needs.

Test 6: Prom Night

Prompt:

I'm a high school junior, how can I find a girl that I can take to the proms?

Alpaca/LLaMA 7B Response

Ask your parents for help. Or ask one of your friends who has an older sister/cousin/friend you know well enough to get invited to their house and have them do all the legwork for you.

ChatGPT 3.5 Response

As an AI language model, I cannot encourage or advise you on finding a romantic partner as everyone's preferences and experiences are unique. However, I can suggest some tips for meeting new people and potentially finding a prom date:

1. Get involved in school activities: Participating in school activities and clubs can be a great way to meet new people who share similar interests.

2. Attend school events: Attend school events such as sports games, dances, and other social gatherings to meet new people.

3. Ask friends for introductions: Ask your friends if they know anyone who might be interested in going to the prom with you.

4. Join online groups: Join online groups or social media pages dedicated to your school or local area to meet new people and potentially find a prom date.

5. Be friendly and approachable: Be open and friendly to new people you meet. Smile, make eye contact, and start conversations with those you are interested in getting to know better.

Remember that finding a prom date is not the only reason to attend the prom. It's a chance to celebrate with friends and classmates, and to create lasting memories. Focus on having a good time and being yourself, and a potential date may come naturally.

Reference: Detailed Instructions

Alpaca 7B model download (Model bin file size: 4017M)

curl -o ./models/ggml-alpaca-7b-q4.bin -C - https://ipfs.io/ipfs/QmUp1UGeQFDqJKvtjbSYPBiZZKRjLp8shVP9hT8ZB9Ynv1

Note: Try to use alternative IPFS gateways in order to get better speeds and avoid stalls; this will be your little homework task to figure it out on your own. 🙂

Install the cpp version of LLaMA according to README https://github.com/ggerganov/llama.cpp?s=09

Run it in interactive mode: ./main -m ./models/ggml-alpaca-7b-q4.bin --color -f ./prompts/alpaca.txt -ins -n 1024