How To Set Up The Environment for RISCV-64 Linux Kernel Development In Ubuntu 20.04

I have read some articles related to set up a QEMU environment for enabling RISC-V Linux kernel development. Unfortunately, none of them describes all the required steps to get a fully prepared RISC-V 64bit system.

In this article, I am describing how to build the latest (as of the moment of writing) Linux Kernel for RISC-V 64 architecture and deploy it along with minimal environment: busybox command-line.

PREREQUISITES

I use Ubuntu as my main desktop OS, so all the described steps related to it. Probably it could be reproduced on other Debian-based distributions.

My precise Ubuntu version is:

hedin@home:~/projects/linux/riscv64-linux$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

The following is the project's catalog structure:

.....
hedin@home:~/projects/linux/riscv64-linux$ tree -L 1
.
├── busybox
├── initramfs
└── qemu

Their intention is following:

busybox - contains minimal command-line environment to work with the Kernel.
initramfs - initial RAM disk for the Kernel.
qemu - emulator I run the RISC-V enabled Kernel on.

Note 1: From now on I am using "$" instead of the full path to save space.

Note 2: I intentionally didn't include Linux Kernel in this project as it is a huge amount of source code and it makes no sense to copy it to every project. It is better to have only one shared Kernel repository for all the related projects and use Git branches for each of them.

SETUP

Firstly, we need to install common development tools:

$ sudo apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev git

And RISC-V compiler:

$ sudo apt-get install -y gcc-riscv64-linux-gnu

Secondly, we need to install Qemu from the source. The reason is that most Linux distributions don't contain the latest versions and also don't contain (or uses limited) RISC-V support.

$ git clone https://github.com/qemu/qemu
$ cd qemu
$ git checkout v5.2.0
$ ./configure --target-list=riscv64-softmmu
ERROR: Cannot find Ninja

Something went wrong. Qemu requires a Ninja-build package. Let's install it:

$ sudo apt install ninja-build

And try again:

$ ./configure --target-list=riscv64-softmmu
$ make -j $(nproc)
$ sudo make install

OK, now it is built. Let's return to the project catalog and obtain the Linux Kernel:

$ cd ..

Get the Linux tree:

$ git clone https://github.com/torvalds/linux

Check out the latest (as of this moment of writing) kernel version:

$ git checkout -b risc-v v5.11

Make default config:

$ make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- defconfig

And get a new error message:

compiler 'riscv64-unknown-linux-gnu-gcc' not found

The issue is that there is no standard toolchain that required my meets:

$ sudo apt search riscv64-unknown-
Sorting... Done
Full Text Search... Done
binutils-riscv64-unknown-elf/focal,now 2.34-0ubuntu1 amd64 [installed,automatic]
  GNU assembler, linker and binary utilities for RISC-V processors
gcc-riscv64-unknown-elf/focal,now 9.3.0-0ubuntu1 amd64 [installed]
  GCC compiler for embedded RISC-V chips

There are 2 approaches for solving this issue:

Compile it manually as described in the following article: https://programmersought.com/article/4626958334/
Get the latest build from bootlin.

I will use the latter approach:

Goto https://toolchains.bootlin.com/
Select arch - riscv64
Select libc - glibc
Copy the link and download it somewhere (not in my project catalog):

$ wget https://toolchains.bootlin.com/downloads/releases/toolchains/riscv64/tarballs/riscv64--glibc--bleeding-edge-2020.08-1.tar.bz2

 sudo mkdir -p /opt/bootlin

sudo tar jxf riscv64--glibc--bleeding-edge-2020.08-1.tar.bz2 -C /opt/bootlin/

Now we are ready to compile the kernel again:

$ make ARCH=riscv CROSS_COMPILE=/opt/bootlin/riscv64--glibc--bleeding-edge-2020.08-1/bin/riscv64-buildroot-linux-gnu- defconfig
$ make ARCH=riscv CROSS_COMPILE=/opt/bootlin/riscv64--glibc--bleeding-edge-2020.08-1/bin/riscv64-buildroot-linux-gnu- -j $(nproc)
...............................................................
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  SYSMAP  System.map
  MODPOST Module.symvers
  OBJCOPY arch/riscv/boot/Image
  CC [M]  fs/efivarfs/efivarfs.mod.o
  CC [M]  fs/nfs/flexfilelayout/nfs_layout_flexfiles.mod.o
  GZIP    arch/riscv/boot/Image.gz
  LD [M]  fs/efivarfs/efivarfs.ko
  LD [M]  fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko
  Kernel: arch/riscv/boot/Image.gz is ready

The kernel is ready. It's time to compile BusyBox:

Apply default config:

$ CROSS_COMPILE=/opt/bootlin/riscv64--glibc--bleeding-edge-2020.08-1/bin/riscv64-buildroot-linux-gnu- make defconfig

Adjust one setting. I want a single monolith executable file without dependencies to dynamic libraries:

$ CROSS_COMPILE=/opt/bootlin/riscv64--glibc--bleeding-edge-2020.08-1/bin/riscv64-buildroot-linux-gnu- make menuconfig

In the configuration change the following setting:

	Build static binary (CONFIG_STATIC=y)

Build:

$ CROSS_COMPILE=/opt/bootlin/riscv64--glibc--bleeding-edge-2020.08-1/bin/riscv64-buildroot-linux-gnu- make -j $(nproc)

Install:

$ CROSS_COMPILE=/opt/bootlin/riscv64--glibc--bleeding-edge-2020.08-1/bin/riscv64-buildroot-linux-gnu- make install

The last thing we need to do is to create an initial RAM disk. And again I create it out of the project's tree as it will need for my purposes only once. After completion, we can remove it. So let's create it somewhere:

$ mkdir _install
$ cd _install

Create device tree catalog:

$ mkdir -p dev

And two required devices - console and ram:

$ sudo mknod dev/console c 5 1
$ sudo mknod dev/ram b 1 0

And Init script, the first userspace process that the Kernel will run. This script creates required catalogs and mounts special-purpose kernel filesystems to them. Also, it measures how long this particular boot process took. And in the end, it runs a command interpreter, that is, bash-like busybox.

$ vim init

File content:

#!/bin/sh
echo "### INIT SCRIPT ###"
mkdir /proc /sys /tmp
mount -t proc none /proc
mount -t sysfs none /sys
mount -t tmpfs none /tmp
echo -e "\nThis boot took $(cut -d' ' -f1 /proc/uptime) seconds\n"
exec /bin/sh

Let's pack file structure to the format the Kernel recognizes:

find -print0 | cpio -0oH newc | gzip -9 > <PATH_TO_OUR_PROJECT>/initramfs/initramfs.cpio.gz

RUN

Run qemu passing it the kernel, init RAM disk, and appropriate parameters:

$ sudo qemu-system-riscv64 -nographic -machine virt -kernel ../linux-torvalds-tree/arch/riscv/boot/Image -initrd initramfs/initramfs.cpio.gz -append "root=/dev/vda ro console=ttyS0"

...................................................................................................................................

I am in:

/ # ls
bin      init     proc     sbin     tmp
dev      linuxrc  root     sys      usr
/ # uname -a
Linux (none) 5.11.0 #2 SMP Fri Mar 12 09:36:08 MSK 2021 riscv64 GNU/Linux

References: