Building a custom Jetson Nano kernel to enable module support
I recently tried to install an Intel AC 9260 wireless card into a Jetson nano developer kit. Unfortunately, NVIDIA has what appears to be a well deserved reputation of terrible software support. The kernel they supply is woefully out of date and they don't provided kernel modules for the AC-9260. To make matters worse, the kernel they do provide doesn't support module signing so you can't simply compile and load your own module.
- Fortunately, this is Linux and there is backport code available so you can build your own device module
- Unfortunately, this is NVIDIA and they have disabled module signing so attempting to build your own module fails.
- Fortunately, this is still Linux and if you're willing to build your own kernel you can correct what NVIDIA has broken
I did this on an x86_64 system running Linux Mint. To do the same, i.e. cross-compile a kernel, you'll need
- working build system
- cross compiler
- kernel source
- L4T Linux driver package
- Note: this has a tool to aid in getting the kernel source
Prepping the build system
General setup
Start by creating a working space for the code, build-tools and anything else we may need
export L4T_HOME=${HOME}/JetsonNano/L4T
mkdir -p ${L4T_HOME}/Downloads
General build tools
You'll need the basic build tools installed on your system, e.g. make, dev libs etc. Getting this setup isn't difficult but it's beyond the scope of this doc. In short, if you're not sure how to do that you may want to think twice about building a kernel. The short version, apt-get install build-essential is your friend.
GCC Cross compiler
Retrieve the Linaro x86_64 to aarch64 cross-compiler from Linaro Releases
cd ${L4T_HOME}/Downloads
curl -LO https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz
curl -LO https://releases.linaro.org/components/toolchain/binaries/latest-7/aarch64-linux-gnu/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz.asc
md5sum -c gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz.asc
cd ${L4T_HOME}
xzcat Downloads/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz |tar -xv
Misc packages
You will also need:
- For kernel build
- libssl-dev
- When building a root FS
- qemu-user-static
- When creating a custom disk image
- apt-get install libxml2-utils
apt-get install libssl-dev qemu-user-static libxml2-utils
Jetson Linux source code
L4T Driver Package
The Jetson Linux R32.6.1 Release Page has a link to the L4T Driver Package (BSP) package. This package has the sync script we need to retrieve the kernel sources so we'll start with it.
cd ${L4T_HOME}/Downloads
curl -LO https://developer.nvidia.com/embedded/l4t/r32_release_v6.1/t210/jetson-210_linux_r32.6.1_aarch64.tbz2
cd ${L4T_HOME}
bunzip2 -c Downloads/jetson-210_linux_r32.6.1_aarch64.tbz2 |tar -xv
Kernel source
Now we'll use the source_sync.sh script to retrieve the Linux kernel code. See the NVIDIA Jetson Linux Driver Package Software Features page. Specifically, the Kernel Customization section. Note that we're syncing to the tegra-l4t-r32.6.1 tag
> cd ${L4T_HOME}/Linux_for_Tegra
> ./source_sync.sh
Downloading default kernel/kernel-4.9 source...
Cloning into '/home/demo/JetsonNano/L4T/Linux_for_Tegra/sources/kernel/kernel-4.9'...
remote: Enumerating objects: 5215135, done.
remote: Counting objects: 100% (5215135/5215135), done.
remote: Compressing objects: 100% (798844/798844), done.
Receiving objects: 100% (5215135/5215135), 937.79 MiB | 8.03 MiB/s, done.
Resolving deltas: 100% (4375648/4375648), done.
The default kernel/kernel-4.9 source is downloaded in: /home/demo/JetsonNano/L4T/Linux_for_Tegra/sources/kernel/kernel-4.9
Please enter a tag to sync /home/demo/JetsonNano/L4T/Linux_for_Tegra/sources/kernel/kernel-4.9 source to
(enter nothing to skip): tegra-l4t-r32.6.1
Syncing up with tag tegra-l4t-r32.6.1...
Updating files: 100% (57317/57317), done.
Switched to a new branch 'mybranch_2021-11-06-1636231323'
/home/demo/JetsonNano/L4T/Linux_for_Tegra/sources/kernel/kernel-4.9 source sync'ed to tag tegra-l4t-r32.6.1 successfully!
...
Downloading default kernel/nvgpu source...
Downloading default kernel/nvidia source...
Downloading default hardware/nvidia/platform/tegra/common source...
Downloading default hardware/nvidia/platform/t18x/common source...
Downloading default hardware/nvidia/platform/t18x/quill source...
Downloading default hardware/nvidia/platform/t18x/lanai source...
Downloading default hardware/nvidia/soc/t210 source...
Downloading default hardware/nvidia/platform/t210/common source...
Downloading default hardware/nvidia/platform/t210/jetson source...
Downloading default hardware/nvidia/platform/t210/porg source...
Downloading default hardware/nvidia/platform/t210/batuu/kernel-dts source...
Downloading default hardware/nvidia/soc/t19x source...
Downloading default hardware/nvidia/platform/t19x/common source...
Downloading default hardware/nvidia/platform/t19x/galen/kernel-dts source...
Downloading default hardware/nvidia/platform/t19x/jakku/kernel-dts source...
Downloading default hardware/nvidia/platform/t19x/mccoy/kernel-dts source...
Downloading default hardware/nvidia/soc/tegra source...
...
Downloading default u-boot source...
...
/home/demo/JetsonNano/L4T/Linux_for_Tegra/sources/u-boot source sync'ed to tag tegra-l4t-r32.6.1 successfully!
Building the kernel
Required lib package
You'll need libssl-dev package: apt-get install libssl-dev
Restoring the required kernel option
For some reason which is not at all clear to me, NVIDIA has disabled the module signing option. This is what prevents us from building the backport driver module. Restore it by setting the bool for SYSTEM_DATA_VERIFICATION in Kconfig to 'y' per the following:
> vi ${L4T_HOME}/Linux_for_Tegra/sources/kernel/kernel-4.9/init/Kconfig
> git diff
diff --git a/init/Kconfig b/init/Kconfig
index cefd06480226..7b41356c7dc5 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1999,7 +1999,7 @@ config MMAP_ALLOW_UNINITIALIZED
See Documentation/nommu-mmap.txt for more information.
config SYSTEM_DATA_VERIFICATION
- def_bool n
+ def_bool y
select SYSTEM_TRUSTED_KEYRING
select KEYS
Run the build
export L4T_HOME=${HOME}/JetsonNano/L4T
export CROSS_COMPILE="${L4T_HOME}/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-"
export LOCALVERSION="-tegra"
export TEGRA_KERNEL_OUT="${L4T_HOME}/Linux_for_Tegra/sources/kernel/kernel-4.9/build_out"
mkdir -p $TEGRA_KERNEL_OUT
cd ${L4T_HOME}/Linux_for_Tegra/sources/kernel/kernel-4.9
make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig
grep SYSTEM_DATA ${TEGRA_KERNEL_OUT}/.config
make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4
Building a new root FS image
Prep for the new FS image
NVIDIA's docs tell you to use an example FS image from their site. I found that to not work as it was missing some programs their tools were looking for, e.g. zstd. (See above re terrible software support.) I found it simpler to just use the default SD card image used to boot the Dev kit. You know, the one they provide which has module signing enabled and is the reason we're wasting all this time building a new kernel.
Note that I performed the following steps as root because the process requires mounting/unmounting the SD card and installing packages to the Root FS image. Note changing users changes the environment as well so you'll have to set these vars again as the root user.
Start with the SD card inserted in the host system. If it was automounted, unmount it. Don't simply "eject" it with the GUI as this will drop the device too. Simply unmount it. e.g. umount /media/demo/0fc4f9cc-1766-4711-bed9-ac90b326f4e0
Warning: Don't blindly cut and paste! Your SD card device may not be the same and you will most likely be sad if you trash your host's file system because you didn't check device names.
If you're building a root FS per the docs at NVIDIA Jetson Linux Driver Package Software Features we're at step 5 of Building the NVIDIA Kernel where the files are copied back up to the Linux_for_Tegra tree.
mount /dev/sda1 ${L4T_HOME}/Linux_for_Tegra/rootfs
cd ${L4T_HOME}/Linux_for_Tegra/sources/kernel/kernel-4.9
# Step 5
mv ${L4T_HOME}/Linux_for_Tegra/kernel/Image{,.ORIG}
cp $TEGRA_KERNEL_OUT/arch/arm64/boot/Image ${L4T_HOME}/Linux_for_Tegra/kernel/
# Step 6
mv ${L4T_HOME}/Linux_for_Tegra/kernel/dtb{,.ORIG}
cp -r --preserve=mode ${TEGRA_KERNEL_OUT}/arch/arm64/boot/dts ${L4T_HOME}/Linux_for_Tegra/kernel
mv ${L4T_HOME}/Linux_for_Tegra/kernel/{dts,dtb}
# Step 7
make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=${L4T_HOME}/Linux_for_Tegra/rootfs/
Build the new FS image
Again, if you're following along with NVIDIA docs, we're in the section Setting Up Your File System
cd ${L4T_HOME}/Linux_for_Tegra
echo "Custom kernel on top of standard RootFS Image" > rootfs/README.txt
./apply_binaries.sh
Creating a new SD card
We're now in the section of of the NVIDIA docs related to Flashing to an SD Card
tegra:/home/demo/JetsonNano/L4T/Linux_for_Tegra# cd ${L4T_HOME}/Linux_for_Tegra/tools
tegra:/home/demo/JetsonNano/L4T/Linux_for_Tegra/tools# ./jetson-disk-image-creator.sh -o sd-nano.img -b jetson-nano -r 300
********************************************
Jetson Disk Image Creation Tool
********************************************
jetson-disk-image-creator.sh - creating signed images
/home/demo/JetsonNano/L4T/Linux_for_Tegra /home/demo/JetsonNano/L4T/Linux_for_Tegra/tools
###############################################################################
# L4T BSP Information:
# R32 , REVISION: 6.1
###############################################################################
Board ID(3448) version(300)
...
13657702400 bytes (14 GB, 13 GiB) copied, 122.318 s, 112 MB/s
********************************************
Jetson Disk Image Creation Complete
********************************************
Flash the image to a new SD card
Assuming all went well, you can flash the image to a new SD card and boot your Nano Dev Kit. Note this version doesn't auto extend the root file system partition to use the all the free space. So, once you've verified it works by booting your nano dev kit you can extend the partition/file system. I did this on my host system but I expect you should be able to do it directly on the dev kit. If you're going to do this directly on the dev kit you should be able to jump right to step #3
The process I used is:
- Insert the SD card into the host
- Unmount it (assuming it was automounted)
- Resize the root partition to use all available space
- Resize the file system on the root partition
# umount /media/demo/0fc4f9cc-1766-4711-bed9-ac90b326f4e0
# parted /dev/sda print
Model: Multiple Card Reader (scsi)
Disk /dev/sda: 31.1GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
2 1049kB 1180kB 131kB TBC
...
1 14.7MB 13.7GB 13.7GB ext4 APP
# parted /dev/sda resizepart 1 100%
Information: You may need to update /etc/fstab.
# parted /dev/sda print
Model: Multiple Card Reader (scsi)
Disk /dev/sda: 31.1GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
2 1049kB 1180kB 131kB TBC
...
1 14.7MB 31.1GB 31.1GB ext4 APP
# e2fsck -f /dev/sda1
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 186099/833952 files (0.1% non-contiguous), 3168273/3334656 blocks
# resize2fs /dev/sda1
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/sda1 to 7590395 (4k) blocks.
The filesystem on /dev/sda1 is now 7590395 (4k) blocks long.