I recently found myself trying to get encryption working with adhoc mode under OpenWRT and found that info in the space is fairly fragmented. Since I had to trawl a lot of old mailing lists and stare at code for a while to figure this out, I wanted to outline the important parts in one place.

First, to be clear, we're talking about mode 'adhoc' as it appears in a wifi-iface block, like so: 

orc@fuyuki ~ $ cat wireless.adhoc
config wifi-iface
        option device 'radio0'
        option network 'wifi'
        option mode 'adhoc'
        option ssid 'foobar'
        option encryption 'psk2'
        option key 'strongpassword'

This is very emphatically not mode 'mesh': that is 802.11s and a totally different thing.

However, if you drop a config block like the above in your OpenWRT wireless config, you probably won't be able to get anything to associate with the network. There's a missing element:

What we're after is called IBSS-RSN. The issue is that some of the common OpenWRT wpa_supplicant family packages aren't by default built with support for IBSS-RSN:

orc@nira ~ $ grep -ri "CONFIG_IBSS_RSN .
./package/network/services/hostapd/files/wpa_supplicant-full.config:CONFIG_IBSS_RSN=y                                                                                                                                                                                          
./package/network/services/hostapd/files/wpa_supplicant-mini.config:#CONFIG_IBSS_RSN=y                                                                                                                                                                                         
./package/network/services/hostapd/files/wpa_supplicant-basic.config:#CONFIG_IBSS_RSN=y                                                                                                                                                                                        
./package/network/services/hostapd/files/wpa_supplicant-p2p.config:CONFIG_IBSS_RSN=y                                                                                                                                                                                           
./package/network/services/hostapd/patches/200-multicall.patch: ifdef CONFIG_IBSS_RSN                                                                                                                                                                                          
./package/network/services/hostapd/patches/200-multicall.patch: CFLAGS += -DCONFIG_IBSS_RSN

So, either swap to the full or p2p versions of wpa_supplicant, or enable the cflag and build it yourself.

In my case, once rebuilt, I can see the expected string in the emitted binary:

orc@nira ~/openwrt/build/wpad $ strings usr/sbin/wpad | grep IBSS_RSN
IBSS_RSN

Once a supplicant supporting IBSS-RSN is enabled, it should just werk. I can confirm this is the case on OpenWRT v19.07.8; dropping in the above wifi config allows two Ubiquiti Rocket 2.4 (XM) devices to talk.

TODO: examples of working supplicant