FreeNAS - Feature #1974 VirtualBox can't be installed inside a Jail due to kernel module dependency 12/29/2012 04:30 PM - Marcus Ahlberg Status: Resolved Estimated time: 0.00 hour Priority: Nice to have Assignee: John Hixson Category: Target version: 9.2.1.6-RELEASE Severity: New Needs Merging: Yes Reason for Closing: Needs Automation: No Reason for Blocked: Support Suite Ticket: n/a Needs QA: Yes Hardware Configuration: Needs Doc: Yes Description I have been trying to find a solution on how to install [[VirtualBox]] on a [[FreeNAS]] host. With memory so cheap, the power of a [[FreeNAS]] box should be enough for a couple of virtual servers. The problem I have come across is the fact that [[FreeBSD]] jails share the kernel with the host and Virtualbox needs to load a module into the kernel. I have found a possible solution for installing the kernel module on the host and then installing virtualbox inside a jail. This solution however assumes a full [[FreeBSD]] host where the ports tree can be used. http://forums.freebsd.org/showthread.php?t=21897 I quickly tried to install virtualbox-ose-kmod-legacy using pkg_add on the host, but I couldn't load it due to "KLD vboxnetflt.ko: depends on netgraph - not available or version missmatch". I haven't dug any deeper into this problem. What I call for is support for software add-ons that require kernel modules to be loaded on the host. History #1 - 12/29/2012 04:57 PM - Josh Paetzel The first stopper you are hitting here is the lack of the netgraph kernel module on [[FreeNAS]]. I'll think about this some, there are lots of hidden gotchyas. Perhaps the best solution is shipping with the vbox modules, that seems like a reasonable thing to me. #2 - 12/29/2012 05:27 PM - Josh Paetzel I've looked at this a bit more indepth. Running virtualbox in a jail is a unique case, but it's very useful functionality. What I'm going to do is set [[FreeNAS]] up so this just works. #3 - 04/09/2013 04:01 PM - jlohiser - Is this still in the works or is this issue dead? I would like to run a Linux + Asterisk server as a guest on top of [[FreeNAS]]. #4 - 05/31/2013 03:17 PM - Marcus Ahlberg I would also like an update on this. #5 - 06/02/2013 05:26 PM - 8ender - I'd also love to see this as well. I tried setting up a proxmox server and running Freenas under that but its just not as good as Freenas bare metal. I have lots of spare cycles on this server and it makes so much more sense to have Freenas run bare metal and then have a Virtualbox jail to spawn 10/01/2021 1/9 other servers with, using the freenas mounts for storage. #6 - 10/11/2013 03:14 PM - Patrick Private I'm running VirtualBox inside 9.x jail (also did that in 8.x) but lacking netgraph brings lot of issues. You can't use bridged mode for your VM's - and that's a big deal. Missing netgraph module that is needed as dependency for native VB modules prevents from using bridged mode interfaces in VM's and forces to use NAT on jailed bridge interface. This is sub-optimal solution with multiple limitations to "FreeNass + Virtualization" as whole. Simply adding/ticking netgraph kernel compile option, would helped lot of us - that wants to stick to single-box solution that FN is perfectly suited to + using jails for simpler tasks and VM's for home lab/more advanced scenarios. The obvious reason for FreeNas to support VB is that, sometimes powerful boxes that runs FreeNas could be easily re-used to do some visualization as well. Adding internal storage mapping that eliminate need for NFS/iSCSI makes FreeNas perfect single box, green IT solution for SMB/home. #7 - 10/11/2013 05:03 PM - Josh Paetzel Testing a build with netgraph enabled in the kernel. #8 - 11/02/2013 12:57 AM - Albert Marin Hi Patrick, could you post the procedure to implement VirtualBox inside 9.x jail please? #9 - 11/13/2013 07:45 PM - Marcus Ahlberg Patrick, I would love to hear about your experiences. With or without netgraph. #10 - 11/13/2013 10:48 PM - Patrick Private Albert, Marcus, Recently downloaded newest 9.2 alpha and there are some good news and some bad :D Good news: on host system VB network driver needed for bridged mode loads without any complains. That's major step forward. The bad news is that guest starting from jail cause kernel dump (.!.). I've checked some options around but it was quite hopeless [multiple dumps :D]. Now I need to do testing on real hardware as I was testing VB inside guest in jail on FreeNas host that was running inside VB guest on my Windows. That can be a reason for my failure. So next tests will be on physical PC. @Albert: I've applied multiple hacks in addition to guides available on web (there are some good howtos) to make is running for me. There are two major steps not covered in any guide that is: - allow.sysvipc=1 you should put on jail meta "jail.flags" file not on host directly as guides show - doing NAT'ed VB you need to go 'deep' into VB command line guides to set up proper port forwarding in order to be able access your virtual PC. (I did it once and unfortunately I can't recall exact settings now) I'll post on progress once done. #11 - 11/17/2013 11:23 PM - Jordan Hubbard - Category set to 18 - Status changed from Unscreened to Screened - Target version set to 49 10/01/2021 2/9 #12 - 01/02/2014 08:42 PM - Patrick Private Found some time to check it against 9.2 stable. As it worked before in 9.2 RC build (but only without bridged mode in VB guest) now whole VB kernel support seems to be broken 100%. Where you could load vboxdrv.ko, it's no more possible. Looks like already reported here: https://bugs.freenas.org/issues/3701. For those using VB you just need to roll-back to 9.1 :(. #13 - 01/05/2014 01:52 PM - Patrick Private Update: kernel modules provided by Josh Paetzel here: https://bugs.freenas.org/issues/3701 works and indeed system crash issue is resolved now (still not in stock 9.2). But as pointed out later in #3701 bridged adapters doesn't work. On interface you can see: - in guest: some (strange but not all?) outgoing arps - at host on bridged IF: arp requests WITH arp replies :), -- also legitymate outgoing packets like ICMP queries but without replies What is interesting while trying to do ngctl to check hooks etc. you get "ngctl: can't create node: No such file or directory". In google you get only ARM related topics related to such error so I guess it's some kind of missing dependency. #14 - 01/09/2014 01:01 AM - Ian Pitcher I've seen similar network traffic with the drivers compiled with the VIMAGE patch/option, but not crashing is good! I got ngctl to run by downloading the kernel source at https://github.com/trueos/trueos/releases/tag/9.2.0-RELEASE and replacing /usr/src in my portjail with its contents (this also helps if you want to compile your own vbox modules). I determined that ngctl needed ng_socket.ko loaded, so I did: Within port jail: cd /usr/src/sys/modules/netgraph make; make install Exit the jail and loaded the module on the host: kldload /[path to root of jail]/boot/kernel/ng_socket.ko Re-enter jail, start VM with bridged NIC, and check ngctl output: [root@box /]# ngctl list There are 2 total nodes: Name: vboxnetflt_epair1b Type: vboxnetflt ID: 00000003 Num hooks: 0 Name: ngctl29256 Type: socket ID: 00000004 Num hooks: 0 I don't know anything about netgraph, so I don't know what I should expect to see there. I did compile if_tap.ko and get bridged networking going with that, but it has a major downside: whenever the application accessing the tunnel device quits (ie. VirtualBox), it crashes the kernel. I suspect it's also VIMAGE related, as I saw a PR describing the same behavior here: http://www.freebsd.org/cgi/query-pr.cgi?pr=158686 -- perhaps this is a regression? And since I'm throwing whatever garbage I can at the wall to see what sticks (sorry), there's also this: http://lists.freebsd.org/pipermail/freebsd-questions/2013-May/251160.html I did try to load ng_ether.ko and got "link_elf_obj: symbol ifnet undefined, linker_load_file: Unsupported file type" so maybe the kernel needs to be complied with options NETGRAPH_ETHER 10/01/2021 3/9 #15 - 01/15/2014 07:37 PM - Ian Pitcher - File ng_ether.ko added - File ng_socket.ko added After much horsing around, I can say for certain now that NETGRAPH_ETHER (or ng_ether.ko) are necessary for this to work, and it works with or without VIMAGE enabled for the jail. Here's the output from ngctl in the jail while VBoxHeadless is running a VM: [root@box /]# ngctl list There are 3 total nodes: Name: epair0b Type: ether ID: 00000001 Num hooks: 2 Name: vboxnetflt_epair0b Type: vboxnetflt ID: 00000002 Num hooks: 2 Name: ngctl6630 Type: socket ID: 00000006 Num hooks: 0 For good measure, here's what it looks like on the host itself: antarctica# ngctl list There are 6 total nodes: Name: re0 Type: ether ID: 00000001 Num hooks: 0 Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0 Name: bridge0 Type: ether ID: 00000003 Num hooks: 0 Name: epair0a Type: ether ID: 00000004 Num hooks: 0 Name: epair1a Type: ether ID: 00000006 Num hooks: 0 Name: ngctl6632 Type: socket ID: 0000000c Num hooks: 0 In case anyone else would like to test, I'm attaching my ng_socket.ko and ng_ether.ko modules for FreeNAS 9.2.0-RELEASE here.