FreeNAS - Feature #1974 VirtualBox can't be installed inside a Jail due to kernel module dependency 12/29/2012 04:30 PM - Marcus Ahlberg

Status: Resolved Estimated time: 0.00 hour Priority: Nice to have Assignee: John Hixson Category: Target version: 9.2.1.6-RELEASE Severity: New Needs Merging: Yes Reason for Closing: Needs Automation: No Reason for Blocked: Support Suite Ticket: n/a Needs QA: Yes Hardware Configuration: Needs Doc: Yes Description I have been trying to find a solution on how to install [[VirtualBox]] on a [[FreeNAS]] host. With memory so cheap, the power of a [[FreeNAS]] box should be enough for a couple of virtual servers.

The problem I have come across is the fact that [[FreeBSD]] jails share the kernel with the host and Virtualbox needs to load a module into the kernel.

I have found a possible solution for installing the kernel module on the host and then installing virtualbox inside a jail. This solution however assumes a full [[FreeBSD]] host where the ports tree can be used. http://forums.freebsd.org/showthread.php?t=21897

I quickly tried to install virtualbox-ose-kmod-legacy using pkg_add on the host, but I couldn't load it due to "KLD vboxnetflt.ko: depends on - not available or version missmatch". I haven't dug any deeper into this problem.

What I call for is support for software add-ons that require kernel modules to be loaded on the host.

History #1 - 12/29/2012 04:57 PM - Josh Paetzel The first stopper you are hitting here is the lack of the netgraph kernel module on [[FreeNAS]]. I'll think about this some, there are lots of hidden gotchyas.

Perhaps the best solution is shipping with the vbox modules, that seems like a reasonable thing to me.

#2 - 12/29/2012 05:27 PM - Josh Paetzel I've looked at this a bit more indepth. Running virtualbox in a jail is a unique case, but it's very useful functionality. What I'm going to do is set [[FreeNAS]] up so this just works.

#3 - 04/09/2013 04:01 PM - jlohiser - Is this still in the works or is this issue dead? I would like to run a + Asterisk server as a guest on top of [[FreeNAS]].

#4 - 05/31/2013 03:17 PM - Marcus Ahlberg I would also like an update on this.

#5 - 06/02/2013 05:26 PM - 8ender - I'd also love to see this as well. I tried setting up a proxmox server and running Freenas under that but its just not as good as Freenas bare metal. I have lots of spare cycles on this server and it makes so much more sense to have Freenas run bare metal and then have a Virtualbox jail to spawn

10/01/2021 1/9 other servers with, using the freenas mounts for storage.

#6 - 10/11/2013 03:14 PM - Patrick Private I'm running VirtualBox inside 9.x jail (also did that in 8.x) but lacking netgraph brings lot of issues. You can't use bridged mode for your VM's - and that's a big deal. Missing netgraph module that is needed as dependency for native VB modules prevents from using bridged mode interfaces in VM's and forces to use NAT on jailed bridge interface. This is sub-optimal solution with multiple limitations to "FreeNass + Virtualization" as whole.

Simply adding/ticking netgraph kernel compile option, would helped lot of us - that wants to stick to single-box solution that FN is perfectly suited to + using jails for simpler tasks and VM's for home lab/more advanced scenarios.

The obvious reason for FreeNas to support VB is that, sometimes powerful boxes that runs FreeNas could be easily re-used to do some visualization as well. Adding internal storage mapping that eliminate need for NFS/iSCSI makes FreeNas perfect single box, green IT solution for SMB/home.

#7 - 10/11/2013 05:03 PM - Josh Paetzel Testing a build with netgraph enabled in the kernel.

#8 - 11/02/2013 12:57 AM - Albert Marin Hi Patrick, could you post the procedure to implement VirtualBox inside 9.x jail please?

#9 - 11/13/2013 07:45 PM - Marcus Ahlberg Patrick, I would love to hear about your experiences. With or without netgraph.

#10 - 11/13/2013 10:48 PM - Patrick Private Albert, Marcus, Recently downloaded newest 9.2 alpha and there are some good news and some bad :D Good news: on host system VB network driver needed for bridged mode loads without any complains. That's major step forward. The bad news is that guest starting from jail cause kernel dump (.!.).

I've checked some options around but it was quite hopeless [multiple dumps :D]. Now I need to do testing on real hardware as I was testing VB inside guest in jail on FreeNas host that was running inside VB guest on my Windows. That can be a reason for my failure. So next tests will be on physical PC.

@Albert: I've applied multiple hacks in addition to guides available on web (there are some good howtos) to make is running for me. There are two major steps not covered in any guide that is: - allow.sysvipc=1 you should put on jail meta "jail.flags" file not on host directly as guides show - doing NAT'ed VB you need to go 'deep' into VB command line guides to set up proper port forwarding in order to be able access your virtual PC. (I did it once and unfortunately I can't recall exact settings now) I'll post on progress once done.

#11 - 11/17/2013 11:23 PM - - Category set to 18 - Status changed from Unscreened to Screened - Target version set to 49

10/01/2021 2/9 #12 - 01/02/2014 08:42 PM - Patrick Private Found some time to check it against 9.2 stable.

As it worked before in 9.2 RC build (but only without bridged mode in VB guest) now whole VB kernel support seems to be broken 100%. Where you could load vboxdrv.ko, it's no more possible. Looks like already reported here: https://bugs.freenas.org/issues/3701. For those using VB you just need to roll-back to 9.1 :(.

#13 - 01/05/2014 01:52 PM - Patrick Private Update: kernel modules provided by Josh Paetzel here: https://bugs.freenas.org/issues/3701 works and indeed system crash issue is resolved now (still not in stock 9.2). But as pointed out later in #3701 bridged adapters doesn't work. On interface you can see: - in guest: some (strange but not all?) outgoing arps - at host on bridged IF: arp requests WITH arp replies :), -- also legitymate outgoing packets like ICMP queries but without replies

What is interesting while trying to do ngctl to check hooks etc. you get "ngctl: can't create node: No such file or directory". In google you get only ARM related topics related to such error so I guess it's some kind of missing dependency.

#14 - 01/09/2014 01:01 AM - Ian Pitcher I've seen similar network traffic with the drivers compiled with the VIMAGE patch/option, but not crashing is good!

I got ngctl to run by downloading the kernel source at https://github.com/trueos/trueos/releases/tag/9.2.0-RELEASE and replacing /usr/src in my portjail with its contents (this also helps if you want to compile your own vbox modules). I determined that ngctl needed ng_socket.ko loaded, so I did:

Within port jail: cd /usr/src/sys/modules/netgraph make; make install

Exit the jail and loaded the module on the host: kldload /[path to root of jail]/boot/kernel/ng_socket.ko

Re-enter jail, start VM with bridged NIC, and check ngctl output:

[root@box /]# ngctl list

There are 2 total nodes: Name: vboxnetflt_epair1b Type: vboxnetflt ID: 00000003 Num hooks: 0 Name: ngctl29256 Type: socket ID: 00000004 Num hooks: 0

I don't know anything about netgraph, so I don't know what I should expect to see there. I did compile if_tap.ko and get bridged networking going with that, but it has a major downside: whenever the application accessing the tunnel device quits (ie. VirtualBox), it crashes the kernel. I suspect it's also VIMAGE related, as I saw a PR describing the same behavior here: http://www.freebsd.org/cgi/query-pr.cgi?pr=158686 -- perhaps this is a regression?

And since I'm throwing whatever garbage I can at the wall to see what sticks (sorry), there's also this: http://lists.freebsd.org/pipermail/freebsd-questions/2013-May/251160.html

I did try to load ng_ether.ko and got "link_elf_obj: symbol ifnet undefined, linker_load_file: Unsupported file type" so maybe the kernel needs to be complied with options NETGRAPH_ETHER

10/01/2021 3/9 #15 - 01/15/2014 07:37 PM - Ian Pitcher - File ng_ether.ko added - File ng_socket.ko added

After much horsing around, I can say for certain now that NETGRAPH_ETHER (or ng_ether.ko) are necessary for this to work, and it works with or without VIMAGE enabled for the jail.

Here's the output from ngctl in the jail while VBoxHeadless is running a VM:

[root@box /]# ngctl list There are 3 total nodes: Name: epair0b Type: ether ID: 00000001 Num hooks: 2 Name: vboxnetflt_epair0b Type: vboxnetflt ID: 00000002 Num hooks: 2 Name: ngctl6630 Type: socket ID: 00000006 Num hooks: 0

For good measure, here's what it looks like on the host itself: antarctica# ngctl list There are 6 total nodes: Name: re0 Type: ether ID: 00000001 Num hooks: 0 Name: ipfw0 Type: ether ID: 00000002 Num hooks: 0 Name: bridge0 Type: ether ID: 00000003 Num hooks: 0 Name: epair0a Type: ether ID: 00000004 Num hooks: 0 Name: epair1a Type: ether ID: 00000006 Num hooks: 0 Name: ngctl6632 Type: socket ID: 0000000c Num hooks: 0

In case anyone else would like to test, I'm attaching my ng_socket.ko and ng_ether.ko modules for FreeNAS 9.2.0-RELEASE here.

#16 - 01/16/2014 12:14 AM - Patrick Private I've made some progress there. Using repo suggested by Ian (https://github.com/trueos/trueos/releases/tag/9.2.0-RELEASE) and following netgraph compile procedure I've produced all ng_*.ko modules. Easy way to get all src in place is to download https://github.com/trueos/trueos/archive/9.2.0-RELEASE.tar.gz and than unpack it to /usr/src .

Those attached above by Ian works, but I was not making any progress in terms of talking on bridged IF. I've added other ng_* modules judging by name:

22 1 0xffffffff81a31000 2f249 if_cxgbe.ko 24 3 0xffffffff81a61000 32652 vboxdrv.ko 25 2 0xffffffff81a94000 2bb1 vboxnetflt.ko 26 1 0xffffffff81a97000 400a vboxnetadp.ko 27 1 0xffffffff81b32000 17bd ng_ether.ko 28 1 0xffffffff81b34000 1e1d ng_socket.ko 29 1 0xffffffff81b36000 1aed ng_bridge.ko 30 1 0xffffffff81b38000 f15 ng_eiface.ko

10/01/2021 4/9 31 1 0xffffffff81b39000 72d ng_ipfw.ko 32 1 0xffffffff81b3a000 13b5 ng_iface.ko 33 1 0xffffffff81b3c000 3ed ng_split.ko 34 1 0xffffffff81b3d000 d8d ng_vlan.ko 35 1 0xffffffff81b3e000 e0d ng_one2many.ko 36 1 0xffffffff81b3f000 1aff ng_nat.ko 37 1 0xffffffff81b41000 24d ng_ip_input.ko 38 1 0xffffffff81b42000 b35 ng_device.ko 39 1 0xffffffff81a9c000 54d ng_UI.ko 40 1 0xffffffff81a9d000 13cd ng_car.ko 41 1 0xffffffff81a9f000 2ad ng_echo.ko 42 1 0xffffffff81aa0000 a8d ng_etf.ko 43 1 0xffffffff81aa1000 34d ng_ether_echo.ko 44 1 0xffffffff81aa2000 1acd ng_fec.ko 45 1 0xffffffff81aa4000 4cd ng_hub.ko 46 1 0xffffffff81aa5000 2041 ng_ksocket.ko 47 1 0xffffffff81aa8000 2185 ng_pipe.ko 48 1 0xffffffff81aab000 d4d ng_rfc1490.ko 49 1 0xffffffff81aac000 e2d ng_tag.ko 50 1 0xffffffff81aad000 a0d ng_tcpmss.ko 51 1 0xffffffff81aae000 1efa ng_vjc.ko and it WORKED! (almost:) Disclaimer: I have no idea what is correct set of modules to load - so further investigation is needed, it require to start-test-stop VB guest each time to test :)

Both in jail or directly in host (via ) you can configure your nic as bridge (I've tested on both Ethernet ifs I have on host) and those works perfectly. With exception that I was unable to talk back to my freenas host. On host interface I saw ARP requests from VB guests along with replies but that was just it.

So for guest<->host I've made extra IF and set it to NAT + route configuration on guest to talk to host directly via NAT'ed if. For all other it all talks now on bridged one in both direction.

Multiple jail restarts and multiple VB restarts and so far very stable.

#17 - 01/16/2014 01:36 AM - Ian Pitcher Not sure I understand your network config, but it sounds like ng_bridge.ko and ng_ether.ko, at least, are part of the solution for you. Looking at the netgraph hooks should help narrow it down. Post the output of "ngctl list" run from within your jail while your VM is running. I have two Windows VMs, each running virtio interfaces bridged to the epair interface in my VIMAGE jail, and network access is fine to/from my entire subnet and the Internet.

I should mention that I compiled ng_ether.ko by cobbling together a working patch from this post:

10/01/2021 5/9 https://groups.google.com/forum/#!msg/mailing.freebsd.bugs/eJoUdTeBUvk/kOfLv0r1fSIJ to get around the "link_elf_obj: symbol ifnet undefined, linker_load_file: Unsupported file type" issue that I mentioned above.

#18 - 01/16/2014 02:22 AM - Patrick Private Yes, those files were uploaded at first and they seem to be part of solution. Than I've checked but bridging didn't worked. Later adding (blindly) further modules enabled em0 bridged VB guest (VB headless running from jail) to communicate with all other IPs but FNAS server itself (em0 host IP).

Traces:

From jail: ngctl: can't create node: Operation not permitted

From host (FNAS): There are 10 total nodes: Name: em0 Type: ether ID: 0000000e Num hooks: 2 Name: em1 Type: ether ID: 0000000f Num hooks: 0 Name: bge0 Type: ether ID: 00000010 Num hooks: 0 Name: ipfw0 Type: ether ID: 00000011 Num hooks: 0 Name: epair0a Type: ether ID: 00000012 Num hooks: 0 Name: epair0b Type: ether ID: 00000013 Num hooks: 0 Name: epair1a Type: ether ID: 00000015 Num hooks: 0 Name: vboxnetflt_em0 Type: vboxnetflt ID: 00000038 Num hooks: 2 Name: ngctl84963 Type: socket ID: 00000039 Num hooks: 0 Name: ipfw Type: ipfw ID: 0000001c Num hooks: 0

VBoxManage showvminfo (run in jail):

NIC 1: MAC: 080027ACB5D2, Attachment: Bridged Interface 'em0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-vms, Bandwidth group: none NIC 2: MAC: 080027E6B044, Attachment: NAT, Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-vms, Bandwidth group: none NIC 2 Settings: MTU: 0, Socket (send: 64, receive: 64), TCP Window (send:64, receive: 64)

#19 - 01/18/2014 05:24 PM - Ian Pitcher

10/01/2021 6/9 Sorry, I misunderstood; when you said "bridge" I thought that you were referring to an "if_bridge" interface, not the bridged interface from VirtualBox. There's probably some critical difference between your setup and mine, but I don't know what it could be. The netgraph hooks between em0 and vboxnetflt_em0 from your ngctl output are what I would expect to see based on my limited understanding of netgraph. All I can say is that I know that those netgraph hooks weren't set up properly on my system before ng_ether.ko was loaded, and now they are. Nothing else was required as far as I know.

Has anyone else tried those modules?

EDIT: In an attempt to replicate your results, I just restarted the jail with VIMAGE disabled, and attached re0 (my main ethernet interface on the host) to one of my VirtualBox VMs. I was able to replicate the results you had where you could not communicate back to the host from within the VM, but communication to other hosts on the local subnet and the Internet works perfectly. With VIMAGE enabled, network communications on the VM work perfectly, including to and from the FreeNAS host.

#20 - 01/19/2014 06:21 PM - Patrick Private Ian, you right: I was referring to bridge as VB bridge ie. plugging into real host interface. That was my main goal - to be able to create vm's and attach them to whatever IF I want (incl. vlans defined on host). I've changed kmem options to limit memory usage by FNAS to make safe space for VBox. That + phpvirtualbox makes most close experience to ESXi server, on small scale of course but sufficient for home lab.

Underlying reason for all that is to replace my former ESXi 5.4 setup as VMware began it's journey to get rid of 'free ESXi' users by limiting version 10 Virtual Machines manage (you can only use trial vSphere on Windows Server [!!!] to manage new guests) usage in new 5.5 ESXi edition. And here it goes: FNAS + VirtualBox + phpvirtualbox as a great replacement.

#21 - 01/21/2014 03:40 PM - Michael Fayez Dears... I want to say it's working.... yes... FreeNAS as HOST & VirtualBOX(Headless) & PhpVirtualBox inside jail with bridged mode for CentOS as guest workfine as a charm :)

Thanks a lot, thanks very much... it will be great if included in future releases.

#22 - 01/31/2014 02:14 AM - Kenneth Langga I also hope that this would be included in the future releases.

In the meantime, could you guys post the necessary steps to replicate your setup (FreeNAS + VirtualBox + phpvirtualbox)? Thanks.

#23 - 02/17/2014 04:57 PM - Patrick Private - File boot_modules.zip added

Just moved to 9.2.1 and since I've had to re-do all VB, than I've created short summary.

Use one of your jails to do all ports and src stuff. Inside jail: - get fresh trueos-9.2.1-RELEASE (for a /usr/src) - update/get ports

On /usr/src apply patch from https://groups.google.com/forum/#!msg/mailing.freebsd.bugs/eJoUdTeBUvk/kOfLv0r1fSIJ

10/01/2021 7/9 (I got errors on ".if ${MK_VIMAGE} != "no" " so I cut it off and left only "+CFLAGS+= -DVIMAGE" in "sys/modules/netgraph/ether/Makefile" section.

With above patch - compile netgraph modules and copy ng_ether and ng_socket to host (FNAS) /kernel/modules In ports, compile VB kmod (configure to use VIMAGE) and copy modules vboxdrv vboxnetflt vboxnetflt to host (FNAS) /kernel/modules

Load all of them + usual virtualbox install, and edit rc.conf and loader.conf (as in all VB tutorials). To my surprise there is new 4.3.6 version of VB in ports but modules works well (10 minutes now) with old 4.2 VB binaries (to my suprise).

After compiling VB 4.3.6 form ports, it boots and works, but you need to redo phpvirtualbox install since it becomes incompatible. I got (still) issues when bridging to multiple IF's along with other jails + reaching host from within guest, but those are minor cases :D HTH

#24 - 05/03/2014 07:24 PM - Josh Paetzel - Status changed from Screened to Resolved

Changing this to resolved. We basically have this working and will be surfacing a plugin that exposes this functionality.

#25 - 05/03/2014 07:48 PM - Josh Paetzel - Status changed from Resolved to Fix In Progress - Assignee changed from Josh Paetzel to John Hixson

It turns out there are a few small pieces for this yet to call it working, however we are close!

#26 - 05/03/2014 08:23 PM - Jordan Hubbard Yes, we need the following three tunables set whenever this jail type is active: ng_ether_load=YES ng_gif_load=YES vboxdrv_load=YES

And of course the box needs to boot with them loaded (or kldload them manually before starting the virtualbox jail) or Bad Things™ can happen while trying to run VirtualBox in the jail.

Ideally, I think this should actually be a plugin rather than a template. The template was easy to do, and certainly advanced the "science experiment" past the point where it was clearly proven workable, but I think a plugin (even a very simple one) would accomplish three key things:

1. It would raise the visibility of this feature since the Plugin menu is more prominent than burying a template in the Jail type drop-down, where you have to actually know to go look for it. 2. It would give us the ability to have a plex / owncloud / maraschino style "trampoline" dialog which the user could click a link on to go straight to the phpvirtualbox UI. It's a small thing, but some users have trouble figuring out that the jail IP is what they need to go to, and where to find it. 3. Since the plugin installation process is a bit more "intelligent", we could add the tuneables from the plugin's install procedure vs having to figure out how to hook into the jail startup code.

10/01/2021 8/9 #27 - 05/03/2014 11:49 PM - Marcus Ahlberg Great news everyone! I can't wait to try this out.

#28 - 07/01/2014 03:09 PM - Kevin Martin Are all three of these tunables needed with the current jail template in 9.2.1.6RC2? Because I find that VB will crash when using a VM. I am only using ng_ether_load=YES.

#29 - 07/06/2014 04:58 PM - Corvin Wimmer The jail template is working perfectly in 9.2.1.6-RELEASE. By the way, the default username and password for phpVirtualBox is admin.

This is absolutely brilliant work! A big thank you to everyone that made this possible.

#30 - 07/06/2014 05:09 PM - Jordan Hubbard - Status changed from Fix In Progress to Resolved - Target version changed from 49 to 9.2.1.6-RELEASE

#31 - 07/07/2014 11:51 AM - Marcus Ahlberg Great work! I haven't had the time to upgrade yet but I'm looking forward to test it.

#32 - 12/09/2017 10:12 AM - - File deleted (ng_ether.ko)

#33 - 12/09/2017 10:12 AM - Dru Lavigne - File deleted (ng_socket.ko)

#34 - 12/09/2017 10:12 AM - Dru Lavigne - File deleted (boot_modules.zip)

10/01/2021 9/9

Powered by TCPDF (www.tcpdf.org)