Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • World
  • Users
  • Groups
Skins
  • Light
  • Brite
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (Cyborg)
  • No Skin
Collapse
Brand Logo

CIRCLE WITH A DOT

  1. Home
  2. Uncategorized
  3. Hey #freebsd #bhyve users!

Hey #freebsd #bhyve users!

Scheduled Pinned Locked Moved Uncategorized
freebsdbhyveopenzfsmastery
24 Posts 10 Posters 3 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • dch@bsd.networkD dch@bsd.network

    @mwl have you tried this with NVMe type as disk? s/virtio-blk/nvme/ for the type.

    also maybe it runs out of “pci slots” try adding this:

    ahci_device_limit="8"

    I’ll try this tomorrow and see what comes up

    mwl@io.mwl.ioM This user is from outside of this forum
    mwl@io.mwl.ioM This user is from outside of this forum
    mwl@io.mwl.io
    wrote last edited by
    #9

    @dch I have not, good idea! maybe virtio as well? Gonna play around some this weekend...

    1 Reply Last reply
    0
    • ctuffli@mastodon.onlineC This user is from outside of this forum
      ctuffli@mastodon.onlineC This user is from outside of this forum
      ctuffli@mastodon.online
      wrote last edited by
      #10

      @Tubsta @dch @mwl hopefully goes without saying, but yell at me if a virtual shelf of N NVMe disks in bhyve doesn’t work

      1 Reply Last reply
      0
      • ctuffli@mastodon.onlineC This user is from outside of this forum
        ctuffli@mastodon.onlineC This user is from outside of this forum
        ctuffli@mastodon.online
        wrote last edited by
        #11

        @Tubsta @dch @mwl bhyve "slot" specification in the max is bus:pcislot:function where bus is 0-255, pcislot is 0-31, and function is 0-7. If the maths don't fail me, that is around 64k-ish devices

        dexter@bsd.networkD 1 Reply Last reply
        0
        • ctuffli@mastodon.onlineC ctuffli@mastodon.online

          @Tubsta @dch @mwl bhyve "slot" specification in the max is bus:pcislot:function where bus is 0-255, pcislot is 0-31, and function is 0-7. If the maths don't fail me, that is around 64k-ish devices

          dexter@bsd.networkD This user is from outside of this forum
          dexter@bsd.networkD This user is from outside of this forum
          dexter@bsd.network
          wrote last edited by
          #12

          @ctuffli @Tubsta @dch @mwl So, we talking bhyve or vm-bhyve?

          I would not be surprised if vm-bhyve doesn’t assume that, “no one would ever want more than 8 devices”, or at least devices of one type.

          Have you tried raw bhyve?

          mwl@io.mwl.ioM 1 Reply Last reply
          0
          • mwl@io.mwl.ioM mwl@io.mwl.io

            @ParadeGrotesque

            I already freeload sufficiently, thank you!

            and I'm not gonna ask for expensive kit just to write one chapter.

            drscriptt@oldbytes.spaceD This user is from outside of this forum
            drscriptt@oldbytes.spaceD This user is from outside of this forum
            drscriptt@oldbytes.space
            wrote last edited by
            #13

            @mwl @ParadeGrotesque if we were close I have EMC Fibre Channel disks in shelf that I could loan to you.

            1 Reply Last reply
            0
            • mwl@io.mwl.ioM This user is from outside of this forum
              mwl@io.mwl.ioM This user is from outside of this forum
              mwl@io.mwl.io
              wrote last edited by
              #14

              @Tubsta @dch @dexter @ctuffli thank you!!!

              time zones rock, I go to sleep and someone else fixes my problems. 😉

              1 Reply Last reply
              0
              • dexter@bsd.networkD dexter@bsd.network

                @ctuffli @Tubsta @dch @mwl So, we talking bhyve or vm-bhyve?

                I would not be surprised if vm-bhyve doesn’t assume that, “no one would ever want more than 8 devices”, or at least devices of one type.

                Have you tried raw bhyve?

                mwl@io.mwl.ioM This user is from outside of this forum
                mwl@io.mwl.ioM This user is from outside of this forum
                mwl@io.mwl.io
                wrote last edited by
                #15

                @dexter @ctuffli @Tubsta @dch vm-bhyve, because I'm lazy

                1 Reply Last reply
                0
                • mwl@io.mwl.ioM This user is from outside of this forum
                  mwl@io.mwl.ioM This user is from outside of this forum
                  mwl@io.mwl.io
                  wrote last edited by
                  #16

                  @Tubsta @dch @dexter @ctuffli

                  bhyve seems to work well, but yeah, those command lines are

                  dexter@bsd.networkD 1 Reply Last reply
                  0
                  • mwl@io.mwl.ioM mwl@io.mwl.io

                    @Tubsta @dch @dexter @ctuffli

                    bhyve seems to work well, but yeah, those command lines are

                    dexter@bsd.networkD This user is from outside of this forum
                    dexter@bsd.networkD This user is from outside of this forum
                    dexter@bsd.network
                    wrote last edited by
                    #17

                    @mwl @Tubsta @dch @ctuffli I am speaking purely from a diagnostics perspective. I believe, to their credit, their log shows the command that was executed. Which is kinda important, given the many “bhyve” bugs along the lines of “it’s treating my boot drive as if it is a CD-ROM drive …” #TrueStory!

                    So yeah. Share the underlying actual stuff please because very few vm-bhyve bugs are bhyve bugs.

                    dexter@bsd.networkD 1 Reply Last reply
                    0
                    • dexter@bsd.networkD dexter@bsd.network

                      @mwl @Tubsta @dch @ctuffli I am speaking purely from a diagnostics perspective. I believe, to their credit, their log shows the command that was executed. Which is kinda important, given the many “bhyve” bugs along the lines of “it’s treating my boot drive as if it is a CD-ROM drive …” #TrueStory!

                      So yeah. Share the underlying actual stuff please because very few vm-bhyve bugs are bhyve bugs.

                      dexter@bsd.networkD This user is from outside of this forum
                      dexter@bsd.networkD This user is from outside of this forum
                      dexter@bsd.network
                      wrote last edited by
                      #18

                      @mwl @Tubsta @dch @ctuffli I mean like, I didn’t write an OpenBSD bug report when I found that typo in Absolute OpenBSD. In of course the one chapter you didn’t trust me to have in advance. Nosiree. THEN you tasked me with raising over $150K to date (USD no less, not that Looney stuff) for a conference.

                      But yeah sure, blame bhyve.

                      Until of course it comes time to write a book on the subject… ahem!

                      mwl@io.mwl.ioM 1 Reply Last reply
                      0
                      • dexter@bsd.networkD dexter@bsd.network

                        @mwl @Tubsta @dch @ctuffli I mean like, I didn’t write an OpenBSD bug report when I found that typo in Absolute OpenBSD. In of course the one chapter you didn’t trust me to have in advance. Nosiree. THEN you tasked me with raising over $150K to date (USD no less, not that Looney stuff) for a conference.

                        But yeah sure, blame bhyve.

                        Until of course it comes time to write a book on the subject… ahem!

                        mwl@io.mwl.ioM This user is from outside of this forum
                        mwl@io.mwl.ioM This user is from outside of this forum
                        mwl@io.mwl.io
                        wrote last edited by
                        #19

                        @dexter @Tubsta @dch @ctuffli

                        Sure, I ask if something's weird and file a bug.

                        But sometimes, I'm doing something actively daft. If bhyve-vm hangs trying to launch a host with 75 drives, my first instinct is to say "well, this would be because I'm a damn idiot trying to use 75 drives on one VM..."

                        dexter@bsd.networkD 1 Reply Last reply
                        0
                        • mwl@io.mwl.ioM mwl@io.mwl.io

                          @dexter @Tubsta @dch @ctuffli

                          Sure, I ask if something's weird and file a bug.

                          But sometimes, I'm doing something actively daft. If bhyve-vm hangs trying to launch a host with 75 drives, my first instinct is to say "well, this would be because I'm a damn idiot trying to use 75 drives on one VM..."

                          dexter@bsd.networkD This user is from outside of this forum
                          dexter@bsd.networkD This user is from outside of this forum
                          dexter@bsd.network
                          wrote last edited by
                          #20

                          @mwl @Tubsta @dch @ctuffli Bruh. bhyve now supports NUMA domains, because test ALL THE THINGS. 75 drives is like… a good start. All scaling issues should be explored to their unholy extremes.

                          U got this!

                          mwl@io.mwl.ioM 1 Reply Last reply
                          0
                          • dexter@bsd.networkD This user is from outside of this forum
                            dexter@bsd.networkD This user is from outside of this forum
                            dexter@bsd.network
                            wrote last edited by
                            #21

                            @Tubsta @dch @mwl @ctuffli Thar u go.

                            1 Reply Last reply
                            0
                            • dexter@bsd.networkD dexter@bsd.network

                              @mwl @Tubsta @dch @ctuffli Bruh. bhyve now supports NUMA domains, because test ALL THE THINGS. 75 drives is like… a good start. All scaling issues should be explored to their unholy extremes.

                              U got this!

                              mwl@io.mwl.ioM This user is from outside of this forum
                              mwl@io.mwl.ioM This user is from outside of this forum
                              mwl@io.mwl.io
                              wrote last edited by
                              #22

                              @dexter @Tubsta @dch @ctuffli

                              okay then, I'll send you my stupid failures.

                              1 Reply Last reply
                              0
                              • mwl@io.mwl.ioM mwl@io.mwl.io

                                Hey #freebsd #bhyve users!

                                I'm trying to test draid virtually for #openzfsmastery, because nobody's gonna give me a shelf of 60 disks to play with. I set everything up like:

                                disk35_name="disk35"
                                disk35_type="ahci-hd"
                                disk35_dev="sparse-zvol"

                                Seems that if I have 35 ahci-hd entries, disk0 through disk34, bhyve and freebsd works. At disk35, the host panics on boot.

                                Is this expected with bhyve? Or does FreeBSD need a special tweak with 36 disks?

                                dgeo@mamot.frD This user is from outside of this forum
                                dgeo@mamot.frD This user is from outside of this forum
                                dgeo@mamot.fr
                                wrote last edited by
                                #23

                                @mwl
                                I use 48+ disks zpools on bare-metal hardware, did not see any limit here...

                                1 Reply Last reply
                                0
                                • mwl@io.mwl.ioM mwl@io.mwl.io

                                  Hey #freebsd #bhyve users!

                                  I'm trying to test draid virtually for #openzfsmastery, because nobody's gonna give me a shelf of 60 disks to play with. I set everything up like:

                                  disk35_name="disk35"
                                  disk35_type="ahci-hd"
                                  disk35_dev="sparse-zvol"

                                  Seems that if I have 35 ahci-hd entries, disk0 through disk34, bhyve and freebsd works. At disk35, the host panics on boot.

                                  Is this expected with bhyve? Or does FreeBSD need a special tweak with 36 disks?

                                  mwl@io.mwl.ioM This user is from outside of this forum
                                  mwl@io.mwl.ioM This user is from outside of this forum
                                  mwl@io.mwl.io
                                  wrote last edited by
                                  #24

                                  Finally getting back to my 72-disk draid bhyve VM for #openzfsmastery.

                                  Turns out that setting ahci_device_limit="8" in the vm-bhyve config let it boot just fine.

                                  Thank you everyone for your suggestions, but especially @dch for this one!

                                  1 Reply Last reply
                                  1
                                  0
                                  Reply
                                  • Reply as topic
                                  Log in to reply
                                  • Oldest to Newest
                                  • Newest to Oldest
                                  • Most Votes


                                  • Login

                                  • Login or register to search.
                                  • First post
                                    Last post
                                  0
                                  • Categories
                                  • Recent
                                  • Tags
                                  • Popular
                                  • World
                                  • Users
                                  • Groups