Render Farm (Tech Stuff)



  • Hi folks,

    Firstly, please keep in mind, at all times, that I don't know a damn thing about any of this and, as such, you are teaching your 90-year-old, half blind, mostly deaf Gran to use a holographic, voice-activated Stark Industries smartphone! Please keep that in mind all the time, so as to avoid confusion.

    Ok... so... I have, like, ten computers all cluttering up my flat. Some are of the same specs, thanks to a local school upgrading and ditching their old ones, some are of differing specs, as they have come from individuals. I think I have four or five which I could get running the same spec. That said, all are relatively low spec systems. Not exactly Win95, struggle to play Doom kind of low, but probably the sort of thing I started out using Poser on, many, many years ago. They aren't even on the same playing field as my main PC, the specs of which can be seen in my sig.

    I would like to link these computers together to create a 'render farm', or something. I want to be able to churn out renders pretty much non-stop, as I'm tired of waiting roughly 24 hours for a decent render to complete, then having another 15 scenes in the pipeline, which I've created since beginning this render, waiting to be rendered. That's if I can create scenes at all while a render is ongoing... sometimes, I can't even run Task Manager to keep an eye on the performance, as even that crashes! No, I would like to keep my main PC running freely and dish out renders to others to get on with, leaving me free to create other scenes, watch Netflix or even make some music in my DAW.

    So, now you know what I want to do... how do I do it?! Please keep in mind the first paragraph; telling me something like "link them together and run them through a whatsit system" means totally nothing... how do I link them, which bits do I link with which, how do I run that, what, when, where, who? Totally clueless!

    Thank you folks, I have faith in you! I need to do something with these computers, I can build a den with them... you know you've gone full-on 'hoarder' when you can build a den with your junk! I hope they're not junk either!



  • Unfortunately, although you can network render with Poser using Queue Manager, each pc will still only render one frame or single image at a time. So you can't distribute a single image to multiple computers and get them all working on that same one.
    @seachnasaigh will be able to inform better on this, he has a pretty monster setup, as seen in this post and the next couple! https://forum.smithmicro.com/post/24953


  • Poser Ambassadors

    That would be an excellent way to make use of those extra machines.

    Caveats:


    You can also send animation renders to queue; QM will assign frames to each remote which is online.
    Queue will ^not^ distribute a single render among the remotes. I hope SM will add this capability to the next Poser Pro.


    My C1100 slaves use 270W (2.25 Amps) each when rendering at 100% CPU; a big dual Xeon workstation draws 744W (6.20 Amps) at 100% CPU.

    That said, this is a diagram of how the network is connected:
    0_1508942743364_network diagram.png

    OK, you don't want to have ten keyboards, ten mice, and ten monitors for the remotes. Instead, you scrounge one keyboard, one mouse, and one monitor. This can be the workstation's, if the remotes will be located near the workstation.

    Now, you need to buy a switching box called a KVM (Keyboard/Video/Mouse). It will need enough ports to accomodate all of the remotes (and the workstation, if you use the workstation's keyboard, monitor, and mouse). My TrendNet 16-port KVM is $187 on Amazom, $195 at Newegg
    The KVM box will use siamesed VGA/USB cables, so each remote gets a VGA and USB connection to the KVM box.
    The keyboard, monitor, and mouse plug into the KVM box.
    The KVM box has a row of "radio" buttons (exactly one is active at any time); you can view and control any machine by selecting it's KVM button.

    This is a KVM box; when the render slaves are online, there is a red LED lit for each machine; exactly one will have a green LED also; that indicates which unit is currently using the keyboard/monitor/mouse. This is a 16-port KVM; that's the size you'll need.
    0_1508944830174_KVM - front 1200p.JPG
    The units below the KVM are surge suppressors ($48 each for 15Amp (shown), $67 each for 20Amp model). You don't need these, though.

    The back of the KVM box has a gaggle of VGA cables, and on the photo left are the ports for the keyboard, monitor, and mouse.
    0_1508944933371_KVM - rear 1200p.JPG

    Using the KVM box, you'll get all of the remotes booted up, install/run SmithMicro's DLM, and insert the Queue Manager serial (^not^ the Poser serial). DLM will download only Queue Manager for each remote machine.
    When you want to batch render, or render an animation, boot each remote, and launch Queue on each remote.

    The first time you launch Queue Manager on each remote, watch for your security software to pop up a query about firewall permissions; you must give Queue Manager permissions, so that the remote can talk to the master workstation, and vice versa.

    You are also going to need an "unmanaged" switch (a router is a switch plus other stuff you don't need). The switch will need enough ports for every computer plus your modem. $60 for a 16-port Gigabit unmanaged switch, $80 for a 24-port.
    Every computer will connect to the switch via a Cat6 (aethernet, RJ45, LAN) cable.
    I recommend using different colors for the Cat cables, to facilitate identifying which cable is for which computer. Label tags are also helpful. This pic shows a 24-port switch; it's the box with a whole gaggle of Cat cables plugged in:
    0_1508946027400_switch router modem.jpg
    One of those switch ports will connect to your current modem.
    Switches come in 10/100MB/s and Gigabit speeds; Gigabit is faster.


  • Poser Ambassadors

    @Glen85 said in Render Farm (Tech Stuff):

    I would like to keep my main PC running freely and dish out renders to others to get on with, leaving me free to create other scenes, watch Netflix or even make some music in my DAW.

    You can set your master machine's Queue to ^not^ process jobs locally; it will send all renders out to remotes and leave your workstation free.

    0_1508952121894_QM set to not process locally.png

    In any case, once you've sent a render to queue, you're free to use Poser for some other scene.


  • Poser Ambassadors

    Probably the most common cause of problems is failure to give firewall permissions to Queue Manager.

    Another possible cause is differing build numbers; All machines must have the same build number for Poser Pro (master machine) and all remotes (Queue Manager). You can check a remote's Queue build number using Help:About
    0_1508963468920_check build number of Queue.png
    So if you run DLM on the master and update Poser Pro, then you need to run DLM on each remote and update them also.

    If you still have older Poser Pro versions on your master machine, you can install older versions of Queue on the remotes.
    The Queue icons look the same, so you may want to rename the shortcuts as Queue 11, Queue 2014, Queue 2012, Queue 2010.


  • Poser Ambassadors

    Simpler, clearer pictures of the hardware...

    KVM box (16-port):
    front...
    0_1508973587389_TrendNet 16-port KVM.jpg

    rear of KVM box...
    0_1508973629091_TrendNet 16-port KVM rear ports.jpg
    Silver section is where you hook up keyboard, monitor, and mouse. keyboard and mouse can use either the USB ports or use the PlayStation2 ports.
    Each computer plugs into one of the VGA ports in the black section.

    The KVM cables have a blue VGA and a USB at one end for the computer, and on the KVM box end a black VGA (which has the USB siamesed in)...
    0_1508973859640_TrendNet KVM cable.jpg

    This is an unmanaged 24-port Gigabit switch...
    0_1508973912154_Cisco 24-port Gigabit unmanaged switch 747p.jpg
    ...into which you connect all of your computers (and your internet modem) using Cat6 (aethernet) cables...
    0_1508973986621_Cat6 cables.jpg

    I recommend renaming your computers to give them unique names (from the default "my computer"), so that you can identify them easily.



  • While on the subject, I used to have a few PC hooked up through a relatively decent router, but the Render Q would constantly crash when rendering animated frames over the network. It got to the point where the network thing was ditched and the same file was opened on several PCs for individual rendering.

    Has this bug been squashed since Poser 7-ish? Is the RQ more more stable for network rendering these days?



  • With re-using old hardware in mind I did ponder how it would be to make a Unix compile of the QueueManager application (it does nothing necessarily graphic so should not be difficult) and pack that with a minimal barebone Unix (or DOS64) on a bootable diskette or USB stick. Plug it in. Start system. No need to waste memory on anything else than what you need. SystemHDD can go also.
    And oh yes, switch off room heating. LOL!


  • Poser Ambassadors

    @krios said in Render Farm (Tech Stuff):

    While on the subject, I used to have a few PC hooked up through a relatively decent router, but the Render Q would constantly crash when rendering animated frames over the network. It got to the point where the network thing was ditched and the same file was opened on several PCs for individual rendering.

    Has this bug been squashed since Poser 7-ish? Is the RQ more more stable for network rendering these days?

    If I recall correctly, the early Queue would accumulate in RAM and eventually bog the system down or crash. That was fixed some time ago.

    @fverbaas said in Render Farm (Tech Stuff):

    With re-using old hardware in mind I did ponder how it would be to make a Unix compile of the QueueManager application (it does nothing necessarily graphic so should not be difficult) and pack that with a minimal barebone Unix (or DOS64) on a bootable diskette or USB stick. Plug it in. Start system. No need to waste memory on anything else than what you need. SystemHDD can go also.
    And oh yes, switch off room heating. LOL!

    I wouldn't know if it is feasible to run Queue on Unix or Linux. @shvrdavid might know. If your geek mojo is strong, then you could run the servers on Unix/Linux, and PiXiE boot them all with one command from the workstation.

    Note to all: The hardware I suggested is not needed if you only have three or four machines; in that case, you just plug them all into your modem/router (most have four ports)
    For the keyboard,mouse, and monitor, you can use a KVM squid ($38) like this:
    0_1509025161201_squid 4-port KVM.jpg 0_1509025170326_squid 4-port KVM diagram.jpg


  • Poser Ambassadors

    @Glen85 Do these 10 (?) castoff machines have an OS license? That is, do they have Windows 7 (or Mac OSX) installed and working? If so, is it 32bit or 64bit?

    Or, do they have an earlier OS (Win Vista, WinXP, Mac OS 9.6)?

    Or, do is the holographic Windows sticker sratched/X'ed out, and there is no operating system installed?

    Next question: How much memory is in each machine?

    What processor do they have? How many cores? HyperThreaded? Clockspeed of the processor (2.66GHz, e.g.)?



  • @seachnasaigh said in Render Farm (Tech Stuff):

    If I recall correctly, the early Queue would accumulate in RAM and eventually bog the system down or crash. That was fixed some time ago.

    Good to know. Thank you!



  • ...what about allocating virtual RAM for rendering?

    Got a scene that clocks in at ~13GB of RAM. The workstation has enough RAM and the CPU is at a constant 100% capacity:

    0_1509032469735_100.jpg

    The slave PC got only 8 GB RAM, and for some reason the CPU is dragging its feet.

    0_1509032529560_11.jpg

    Is there a way to remedy this with some hard drive RAM? Or does the Poser scene need to be split up to get it under 8 GB RAM?


  • Poser Ambassadors

    The workstation's graph looks good. You want the CPU at 100%.

    The slave would do well with more RAM. It is the lack of RAM which is leaving the CPU cores standing around idle. It also puts considerable wear on the hard drive.

    Whether you can add RAM is controlled by the motherboard's main chipset; many motherboards are limited to 8GB, some are limited to 24GB, etc.
    Server/workstation motherboards tend to have more generous RAM capabilities; even a little midtower ATX server board (like my Urania, an HP z600) can accommodate 48GB. E-ATX boards usually allow 96GB, and 192GB with a RAM riser.
    The point is, you need to look at the mobo specs to see how much RAM the mobo can read; more RAM may physically fit in the DIMM slots, but if the main chipset is wimpy, you won't be able to use the extra memory.

    Another potential limiter of RAM is the version of Windows. Of course, anything 32bit is severely limited (4GB total, with the video card's onboard memory being subtracted from that 4GB).
    Of the 64bit versions, Win7Pro and Win7Ultimate will allow lots of RAM, but the "home" versions not so much. 8GB for Win7 Home Basic, and 16GB for Win7 Home Premium.
    0_1509033977357_Win7 memory limits.PNG

    You might try smaller bucket size; that will reduce the memory use somewhat (but probably not enough).

    You need more RAM. You'll need to look up the mobo's specs to see if it can accept more RAM. Either from the manufacturer's website, or run the free utility Speccy, and in the motherboard tab, look at the chipset vendor & model.
    For Urania, the main chipset is an Intel 5520.



  • @seachnasaigh

    Thanks for the pointers... problem is that the slave is a laptop, and 8 gigs of ram is all it'll eat.
    Might have to sacrifice the HD for virtual RAM. Possible? Will Poser see it?


  • Poser Ambassadors

    @krios I don't know; I've never tried. If the laptop has a second HD slot, and if the second HD can be specified as the virtual RAM drive, I'd get a 250GB SSD ($80) and use that for the virtual RAM, and so spare the OS hard drive.

    Anybody else know about using virtual RAM for this?



  • Ooh, lots to take in, thank you very much!

    The machines, at the moment, are totally clean, no OS or anything, most didn't come with hard drives even, so I'm going to use some of the drives I have kept over the years and try moving data around or deleting it, as they all have a lot of stuff on at the moment. RAM wise, I don't know off the top of my head, but as well as these machines, I have an electrostatic bag full of memory sticks to play around with.

    I'll try and get as much info on the machines as I can, but they're stashed away at the moment and it involves me turning my tiny flat upside down to have them all out, so it will take some time... maybe weekend. I'm not in a rush to get this thing up and going, just need to do it right. I'm glad it seems like a possibility!

    Thank you, all!


  • Poser Ambassadors

    @Glen85 Ah, well, the question is, do they have an (intact, not canceled) holographic Windows sticker?
    If they have a 64bit Win7 sticker (which version?), you can re-install onto a new hard drive.
    If they have no sticker, or the stickers have been X-ed out, then you would need to add the cost of buying an OEM Win7 (or newer) license. These cost about $140 nowadays. The OEM license is cheaper than the retail license, but does not offer Microsoft support getting the OS installed. The install disc runs itself, so I've never had need of MS support.

    Before making any decision, you'll want to identify the motherboard. From your description, they aren't operational, so you can't run Speccy. The task is then to open the case side panel, and -with bright lighting and reading glasses- find the model number on the motherboard. We can then look up the specs to tell us that machine's potential.

    Regarding memory, if these are a batch of sister machines (same make/model/vintage), then you could cannibalize memory from one to load up another. That option may not be there if the DIMM slots were filled with low-capacity memory sticks. In a case like that, you have to remove the low capacity DIMMs and replace them with higher capacity DIMMs, of the correct type. The RAM sticks must be the type intended for that motherboard, and all of the DIMMs in a machine should have the same clockspeed and timing.

    You can look on one of the existing DIMMs, and it will say something like PC3-10600, and there may be a group of numbers like 5,5,5,6. PC3 means DDR3, 10600 is the clockspeed in MHz, and the 5,5,5,6 bit is the timing.
    0_1509061229015_Samsung_DDR3_4GB_2Rx8 PC3-10600.jpg

    Server/workstation mobos can take either regular memory or registered memory; registered memory does some of its own controller/routing chores, and so a mobo might be specced to take 24GB of regular desktop memory, or 48GB registered memory.

    Regular desktop motherboards can only handle the non-registered type memory, and their capacity will be limited, though if it can take 12GB, 16GB or 24GB, then you'll be happy. The remote doesn't need quite as much memory as the workstation; the remote only has to handle rendering one pre-set frame.

    Another valuable piece of evaluation data is the type of processor and the type of CPU socket. If these machines have a simple dual-core (two render threads) of modest speed, then that won't offer much render speed. If it has a high clockspeed HyperThreaded hex-core processor (twelve render threads), it will be very helpful as a render slave.
    It is often possible to replace a mediocre CPU with a higher performance CPU made for the same socket. Not always, because the motherboard's CPU support features may not be capable of feeding the faster CPU's greater power appetite.

    The case may a decal on the grill indicating which series of processor you have (Pentium D, Core two Duo, Core i5, Xeon)
    With the machine not running, the only way to find out which processor those castoff machines have is to remove the fin&fan cooler from atop the CPU. Only do this is you're willing to buy a syringe of thermal paste for re-installation. Clean out dust and cat hair while you're at it.
    Do ^^not^^ remove the CPU itself.
    Use alcohol wipes to clean the grey thermal paste from the CPU.
    With a bright light and reading glasses, try to read the processor model number.
    0_1509059782741__35.JPG
    Now to re-install the heat sink fin&fan, squeeze out a bit of thermal paste the size of a grain of rice onto the center of the CPU, then replace the heatsink.

    If it turns out that the school castoffs don't offer much promise, then you might consider another option, such as buying a refurb server blade.
    This Dell C1100 has two HyperThreaded hex-core Xeons -twenty four render threads at 2.66GHz (3.06GHz turbo)- and 72GB of registered RAM, for $351+$35sh.
    Install a new hard drive and a Win7Pro OEM license, and you have serious render slave muscle.


  • Poser Ambassadors

    Just quickly scanned eBay for "2x X5650"; here's a blade with dual X5650 Xeons (twenty four render threads at 2.66GHz) and 16GB RAM for $150+sh.



  • That's some really good (if not, in parts, slightly confusing... but don't worry, I'll do some homework) information, thank you. I'll get all the info I can and probably take some pics too, but I'll have to overhaul my flat to get everything out and have room to work, as well as having room to exist in such a place with such clutter, lol! I'll probably number or letter each machine to make it easier to identify them. I'm totally open to gutting everything and having piles of components to make a farm from... more on that idea:

    I live in a studio flat, quite small, storage space is hardly existent. My living 'room' and bed'room' are one, but I'd like to separate them. I've designed a modular system which features a built-in bed, divided completely from the rest of the flat... I call it my 'passion pit' or 'pit' for short.

    This will be built out of MDF, and it's all entirely doable and wouldn't take all that much work, really. Much of it will be made from interlocking 'boxes', and, as such, could be dismantled in minutes and loaded into a van for moving day, then be rebuilt as a similar kind of thing somewhere else. The 'footboard' of the bed will have a decent-sized television mounted on it, with my studio monitors on each side. The base unit of this is something I've put a lot of thought into... there will be three or four of these 'boxes' stacked on top of each other to make the floor-to-ceiling 'footboard'. The bottom section could, if adequately ventilated, be used to mount the motherboards etc. Cables could be routed one direction or another, keeping it tidy, and ports could be routed to a panel fairly easily, with extensions, if needed.

    So, you see, if I vented it all properly and made sure it was all done right, I could have a render farm in one box, with no need for cases, cables etc. If I went for that option, obviously, all the machines would need gutting anyway.

    With regards to server blades, I hadn't thought much about that, but that's a decent idea! I have a friend who knows about all that stuff... she spends a reasonable amount of time messing about with servers for work and is Microsoft accredited (or a partner, I don't know, something along those lines), so maybe she could advise too... I'll link her to this thread.

    I should just mention that I'm in the UK, though, so getting something shipped could be interesting... not to mention my bank charging me for making payments in non-GBP currency... funny guys!

    Thanks for the help here, I'm really pleased... I knew my faith in you lot wasn't misplaced! ^_^



  • I just fished out some of my hard drives, memory and cards. I do have more but this is what is to hand right now.

    0_1509066731758_DSC_0212.JPG 0_1509066759650_DSC_0213.JPG 0_1509066791977_DSC_0214.JPG 0_1509066814369_DSC_0215.JPG

    Did I mention that I have never, once, 'thrown away' a hard drive... only took one apart when it died? I still have the drive from my very first computer somewhere... it's enormous, lol! And yet, most are full of all kinds of data, from music projects to Poser to po... well, I'll leave it at that, shall I? Lol!

    Suffice it to say, I think I'll have plenty of storage space, coupled with the four or five terabytes I'm currently using, for plenty of renders and, if I can figure out how to animate, animations! ^_^