complex scene makes 2 GPU render crash



  • Hey folks

    Just got me new system together and pulled out a scen I had been working on previously with my old computer. The scene renders fine on either card by itself but when I combine the two cards I get a crash. I'm running a Ryzen 5 1600x with 16gb ram no over clocking and the GPU's are a GTX 970 and GTX 980. Windows 10 and I think the cards have the latest drivers because I did not fiddle with that when installing: just downloaded whatever they gave me. Here is the scene.
    0_1497670653162_Girlfriends 1.jpg


  • Poser Ambassadors

    @ghostship
    Did you set tdr_delay to a higher value on your new system?
    See https://docs.microsoft.com/en-us/windows-hardware/drivers/display/tdr-registry-keys
    The default value may be too low for one of your cards - also depends on bucket size.



  • @wimvdb Thanks! I'll look that over. I had no idea I needed to do something with it.



  • @wimvdb OK, I looked at that page. I've never messed with the registry on a Windows PC. How, what, where would I do this at?


  • Poser Ambassadors

    @ghostship
    Look at this page - method 3: https://support.microsoft.com/en-us/help/2665946/-display-driver-stopped-responding-and-has-recovered-error-in-windows-7-or-windows-vista
    I have set it to 10 instead of 8 but for the rest it is the same as how i do i't.
    What it actually does is to increase the number of seconds for the graphics driver to respond before Windows decides the graphics driver has crashed. If the render buckets are large it often happens that it goes over the default time of 2 seconds.



  • @wimvdb In the instructions is says to close all windows programs. Will it cause a problem if my browser is open as I am reading the instructions off of the page?


  • Poser Ambassadors

    @ghostship
    No, that is no problem



  • @ghostship

    Hi there

    Looks like @wimvdb said can be down to TDR, for manipulation with TDR registry I use this, have used this on Win7 or Win10

    https://www.wagnardsoft.com/forums/viewtopic.php?f=8&t=755

    Sometimes downgrading drivers can help, in my case on Win10 I used fairly old drivers as with new drivers I have got crashes in Poser or IRAY

    Some new drivers cause several issues and I can recommend for earlier cards to use 373.06 which has been spotless there in Poser or any other rendering SW which do use CUDA

    Hope this helps

    Thanks, Jura



  • @jura11 Downgraded driver to the version you suggested. @wimvdb changed the registry setting. It said "add 8," and that got me a bit confused so I just typed in 8 in the field. Should I have typed 10"

    Also running the render a few times to see if I can read the erro (it goes by very quickly) The error says CUDA error out of memory or something similar.


  • Poser Ambassadors

    @ghostship
    It is a hex value, so 10 would have been 16 seconds, now it is set to 8 seconds.
    That is a lot better as the 2 seconds which were the default.
    You 980 has 4GB memory which is usually sufficient to get everything in VRAM. But it might stretch it if you use HDR maps of 10k by 5K (which will cost you around 1GB or VRAM).
    Remember also that a V4 figure (or others) often use 4K maps for the head, torso, limbs, eyes, innermouth, etc. Add the bump and specular maps as well and you often use 10 or more 4K textures, each taking up 32MB. And with multiple figures, that adds up pretty quickly. Also the OS takes about 300MB of VRAM memory for its own purposes.
    So it is possible that you ran our of VRAM.



  • @ghostship said in complex scene makes 2 GPU render crash:

    @jura11 Downgraded driver to the version you suggested. @wimvdb changed the registry setting. It said "add 8," and that got me a bit confused so I just typed in 8 in the field. Should I have typed 10"

    Also running the render a few times to see if I can read the erro (it goes by very quickly) The error says CUDA error out of memory or something similar.

    Hi @ghostship

    This error I've got several times and this has been mostly down to TDR and "8" should be OK, I've set 10 which has been enough

    But sometimes this didn't make any difference as error will return back

    Can you check how much yours scene taking from VRAM?

    Hope this helps

    Thanks, Jura



  • @jura11 @wimvdb elimination of just one of the hair models (does not matter which) allows it to render.



  • @ghostship said in complex scene makes 2 GPU render crash:

    @jura11 @wimvdb elimination of just one of the hair models (does not matter which) allows it to render.

    @ghostship

    If you eliminate one hair model how much VRAM is taking Poser?

    Assume you are rendering with not enabled BPT? Or do you have enabled BPT during the rendering?

    Hope this helps

    Thanks, Jura



  • switching out the HDRI in the skydome for a low-res jpg allows it to render as well.



  • This more likely looking like you are running out of VRAM

    4GB is not so much if you are looking for rendering several figures at once plus few props etc

    I would try edit HDRi and lower resolution this should help or at least you can try

    Hope this helps

    Thanks, Jura



  • @jura11 BPT is off. Looking through my log...


  • Poser Ambassadors

    @ghostship said in complex scene makes 2 GPU render crash:

    switching out the HDRI in the skydome for a low-res jpg allows it to render as well.

    Do the following:
    Make a low resolution of the HDR file (use an image editor which will keep it an HDR, like photoshop). Use that one as HDRI. If you use EZDome, you can make an additonal copy of the HDR in JPG format and use that in the inner dome as background. The JPG takes a lot less VRAM and the resulution will be the same. The HDR in the outerdome can be lower resolution since you only use the intensity and color values for the light.

    Another thing you can do is to lower the resolution of the hair textures. If they are 4K, then make them 2K - it will then use up a only a quarter of the memory as before. Hair textures are often oversized.



  • @wimvdb Already ahead of you there! This image was using ezdome so I just deleted that and loaded Envirosphere with just the HDRI image and I got it to render just fine.

    Here is the error I was getting:
    0_1497742358747_CUDA erro.jpg

    eliminating one texture map gets me down to about 5300 MB and it renders at that point.



  • @ghostship said in complex scene makes 2 GPU render crash:

    @wimvdb Already ahead of you there! This image was using ezdome so I just deleted that and loaded Envirosphere with just the HDRI image and I got it to render just fine.

    Here is the error I was getting:
    0_1497742358747_CUDA erro.jpg

    eliminating one texture map gets me down to about 5300 MB and it renders at that point.

    Hi @ghostship

    Can you try render that with single GPU if you will get that error, I would try render that with GTX980

    This 5300MB looks more likely you are not running out of VRAM, I suspect with single GPU you will see 2650-2800MB maybe close to 3GB as Poser SuperFly works like that and combine VRAM in single figure when you render, have look in my case my scene takes around 6GB VRAM on 3*GPU but every GPU is taking from VRAM just 2GB as max

    I will do later on quick render to show you what I mean but this error which you are getting is error which I saw several times and still I can't point out why I was getting as my VRAM usage never been so high and I never touched full VRAM or been close to VRAM limit

    This error I've seen too in Cycles last time and not sure what is causing this, HDRi or textures hard to say

    If you will look on the few forums you will find these errors popping up

    Hope this helps

    Thanks, Jura



  • @jura11 it is strange indeed! Before the new computer I was just using the 970 and this scene rendered fine on that. Here on the new computer this scene renders fine using either card by itself but combined I get the memory error.