OpenEXR cache renders and area renders



  • It's great that the Poser devs have chosen an open standard for their cached renders, but their doesn't seem to be any documentation about the file format of area renders, which are stored as .tmp files in the RenderCache folder within poser.PrefsLocation(). Poser seems well able to display the cumulative results of an initial, complete render, plus any number of subsequent area renders, but doesn't (currently, though I know previous versions used to) offer the ability to export such combined renders as a single OpenEXR format file.

    I'm only seeing these options for Export Image in Poser Pro 11.1.0.34764 on macOS:
    0_1538544275035_Screen Shot 2018-10-03 at 3.10.35 pm.png
    whereas I definitely used to be able to export them in Poser Pro 2012 (I can't run PP2014 as I upgraded my boot SSD and it's licensing got lost somehow) and still can:
    0_1538544291371_Screen Shot 2018-10-03 at 3.12.40 pm.png
    I wonder why we've lost that functionality recently?

    I also note that there is no OpenEXR module installed with Poser Python, and the [utterly broken on macOS] PIL (Python Image Libraries) bundled with Poser (can't save JPEG files) which I had to manually replace with Pillow to get ShaderWorks' Postwork Manager to save renders with my Sig, doesn't support OpenEXR files at all, so I can't even parse the cached renders or explore the file format without major shenanigans (I will probably have to compile and manually install the OpenEXR module myself).

    Has anyone made use of these area renders outside of Poser? I don't have anything which can identify or open them at all, and I've tried GraphicConverter (which displays OpenEXR but doesn't handle the HDR well) and Photoshop (preCC), which has no trouble with the .exr files but can't open them until the render completes and doesn't recognise the .tmp area renders.



  • @anomalaus whelp, just reminded myself that I'd had this discussion 6 months ago, and that the discrepancy in available export formats is due to having selected a partial area render (stored as a .tmp file in the RenderCache) to try to export. Poser does not currently support this (WHY NOT?)! When I select a full render, I do get the HDR and OpenEXR export options in Poser Pro 11.1.0.34764 on macOS.

    I guess I should just delete this topic, except that I want to explore the options for adding EXIF tags like render time, render memory and author to exported renders, which ought to be achievable, since the files already contain a "Poser Gamma" EXIFtag and the value 2.20, according to one of my image viewer apps, GraphicConverter.

    Must be time to go and bash my head against PIP in Poser Python again. X-/



  • @anomalaus OH DEAR! stymied again.

    >>> import pip
    >>> pip.main(['install','OpenEXR'])
    Collecting OpenEXR
    /Applications/Poser 11/Poser.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
      SNIMissingWarning
    /Applications/Poser 11/Poser.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
      Could not fetch URL https://pypi.python.org/simple/openexr/: There was a problem confirming the ssl certificate: [Errno 1] _ssl.c:503: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version - skipping
      Could not find a version that satisfies the requirement OpenEXR (from versions: )
    No matching distribution found for OpenEXR
    /Applications/Poser 11/Poser.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
    1
    

    the Poser Python 2.7 bundled OPENSSL_VERSION reports:

    >>> import ssl
    >>> ssl.OPENSSL_VERSION
    'OpenSSL 0.9.8zh 14 Jan 2016'
    

    unfortunately, this means that the bundled TLSv1.0, used for certification of packages being downloaded for installation via pip is now on the permanent, mandatory deprecation list, so pip will never work again in Poser Python until a newer distribution is bundled, since the package servers now have a mandatory requirement to refuse connections from anything prior to TLS 1.2!!!!!!

    Utterly SC@#$*&^%D



  • I'm going to continue posting my stumbling progress here, in the vain hope that others will have relevant experience to broaden the path.

    So, Poser's bundled Python 2.7.2 is now essentially both crippled and orphaned on macOS, since any module installation or individual module/package updates are blocked from their normal installation modes due to the Package servers blocking connection requests from old, insecure versions of pip, as are included with not only Poser's Python, but the macOS Sierra 10.12 standard Python 2.7.14 installation, which also includes the same pip v9.0.1, now permanently blocked since April 2018 due to it's reliance on openSSL's TLSv1.0.

    There do appear to have been workarounds for the macOS python installation, since pip v9.0.1 can no longer upgrade itself. The Unix shell "curl" command on the first line below:

    curl https://bootstrap.pypa.io/get-pip.py | python
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 1604k  100 1604k    0     0  2273k      0 --:--:-- --:--:-- --:--:-- 2271k
    Collecting pip
      Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
        100% |████████████████████████████████| 1.3MB 2.3MB/s 
    Installing collected packages: pip
      Found existing installation: pip 9.0.1
        Uninstalling pip-9.0.1:
          Successfully uninstalled pip-9.0.1
    Successfully installed pip-18.0
    
    

    does appear to have successfully updated the macOS Sierra python 2.7.14 pip module to v18.0, which uses TLSv1.2 or higher and is not blocked from installing python packages.

    I have yet to determine whether I can replace Poser's pip hierarchy with the files installed in macOS' Python and have it work (I've duplicated the Poser pip folder and used it to uninstall itself, which worked and removed the original folder. I'll report later whether that is successful, and whether renaming my duplicate folder restores Poser's awareness of the previous pip 9.0.1 package.

    On the OpenEXR front, now that pip is updated in macOS python, the first install attempt failed with a missing or unlocated ImathBox.h include file.

    pip install OpenEXR
    Collecting OpenEXR
      Using cached https://files.pythonhosted.org/packages/7c/c4/76bf884f59d3137847edf8b93aaf40f6257d8315d0064e8b1a606ad80b1b/OpenEXR-1.3.2.tar.gz
    Building wheels for collected packages: OpenEXR
      Running setup.py bdist_wheel for OpenEXR ... error
      Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/7k/xt8wllys2_lck7zcxm5xgtkm0000gn/T/pip-install-OOnC_9/OpenEXR/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/7k/xt8wllys2_lck7zcxm5xgtkm0000gn/T/pip-wheel-S4TP4l --python-tag cp27:
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.12-x86_64-2.7
      copying Imath.py -> build/lib.macosx-10.12-x86_64-2.7
      running build_ext
      building 'OpenEXR' extension
      creating build/temp.macosx-10.12-x86_64-2.7
      clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/OpenEXR -I/usr/local/include/OpenEXR -I/opt/local/include/OpenEXR -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c OpenEXR.cpp -o build/temp.macosx-10.12-x86_64-2.7/OpenEXR.o -g -DVERSION="1.3.2"
      OpenEXR.cpp:36:10: fatal error: 'ImathBox.h' file not found
      #include <ImathBox.h>
               ^~~~~~~~~~~~
      1 error generated.
      error: command 'clang' failed with exit status 1
      
      ----------------------------------------
      Failed building wheel for OpenEXR
      Running setup.py clean for OpenEXR
    Failed to build OpenEXR
    Installing collected packages: OpenEXR
      Running setup.py install for OpenEXR ... error
        Complete output from command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/7k/xt8wllys2_lck7zcxm5xgtkm0000gn/T/pip-install-OOnC_9/OpenEXR/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/7k/xt8wllys2_lck7zcxm5xgtkm0000gn/T/pip-record-1l3Fvg/install-record.txt --single-version-externally-managed --compile:
        running install
        running build
        running build_py
        creating build
        creating build/lib.macosx-10.12-x86_64-2.7
        copying Imath.py -> build/lib.macosx-10.12-x86_64-2.7
        running build_ext
        building 'OpenEXR' extension
        creating build/temp.macosx-10.12-x86_64-2.7
        clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include/OpenEXR -I/usr/local/include/OpenEXR -I/opt/local/include/OpenEXR -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/tcl-tk/include -I/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c OpenEXR.cpp -o build/temp.macosx-10.12-x86_64-2.7/OpenEXR.o -g -DVERSION="1.3.2"
        OpenEXR.cpp:36:10: fatal error: 'ImathBox.h' file not found
        #include <ImathBox.h>
                 ^~~~~~~~~~~~
        1 error generated.
        error: command 'clang' failed with exit status 1
        
        ----------------------------------------
    Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/7k/xt8wllys2_lck7zcxm5xgtkm0000gn/T/pip-install-OOnC_9/OpenEXR/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/7k/xt8wllys2_lck7zcxm5xgtkm0000gn/T/pip-record-1l3Fvg/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/7k/xt8wllys2_lck7zcxm5xgtkm0000gn/T/pip-install-OOnC_9/OpenEXR/
    

    Many others have experienced that problem, but the reported macOS workaround depends on having the HomeBrew package manager installed (which I fortunately already had, so much time saved there):

    % brew install openexr
    Updating Homebrew...
    ...
    <much extraneous Homebrew data omitted>
    ...
    ==> Installing dependencies for openexr: ilmbase
    ==> Installing openexr dependency: ilmbase
    ==> Downloading https://homebrew.bintray.com/bottles/ilmbase-2.2.1.sierra.bottle.tar.gz
    ######################################################################## 100.0%
    ==> Pouring ilmbase-2.2.1.sierra.bottle.tar.gz
    🍺  /usr/local/Cellar/ilmbase/2.2.1: 353 files, 5.6MB
    ==> Installing openexr
    ==> Downloading https://homebrew.bintray.com/bottles/openexr-2.2.0_1.sierra.bottle.tar.gz
    ######################################################################## 100.0%
    ==> Pouring openexr-2.2.0_1.sierra.bottle.tar.gz
    🍺  /usr/local/Cellar/openexr/2.2.0_1: 132 files, 11.1MB
    
    
    
    % sudo pip install openexr
    Password:
    ... <more irrelevant omissions> ...
    Collecting openexr
      Downloading https://files.pythonhosted.org/packages/7c/c4/76bf884f59d3137847edf8b93aaf40f6257d8315d0064e8b1a606ad80b1b/OpenEXR-1.3.2.tar.gz
    Installing collected packages: openexr
      Running setup.py install for openexr ... done
    Successfully installed openexr-1.3.2
    

    The macOS Sierra Python2.7.14 installation can now see the OpenEXR module.

    The major difficulty I see is in leveraging what is in the OS Python to repair Poser's Python bundle, since the shell commands have no immediate parallel within Poser's Python shell, so I'll probably attempt to copy individual package folders from one site-packages location to the other and see what happens.

    I was trying to use:

    >>> import pip
    >>> pip.main(['install','-U','pip'])
    

    to update pip itself, but that cannot happen with pip 9.0.1. Nor can OpenEXR be built that way, since the Homebrew must have fixed some broken or missing search paths for the build to complete, and I can't do the "sudo python" trick with Poser's bundled python.



  • Oh, I should add that I discovered that the Python 2.7 bundled with macOS High Sierra (10.13) has a recent enough version of pip (at least 9.0.3) that it supports the TLSv1.2 protocol and is does not have connections blocked by the Python package servers, though that doesn't help Poser Python on macOS High Sierra yet.



  • I only use area renders for two things: Spot checking settings or changes and for fixing small mistakes that I spot after doing a render. I can export the fixed image as one file in any format I choose.



  • @ghostship except that you can't then export the combined full and area render as HDR or OpenEXR, since those options aren't offered when the area render is displayed, only when a full render is selected.

    Poser knows how to seamlessly display the area render(s) on top of a full render, but won't let you combine them for export in the original format of OpenEXR.



  • Daft question, but if you rename the .tmp extension to .EXR does that then allow EXR export?



  • @amethystpendant afraid not. That's the first thing I tried. If I try to open the file with GraphicConverter 10, it can't identify the file type and asks me if I want to try a RAW import. Now that shows me the opening bytes of the file, which obviously contains the dimensions of the main render (rather than the dimensions and offsets of the area render), but since OpenEXR uses half precision floating point for it's RGB info, it can't make much sense of the file (though some settings seem to show previews that look like the right colours, but no X & Y resolutions I've tried give a meaningful image.

    The data's obviouisly there, since Poser can composite the full render and any number of subsequent area renders perfectly well on screen, but I can't fathom why they left out the bit where you combine them and export them in the base HDR format of OpenEXR!



  • @ghostship even area renders take significantly long times on my system. Having to do a complete re-render of the entire scene just seems wasteful. I keep all of the render cache entries (except for aborted renders) in an archive folder, in case I want to put them back in Poser's RenderCache and export them, but I can't preview any of the area renders outside Poser, which is frustrating.



  • @anomalaus OK, after checking it it doesn't give you the option for open EXR. interesting.



  • @ghostship I will admit that TIFF does support HDR, but the files are so much larger, since it uses full floating point precision, which is why ILM invented OpenEXR, to give the precision required without excessive storage requirements. To do so, they had to invent the half floating precision format, which apparently is also used internally by nVidia GPUs, though I'm just regurgitating what I've recently read, rather than claiming any TechnoMage Lore status.



  • OK, I'm making some progress here.

    Given that I was able to overcome the issue of old versions of pip being prevented from contacting the PyPI servers in my macOS Python 2.7 installation, I had proceeded to install the OpenEXR package there. While trawling through my development folder, I came across the breadcrumb that I'd left myself on how to escape the labyrinth install pillow as a replacement for the PIL (Python Image Libraries) package which is broken on mac, preventing scripts like Shaderworks' Postwork Manager (which relies on PIL) from saving renders as JPG files.

    The fix didn't involve directly adding packages within Poser's own Python bundle, but merely renaming the existing PIL package folder and soft-linking in the Pillow upgraded one from the macOS Python 2.7 installation hierarchy (plus updating the PIL.pth file to point to the replacement PIL package folder external to Poser. The same technique just worked for the OpenEXR package (which just involved three soft links to Imath.py, Imath.pyc and OpenEXR.so, with no .pth file required), successfully allowing me to issue:

    >>> import OpenEXR
    >>> import os
    >>> cachedir = os.path.join(poser.PrefsLocation(),'RenderCache')
    >>> lastrender = os.path.join(cachedir,os.listdir(cachedir)[-1])
    >>> lastrender
    u'/Users/GeoffIX/Library/Application Support/Poser Pro/11/RenderCache/2018-10-10_20h46m45s.exr'
    >>> OpenEXR.isOpenExrFile(lastrender)
    True
    

    in the Poser Python shell and not have it complain.



  • @amethystpendant & @ghostship first response from SM Support is that the TMP files in the render cache are an augmented form of BMP file. I was not aware that BMP supported half floating point storage of RGB channels which OpenEXR uses to support HDR, without the extreme, full floating point storage required by TIFF. Regardless, none of my image editors (PhotoShop and GraphicConverter) recognise the .tmp files when they're renamed to .bmp, so that's a bust.

    I've requested more detailed information on the file format, but I imagine the worst case scenario is that area renders do not contain or retain the HDR information that full renders do (pure speculation), so area renders will not prove to be a useful shortcut to fix render problems without HDR.


  • Poser Ambassadors

    @anomalaus

    The render cache temp files are some form of a ProEXR file, look at the header info in them...



  • @shvrdavid is that the case on macOS as well? Do you have an application that can open them directly?


  • Poser Ambassadors

    There are plugins for Adobe stuff to open them, but I do not know if SM put there own spin on them to make that not the case.

    I never really tried and don't use much Adobe stuff anymore.

    The plugins available for it adobe wise
    https://www.fnordware.com/ProEXR/
    https://github.com/fnordware/ProEXR
    http://fnordware.blogspot.com/2017/01/proexr-is-free.html



  • @shvrdavid the headers that I saw did, in the first instance, seem to conform to the BMP header, whose format I have since looked up. Whether or not that's the case, and trying to reverse engineer a file format which should be well documented (at least in a source code header file somewhere, if the developers were earning their pay properly, which I have no reason to doubt), is not my idea of fun, most days.

    If what you say is correct, why would I receive a reply specifying BMP with additions?


  • Poser Ambassadors

    Technically, it is a layered 16 or 32 bit map file. Other than that, no idea why



  • @shvrdavid thanks for the links. I have PhotoShop CS5.1, and it certainly seems to recognise the plugin, but while ProEXR EZ will recognise and open the OpenEXR render cache files with a .exr extension, it greys out the .tmp files. Duplicating one of those and changing the extension to .exr doesn't give previews and attempting to open one complains the file is corrupted or unrecognisable. Exactly the same behaviour as the OpenEXR plugin bundled with PhotoShop.