2017-12-18 23:10 CET

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0001457Orfeo Toolbox (OTB)Generalpublic2017-10-16 11:02
Reporterjulien 
Assigned Togpasero 
PriorityurgentSeverityminorReproducibilityalways
StatusresolvedResolutionfixed 
PlatformOSRedhat 6OS Version
Summary0001457: Can not build OTB >= 6.2 superbuild on systems where a hdf5 system lib exist and is tool old
DescriptionEven if the superbuild hdf5 libs are found, the version check in FindHDF5.cmake uses the system hdf5 headers, and report their version. As a result, HDF5 configure step aborts with error message:

CMake Error at CMakeLists.txt:592 (MESSAGE):
  netCDF requires at least HDF5 1.8.10. Found 1.8.5.1.
Additional InformationCmake version: 3.7.2
TagsNo tags attached.
Attached Files

-Relationships
+Relationships

-Notes

~0004345

gpasero (administrator)

In the description, I guess you meant "netCDF configure step aborts"

I have a potential solution here : since the find_package() should not use the "Module" mode, let's make it use the "Config" mode (i.e. : search and find HDF5Config.cmake). In netCDF sources, can you try the following patch :

CMakeLists.txt:572
  -FIND_PACKAGE(HDF5 COMPONENTS C HL REQUIRED)
  +FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED)

~0004346

julien (administrator)

Tried it:

CMake Error at CMakeLists.txt:571 (FIND_PACKAGE):
  Could not find a package configuration file provided by "HDF5" with any of
  the following names:

    HDF5Config.cmake
    hdf5-config.cmake

  Add the installation prefix of "HDF5" to CMAKE_PREFIX_PATH or set
  "HDF5_DIR" to a directory containing one of the above files. If "HDF5"
  provides a separate development package or SDK, be sure it has been
  installed.

~0004347

gpasero (administrator)

Right, next step is :

SuperBuild/CMake/External_netcdf.cmake:28
-set(NETCDF_CONFIG)
+set(NETCDF_CONFIG -DHDF5_DIR:PATH=${SB_INSTALL_PREFIX}/share/cmake)

The install location of HDF5 is not detected by default.

~0004348

julien (administrator)

Still not working with current patch:

$ git diff --cached
diff --git a/SuperBuild/CMake/External_netcdf.cmake b/SuperBuild/CMake/External_netcdf.cmake
index 01d7a82d0c..e4e67673f6 100644
--- a/SuperBuild/CMake/External_netcdf.cmake
+++ b/SuperBuild/CMake/External_netcdf.cmake
@@ -25,7 +25,7 @@ SETUP_SUPERBUILD(NETCDF)
 # declare dependencies
 ADDTO_DEPENDENCIES_IF_NOT_SYSTEM(NETCDF HDF5 ZLIB CURL HDF4 JPEG)
 
-set(NETCDF_SB_CONFIG)
+set(NETCDF_SB_CONFIG ${NETCDF_SB_CONFIG} -DHDF5_DIR:PATH=${SB_INSTALL_PREFIX}/share/cmake)
 
 ExternalProject_Add(NETCDF
   PREFIX NETCDF
diff --git a/SuperBuild/patches/NETCDF/netcdf-2-find_hdf5-all.diff b/SuperBuild/patches/NETCDF/netcdf-2-find_hdf5-all.diff
new file mode 100644
index 0000000000..5664420a7c
--- /dev/null
+++ b/SuperBuild/patches/NETCDF/netcdf-2-find_hdf5-all.diff
@@ -0,0 +1,11 @@
+--- NETCDF/CMakeLists.txt 2017-09-28 10:04:42.483388000 +0000
++++ NETCDF/CMakeLists.txt 2017-10-04 07:35:57.766792000 +0000
+@@ -568,7 +568,7 @@
+ SET(SEARCH_PACKAGE_NAME ${HDF5_PACKAGE_NAME})
+ FIND_PACKAGE(HDF5 NAMES ${SEARCH_PACKAGE_NAME} COMPONENTS C HL NO_MODULES REQUIRED ${NC_HDF5_LINK_TYPE})
+ ELSE(MSVC)
+- FIND_PACKAGE(HDF5 COMPONENTS C HL REQUIRED)
++ FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED)
+ ENDIF(MSVC)
+
+ ##

Configure step now succeeds, but the build step fails with:

CMakeFiles/ncgen.dir/util.c.o: In function `check_err2.part.1':
util.c:(.text.unlikely+0x6b): undefined reference to `H5Eprint1'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Tget_member_type'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5T_NATIVE_SHORT_g'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Pcreate'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Tcopy'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Aexists'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Pget_filter2'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Aget_space'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Sget_simple_extent_dims'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5P_CLS_DATASET_ACCESS_ID_g'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Eset_auto2'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Sget_simple_extent_ndims'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Oclose'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Aget_type'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Fcreate'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5Pset_layout'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5T_STD_U8BE_g'
../liblib/libnetcdf.so.11.4.0: undefined reference to `H5T_STD_I32BE_g'

...

What is even more strange, is that CMakeCache.txt only contains HDF5_DIR:PATH variable, and no hdf5 libraries path as it used to contain before the patch.

~0004349

gpasero (administrator)

Could you check if the layout of the install directory is fine : all libraries should be installed in $INSTALL_PREFIX/lib

I think it is normal if the CMakeCache.txt only contains the HDF5_DIR. When using the XXXConfig.cmake, the new variables are not exposed in the cache (just like when doing find_package(OTB)).

~0004350

julien (administrator)

$ ls /work/logiciels/otb/rh6/develop-20170929-123122/lib/
cmake libgeos_c.so.1 libhdf5_hl_cpp.so.1.10.1 libITKMesh-4.12.so.1 libitkvcl-4.12.so libopenjp2.so libQtOpenGL.la
engines libgeos_c.so.1.10.1 libhdf5_hl.so libITKMetaIO-4.12.so libitkvcl-4.12.so.1 libopenjp2.so.2.2.0 libQtOpenGL.prl
libcrypto.so libgeos.so libhdf5_hl.so.100.1.0 libITKMetaIO-4.12.so.1 libitkvnl-4.12.so libopenjp2.so.7 libQtOpenGL.so
libcrypto.so.1.0.0 libgeos.so.3.6.1 libhdf5_hl.so.1.10.1 libitknetlib-4.12.so libitkvnl-4.12.so.1 libOpenThreads.so libQtOpenGL.so.4
libcurl.so libgeotiff.a libhdf5.settings libitknetlib-4.12.so.1 libitkvnl_algo-4.12.so libOpenThreads.so.20 libQtOpenGL.so.4.8
libdfalt.a libgeotiff.so libhdf5.so libitkNetlibSlatec-4.12.so libitkvnl_algo-4.12.so.1 libOpenThreads.so.3.3.0 libQtOpenGL.so.4.8.7
libdfalt.so libgeotiff.so.2 libhdf5.so.100.1.0 libitkNetlibSlatec-4.12.so.1 libITKVNLInstantiation-4.12.so libossim.so libshark.so
libdfalt.so.4.2.13 libgeotiff.so.2.1.2 libhdf5.so.1.10.1 libITKOptimizers-4.12.so libITKVNLInstantiation-4.12.so.1 libossim.so.1 libshark.so.0
libexpat.so libGLEWmx.so libhdf5_tools.a libITKOptimizers-4.12.so.1 libITKWatersheds-4.12.so libossim.so.1.8.20 libshark.so.3.0.0
libfftw3f.la libGLEW.so libhdf5_tools.so libITKOptimizersv4-4.12.so libITKWatersheds-4.12.so.1 libpng.so libsqlite3.so
libfftw3f.so libglfw.so libhdf5_tools.so.100.1.0 libITKOptimizersv4-4.12.so.1 libjpeg.la libpng.so.1 libssl.so
libfftw3f.so.3 libglfw.so.3 libhdf5_tools.so.1.10.1 libITKPath-4.12.so libjpeg.so libpng.so.1.6.16 libssl.so.1.0.0
libfftw3f.so.3.4.4 libglfw.so.3.1 libITKBiasCorrection-4.12.so libITKPath-4.12.so.1 libjpeg.so.62 libproj.so libsvm.so
libfftw3f_threads.la libglut.la libITKBiasCorrection-4.12.so.1 libITKPolynomials-4.12.so libjpeg.so.62.1.0 libproj.so.12 libtiff.so
libfftw3f_threads.so libglut.so libITKCommon-4.12.so libITKPolynomials-4.12.so.1 libmfhdfalt.a libproj.so.12.0.0 libtiff.so.5
libfftw3f_threads.so.3 libglut.so.3 libITKCommon-4.12.so.1 libITKQuadEdgeMesh-4.12.so libmfhdfalt.so libQtCore.la libtiff.so.5.2.6
libfftw3f_threads.so.3.4.4 libglut.so.3.9.0 libitkdouble-conversion-4.12.so libITKQuadEdgeMesh-4.12.so.1 libmfhdfalt.so.4.2.13 libQtCore.prl libtinyxml.so
libfftw3.la libhdf4.settings libitkdouble-conversion-4.12.so.1 libITKSpatialObjects-4.12.so libmuparser.so libQtCore.so libturbojpeg.la
libfftw3.so libhdf5.a libITKFFT-4.12.so libITKSpatialObjects-4.12.so.1 libmuparserx.so libQtCore.so.4 libturbojpeg.so
libfftw3.so.3 libhdf5_cpp.a libITKFFT-4.12.so.1 libITKStatistics-4.12.so libmuparserx.so.4.0.7 libQtCore.so.4.8 libturbojpeg.so.0
libfftw3.so.3.4.4 libhdf5_cpp.so libITKIOImageBase-4.12.so libITKStatistics-4.12.so.1 libopencv_core.so libQtCore.so.4.8.7 libturbojpeg.so.0.1.0
libfftw3_threads.la libhdf5_cpp.so.100.1.0 libITKIOImageBase-4.12.so.1 libitksys-4.12.so libopencv_core.so.2.4 libQtGui.la libz.so
libfftw3_threads.so libhdf5_cpp.so.1.10.1 libITKKLMRegionGrowing-4.12.so libitksys-4.12.so.1 libopencv_core.so.2.4.13 libQtGui.prl libz.so.1
libfftw3_threads.so.3 libhdf5_hl.a libITKKLMRegionGrowing-4.12.so.1 libITKTransform-4.12.so libopencv_ml.so libQtGui.so libz.so.1.2.8
libfftw3_threads.so.3.4.4 libhdf5_hl_cpp.a libITKLabelMap-4.12.so libITKTransform-4.12.so.1 libopencv_ml.so.2.4 libQtGui.so.4 openjpeg-2.2
libfreetype.so libhdf5_hl_cpp.so libITKLabelMap-4.12.so.1 libitkv3p_netlib-4.12.so libopencv_ml.so.2.4.13 libQtGui.so.4.8 pkgconfig
libgeos_c.so libhdf5_hl_cpp.so.100.1.0 libITKMesh-4.12.so libitkv3p_netlib-4.12.so.1 libopenjp2.a libQtGui.so.4.8.7

~0004351

gpasero (administrator)

Maybe something to try : like for MSVC case, can you add the type of library requested. In netCDF:

CMakeLists.txt:572
  -FIND_PACKAGE(HDF5 COMPONENTS C HL REQUIRED)
  +FIND_PACKAGE(HDF5 COMPONENTS C HL NO_MODULE REQUIRED ${NC_HDF5_LINK_TYPE})

In fact, the MSVC case also uses the hdf5-config.cmake, so we may find clues by looking at the MSVC specific parts of this file.

~0004352

julien (administrator)

I finally got it to build with the following patch:

diff --git a/SuperBuild/CMake/External_netcdf.cmake b/SuperBuild/CMake/External_netcdf.cmake
index 01d7a82d0c..27f14dd74b 100644
--- a/SuperBuild/CMake/External_netcdf.cmake
+++ b/SuperBuild/CMake/External_netcdf.cmake
@@ -25,7 +25,8 @@ SETUP_SUPERBUILD(NETCDF)
 # declare dependencies
 ADDTO_DEPENDENCIES_IF_NOT_SYSTEM(NETCDF HDF5 ZLIB CURL HDF4 JPEG)
 
-set(NETCDF_SB_CONFIG)
+set(NETCDF_SB_CONFIG ${NETCDF_SB_CONFIG} -DHDF5_C_LIBRARY:PATH=${SB_INSTALL_PREFIX}/lib/libhdf5.so
+ -DHDF5_HL_LIBRARY:PATH=${SB_INSTALL_PREFIX}/lib/libhdf5_hl.so -DHDF5_INCLUDE_DIR:PATH=${SB_INSTALL_PREFIX}/include)
 
 ExternalProject_Add(NETCDF
   PREFIX NETCDF

This skips the find_package(hdf5) entirely. If it is ok I can commit this as a bugfix in 6.2 branch.

~0004353

rashadkm (developer)

a better fix is to get rid of FindHDF.cmake from CMake's distribution and keep it in OTB's CMake/ directory. You know like shark, glew, opencv etc..

otherwise, we may or may not revisit this bug for each upgrade of plaform/cmake version combination.

~0004354

julien (administrator)

The thing is it is not OTB that tries to find HDF5, it is NetCDF. So technically the new FindHDF.cmake should be in NETCDF cmake/ directory ...

~0004356

rashadkm (developer)

so add both NetCDF and HDF as they work together in someway

~0004357

gpasero (administrator)

This fix will do, but we should maybe leave a comment in the code to warn that a better fix may exist if we look into netCDF CMakeLists.txt.

~0004371

julien (administrator)

https://git.orfeo-toolbox.org/otb.git/commitdiff/12722ef7d98a8949d79a10d287758054a780b114

~0004382

gpasero (administrator)

SuperBuild broken on Windows.
I will limit the previous patch to UNIX case.

~0004383

gpasero (administrator)

Pushed another fix on release-6.2:
https://git.orfeo-toolbox.org/otb.git/commit/9b4696c74a24810c1c3d1fcac31af7cb27a09b9f
+Notes

-Issue History
Date Modified Username Field Change
2017-10-03 15:23 julien New Issue
2017-10-03 17:28 gpasero Note Added: 0004345
2017-10-03 17:28 gpasero Assigned To => gpasero
2017-10-03 17:28 gpasero Status new => feedback
2017-10-04 09:37 julien Note Added: 0004346
2017-10-04 09:37 julien Status feedback => assigned
2017-10-04 10:00 gpasero Note Added: 0004347
2017-10-05 11:32 julien Note Added: 0004348
2017-10-05 12:29 gpasero Note Added: 0004349
2017-10-05 12:58 julien Note Added: 0004350
2017-10-05 14:36 gpasero Note Added: 0004351
2017-10-06 10:30 julien Note Added: 0004352
2017-10-06 12:12 rashadkm Note Added: 0004353
2017-10-06 14:59 julien Note Added: 0004354
2017-10-09 15:16 rashadkm Note Added: 0004356
2017-10-09 18:38 gpasero Note Added: 0004357
2017-10-11 09:35 grizonnetm Priority normal => urgent
2017-10-13 10:41 julien Note Added: 0004371
2017-10-13 10:41 julien Status assigned => resolved
2017-10-13 10:41 julien Resolution open => fixed
2017-10-16 10:40 gpasero Note Added: 0004382
2017-10-16 10:40 gpasero Status resolved => assigned
2017-10-16 11:02 gpasero Note Added: 0004383
2017-10-16 11:02 gpasero Status assigned => resolved
+Issue History