2017-12-18 23:11 CET

View Issue Details Jump to Notes ]
IDProjectCategoryView StatusLast Update
0000219OTB-libGeneralpublic2011-11-18 11:39
Reporterchmoisy 
Assigned Tomickael 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
Summary0000219: HDF dataset not openable with ImageFileReader
DescriptionImageFileReader file existence test :
l496 (otbImageFileReader.txx)
  // Test if the file exists.
  if (!itksys::SystemTools::FileExists(this->m_FileName.c_str()))


prevents HDF dataset to be read because the filename is not a filesystem name
HDF4_EOS:EOS_GRID:"/home/chmoisy/MCD12Q1.A2008001.h13v11.005.2009338160210.hdf":MOD12Q1:Land_Cover_Type_1


if we try to open the HDF file, there is no bands, so we are blocked with otbGDALImageIO dataset->GetRasterCount() 0 value returned



can we use a GDALImageIO to be plugged with ImageFileReader so we can set the Filename of the Dataset directly ?
Additional InformationOTB v3.8
TagsNo tags attached.
Attached Files

-Relationships
related to 0000312closedjulienm Systematic crash when opening images with GDAL (tif, png, jpeg, etc) 
+Relationships

-Notes

~0000396

christop (administrator)

I've just added the support for hdf last week-end:
http://hg.orfeo-toolbox.org/OTB/rev/5369ae0f2d44
http://hg.orfeo-toolbox.org/OTB/rev/cbdb47182fab

Can you try with the dev version? Use just the hdf as filename: /home/chmoisy/MCD12Q1.A2008001.h13v11.005.2009338160210.hdf
and pass the dataset number using the SetDatasetNumber() method.

Let us know if you have any feedback, other suggestions for interface, etc. This could change until the next release if we find some more practical interface.

~0000400

julienm (developer)

We planned to make an evolution on this, so that we can also read an hdf file by setting the filename followed by ":#dataset", like :

/home/chmoisy/MCD12Q1.A2008001.h13v11.005.2009338160210.hdf:0

to select the first dataset of MCD12Q1.A2008001.h13v11.005.2009338160210.hdf


This will allow all existing applications to work with hdf files without needing to change the code and add the SetDatasetNumber() call, and will save the specific GUI evolutions.

~0000403

christop (administrator)

Sounds good, we could plan the access using the dataset name as well.
Julien, if you push the tests, I'll try to make them pass...

~0000404

chmoisy (reporter)

Seems to be working good
for raster resolution, do you use metadata DATAROWS and DATACOLUMNS ?
they seem more reliable than GDALDataset->GetRasterXSize() and GetRasterYSize()

I'll add a bunch of feature requests and suggestion for HDF, especially Modis HDF

~0000410

christop (administrator)

we use the gdal methods. Do you have a dataset example where the gdal methods fail to return the proper information?

~0000412

chmoisy (reporter)

the file is too big to be uploaded
it is the product Modis MOD13Q1

here is the beginning of the result from gdalinfo on the HDF file :
river: HDF4/Hierarchical Data Format Release 4
Files: MOD13Q1.hdf
Size is 512, 512
Coordinate System is `'
Metadata:
  HDFEOSVersion=HDFEOS_V2.9
  LOCALGRANULEID=MOD13Q1.A2010001.h17v05.005.2010028003734.hdf
  PRODUCTIONDATETIME=2010-01-28T05:37:34.000Z
  DAYNIGHTFLAG=Day
  REPROCESSINGACTUAL=reprocessed
  LOCALVERSIONID=5.2.1
  REPROCESSINGPLANNED=further update is anticipated
  SCIENCEQUALITYFLAG=Not Investigated
.....
 CHARACTERISTICBINSIZE=231.656358263889
  DATACOLUMNS=4800
  DATAROWS=4800
  GLOBALGRIDCOLUMNS=172800
  GLOBALGRIDROWS=86400
...
Subdatasets:
  SUBDATASET_1_NAME=HDF4_EOS:EOS_GRID:"MOD13Q1.hdf":MODIS_Grid_16DAY_250m_500m_V
I:250m 16 days NDVI
  SUBDATASET_1_DESC=[4800x4800] 250m 16 days NDVI MODIS_Grid_16DAY_250m_500m_VI
(16-bit integer)
...


and the result of gdalinfo on the first dataset ;
Driver: HDF4Image/HDF4 Dataset
Files: MOD13Q1.hdf
Size is 0, 0
Coordinate System is `'
Metadata:
  HDFEOSVersion=HDFEOS_V2.9
  LOCALGRANULEID=MOD13Q1.A2010001.h17v05.005.2010028003734.hdf
  PRODUCTIONDATETIME=2010-01-28T05:37:34.000Z
  DAYNIGHTFLAG=Day
  REPROCESSINGACTUAL=reprocessed
  LOCALVERSIONID=5.2.1
.....
  CHARACTERISTICBINSIZE=231.656358263889
  DATACOLUMNS=4800
  DATAROWS=4800
  GLOBALGRIDCOLUMNS=172800
....



on the HDF the size is wrong
on the dataset, the size is unknown



on the product MCD12Q1 (I will try to upload it), there is the same problem on the HDF file but it is working with the dataset :


Driver: HDF4Image/HDF4 Dataset
Files: MCD12Q1.A2008001.h13v11.005.2009338160210.hdf
Size is 2400, 2400
...
Metadata:

  HDF EOSVersion=HDFEOS_V2.9
....
  DATACOLUMNS=2400
  DATAROWS=2400
....

~0000414

christop (administrator)

Would it be possible to make the file available for download for a short while?
http://demo.ovh.com/ for example

You can send me the link by mail if necessary.

Otherwise, is there a public link for those data?

~0000415

chmoisy (reporter)

you can download it anonymously here : ftp://e4ftl01u.ecs.nasa.gov/MODIS_Composites/MOLT/MOD13Q1.005/2010.01.01/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf

~0000416

christop (administrator)

Hi,
Just implemented the MCD12Q1.A2008001.h13v11.005.2009338160210.hdf:0 notation for the dataset (http://hg.orfeo-toolbox.org/OTB/rev/27855e5e3405).
It seems to work with the MOD13Q1.A2010001.h17v05.005.2010028003734.hdf correctly.

If you compile OTB with the examples (BUILD_EXAMPLES ON), bin/SimpleViewer MOD13Q1.A2010001.h17v05.005.2010028003734.hdf:0
bin/SimpleViewer MOD13Q1.A2010001.h17v05.005.2010028003734.hdf:9
enable you to see different products.

Could you confirm that it's working fine for you?

~0000417

chmoisy (reporter)

With today's update, I've got some error messages :

/opt/OTB-dev/OTB-Binary/bin/SimpleViewer /data/MCD12Q1.005/2008.01.01/MCD12Q1.A2008001.h13v11.005.2009338160210.hdf:1
ERROR 1: GDreadfield() failed for block.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
terminate called after throwing an instance of 'itk::ExceptionObject'
  what(): /opt/OTB-dev/OTB/Code/IO/otbGDALImageIO.cxx:368:
itk::ERROR: GDALImageIO(0x1a0ac90): Error while reading image (GDAL format) /data/MCD12Q1.005/2008.01.01/MCD12Q1.A2008001.h13v11.005.2009338160210.hdf.

same error with a different dataset number and products MOD13Q1 et MOD13A1

~0000418

christop (administrator)

There might be a link with the version of gdal:
- what version are you using: gdalinfo --version
(I have GDAL 1.7.3, released 2010/11/10)

- what hdf does it supports: gdalinfo --formats
( Among other stuff, I have:
  HDF4 (ro): Hierarchical Data Format Release 4
  HDF4Image (rw+): HDF4 Dataset
  HDF5 (ro): Hierarchical Data Format Release 5
  HDF5Image (ro): HDF5 Dataset
)

~0000419

chmoisy (reporter)

this is a debian server
~> gdalinfo --formats | grep HDF
  HDF4 (ro): Hierarchical Data Format Release 4
  HDF4Image (rw+): HDF4 Dataset
  HDF5 (ro): Hierarchical Data Format Release 5
  HDF5Image (ro): HDF5 Dataset

~> gdalinfo --version
GDAL 1.7.2, released 2010/04/23

I will try on a ubuntu 10.10

~0000420

julienm (developer)

It works OK on my Ubuntu 10.10.

However, Emmanuel, you mentioned a potential issue in Monteverdi ?
Is it a performance issue (difficult to navigate in the image) ?

I noticed GDAL 1.8 RC2 (which is in the process of being accepted as the next 1.8) contains several improvements in the way HDF4 files are read (and maybe written).
The performance issues I see with gdal 173 are not reproduced when compiling with gdal 18
otbConvert (hdf -> tif) dropped down from several minutes to a few seconds, with the exact same OTB code involved, and the same goes for navigation in Monteverdi (even with a concatenated image made up from several subdatasets of the same file)

~0000421

christop (administrator)

Julien,
The issue (Monteverdi reader crash) I had with Monteverdi was when using the -in option:
monteverdi -in file.hdf:5
for example. Can you confirm whether it works for you?
Good to know that gdal 1.8 improve performances, the quicklook generation seemed a bit long.

Christophe,
I'm also on a Debian system. Do you have a 32 bit or 64 bits OS? The gdal version is slightly different as well, 1.7.2 vs 1.7.3. The gdal release note for 1.7.3 mention a fix about HDF (http://trac.osgeo.org/gdal/wiki/Release/1.7.3-News) but I'm not sure if it is related.
You can try with a gdal compiled in debug mode and try to get the error in gdb.

~0000422

chmoisy (reporter)

I have tested last night update on Ubuntu 10.10 and GDAL 1.8.0 RC2 and it is working.
I was able to have a look at MOD13Q1 product, HDF access seems to have been improved but the SimpleViewer is a but slow for this 4800*4800 pixels raster
A notice : access of the dataset is from 0 to number of datasets -1
gdalinfo gives the number from 1 to the number of datasets, so we have to care about taking the right dataset

~0000423

chmoisy (reporter)

my debian OS is a 64 bits one and my gdal version is 1.7.2
I will update this system to check

~0000424

chmoisy (reporter)

on debian I still got the error with GDAL 1.8.0 :
here is the ouput from gdb :
gdb) run /data/MOD13Q1.005/2010.01.01/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf
Starting program: /opt/OTB-dev/OTB-Binary/bin/SimpleViewer /data/MOD13Q1.005/2010.01.01/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf
[Thread debugging using libthread_db enabled]
warning: Lowest section in /usr/lib/libicudata.so.38 is .hash at 0000000000000120
[New Thread 0x7f4d17e6a730 (LWP 14901)]
ERROR 1: GDreadfield() failed for block.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
terminate called after throwing an instance of 'itk::ExceptionObject'
  what(): /opt/OTB-dev/OTB/Code/IO/otbGDALImageIO.cxx:368:
itk::ERROR: GDALImageIO(0x1ad8890): Error while reading image (GDAL format) /data/MOD13Q1.005/2010.01.01/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7f4d17e6a730 (LWP 14901)]
0x00007f4d0e1dfd25 in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007f4d0e1dfd25 in raise () from /lib/libc.so.6
#1 0x00007f4d0e1e2de1 in abort () from /lib/libc.so.6
#2 0x00007f4d0e7dd975 in __gnu_cxx::__verbose_terminate_handler ()
   from /usr/lib/libstdc++.so.6
#3 0x00007f4d0e7dbda6 in ?? () from /usr/lib/libstdc++.so.6
#4 0x00007f4d0e7dbdd3 in std::terminate () from /usr/lib/libstdc++.so.6
0000005 0x00007f4d0e7dbece in __cxa_throw () from /usr/lib/libstdc++.so.6
#6 0x00007f4d14bbea2f in itk::ProcessObject::UpdateOutputData ()
   from /opt/OTB-dev/OTB-Binary/bin/libITKCommon.so.3.20
#7 0x00007f4d14b79106 in itk::DataObject::UpdateOutputData ()
   from /opt/OTB-dev/OTB-Binary/bin/libITKCommon.so.3.20
#8 0x000000000052a110 in itk::ImageBase<2u>::UpdateOutputData ()
#9 0x00007f4d14bbe464 in itk::ProcessObject::UpdateOutputData ()
   from /opt/OTB-dev/OTB-Binary/bin/libITKCommon.so.3.20
0000010 0x00007f4d14b79106 in itk::DataObject::UpdateOutputData ()
   from /opt/OTB-dev/OTB-Binary/bin/libITKCommon.so.3.20
0000011 0x000000000052a110 in itk::ImageBase<2u>::UpdateOutputData ()
0000012 0x0000000000567ed7 in otb::StreamingShrinkImageFilter<otb::Image<double, 2u>, otb::Image<double, 2u> >::UpdateOutputData ()
0000013 0x00007f4d14b79106 in itk::DataObject::UpdateOutputData ()
   from /opt/OTB-dev/OTB-Binary/bin/libITKCommon.so.3.20
0000014 0x000000000052a110 in itk::ImageBase<2u>::UpdateOutputData ()
0000015 0x00007f4d14b7902a in itk::DataObject::Update ()
   from /opt/OTB-dev/OTB-Binary/bin/libITKCommon.so.3.20
0000016 0x00007f4d14bbd3ef in itk::ProcessObject::Update ()
   from /opt/OTB-dev/OTB-Binary/bin/libITKCommon.so.3.20
0000017 0x0000000000542a43 in otb::ImageLayerGenerator<otb::ImageLayer<otb::Image<double, 2u>, otb::Image<itk::RGBAPixel<unsigned char>, 2u> > >::GenerateQuicklook ()
0000018 0x00000000004b097b in otb::ImageLayerGenerator<otb::ImageLayer<otb::Image<double, 2u>, otb::Image<itk::RGBAPixel<unsigned char>, 2u> > >::GenerateLayer ()
#19 0x0000000000565ef6 in otb::StandardImageViewer<otb::Image<double, 2u>, otb::VectorD---Type <return> to continue, or q <return> to quit---
ata<double, 2u, double>, otb::PackedWidgetManager>::Update ()
0000020 0x00000000004ada23 in main ()


gdal is compiled with enable-debug option

~0000425

christop (administrator)

Can you locate in gdal where these error appear:
ERROR 1: GDreadfield() failed for block.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
(that's what is triggering the otb exception after the gdal function fails)

Index from 0 to n-1 is deliberate (we try to use that all over inside the library), the user interface in Monteverdi might present it in a way more suitable to the users (showing the dataset name as well?).

~0000426

julienm (developer)

monteverdi -in file.hdf:5
monteverdi -iml file.hdf:5 file.hdf:6 file.hdf:7

both works OK for me.


we are currently working on a better Monteverdi integration.
GDALImageIO has been modified to provide access to the list of dataset names/numbers, just need to put this in the OpenFile dialog when a hdf file is selected

~0000427

christop (administrator)

Julien,
Monteverdi seems fine to me as well (might have been a partial compilation).

Few points:

1. Could we add the dataset
ftp://e4ftl01u.ecs.nasa.gov/MODIS_Composites/MOLT/MOD13Q1.005/2010.01.01/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf
in LargeInput/MODIS/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf to have a few tests on a real image (some performance tests?)

2. Do we have some generic tests for pixel access for a wide range of images? We talked about it last september (something in the CMakeLists with an image list...)

3. We should provide access to the dataset names (via the ImageMetadata?) that would be a great information to display in Monteverdi...

~0000428

mickael (administrator)

With Ubuntu 8.04 and gdal 1.7.2
SimpleViewer and Monteverdi crash with the following output :
We think that it is a segfault in hdf library.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f84ce19b740 (LWP 11390)]
0x00007f84c0131323 in mcache_close () from /usr/lib/libdf.so.4
(gdb) bt
#0 0x00007f84c0131323 in mcache_close () from /usr/lib/libdf.so.4
#1 0x00007f84c012d822 in HMCPcloseAID () from /usr/lib/libdf.so.4
#2 0x00007f84c012da29 in HMCPendaccess () from /usr/lib/libdf.so.4
#3 0x00007f84c0116f93 in Hendaccess () from /usr/lib/libdf.so.4
#4 0x00007f84c03c12ab in ?? () from /usr/lib/libmfhdf.so.4
0000005 0x00007f84c55ae9b7 in GDdetach (gridID=4194304) at GDapi.c:5641
#6 0x00007f84c568ce2f in HDF4ImageRasterBand::IReadBlock (this=0xf33000, nBlockXOff=0, nBlockYOff=<value optimized out>,
    pImage=0xeebde0) at hdf4imagedataset.cpp:406
#7 0x00007f84c5846bfa in GDALRasterBand::GetLockedBlockRef (this=0xf33000, nXBlockOff=0, nYBlockOff=0, bJustInitialize=0)
    at gdalrasterband.cpp:1249
#8 0x00007f84c5851a8b in GDALRasterBand::IRasterIO (this=0xf33000, eRWFlag=GF_Read, nXOff=0, nYOff=0, nXSize=4800,
    nYSize=<value optimized out>, pData=0xfaf440, nBufXSize=4800, nBufYSize=2, eBufType=GDT_Int16, nPixelSpace=2, nLineSpace=9600)
    at rasterio.cpp:100
#9 0x00007f84cdaf115f in otb::GDALImageIO::Read (this=0x7f3cd0, buffer=0xfaa930)
    at /home/mickael/dev/src/OTB/Code/IO/otbGDALImageIO.cxx:365
0000010 0x000000000051d9b9 in otb::ImageFileReader<otb::Image<double, 2u> >::GenerateData (this=0x7e2d80)
    at /home/mickael/dev/src/OTB/Code/IO/otbImageFileReader.txx:177
0000011 0x00007f84c81cc6f7 in itk::ProcessObject::UpdateOutputData (this=0x7e2d80)
    at /home/mickael/dev/src/OTB/Utilities/ITK/Code/Common/itkProcessObject.cxx:987
0000012 0x00007f84c818711a in itk::DataObject::UpdateOutputData (this=0x7e71a0)
    at /home/mickael/dev/src/OTB/Utilities/ITK/Code/Common/itkDataObject.cxx:420
0000013 0x000000000052c86a in itk::ImageBase<2u>::UpdateOutputData (this=0x7e71a0)
    at /home/mickael/dev/src/OTB/Utilities/ITK/Code/Common/itkImageBase.txx:280
0000014 0x00007f84c81cc382 in itk::ProcessObject::UpdateOutputData (this=0x7e74b0)
    at /home/mickael/dev/src/OTB/Utilities/ITK/Code/Common/itkProcessObject.cxx:936
0000015 0x00007f84c818711a in itk::DataObject::UpdateOutputData (this=0x7eb820)
    at /home/mickael/dev/src/OTB/Utilities/ITK/Code/Common/itkDataObject.cxx:420
0000016 0x000000000052c86a in itk::ImageBase<2u>::UpdateOutputData (this=0x7eb820)
    at /home/mickael/dev/src/OTB/Utilities/ITK/Code/Common/itkImageBase.txx:280
0000017 0x000000000055b700 in otb::StreamingShrinkImageFilter<otb::Image<double, 2u>, otb::Image<double, 2u> >::UpdateOutputData (
    this=0x80af50) at /home/mickael/dev/src/OTB/Code/BasicFilters/otbStreamingShrinkImageFilter.txx:182
0000018 0x00007f84c818711a in itk::DataObject::UpdateOutputData (this=0x80b050)
    at /home/mickael/dev/src/OTB/Utilities/ITK/Code/Common/itkDataObject.cxx:420

~0000429

chmoisy (reporter)

I have displayed the parameter for SimpleVievwer application :
 ./bin/SimpleViewer /tmp/modis/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf
lFirstColumn : 0
lFirstLine : 0
lNbColumns : 4800
lNbLines : 2
ERROR 1: GDreadfield() failed for block.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
terminate called after throwing an instance of 'itk::ExceptionObject'
  what(): /opt/OTB-dev/OTB/Code/IO/otbGDALImageIO.cxx:372:
itk::ERROR: GDALImageIO(0x232c7d0): Error while reading image (GDAL format) /tmp/modis/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf.

The number of lines does not seem to be correct for
 int lNbLines = this->GetIORegion().GetSize()[1];

anyway setting this parameter to the good value does not fix the problem
the values returned by GetRasterXSize and GetRasterYSize are correct

~0000430

julienm (developer)

Last edited: 2011-01-19 14:36

The same segfault happens with gdal_translate, when trying to convert a subdataset to a tif file.

Christophe, can you try using gdal_translate ?
This will show that the problem is not related to the OTB implementation.

With this line command and gdb:
gdal_translate -of GTiff -mo "SUBDATASET_11_NAME=HDF4_EOS:EOS_GRID:"../src/OTB-Data/Input/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf":MODIS_Grid_16DAY_250m_500m_VI:250m 16 days composite day of the year" ../src/OTB-Data/Input/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf out.tif -sds

we obtain this output :
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fa5f29ed710 (LWP 26909)]
0x00007fa5f011a323 in mcache_close () from /usr/lib/libdf.so.4
(gdb) bt
#0 0x00007fa5f011a323 in mcache_close () from /usr/lib/libdf.so.4
#1 0x00007fa5f0116822 in HMCPcloseAID () from /usr/lib/libdf.so.4
#2 0x00007fa5f0116a29 in HMCPendaccess () from /usr/lib/libdf.so.4
#3 0x00007fa5f00fff93 in Hendaccess () from /usr/lib/libdf.so.4
#4 0x00007fa5f03aa2ab in ?? () from /usr/lib/libmfhdf.so.4
0000005 0x00007fa5f1e78f77 in GDdetach (gridID=4194304) at GDapi.c:5641
#6 0x00007fa5f1f72d42 in HDF4ImageRasterBand::IReadBlock (this=0xd18490, nBlockXOff=0, nBlockYOff=0, pImage=0x7fa5eac33010)
    at hdf4imagedataset.cpp:457
#7 0x00007fa5f21078ea in GDALRasterBand::GetLockedBlockRef (this=0xd18490, nXBlockOff=0, nYBlockOff=0, bJustInitialize=0)
    at gdalrasterband.cpp:1279
#8 0x00007fa5f2114b03 in GDALRasterBand::IRasterIO (this=0xd18490, eRWFlag=GF_Read, nXOff=0, nYOff=0, nXSize=4800,
    nYSize=<value optimized out>, pData=0x7fa5eae1b010, nBufXSize=4800, nBufYSize=1024, eBufType=GDT_Int16, nPixelSpace=2,
    nLineSpace=9600) at rasterio.cpp:106
#9 0x00007fa5f20ce737 in VRTSimpleSource::RasterIO (this=0xc013f0, nXOff=<value optimized out>, nYOff=<value optimized out>,
    nXSize=<value optimized out>, nYSize=<value optimized out>, pData=0x7fa5eae1b010, nBufXSize=4800, nBufYSize=1024,
    eBufType=GDT_Int16, nPixelSpace=0, nLineSpace=0) at vrtsources.cpp:754
0000010 0x00007fa5f20cb01d in VRTSourcedRasterBand::IRasterIO (this=0xca1f30, eRWFlag=<value optimized out>, nXOff=0, nYOff=0,
    nXSize=4800, nYSize=1024, pData=0x7fa5eae1b010, nBufXSize=4800, nBufYSize=1024, eBufType=GDT_Int16, nPixelSpace=2,
    nLineSpace=9600) at vrtsourcedrasterband.cpp:219
0000011 0x00007fa5f20ed341 in GDALDataset::IRasterIO (this=0xd18760, eRWFlag=GF_Read, nXOff=0, nYOff=0, nXSize=4800, nYSize=1024,
    pData=0x7fa5eae1b010, nBufXSize=4800, nBufYSize=1024, eBufType=GDT_Int16, nBandCount=1, panBandMap=0x7fff1e44b87c,
    nPixelSpace=2, nLineSpace=9600, nBandSpace=0) at gdaldataset.cpp:1498
0000012 0x00007fa5f20ed070 in GDALDataset::RasterIO (this=0xd18760, eRWFlag=GF_Read, nXOff=0, nYOff=0, nXSize=4800, nYSize=1024,
    pData=0x7fa5eae1b010, nBufXSize=4800, nBufYSize=1024, eBufType=GDT_Int16, nBandCount=1, panBandMap=0x7fff1e44b87c,
    nPixelSpace=2, nLineSpace=9600, nBandSpace=0) at gdaldataset.cpp:1732
0000013 0x00007fa5f21121b5 in GDALDatasetCopyWholeRaster (hSrcDS=0xd18760, hDstDS=0xcf1930, papszOptions=<value optimized out>,
    pfnProgress=0x7fa5f20df230 <GDALScaledProgress>, pProgressData=0xce7580) at rasterio.cpp:2922
0000014 0x00007fa5f1f49dad in GTiffDataset::CreateCopy (pszFilename=0x621700 "out.tif1", poSrcDS=0xd18760, bStrict=0, papszOptions=0x0,
    pfnProgress=0x7fa5f20e13e0 <GDALTermProgress>, pProgressData=0x0) at geotiff.cpp:7916
0000015 0x00007fa5f20f35b4 in GDALDriver::CreateCopy (this=0x608410, pszFilename=0x621700 "out.tif1", poSrcDS=0xd18760, bStrict=0,
    papszOptions=0x0, pfnProgress=0x7fa5f20e13e0 <GDALTermProgress>, pProgressData=0x0) at gdaldriver.cpp:612
0000016 0x00000000004041de in ProxyMain (argc=<value optimized out>, argv=0xc9dd00) at gdal_translate.cpp:1186
0000017 0x00000000004030a0 in ProxyMain (argc=8, argv=0x618d00) at gdal_translate.cpp:541
0000018 0x00007fa5f11401c4 in __libc_start_main () from /lib/libc.so.6
#19 0x00000000004022d9 in _start ()

~0000432

julienm (developer)

Can you report the hdf packages version of your respective system.


On my ubuntu 10.10 - 32bits desktop platform (working platform) :
hdf4 is provided by libhdf4-0-alt (modified version to avoid a NetCDF conflict...)

jmalik@PC8413:~$ aptitude show libhdf4-0-alt
Paquet : libhdf4-0-alt
Nouveau: oui
État: installé
Automatiquement installé: oui
Version : 4.2r4-10
Priorité : supplémentaire
Section : universe/libs
Responsable : Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Taille décompressée : 668k
Dépend: libc6 (>= 2.7), libjpeg62, zlib1g (>= 1:1.1.4)
Suggère: libhdf4-doc, libhdf4-alt-dev, hdf4-tools, libnetcdf4
Description : The Hierarchical Data Format library -- library package
 HDF is a multi-object file format for storing and transferring graphical and numerical data mainly used in scientific computing. HDF
 supports several different data models, including multidimensional arrays, raster images, and tables. Each defines a specific aggregate
 data type and provides an API for reading, writing, and organizing the data and metadata. New data models can be added by the HDF
 developers or users.
 
 This package contains the HDF run-time libraries which do not collide with the NetCDF library namespace. Fortran functions are missing in
 this flavor of the library set.
Site : http://www.hdfgroup.com/



On dora, ubuntu 8.04 64bits server (where we got the segfault) :
hdf4 is provided by libhdf4g (no "-alt" version is available)
jmalik@dora:~$ dpkg -L libhdf4g
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libhdf4g
/usr/share/doc/libhdf4g/copyright
/usr/share/doc/libhdf4g/changelog.Debian.gz
/usr/share/doc/libhdf4g/README.arm-fortran.gz
/usr/share/man
/usr/share/man/man5
/usr/share/man/man5/hdf.5.gz
/usr/lib
/usr/lib/libdf.so.4.1r4
/usr/lib/libmfhdf.so.4.1r4
/usr/lib/libdf.so.4
/usr/lib/libmfhdf.so.4


jmalik@dora:~$ aptitude show libhdf4g
Package: libhdf4g
State: installed
Automatically installed: no
Version: 4.1r4-21
Priority: optionnel
Section: universe/libs
Maintainer: Ubuntu MOTU Developers <ubuntu-motu@lists.ubuntu.com>
Uncompressed Size: 791k
Depends: libc6 (>= 2.5-5), libjpeg62, zlib1g (>= 1:1.2.1)
Suggests: libhdf4g-doc, libhdf4g-dev, hdf4-tools
Conflicts: libhdf4 (< 4.0.2-4), libhdf4g-run (< 4.1r4-20)
Replaces: libhdf4
Description: The Hierarchical Data Format library -- library package
 HDF is a multi-object file format for storing and transferring graphical and numerical data mainly used in scientific computing.
 HDF supports several different data models, including multidimensional arrays, raster images, and tables. Each defines a specific
 aggregate data type and provides an API for reading, writing, and organizing the data and metadata. New data models can be added by the
 HDF developers or users.
 
 This package contains the HDF run-time libraries.
 
 Home page: http://www.hdfgroup.com/



If I get it right, for Debian, Emmanuel has a working platform, and Christophe has the segfault. What are your respective package versions ?
Do you also have several packages which can provide libhdf4.

~0000433

chmoisy (reporter)

for HDF4 packages installed :
sdeeph16:~> aptitude show libhdf4g
Paquet : libhdf4g
État: installé
Automatiquement installé: oui
Version : 4.1r4-22
Priorité : optionnel
Section : libs
Responsable : Debian GIS Project <pkg-grass-devel@lists.alioth.debian.org>
Taille décompressée : 795k
Dépend: libc6 (>= 2.7-1), libjpeg62, zlib1g (>= 1:1.1.4)
Suggère: libhdf4g-doc, libhdf4g-dev, hdf4-tools
Est en conflit: libhdf4 (< 4.0.2-4), libhdf4g-run (< 4.1r4-20)
Remplace: libhdf4
Description : The Hierarchical Data Format library -- library package
 HDF is a multi-object file format for storing and transferring graphical and numerical data
 mainly used in scientific computing. HDF supports several different data models, including
 multidimensional arrays, raster images, and tables. Each defines a specific aggregate data type
 and provides an API for reading, writing, and organizing the data and metadata. New data models can
 be added by the HDF developers or users.
 
 This package contains the HDF run-time libraries.
 
 Home page: http://www.hdfgroup.com/

sdeeph16:~> dpkg -L libhdf4g
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libhdf4g
/usr/share/doc/libhdf4g/copyright
/usr/share/doc/libhdf4g/changelog.Debian.gz
/usr/share/man
/usr/share/man/man5
/usr/share/man/man5/hdf.5.gz
/usr/lib
/usr/lib/libmfhdf.so.4.1r4
/usr/lib/libdf.so.4.1r4
/usr/lib/libmfhdf.so.4
/usr/lib/libdf.so.4

sdeeph16:~> dpkg -l | grep hdf
ii libhdf4-0-alt 4.2r4-6 The Hierarchical Data Format library -- library package
ii libhdf4g 4.1r4-22 The Hierarchical Data Format library -- library package
ii libhdf4g-dev 4.1r4-22 The Hierarchical Data Format library -- development package

~0000434

chmoisy (reporter)

>gdal_translate -of GTiff HDF4_EOS:EOS_GRID:"MOD13Q1.A2010001.h17v05.005.2010028003734.hdf":MODIS_Grid_16DAY_250m_500m_VI:250m mod13q1_translate.tif
Input file size is 0, 1
Input file has no bands, and so cannot be translated.

~0000435

julienm (developer)

Since you have both versions installed, can you also check the dependencies of the gdal library ? Does it depend on libhdf4-0-alt or libhdf4g ?
If you compiled gdal 1.8 yourself, I guess you use libhdf4g-dev and so depend on libhdf4g, which would be the same situation than on our problematic platform.

~0000436

chmoisy (reporter)

yes, in the configuration log of GDAL I see an include dir corresponding to libhdf4g-dev package (/usr/include/hdf)

~0000437

julienm (developer)

Last edited: 2011-01-19 16:50

OK thanks, so this might be because of this libhdf4g not being compatible with GDAL.
Can you find a libhdf4-0-alt-dev package and recompile gdal with it (i have such a package on my system, but it does not seem to exist on old ubuntu versions) ?

A good pointer talking about this issue :
http://trac.osgeo.org/gdal/wiki/HDF

~0000438

christop (administrator)

I also have the gdal error (debian, gdal 1.7.3 compiled myself):
$ gdal_translate -of GTiff HDF4_EOS:EOS_GRID:"MOD13Q1.A2010001.h17v05.005.2010028003734.hdf":MODIS_Grid_16DAY_250m_500m_VI:250m mod13q1_translate.tif
Input file size is 0, 1
Input file has no bands, and so cannot be translated.

This is the gdal used by otb, so the error might not be linked (OTB/Monteverdi read the image correctly).

Few more info:
ldd /usr/local/bin/gdal_translate | grep -i hdf
    libhdf5.so.6 => /usr/lib/libhdf5.so.6 (0x00007fb3c5733000)
    libmfhdfalt.so.0 => /usr/lib/libmfhdfalt.so.0 (0x00007fb3c5511000)
    libhdf5_hl.so.6 => /usr/lib/libhdf5_hl.so.6 (0x00007fb3c251a000)

ldd ~/OTB/Monteverdi-Binary-Debug/bin/monteverdi | grep -i hdf
    libhdf5.so.6 => /usr/lib/libhdf5.so.6 (0x00007f8c0b34b000)
    libmfhdfalt.so.0 => /usr/lib/libmfhdfalt.so.0 (0x00007f8c0b128000)
    libhdf5_hl.so.6 => /usr/lib/libhdf5_hl.so.6 (0x00007f8c062d6000)

I do have the libhdf4-0-alt and libhdf4-alt-dev installed.
Note that I also have libhdf4-0 but NOT libhdf4-dev

I installed gdal as in the FAQ (Debian Linux / Ubuntu specific instructions http://www.orfeo-toolbox.org/FAQ/)
./configure --prefix=INSTALL_DIR --with-png=internal --with-libtiff=internal
    --with-jpeg=internal --with-geotiff=internal
make
sudo make install
sudo cp frmts/gtiff/libgeotiff/*.h INSTALL_DIR/include/
sudo cp frmts/gtiff/libgeotiff/*.inc INSTALL_DIR/include/
sudo cp frmts/gtiff/libtiff/*.h INSTALL_DIR/include/

~0000439

julienm (developer)

The libhdf4-alt-dev is not available in Debian stable and Ubuntu < 10.04 :

http://packages.debian.org/search?keywords=libhdf4-alt-dev

http://packages.ubuntu.com/search?keywords=libhdf4-alt-dev

~0000441

chmoisy (reporter)

Here is the result fro dd
sdeeph16:~> ldd /usr/local/bin/gdal_translate | grep hdf
    libhdf5-1.6.6.so.0 => /usr/lib/libhdf5-1.6.6.so.0 (0x00007f83d4ce8000)
    libmfhdf.so.4 => /usr/lib/libmfhdf.so.4 (0x00007f83d4ac1000)


sdeeph16:~> dpkg -L libhdf4g | grep libmfhdf.so.4
/usr/lib/libmfhdf.so.4.1r4
/usr/lib/libmfhdf.so.4


I have installed libhdf4-alt-dev and recompiled gdal with these options :
 ./configure --enable-debug --with-ecw=/usr/local --with-threads --prefix=/usr/local/bin --with-png=internal --with-libtiff=internal --with-jpeg=internal --with-geotiff=internal

SimpleViewer is working now with MOD13Q1 product
but gdal_translate returns the same error (no band in dataset)

~0000442

chmoisy (reporter)

is there a method to access dataset Scale and Offset ?
with gdal, the GDALds->GetRasterBand(1)->GetScale() and GetOffset()
it seems that this can't be used as no raster is found in the dataset

~0000443

christop (administrator)

To sum up the conclusion:
- whether or not gdal_translate works is irrelevant
- libhdf4-alt-dev is necessary for the hdf of OTB to work on Debian/Ubuntu

is that correct?

~0000444

julienm (developer)

Last edited: 2011-01-21 12:01

The bug in gdal_translate that Christophe do not reproduce is still relevant I think. There was just something missing in the dataset name on the command line. Without proper fix I also get "Input file size is 0, 1"

On the faulty platform (gdal 1.8 built with libhdf4g-dev installed), we used :

gdal_translate -of GTiff -mo "HDF4_EOS:EOS_GRID:\"/the/path/to/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf\":MODIS_Grid_16DAY_250m_500m_VI:250m 16 days relative azimuth angle" -sds /the/path/to/MOD13Q1.A2010001.h17v05.005.2010028003734.hdf out.tif

(the dataset name contains spaces, and note the "-sds" flag)

This leads to a segfault and proves that the OTB code is not involved.

What I think is necessary is that libhdf4-alt-dev is installed when gdal is built, and this is not possible on not-recent-enough systems.

~0000445

chmoisy (reporter)

On debian : I have to use unstable source for sid (I have Lenny) to get libhdf4-alt-dev, I don't know if there is a backport on lenny for this package.

what is strange (but I haven't checked the source code) is that gdal_translate dans gdalinfo do not take the same subdataset name format.

for ldd, we can directly check the gdal library : ldd /usr/local/lib/libgdal.so for the library dependencies

~0000470

mickael (administrator)

Some details about crashs with HDF files :
HDF5 seems to be readable with ImageFileReader (via GDAL)
HDF4 seems exhibit some weird case linked to the version of the libhdf4g used (dev vs alt-dev).
For example with a gdal 1.8.0 built with libhdf4g-dev :

* I can create a hdf4 file from otb_logo.tif with this command : gdal_translate -of HDF4Image -co "RANK=2" otb_logo.tif otb_logo.he4 . Moreover this file can be open by Monteverdi without segfault.
gdalinfo command return this output with this file :
Driver: HDF4/Hierarchical Data Format Release 4
Files: otb_logo.he4
Size is 512, 512
Coordinate System is `'
Metadata:
  Signature=Created with GDAL (http://www.remotesensing.org/gdal/)
  TransformationMatrix=0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
  Comment=Created with The GIMP
Subdatasets:
  SUBDATASET_1_NAME=HDF4_SDS:GDAL_HDF4:"otb_logo.he4":0
  SUBDATASET_1_DESC=[115x150] Band0 (8-bit unsigned integer)
  SUBDATASET_2_NAME=HDF4_SDS:GDAL_HDF4:"otb_logo.he4":1
  SUBDATASET_2_DESC=[115x150] Band1 (8-bit unsigned integer)
  SUBDATASET_3_NAME=HDF4_SDS:GDAL_HDF4:"otb_logo.he4":2
  SUBDATASET_3_DESC=[115x150] Band2 (8-bit unsigned integer)
  SUBDATASET_4_NAME=HDF4_SDS:GDAL_HDF4:"otb_logo.he4":3
  SUBDATASET_4_DESC=[115x150] Band3 (8-bit unsigned integer)
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 512.0)
Upper Right ( 512.0, 0.0)
Lower Right ( 512.0, 512.0)
Center ( 256.0, 256.0)
So everything seems to be ok with HDF4_SDS file.

* If I used a MODIS File (in OTB_DATA, renamed in the following fileHDF4.hdf) I cannot open it in Monteverdi (SEGFAULT). When we try to convert all datasets included in this file with gdal_translate to different TIFF file ( gdal_translate -of GTiff -sds fileHDF4.hdf fileHDF4.tif ), we get this output :
Input file size is 4800, 4800
0ERROR 1: GDreadtile() failed for block.
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 which indicate that this version of GDAL have some problems with HDF4_EOS files (like MODIS).

To resume :
* Everything seems OK to read HDF5 files with OTB or Monteverdi.
* Case of HDF4 seems more difficult and its linked to the version of HDF4 library selected to built GDAL (libhdf4g-dev => KO only with HDF4_EOS files and libhdf4g-alt-dev => OK).

~0000476

mickael (administrator)

Hello,

You can see since one week some warning messages about HDF on the dashboard.
These warnings are linked to:
+ the fact that you have build or not GDAL with HDF library.
+ the fact that your platform exhibits some problems to read HDF4 file via GDAL (cf. previous note).

So in the first case, all tests about HDF are deactivated but a warning message is launched during the configuration step of the OTB build (and kept on the dashboard).
In the second case, only the test of reading data in the HDF4 file is deactivated (idem a warning message is kept on the dashboard).

If you have pull/update your version of OTB since the 17th February, you have the choice:
+ re-build from scratch your OTB (because the internal variable of CMake related to these tests are persistent)
OR
+ delete these variables in the CMakeCache (if they are present): CHECK_GDAL_BUILDED_WITH_HDF:INTERNAL, CHECK_HDF4OPEN_SYMBOL:INTERNAL, CHECK_HDF4OPEN_SYMBOL_COMPILED:INTERNAL, CHECK_HDF4OPEN_SYMBOL_EXITCODE:INTERNAL. After this you must re-configure, re-generate and re-build your project.
The second solution seems to be the best.

If you have any questions, don't hesitate to send an e-mail.

Mickaël

~0000477

mickael (administrator)

For information :
Now Monteverdi do not produce a segfault when we try to display a HDF4 file. This problem was linked to the generation of the QuickLook by the otb::StreamingShrinkImageFilter where it missed a try catch and an AbortEvent when we try to access to the input data and the reader produce an exception.
A message is now reported in the MsgReporter of Monteverdi.

Mickaël
+Notes

-Issue History
Date Modified Username Field Change
2011-01-06 16:46 chmoisy New Issue
2011-01-06 17:44 christop Note Added: 0000396
2011-01-06 17:44 christop Status new => feedback
2011-01-06 18:41 julienm Note Added: 0000400
2011-01-07 06:44 christop Note Added: 0000403
2011-01-07 11:00 chmoisy Note Added: 0000404
2011-01-09 07:21 christop Note Added: 0000410
2011-01-10 09:29 chmoisy Note Added: 0000412
2011-01-10 09:38 christop Note Added: 0000414
2011-01-10 09:51 chmoisy Note Added: 0000415
2011-01-16 07:55 christop Note Added: 0000416
2011-01-17 15:44 chmoisy Note Added: 0000417
2011-01-17 18:52 christop Note Added: 0000418
2011-01-18 08:07 chmoisy Note Added: 0000419
2011-01-18 08:21 julienm Note Added: 0000420
2011-01-18 08:36 christop Note Added: 0000421
2011-01-18 10:44 chmoisy Note Added: 0000422
2011-01-18 10:45 chmoisy Note Added: 0000423
2011-01-18 14:24 chmoisy Note Added: 0000424
2011-01-18 17:52 christop Note Added: 0000425
2011-01-18 19:29 julienm Note Added: 0000426
2011-01-19 06:41 christop Note Added: 0000427
2011-01-19 11:52 mickael Note Added: 0000428
2011-01-19 13:46 chmoisy Note Added: 0000429
2011-01-19 14:31 julienm Note Added: 0000430
2011-01-19 14:36 mickael Note Edited: 0000430
2011-01-19 15:11 julienm Note Added: 0000432
2011-01-19 15:47 chmoisy Note Added: 0000433
2011-01-19 15:47 chmoisy Note Added: 0000434
2011-01-19 16:00 julienm Note Added: 0000435
2011-01-19 16:24 chmoisy Note Added: 0000436
2011-01-19 16:46 julienm Note Added: 0000437
2011-01-19 16:50 julienm Note Edited: 0000437
2011-01-20 07:15 christop Note Added: 0000438
2011-01-20 10:05 julienm Note Added: 0000439
2011-01-20 15:39 chmoisy Note Added: 0000441
2011-01-20 16:30 chmoisy Note Added: 0000442
2011-01-20 17:59 christop Note Added: 0000443
2011-01-20 18:40 julienm Note Added: 0000444
2011-01-21 09:31 chmoisy Note Added: 0000445
2011-01-21 12:01 mickael Note Edited: 0000444
2011-02-15 17:34 mickael Note Added: 0000470
2011-02-25 10:01 mickael Note Added: 0000476
2011-02-25 10:08 mickael Note Added: 0000477
2011-02-25 10:12 mickael Status feedback => resolved
2011-02-25 10:12 mickael Resolution open => fixed
2011-02-25 10:12 mickael Assigned To => mickael
2011-05-24 14:11 julienm Relationship added related to 0000312
2011-11-18 11:39 C Valladeau Status resolved => closed
+Issue History