Is it ok that Get takes too long?

[abkein]

I have BP4 storage, in which "atoms" is 9 x 500 000 array of double and Get call to this variable takes about 7-8 seconds. (AMD EPYC 7351P, same task on 32 thread).

Is it ok or I miss something?

The complete code:

        adios2::ADIOS adios = adios2::ADIOS();
        adios2::IO io   = adios.DeclareIO("DATAREADER");
        io.SetEngine("BP4");
        std::unique_ptr<double[]> buffer(new double[ 9 * 500 000 ]);
        adios2::Variable<double>   varAtoms;
        adios2::Engine reader = io.Open(storage, adios2::Mode::Read);
        varAtoms  = io.InquireVariable<double>(std::string("atoms"));
        reader.Get(varAtoms, Atoms_buf.get());

[pnorbert]

Can you please give us more information?

• ls -l file.bp
• bpls -lD file.bp atoms
• what read time do you get from other formats on your machine?
• did you build adios with Release or RelWithDebInfo? Debug mode really slows down the ADIOS internals.

Thank you

[abkein]

>ls -lh dumps/MAIN4
-rw-r--r-- 1 architect0rr users  90G Dec 14 14:05 data.0
-rw-r--r-- 1 architect0rr users 121M Dec 14 14:05 md.0
-rw-r--r-- 1 architect0rr users 167K Dec 14 14:05 md.idx

2. https://sprunge.us/mZAkIt
3. The same amount of data (500 000 x 9 of np.float64) is read from csv-file with pandas in 3 sec.
4. Build type was Release. The whole CMakeCache.txt

[dmitry-ganyushin]

It would be good to compare times not for a csv file, but for a panda's DataFrame converted to a binary file. I guess the pandas library has a lot of read optimizations like multithreading, so maybe 8 sec against 3 is not that bad.