Support writing of 32bit integers ( instead of 64bit ) from cg API

Description

Our current status in ANSA dsevelopment is building the cgns lib with 64 bit ( ie CG_BUILD_64 = 1 ), upon our users' request. Of course, we need to be able to import both int32 and int64 files. However, as CFD mesh sizes tend to increase, we are concerned about the ever growing size of the generated files. If billion size meshes are around the corner, this would mean that for a mesh of say 2 billion cells, the file size would exceed 320Gb. So we started seeking ways to force 32 bit integers output, in cases when it can correctly describe the underlying mesh of course.

So, we searched and found out that in CGNS-157, in the comments, Scot is proposing what we need as: "(1) Overload the current APIs to add the memory type of the void *data. We do this for some of the mid-level CGNS APIs, so there are a mechanism and precedent for doing this. It also preserves backward compatibility, but it preserves something fundamentally flawed.". Is also mentioned by Scot, that there exists cgio API for read_data_type and write_data_type from cgnslib 3.4. Is this possible that the same API becomes available for the higher level ( i.e. cg ) API?

Note: we are using no compression - it is disabled by the lib -, because the performance is degraded in case of inaccurate tuning of chunk size, cache size, access pattern etc,

Activity

Show:
Mickael PHILIT
April 29, 2020, 3:30 PM
Edited

Thanks for the interesting and detailed report.

cgio API are well designed and less flawed (For instance ParaView rely only on cgio to parse CGNS file efficiently). Having something similar at the higher level is a bit more complicated since they are a lot of existing checks and hard coded type. Maybe having a function to create an empty section with optional element datatype and elementSize would be possible. Something like:

cg_general_section(int fn, int B, int Z, const char *sectionname, CGNS_ENUMT(ElementType_t) type, cgsize_t start, cgsize_t end, int nbndry, int *S, CGNS_ENUMT(DataType_t) datatype, cgsize_t elementSize)

cg_general_element_data_write(int fn, int B, int Z, int S, cgsize_t start, cgsize_t end, CGNS_ENUMT(DataType_t) mem_data_type, void *Elements, void *ElementsOffset)

First, it enables to have a proper creation of an empty section that will be useful for parallel/partial writing of unstructured meshes.

The second function would be common for structured and unstructured mesh.

Existing function could then be refactored to use those two functions leading to better maintenance. Keeping existing function seems necessary to prevent a huge API breakage on the short term.

Note: For billion cells meshes, one can create multiple sections (remember that sections are just containers and you need Family_t to give topological meaning) or split the volume in multiple zone_t.

Mickael PHILIT
May 7, 2020, 6:34 PM

Check branch

Assignee

Unassigned

Reporter

Anonymous

Fix versions

Priority

Major

Components

Configure