You can control the vectorizer triangulation count for shells using the following code:
ModelerModulePtr pM = odrxDynamicLinker()->loadModule(OdModelerGeometryModuleName, false);
wrTriangulationParams TriangulationParams;
TriangulationParams.normalTolerance = 0.5;
TriangulationParams.PointsPerEdge = 1000;
TriangulationParams.BetweenKnots = 100;
TEST_ASSERT(eOk == pM->setTriangulationParams(TriangulationParams));
Also you can check the OdaMfcApp sample: Tools->Options->Triangulation Params.