Overview
Starting with Visualize version 23.1, a VSFX file can be streamed in partial view mode to provide a desktop-similar partial viewing technique in streaming-based solutions. This can be used in different cases, for example:
- Render a more informative (compared to linear streaming) incomplete first frame as fast as possible (compared to regular loading).
- Render big files that cannot be rendered fully due to RAM limitations.
Another advantage of this technology is that it does not create a heavy load on the server; reading parts of binary files using offsets and lengths does not require additional services on a server.
Partial Viewing
Partial view technology allows you to load from a file only service objects and keep all entities and geometry in the file. Each update() call generates a list of entities that are visible on the screen, and only these entities are loaded from the file. Before partial viewing can be applied to the file, a saved database must contain partial indexes—;a list of pairs
Linear Streaming
Both regular and partial-file loading require access to the whole file. However, in some cases this is an inconvenient limitation. For example, a cloud solution may provide access to the whole file only by keeping it in memory because of the time required to send the whole file to the server and how it takes up limited memory.
VSFX files support streaming which allows you to load the file using small parts of the original file. It also provides the ability to incompletely render a streamed file. Before a VSFX file can be streamed, it must be converted to a streaming-compatible version since regular writeVSFX() methods create a streaming-incompatible version. However, regular readVSFX() methods work with both streaming-compatible and streaming-incompatible versions. For details, see Work with Visualize Streaming.
Partial View Streaming
Partial view streaming is a combination of both partial viewing and linear streaming techniques. So, a VSFX file must contain a partial index and be converted to a streaming-compatible version.
The main steps are the following:
- Linear streaming starts.
- When all service data is parsed, linear streaming stops.
- Each update() call generates a list of entities that are visible on the screen.
- The database receiver processes this list and generates a list of records that contain the offset and length of parts of the original file, and sends these records to the server side.
- The server process these records and sends the requested parts of the file to the receiver.
- The database receiver parses this data using the streaming technology, so data (similar to a file in linear-streaming mode) can be processed as a list of small pieces and not as the whole binary data array.
OdTvDatabaseReceiverReactor
Compared to linear streaming, partial view streaming requires implementation of the OdTvDatabaseReceiverReactor interface on the client side.
This reactor interface contains the definition of the OdTvRequestRecord structure&mdasha structure that describes the parts of the file that is requested by the database receiver. When the receiver parses all service sections, it notifies the server using the following reactor method:
virtual bool onServicePartReceived( bool bHasPartialIndex ) = 0;
The bHasPartialIndex parameter specifies whether a database has a partial index. After this callback is invoked, the regular OdTvDatabaseReceiver::doReceive call does not parse data; it returns true and switches the database to the kReceiver_RequestMode state. This means that the receiver generates requests and parses responses using the OdTvDatabaseReceiver::doReceiveResponce() method.
The main reactor method is:
virtual void onRequest( OdUInt32 requestID, OdUInt32 nRecCount, const OdTvRequestRecord* pRecords ) = 0;
This callback is called when the receiver has processed the partial viewing list and a request to the file is generated. The requestID parameter defines the identifier of a request and nRecCount is the number of request records in the pRecords array.
Because a VSFX file consists of compressed pages, each request record describes a single page or a few file pages, not an entity. Several entities can be on the same page, so they all can be requested by a single record. On the other hand, the receiver provides only quick record-request combinations (for performance reasons), so the receiver also may retrieve the same binary data a few times for different entities during the same request. This is why the sum of a record's length may be greater than the file itself, and the number of records does not provide any information about the count of requested entities.
After receiving the callback, the server reads the requested parts of the file and sends them to the receiver using the following method:
virtual bool doReceiveResponce( OdUInt32 requestID, const OdUInt8* pData, OdUInt32 nSize, OdTvResult* rc = NULL ) = 0;
where:
- requestID - Identifier of the request from the callback.
- pData - Part of the read file.
- nSize - Size of the part.
Note: The doReceiveResponce() method supports the streaming technique, so it can be called several times for the same requestID, providing parts of requested data in linear order.
When the request is completely parsed, the receiver calls the doReceiveResponce() method:
virtual bool doReceiveResponce( OdUInt32 requestID, const OdUInt8* pData, OdUInt32 nSize, OdTvResult* rc = NULL ) = 0;
However, if the request is not completely parsed and the view settings change and a new update() call occurs, the receiver generates a new request, but first the receiver needs to cancel the current request using the following callback:
virtual void onRequestAborted( OdUInt32 requestID ) = 0;
Workflow
Client code should create an instance of the OdTvDatabaseReceiver class:
OdTvFactoryId factId = odTvGetFactory();
OdTvDatabaseReceiverPtr pDbReceiver = factId.createDatabaseReceiver();
Then, switch the receiver to partial mode using a client implementation of the OdTvDatabaseReceiverReactor interface:
m_pDbReceiver->enablePartialMode( true, pReactor );
Linear streaming starts until doReceive() returns true and the receiver state becomes kReceiver_RequestMode:
bool bDone = pDbReceiver->doReceive( pData, nDataSize);
OdTvDatabaseReceiver::DatabaseReceivingState state = pDbReceiver->state();
if( bDone && state != OdTvDatabaseReceiver::kReceiver_RequestMode ) throw OD_T(“Internal error” );
if( bDone ) break; //stop linear streaming, start partial-viewing streaming
When this happens, a client implementation of OdTvDatabaseReceiverReactor also receives an onServicePartReceived() notification.
The server waits, and when OdTvDatabaseReceiverReactor::onRequest() occurs, the server sends a response for the request:
if( !m_pDbReceiver->doReceiveResponce(requestID, binData.asArrayPtr(), binData.size() ) )
throw OD_T(“Response not parsed” );
Additional Management
Partial view streaming supports additional features, which are managed using the OdTvDatabaseReceiverManagementOptions class and OdTvDatabaseReceiver method:
virtual void setManagementOptions( const OdTvDatabaseReceiverManagementOptions& options ) = 0;
Limit Manager
The limit manager for partial view streaming is helpful in the following cases:
- Similar to regular partial viewing, the receiver is used before loading each requested entity; if a limit is reached, the receiver does not load subsequent entities.
- The receiver can use the limit manager when generating requests: it estimates the memory required for entity loading using entity length in the file. If the limit manager indicates that the estimate is beyond the limit, the receiver does not include these entities in the request. For this purpose, the receiver uses the OdTvDatabaseReceiverManagementOptions::memoryEstimationMultiplier() method. However, the actual value of this multiplier is purely heuristic and can vary for different files. The receiver statistic that is described later in this topic can help define this value.
An instance of OdTvLimitManager can be specified using the following factory method:
void setLimitManager( OdTvLimitManager* pManager );
Unload Objects
Similar to regular partial viewing, partial view streaming supports unloading objects that were loaded but currently off-screen. By default, if unloading is enabled, all off-screen objects become unloaded. However, since this operation decreases rendering performance, a scheduler limit can be specified. In this case, the receiver spends no more time for unloading during each update() call than the specified unloadTimeLimit parameter. If it does not have time to unload all objects, the remaining objects are unloaded during the next update() call.
Object unloading is managed by the following OdTvDatabaseReceiverManagementOptions method:
void setObjectsUnloading( bool bUnload, OdUInt32 unloadTimeLimit = 0 );
where:
- bUnload — Flag that enables unloading.
- unloadTimeLimit — Time limit for unloading (0 means "no limit").
Generate Statistics
Generating statistics can be managed using the following OdTvDatabaseReceiverManagementOptions method:
void setEnableStatistic( bool bEnable, bool bOnlyRequestedItems = true, bool bCountUnloadObjects = false );
where:
- bEnable — Enables gathering of statistics.
- bOnlyRequestedItems — Specifies that only statistics for the request (not for service objects from the linear streaming workflow) are gathered.
- bCountUnloadObjects — Specifies that unloaded objects should be included in the statistics.
If statistics are enabled, the results can be obtained using OdTvDatabaseReceiver::StatisticTool from the following OdTvDatabaseReceiver method:
virtual const StatisticTool* statisticTool() const = 0;
Statistics include information about the number and length (raw, non-compressed) of parsed and unloaded objects.
If the client side supports calculation of consumed memory, it can use this tool on different iterations to estimate a custom value of OdTvDatabaseReceiverManagementOptions::memoryEstimationMultiplier().