Data Sources

It’s an infrastructure for holding the data and streaming the data to the client. It will enable faster page loads as well as the potential for asynchronous server processing of results. The basic strategy is to return large result sets to clients in chunks, along with an ID that can be used to get more chunks. From the UI perspective it looks like the data is streaming into the client. This approach is different from DB paging, where we persist the big result set in the database. With DataSources, the result lives in memory only for as long as it takes the client to request it in chunks. In addition, there will be support for the server to add chunks to the datasource in a separate thread, so one can request the datasource and spin off a thread to work on getting data for it, without the client waiting.

Synchronous: This is useful if you have all the data that need to be send to client with you already and want to put the data in DataSource so that it can be send to the clients in chunks. The DataSource session takes in a list of objects that you want sent to the client and returns you a DataChunk object, which contains the first chunk of data to be send to the client. If the List of data could fit in one chunk, then the session doesn't even create a DataSource, it just returns you the chunk. Otherwise a DataSource is created with the data, which can be polled later for additional chunks. The DataSource will be closed so that no more data can be added.

Asynchronous: In this case, you haven't got your data yet. You want to submit a task to get the data in a background thread and have the data returned to the client later. We leverage the capabilities of Executor framework provided by Java.(for more details refer JavaDoc of Executor) To create an asynchronous DataSource, the DataSource session takes a DataSourceTask which will be run by in a background thread sometime in the future. It returns a DataSourceFuture object, so that you have a way to access the DataSource information.

Asynchronous data sources with Synchronous data acquisition: Uses synchronous data acquisition (all data comes from the database in one chunk) and streams chunks to the client. The first chunk sent to the client contains no table data but enables the table to be rendered before a full data chunk is available. This gives the user control of the table and some of its actions sooner. The data chunks then stream in behind the blank chunk as they are processed.

Asynchronous data sources with Asynchronous data acquisition: Uses asynchronous data acquisition (data is obtained from the database in chunks) and streams chunks to the client. The first chunk sent to the client contains no table data but enables the table to be rendered before a full data chunk is available. This gives the user control of the table and some of its actions sooner. The data chunks then stream in behind the blank chunk as they are processed. In this case, we have the potential to see chunks of data rendered in the client while data is still being acquired from the database.

Parameter	Description
ActiveDataSourceCount	The number of datasources that currently are active in all user sessions
TotalDataSourcesCreated	The total number of DataSources that have been created for all user sessions
TotalDataSourcesDestroyed	The total number of DataSources that have been destroyed for all user sessions

Parameter	Description
cancelLongRunningDataSources	Cancel any datasources that have been running for at least the given amount of milliseconds, in all user sessions. This will cancel both active and inactive DataSources

Parameter	Description
AllEmptyPollingTimeout	The timeout in milliseconds to wait for data chunks when the client polls for chunks and none of the ids it asked for have chunks available
ChunkBlockingLimit	Controls the number of chunks a DataSource can hold before it blocks the addition of more items until something polls it or it times out. The actual number of items this resolves to is ChunkBlockingLimit*MaxChunkSize.
DataSourceTaskPoolKeepAliveTime	The amount of time in milliseconds to keep idle threads alive in the DataSource task pool
DataSourceTaskPoolSize	The maximum number of threads allowed in the DataSource task pool
DataSourceTimeout	The amount of time in milliseconds to keep idle DataSources alive
FeedbackBlockingLimit	The number of feedback items a DataSource can hold before it blocks the addition of more feedback until something polls it or it times out
InitialChunkPollingTimeout	The amount of time in milliseconds to wait to for the initial chunk of data to be available, when using asynchronous datasources. This enables components to try and display some data in the table on initial rendering if possible
MaxPollSize	The maximum size of data that is returned by a DataSource for each poll.
PreferredMinPollSize	The minimum size of data for a poll. This is used when polling asynchronous datasources, to allow the server to decide whether to wait for more data or not
PreferredSerializationSize	The preferred size in bytes of the total response to a polling request. If the initial chunks do not reach this size limit, then additional chunks are added until they are over the limit. This property is not currently used.
ResultLimit	The total number of items that can be added to a DataSource before it fails.
SomeEmptyPollingTimeout	The timeout in milliseconds to wait for data chunks when the client polls for chunks and some of the ids it asked for have chunks available, but not all of them
TreeChildNodeFetchTimeout	The timeout in miliseconds to wait for child data for the tree. The OOTB value is 180000 ms and can be increased to avoid time out errors.

In addition to the MBean properties to adjust different sizes and timeouts, the property numberOfThreadsPerComponent can have an effect on overall performance of a component. This property is used to modify the maximum number of threads allowed by any component to process data out of the database through the Data Utilities. These threads process data of size of the RawDataChunkSize MBean value and are short lived. Increasing the number will not likely help further because the data coming from the database will not be fast enough to need more threads. In most of the cases, the maximum count of threads running concurrently will be two. Given below is an example of the property showing default value: com.ptc.core.components.factory.dataUtilities.numberOfThreadsPerComponent=3

For WAN clients, the maxPollSize attribute might work a little better set to a higher number. LAN clients will work faster with a smaller value because there is a smaller latency penalty for making requests. A balanced value of 500 should work well in both scenarios. The client browser should be using multiple Ajax requests to download the data. This has been limited to three concurrent requests as more requests do not improve download speeds because the data is not available any faster. This property is not configurable (PTC.jca.DataSourceRegistry.MAX_REQUESTS) at this point.