Data sources define interfaces for access and management of data in raw format and out-of-memory data. A data source is closely coupled with the data dictionary that describes the structure of the data associated with the data source. To create the associated data dictionary, you can do one of the following:
- While constructing a data source object, specify whether it should automatically create and initialize the associated data dictionary.
- Call the createDictionaryFromContext() method.
The getDictionary() method returns the dictionary associated with the data source.
Data sources stream and transform raw out-of-memory data into numeric in-memory data accessible through numeric table interfaces. A data source is associated with the corresponding numeric table. To allocate the associated numeric table, you can do one of the following:
- While constructing a data source object, specify whether it should automatically allocate the numeric table.
- Call the allocateNumericTable() method.
The getNumericTable() method returns the numeric table associated with the data source.
To retrieve the number of columns (features) in a raw data set, use the getNumberOfColumns() method. To retrieve the number of rows (observations) available in a raw data set, use the getNumberOfAvailableRows() method. The getStatus() method returns the current status of the data source:
-
readyForLoad - the data is available for the load operation.
-
waitingForData - the data source is waiting for new data to arrive later; designated for data sources that deal with asynchronous data streaming, that is, the data arriving in blocks at different points in time.
-
endOfData- all the data is already loaded.
Because the entire out-of-memory data set may fail to fit into memory, as well as for performance reasons, Intel DAAL implements data loading in blocks. Use the loadDataBlock() method to load the next block of data into the numeric table. This method enables you to load a data block into an internally allocated numeric table or into the provided numeric table. In both cases, you can specify the number of rows or not. The method also recalculates basic statistics associated with this numeric table.
Intel DAAL maintains the list of possible values associated with categorical features to convert them into a numeric form. In this list, a new index is assigned to each new value found in the raw data set. You can get the list of possible values from the possibleValues collection associated with the corresponding feature in the data source. In the case you have several data sets with same data structure and you want to use continuous indexing, do the following:
-
Retrieve the data dictionary from the last data source using the getDictionary() method.
-
Assign this dictionary to the next data source using the setDictionary() method.
-
Repeat these steps for each next data source.
Intel DAAL
implements classes for some popular types of data sources. Each of these
classes takes a feature manager class as the class template parameter. The
feature manager parses, filters, and normalizes the data and converts it into a
numeric format. The following are the data sources and the corresponding
feature manager classes:
- Text file (FileDataSource class), to be used with the CSVFeatureManager class
- ODBC (ODBCDataSource class), to be used with the MySQLFeatureManager class
- In-memory text (StringDataSource class), to be used with the CSVFeatureManager class