US 9,811,524 B2
Distributed data set storage and retrieval
Brian Payton Bowman, Apex, NC (US); Steven E. Krueger, Raleigh, NC (US); Richard Todd Knight, Raleigh, NC (US); and Chih-Wei Ho, Cary, NC (US)
Assigned to SAS Institute Inc., Cary, NC (US)
Filed by SAS Institute Inc., Cary, NC (US)
Filed on Jul. 26, 2016, as Appl. No. 15/220,182.
Application 15/220,182 is a continuation of application No. 15/220,034, filed on Jul. 26, 2016, granted, now 9,619,148.
Claims priority of provisional application 62/197,519, filed on Jul. 27, 2015.
Claims priority of provisional application 62/197,514, filed on Jul. 27, 2015.
Prior Publication US 2017/0031936 A1, Feb. 2, 2017
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 12/00 (2006.01); G06F 13/00 (2006.01); G06F 13/28 (2006.01); G06F 17/30 (2006.01); G06F 9/50 (2006.01); G06F 3/06 (2006.01); G06F 12/02 (2006.01)
CPC G06F 17/30097 (2013.01) [G06F 3/061 (2013.01); G06F 3/0604 (2013.01); G06F 3/064 (2013.01); G06F 3/067 (2013.01); G06F 3/0643 (2013.01); G06F 3/0644 (2013.01); G06F 9/5072 (2013.01); G06F 9/5077 (2013.01); G06F 12/0292 (2013.01); G06F 17/302 (2013.01); G06F 17/30312 (2013.01); G06F 17/30584 (2013.01); G06F 2212/1016 (2013.01); G06F 2212/1056 (2013.01); G06F 2212/154 (2013.01); G06F 2212/262 (2013.01); G06F 2212/263 (2013.01)] 30 Claims
OG exemplary drawing
 
1. An apparatus comprising a processor component and a storage to store instructions that, when executed by the processor component, cause the processor component to perform operations comprising:
provide, to a control device, an indication of being currently available to participate in a performance of a processing task as a node device among multiple node devices;
receive, from the control device, an indication of the processing task to perform with one or more data set portions of multiple data set portions of a data set, wherein the data set comprises data organized in a manner indicated in metadata;
perform the processing task with the one or more data set portions;
provide a request to the control device for a pointer to a location at which to store the one or more data set portions as a data block of multiple data blocks within a data file maintained by one or more storage devices, wherein:
the multiple data blocks are organized within the data file in a manner indicated in map data that comprises multiple map entries; and
each map entry of the multiple map entries corresponds to one or more data blocks of the multiple data blocks;
analyze the metadata to determine whether the metadata indicates that the data set is partitioned;
in response to an indication in the metadata that the data set comprises partitioned data, wherein the data within the data set is organized into multiple partitions that are each distributable to just a single node device, each map entry corresponds to a single data block and the processing task is able to be performed with the data within each partition independently of the data within any other partition of the multiple partitions, the processor component is caused to perform operations comprising:
for each data set portion of the one or more data set portions:
include a data sub-block size indicative of a size of the data set portion in the request;
derive a hashed identifier of a partition label of the partition to which the data set portion belongs of the multiple partitions; and
include the hashed identifier in the request;
receive, from the control device, the requested pointer indicating the location within the data file at which to store the data block; and
after the performance of the processing task with the one or more data set portions, store each data set portion of the one or more data set portions as a data sub-block within the data block starting at the location within the data file; and
in response to a lack of indication in the metadata that the data set comprises partitioned data, the processor component is caused to perform operations comprising:
derive a sum of sizes each data set portion of the one or more data set portions;
include the sum of sizes as a data block size of the data block in the request;
receive, from the control device, the requested pointer indicating the location within the data file at which to store the data block; and
after the performance of the processing task with the one or more data set portions, store the one or more data set portions together as the data block at the location within the data file.