Adam Riley

A workflow includes a sequence of tools and a logic controller. A data file is input into the workflow and processed by the tools. A log file and a metadata file are generated for a tool. The log file includes data describing events that occurred during the operation of the tool in the processing of the data file. The metadata file includes data generated from the performance of the data analytics function of the tool. The logic controller uses the log file to determine whether to trigger a… Show more

A workflow includes a sequence of tools and a logic controller. A data file is input into the workflow and processed by the tools. A log file and a metadata file are generated for a tool. The log file includes data describing events that occurred during the operation of the tool in the processing of the data file. The metadata file includes data generated from the performance of the data analytics function of the tool. The logic controller uses the log file to determine whether to trigger a subsequent tool. Responsive to a determination of trigging the subsequent tool, the metadata file is sent to the subsequent tool and processed by a subset of the sequence of tools to generate a result item. The subset starts with the subsequent tool. The result item represents an outcome of the processing of the data file by the workflow. Show less

A module manages usage of a memory in a computer by scheduling data processing tasks. The module determines whether a memory associated with a computer is available for performing a data processing task. In response to determining that the memory is not available for performing the data processing task, the module generates a factory object for the data processing task. The factory object specifies instructions to produce the data processing task. After the factory object is generated, the… Show more

A module manages usage of a memory in a computer by scheduling data processing tasks. The module determines whether a memory associated with a computer is available for performing a data processing task. In response to determining that the memory is not available for performing the data processing task, the module generates a factory object for the data processing task. The factory object specifies instructions to produce the data processing task. After the factory object is generated, the module receives a request to perform the data processing task. The module produces the data processing task by using the factory object in response to the request and determine again whether the memory is available for performing the data processing task. In response to determining that the memory is now available for performing the data processing task, the module performs the data processing task by using the memory. Show less

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed… Show more

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed state exceeding a release threshold, the module releases at least some of the memory blocks in the ready and compressed state. In response to primary memory usage by the memory blocks in the compressed state exceeding a compressed threshold, the module transfers at least some memory blocks in the compressed state to a secondary memory associated with the computer. Show less

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed… Show more

A module manages memory in a computer. The module monitors usage of a primary memory associated with the computer. The primary memory stores memory blocks in a ready state. In response to primary memory usage by the memory blocks in the ready state exceeding a ready state threshold, the module compresses at least some of the memory blocks in the ready state to form memory blocks in a ready and compressed state. In response to primary memory usage by the memory blocks in the ready and compressed state exceeding a release threshold, the module releases at least some of the memory blocks in the ready and compressed state. In response to primary memory usage by the memory blocks in the compressed state exceeding a compressed threshold, the module transfers at least some memory blocks in the compressed state to a secondary memory associated with the computer. Show less

A data analytics system stores a plurality of sequential data blocks in a block cache. Each data block contains one or more data records. Each data record of a data record contains one or more values and a record marker marking the end of the data record. The data analytics system allocates a data block in the block cache to a worker thread. The worker thread parses the data block to identify the one or more data records contained within the data block. The data analytics system determines… Show more

A data analytics system stores a plurality of sequential data blocks in a block cache. Each data block contains one or more data records. Each data record of a data record contains one or more values and a record marker marking the end of the data record. The data analytics system allocates a data block in the block cache to a worker thread. The worker thread parses the data block to identify the one or more data records contained within the data block. The data analytics system determines whether the data block was correctly parsed. Responsive to determining that the data block was correctly parsed, the data analytics system provides the one or more data records contained within the data block to a data analytics workflow. Show less

Data records are joined using a computer. Data records in a first plurality of data records and a second plurality of data records are hashed. The data records in the first and second pluralities are respectively assigned to first and second groupings based on the hashes. Associated pairs of groupings from the first and second groupings are provided to a thread executing on a computer processor, and different pairs are provided to different threads. The threads operate on the pairs of groupings… Show more

Data records are joined using a computer. Data records in a first plurality of data records and a second plurality of data records are hashed. The data records in the first and second pluralities are respectively assigned to first and second groupings based on the hashes. Associated pairs of groupings from the first and second groupings are provided to a thread executing on a computer processor, and different pairs are provided to different threads. The threads operate on the pairs of groupings in parallel to determine whether to join the records in the groupings. A thread joins two data records under consideration if the hashes associated with the data records match. The joined data records are output. Show less