US 11,809,395 B1
Load balancing, failover, and reliable delivery of data in a data intake and query system
Jeff Fan, Vancouver (CA); Daniel Ferstay, Vancouver (CA); and Denis Vergnes, North Vancouver (CA)
Assigned to Splunk Inc., San Francisco, CA (US)
Filed by Splunk Inc., San Francisco, CA (US)
Filed on Jul. 30, 2021, as Appl. No. 17/444,173.
Claims priority of provisional application 63/203,283, filed on Jul. 15, 2021.
Int. Cl. G06F 16/22 (2019.01)
CPC G06F 16/2228 (2019.01) 23 Claims
OG exemplary drawing
 
1. A method comprising:
receiving at an intermediary system, from an intake system, a start block of unparsed data associated with a group of related blocks from a first data source of a plurality of data sources, wherein the first data source is associated with a first topic of a plurality of topics;
obtaining a partition key for the first topic;
obtaining workload information regarding workloads of individual indexing systems of a plurality of indexing systems, the workload information including associations between individual topics of the plurality of topics and the individual indexing systems;
selecting a first indexing system of the plurality of indexing systems to associate with the partition key;
transmitting, from the intermediary system, the start block of unparsed data to the first indexing system;
receiving at the intermediary system, from the intake system subsequent to transmitting the start block, an end block of unparsed data associated with the group of related blocks from the first data source, wherein the intermediary system is configured to transmit the end block to the first indexing system when the first indexing system is determined to be available;
determining that the first indexing system associated with the partition key is unavailable to receive the end block of unparsed data; and
responsive to determining that the first indexing system associated with the partition key is unavailable to receive the end block of unparsed data, transmitting from the intermediary system to a second indexing system, selected for association with the partition key, the group of related blocks, including both the end block of unparsed data that the first indexing system is determined to be unavailable to receive and the start block of unparsed data previously transmitted from the intermediary system to the first indexing system.