a plurality of network crawlers each including, one or more processors and memory storing one or more modules to be executed by the one or more processors, the one or more modules having instructions for fetching documents from hosts on a network; and

a cookie database shared by the plurality of network crawlers, the cookie database storing cookies and associated information for use by the plurality of network crawlers;

wherein each network crawler of the plurality of network crawlers further includes instructions for retrieving one or more cookies from the cookie database so as to enable access to documents on at least one of the hosts on the network and each of the network crawlers includes instructions for detecting any of a plurality of predefined cookie errors associated with fetching a document by comparing a fetched document with a plurality of predefined cookie error patterns; and

wherein the cookie database includes cookie acquisition information corresponding to each of at least a plurality of the cookies in the cookie database; the cookie acquisition information for a respective cookie enabling a respective network crawler to acquire the cookie from an acquisition URL specified by the cookie acquisition information; wherein the acquisition URL is distinct from a target URL to be accessed using the respective cookie.