The index (i.e just the list of urls) of common crawl is 250gb. The actual crawl is 500tb ish. Then pull out just the text you get down to 30tb ish. And then you need that all to be in a fts index… (something like opensearch/elasticsearch) idk how big the cluster you would need for that is… maybe 2tb per data node max. So 15 pretty beefy vms (8cpu w/64gb memory) with 2tb nvme for data and a few more as controller nodes
The index (i.e just the list of urls) of common crawl is 250gb. The actual crawl is 500tb ish. Then pull out just the text you get down to 30tb ish. And then you need that all to be in a fts index… (something like opensearch/elasticsearch) idk how big the cluster you would need for that is… maybe 2tb per data node max. So 15 pretty beefy vms (8cpu w/64gb memory) with 2tb nvme for data and a few more as controller nodes
Common crawl numbers pulled from this paper
https://aclanthology.org/anthology-files/anthology-files/pdf/L/L18/L18-1286.pdf
Cluster sizing numbers pulled from tea leaf divination