Unique domains

About Dataprovider Research

Dataprovider.com transforms the internet into the largest search engine of structured web data for data-minded professionals so that they can do detailed research, make better decisions, measure the digital economy and fight cybercrime. Creating the right dataset is difficult due to the hundreds of filters and values that each field contains. These recipes are custom prefiltered datasets that help you get the most complete results.

About this recipe

Every day we recieve the official zone files from many registries. We add the domains from these zone files this to our index to crawl the domains. Unfortunatley some TLDs like country TLDs are kept secret. Because we use a crawler we go from hyperlink to hyperlink to create a map of the internet. By doing this for many years we almost know all active domains, even if they are on country TLDs.

All Dataprovider users get access to the complete database that contains every single hostname we index. This includes domains that exist and contain a website but it also includes the dropped and unresponsive domains as well. We’re calling our database complete because in the past we used to remove unreachable ccTLD domains and certain unresponsive domains from our database.

If you want to have the list with unique domains where we are certain that they are registered and extist in a zone file then you can use this recipe. This recipe filters out all the hostnames where we do not know if they are actually registered. For statistics and counts it is very important to use this recipes because it makes sure that you measure the unique existing domains.