Culture

Business websites

.com

Brand protection

Online stores

Hosting

Data privacy

Policy-making

Registrars

Investing & reporting

Internet Basics Series

How to

People

Cybersecurity

Web trends

Tech news

Security

COVID-19

Technology detection

domains

Software

WordPress

TLDs

GDPR

Mathijs Baas

As humans we are much better at remembering names, like dataprovider.com or wikipedia.org than remembering a sequence of numbers. Computer algorithms, however, are much better at processing numbers. When it comes to communication across the internet, each connected device, including the servers that host websites, has a unique numerical identifier called an IP address. So when you put dataprovider.com into your browser's address bar, it needs to be translated into a number. That’s where the DNS comes into play. It is often likened to a massive, ever up-to-date phone book, that translates the domain names into computer-friendly IP addresses.

The DNS relies on a distributed, hierarchical system of servers, each with a specific function in the process of resolving domain names:

Root servers: They are at the base of the DNS hierarchy. Root servers provide the initial starting point for DNS queries, directing them towards the appropriate top-level domain (TLD) servers.

TLD servers: These servers manage collections of domains organized by their TLDs, such as, .com, .uk, .info. In a second step, they offer a more precise direction for the query, narrowing down the search towards the correct authoritative server.

Authoritative servers:  These servers store the DNS records that definitively map domain names to IP addresses. Typically managed by domain owners or their web hosting providers, they provide the final answer to any DNS query.

Recursive resolvers: The workhorses of the DNS. Usually configured on a local device or provided by an internet service provider, they initiate the lookup process, diligently contacting other DNS servers and caching the results for future efficiency.

In Figure 1 you can see the process for the domain name www.abc.com. First, the root servers provide the information for the location of the TLD servers, in this case .com is located at 1.1.1.1. In a second query the .com TLD server is now queried for the location of abc.com which returns the address of the authoritative server, here 2.2.2.2. Lastly, the authoritative server is queried and returns the full address of www.abc.com.

Understanding the difference between authoritative and recursive resolvers is crucial for understanding the workings of DNS. Let's break down their roles.

Think of authoritative servers as holding the master key for specific domains. They store the official DNS records, providing the final, definitive answer when someone ask for a website's address. Because they are managed by domain owners themselves, they ensure traffic is directed to the genuine, real location of the website. But authoritative servers are more than simple address books. They store various DNS records that serve different purposes:

A Record: The cornerstone – this maps a domain name to its IPv4 address (example: www.xyz.com = 231.345.378.39).

AAAA Record: Like an A Record, but for the newer IPv6 addresses.

MX Record: Crucial for e-mail delivery. It indicates which mail servers handle incoming emails for a specific domain.

CNAME Record: Creating aliases – allows multiple domain names to ultimately point to the same IP address. For example, multiple subdomains such as team.abc.com or shop.abc.com will have a CNAME record that points to the root domain abc.com.

TXT Record: Flexible and versatile, used for things like domain ownership verification, security mechanisms, and other custom data.

Recursive resolvers are usually operated by an internet service provider or public services. Their purpose is to speed-up the lookup process. To do so they track down the answer to the DNS query and then store this information temporarily in a cache of recent lookups, making repeat visits to websites much quicker.

A key feature that optimizes DNS performance is DNS caching. DNS caching shortens the look-up process and makes internet browsing quicker and more efficient. It temporarily stores recent DNS query results, enabling faster access to frequently visited websites. When a user requests a specific website, the DNS resolver checks its local cache first. If the IP address is found, the resolver returns it immediately, bypassing the full DNS look-up process. 

Caching can occur at multiple levels, including at your device, at the internet service provider, and the recursive resolver level, each maintaining its own cache for a specific Time-to-Live (TTL). After TTL expiration, the cached record is discarded, and a new DNS look-up is performed. In essence, DNS caching streamlines the look-up process, ensuring quicker website access and reduced load on the DNS infrastructure.

DNS Security Extensions (DNSSEC) plays a crucial role in making the DNS look-up process more secure by preventing malicious activities like DNS spoofing or cache poisoning. For instance, let's say you want to visit "abcbank.com" to access online banking services. In a DNS spoofing attack, the attacker intercepts your DNS query returning a false IP address that points to a fake "abcbank.com" website under their control, ultimately stealing your login credentials. DNSSEC combats this issue by incorporating cryptographic signatures into the DNS system, allowing DNS resolvers to confirm the authenticity and integrity of DNS responses.

When a DNS resolver receives a DNSSEC-enabled response, it can validate that the information originates from the correct authoritative DNS server and hasn't been tampered with during transmission. This ensures that users reach the intended, genuine websites rather than being misled to potentially harmful ones designed for phishing or other nefarious purposes. In essence, DNSSEC bolsters the reliability of the DNS infrastructure and provides a more secure internet browsing experience by verifying DNS responses through cryptographic signatures.

As an individual internet user, understanding the DNS might not seem like a top priority, but it plays a crucial role in your daily online activities. Every time you visit a website, send an email, or access online services, the DNS works behind the scenes to ensure smooth and secure communication. Recognizing the importance of DNS caching, for instance, can help you understand why you might occasionally need to clear your browser's cache or flush your DNS resolver cache to resolve website access issues. Moreover, awareness of DNS security extensions highlights the significance of safeguarding your online activities from malicious attacks, such as DNS spoofing and cache poisoning.

Domains

The internet seems so effortlessly simple; you type in a website's name, and it magically appears. However, behind all of this lies a complex system – the Domain Name System (DNS). In this post we’ll dive into the intricacies of the DNS, what it is and how it works.

Internet Basics #5: How DNS works

Olle Vastbinder

The term internet is the global network itself, without any data. The internet serves as the underlying infrastructure that enables communication and data transmission. And despite people using the web (or World Wide Web) to describe the internet, the web is 'just' a part of the internet. To better understand that, it's important to look at the key components of the internet.

The key components that make up the internet

The internet is the name for the physical components that form the network infrastructure. These components, or hardware, can be any physical object that plays a part in connecting the network. From servers and routers to switches, cables, and even satellites. These elements work together to transmit data packets between devices across the globe.

Protocols are the foundation of internet communication. They define how devices communicate by establishing standardized rules and conventions for data transmission. Protocols like HTTP, TCP/IP (Transmission Control Protocol/Internet Protocol) and WHOIS govern how data is formatted, transmitted, and received, ensuring seamless interoperability across diverse networks. The protocols are often considered part of the web.

The DNS is the internet's address book, translating human-readable domain names known as URLs (e.g., www.dataprovider.com) into numerical IP addresses that are used by computers. This system enables users to navigate the internet using intuitive domain names rather than cumbersome IP addresses. While not an essential part of the internet, DNS makes the internet usable for everyone.

The World Wide Web is all the information that is accessible via the internet. The web consists of the interlinked hypertext documents that allow users to navigate between websites and view the available digitalized information, such as text, images, and videos. The web is crucial for that, as it also describes and standardizes how the information can be shared. The technical part of the web dictates the URL, the HTML, and the HTTP protocol, too. So basically, the web is an almost endless encyclopedia with its contents stored on millions of servers on millions of websites all over the world.


Should you say you are surfing the web or the internet?

Essentially, the internet is the hardware and the protocols that make global communication possible. The World Wide Web is the information itself. While the internet is the whole system of connected computers, the web is just the part you see in your web browser, where you can find everything from news to online shopping, and even websites that offer web data. While the internet and the web are often used synonymously, it's important to understand that the web is purely one of the services that run on the internet. So technically, you could say you are surfing the internet using the web.

Tech

When people are talking about the internet, they often use both the terms 'Internet' and 'World Wide Web' (web or WWW) interchangeably. But while these may seem the same, they describe two distinct concepts. In this blog, we explain the difference between the internet and the World Wide Web.

What is the difference between the internet and the web?

Veronika Vilgis

We already showed you how data centers power the web. But if they are not connected, there is no web. That's where the submarine cables come into play. All your digital information is passing through these undersea cables, travelling hundreds of thousands of kilometers in mere fractions of a second. This infrastructure forms the backbone of our digital lives, but for most people it remains largely unseen and unconsidered. 

Submarine cables are connecting the world

Without this extensive cable network, there wouldn't be a functioning global internet. Undersea fiber optic cables crisscross the seabeds, connecting coastal data centers to ensure fast data transfer around the world. 

Globally, there are currently 485 undersea cable systems in service and another 70 planned for construction (Figure 1). Multiple cables often follow similar routes, providing redundancy in case of cable faults or breaks. The longest cable to date is called the SEA-ME-WE 3. It connects, as the acronym suggests, South-East Asia, the Middle East and Western Europe, and stretches 24,233 miles (39,000 kilometers). 

But even that impressive length will soon be surpassed with the completion of the 2Africa project in 2024. The 27,961 mile (45,000 kilometer) long 2Africa cable will connect 33 countries on 3 continents; Africa, Europe and Asia, linking over three billion people.

Until recently, telecommunications companies were laying down most of the cables. But over the past decade, big tech giants have started taking more control. Google alone privately owns six cable systems and has invested in an additional 19 subsea cables. Amazon, Meta, and Microsoft have invested in their fair share of cable networks too. Planning, building and deploying a new cable system is an enormously complex and expensive undertaking, costing up to US$ 500 million and can take several years. 

Before a cable can be laid, the sea floor needs to be mapped to determine the best possible route. That means avoiding strong currents, volcanic activity, and extreme changes in elevation. While deep down in the ocean protection is generally high, when cables approach shore additional safeguarding may be needed, such as concrete trenches. 

Once a route is decided upon, cable laying vessels are sent out to sea. The loading of the cables alone can take months, as was the case for the Marea cable. That 4,104 mile (6,605 kilometers) long, just over 10 million pounds (4.6 million kilograms) heavy transatlantic cable now connects Virginia Beach in the United States with Sopelana in Spain.

Undersea cables can experience faults and disruptions

Cable faults are more common than you might expect. They can happen for multiple reasons, but the most common culprits by far are fishing vessels and ships. They're responsible for two-thirds of all cable faults, with their anchors and nets dragging across the ocean floor tearing at and ripping up cables. Fish bites, including shark attacks, have occurred in the past. But while mentionable, they are so rare that they don’t really pose a threat to global connectivity. 

Forces of nature have disrupted cables before and will likely keep on doing so. In 2012, hurricane Sandy damaged and disrupted several key exchanges on the US East Coast, where undersea cables link North America and Europe. In 2006, a magnitude-7.0 earthquake off the southwest coast of Taiwan cut eight submarine cables, causing internet outages in several countries in the region. More recently, in 2022, the only international sea cable connecting the island of Tonga was cut off due to the undersea eruption of the Hunga Tonga Hunga Ha'apai volcano. 

Espionage and privacy hidden under the sea

Underwater internet cables have also become strategic targets for espionage and warfare. Historically, countries have tapped into these cables to intercept and record sensitive information. During the Cold War, in Operation Ivy Bells, the US placed listening devices on Soviet undersea communication lines, gathering valuable intelligence about Soviet activity. The operation was compromised in the early 1980s due to a betrayal from within the American intelligence community. But you don’t have to go back as far as the 1980s to find cases of cable interception. 

In 2013, The Guardian revealed that both the British Government Communications Headquarters (GCHQ) and the US National Security Authority (NSA) accessed and processed vast quantities of communications tapping 200 fiber optic cables. The data gathered included recordings of phone calls, email messages, social media entries and browsing histories of individual internet users. Although there has been no recent report of this kind of tapping, you can be sure that with increasing geopolitical tensions, someone is probably listening in somewhere.

In times of war, sabotaging or cutting undersea cables can cause significant communication disruptions. This dates back as far as World War I, when the UK severed Germany's undersea telegraph cables. In recent years, concerns about the vulnerability of undersea cables and the potential for espionage have grown. Russian submarines have been observed operating near crucial undersea cables. While no open-source reporting has confirmed any tampering or tapping, their proximity has sparked unease among Western defense officials about Russia's capability to disrupt or monitor the data flowing through these cables. An interruption in these cables could have dramatic impacts on global financial markets, communications, and internet access. 

For most internet users, undersea cables are out of sight and out of mind. Yet, these unseen conduits form the backbone of global connectivity and are a crucial part of the infrastructure that allows everyone around the world to stay in touch and share information online instantly. The next time you send an email or stream content from servers on another continent, spare a thought for the network of submarine cables that enable your digital experience.

While most of us now largely experience the internet through Wi-Fi and mobile phone data plans, those systems eventually link up with physical cables that carry the information across continents and oceans. Today we’re taking a deep dive to discover the physical infrastructure of the internet, hidden beneath the sea. 

From the ocean floor to your screen: submarine cables and the internet

The Dark Web is the internet's hidden alleyways, tucked away from the bustling streets of the World Wide Web that most of us are familiar with. Unlike the websites you visit every day, the Dark Web is inaccessible via standard web browsers or indexed by traditional search engines, such as Google.

The internet most people know about can be broadly divided into three distinct segments or levels: the Surface Web, the Deep Web, and the Dark Web. The Surface Web and Deep Web account for 94% of all the information on the internet, while the Dark Web is estimated to account for around 6%.

The Surface Web, also known as Clearnet or the visible web, is the portion of the internet that is readily accessible to the general public and is indexed by traditional search engines. When you search for a recipe, read news articles, or visit social media platforms, you are navigating the Surface Web. 

The Deep Web is a much bigger part of the internet, but standard search engines do not index it. This includes databases, private academic libraries, member-only websites, and personal email accounts. You are accessing the Deep Web when you log into your online banking account to check your balance or view transactions. While the term 'Deep Web' might sound mysterious, it’s largely unnoticeable, encompassing content that’s not intended for access without the appropriate authorization or credentials.

Lastly, there's the Dark Web, a small and deliberately hidden part of the internet that is not accessible through regular browsers. It requires special software, such as Tor or I2P, to access it. The Dark Web is notorious for its illegal activities. Although it’s true that illicit activities are prevalent on the Dark Web, it is also a platform where political dissidents can engage in communication in countries with oppressive regimes, or where whistleblowers can divulge information without fear of reprisal. 

The roots of the Dark Web can be traced back to the 1990s, when the U.S. Naval Research Laboratory initiated a project called 'onion routing', a technique for anonymous communication over a computer network. This project ultimately led to the creation of Tor in the mid-2000s. 

The primary objective of Tor was to safeguard the online communications of U.S. intelligence. However, once it was released to the public, the broader community began to recognize its potential for ensuring privacy and resisting censorship. As internet use increased, people started to worry about privacy, surveillance and censorship, which led to Tor becoming more popular. 

By the 2010s, with the emergence of marketplaces like Silk Road, the Dark Web started to gain notoriety as a hub for illicit activities. Although these marketplaces brought attention and controversy to the Dark Web, they also ignited the broader debate regarding internet freedom, privacy, and the challenges of digital governance.

Until you see how Tor works, the onion in the Tor name might seem like an odd choice. The onion represents the concept of working in layers. All traffic in Tor is relayed and encrypted three times as it travels over the Tor network. The data is passed through several of the thousands of Tor relays. There’s no single relay that knows where the data originally came from or where it is going. You can compare it to passing a secret note through a group of people, with each person adding a new layer of code to keep the message hidden.

With Tor, internet traffic is routed and encrypted multiple times, which makes it difficult to track activities or locations. Despite Tor's association with accessing the hidden parts of the internet, such as the Dark Web, it can also be used for regular browsing.

Privacy on the Dark Web: The good & the bad

The concept of privacy is a fundamental characteristic of the Dark Web, and it is one of the primary reasons why individuals and organizations may choose to operate there. The Dark Web is distinguished from other parts of the internet by its unique combination of secrecy, anonymity and association with both legal and illegal activities. The illegal aspect of the Dark Web often prevails when people are referring to it. The origin for that negative sentiment can partly be found in the marketplaces on the Dark Web. 

Marketplaces are places that facilitate trade and communication in a shielded environment. Users can buy, sell and exchange goods and services with a heightened sense of anonymity. Common illegal goods and services on offer include, for example, illicit drugs, stolen data, malware and hacking services, firearms and counterfeit identity documents or money. In 2022, you could buy valid US credit card information for as little as $17 or a New York driver's license for around $70.

Some marketplaces have gained notoriety, the most notable of which being AlphaBay and Silk Road. Silk Road, which debuted in 2011, was a pioneer in the underground digital bazaar scene, offering a diverse range of products, ranging from illicit drugs to counterfeit currency. Its user-friendly interface and reliance on Bitcoin as an anonymous payment method made it a massive hit among Dark Web users. However, its prominence also made it a prime target for law enforcement, leading to its eventual closure in 2013. 

AlphaBay quickly filled the void left behind by Silk Road and expanded the range of illicit activity. These marketplaces, while providing a glimpse into the Dark Web's hidden economy, also highlight the seemingly endless cat-and-mouse game between cybercriminals and global law enforcement agencies. AlphaBay was shut down in 2017 by law enforcement. 
In 2021 it was relaunched. But at the beginning of 2023, the owner seemingly disappeared. Due to the way security was set up, the site was basically rendered unusable and it died a slow death. That Illicit activity is lucrative is not a surprise. A prime example of that is Hydra. In 2022, that marketplace was shut down by German and U.S. authorities. At that time, it was the highest-grossing Dark Web marketplace, with an estimated revenue of €1.23 billion.

The Dark Web is rife with illicit activities and potential dangers, but it also serves as a sanctuary for whistleblowers and those evading political repression. Nevertheless, its reputation is primarily shaped by the trade in illegal goods, services and stolen data. It's important to recognize the dual nature of the Dark Web and remain vigilant about personal and organizational cybersecurity measures. To safeguard against data leaks and the potential dissemination of sensitive information on the Dark Web, businesses should adopt comprehensive cybersecurity strategies. 

You probably have heard of the term “Dark Web” before. And although the Dark Web may seem like something out of a mystery novel, it is real and is a big part of the internet. Let's look at what it is, how it works, and how it differs from conventional web.

What is the Dark Web and how does it work?

A humans.txt is nothing more than a simple text file listing the people working for your organization. It’s a nice way to give them the credits they deserve. Often the file is focused on the people involved in making the website, but sometimes everyone in the company is listed. The choice for which people and which departments to include is entirely yours.

No matter what you put in the humans.txt, make sure people agree to be included. Not everyone wants to share their data that freely, so be sure to make it optional. If you don't want to put in a lot of names or keep the file up to date, you can always choose to only name higher management or founders and list the rest under departments or locations.

Humans.txt is still small, but generating traction 

Our data tells us that midway through 2023, just under 200.000 websites have a humans.txt on their website. That’s not a very impressive number. But there are some very big names among the companies that do have it. Google, Medium, Wired and Netflix are just a few of the well-known companies sporting a humans.txt. With big (tech) embracing the humans.txt, it might be just a matter of time before it is considered best practice to use one for all websites. 

When you take a more in depth look at the numbers in Figure 1 above, it stands out that businesses account for almost half (48%) of all websites using a humans.txt. They are probably eager to show the people involved in making and maintaining their websites. And it won't be a surprise that content related websites account for over a quarter (27%) of all websites with a humans.txt. The reason for that is that content websites are more focussed on recognizing the people creating and making the content for their websites. 

When it comes to creating the file, you can add any information  you want. Keep it simple and create a list of names of people working at your company. Or make it more elaborate and add  their job titles, the departments they work at or even links to LinkedIn profiles. Personalcompany’s email addresses, phone numbers, work locations and much more are also great to add.

You can also include other companies whose products you are using, and maybe even some individuals from those companies. It’s your list, and you can add anything you want.

Setting up your own humans.txt is easy. Just open a text editor and make sure you save the end result as a .txt file. Write a small intro and put in the information you want to share. We’ve listed the main things you can put in your personal humans.txt below.

Names and job descriptions or roles
List the people involved in the website's creation and management. Don't forget to mention their roles and responsibilities to give it a personal touch. If your company is a bit bigger, you can just mention the departments or even create a list for each department in the humans.txt.


Contact Information
You can put everything you want in the humans.txt, such as contact details like email addresses, phone numbers or their social media. Be sure you check with everyone if they are ok with sharing all that info. When in doubt, don’t put in contact information.


Acknowledge external partners
Odds are you are using third-party resources, tools, or maybe even open-source projects. Mentioning them shows you appreciate what they do. Spread the love!


Links
Links won’t be clickable in the humans.txt, but you can put them in. Add your companies LinkedIn or maybe a corporate website where people can see who is working at your business. And if you’re always looking for new staff, be sure to add your vacancies site.

Although a text file with names and departments isn’t the place where you would expect some fun, you can play around with the design. Check out the humans.txt we set up for Dataprovider.com. With some ASCII art, even your company name can look nice in a text file. And don’t worry about designing it yourself, there are plenty of tools out there that generate a good-looking file for you.

Once you're happy with the end result, upload the file to the root directory of your website. That’s the place where the well known robots.txt and ads.txt are located. Where those two files both serve a critical task, the humans.txt is just an extra.

Even though it doesn’t offer any mentionable benefits at this time – although some people claim it may very slightly benefit your SEO efforts – it is nice to just acknowledge the people who are working for you. It may also become important to add your own humans.txt to show your website is created by real people instead of it being generated by AI. The big search engines have not yet indicated they are tracking that, but showing your content is original can certainly help.

With the rising popularity of AI, you would almost forget that people are still doing most of the work in creating and maintaining a website. Programming, developing, designing, writing, creating, at the basis of all that there still are actual humans doing the work. To acknowledge the work those people put in, it is becoming increasingly popular to add a humans.txt file to a website. Although the concept of the humans.txt file has been around for some time, it is becoming increasingly popular to add this simple file to a website. But what is a humans.txt file, should you be using it, and how can you create one yourself? We’re covering all these questions in this blog. 

Why you should add a humans.txt file to your website – and how to make one

Massive data centers powering the internet behind the scenes

Data centers come in all shapes and sizes. The largest data centers, so called hyperscale centers, contain more than                                    5,000 servers and are generally larger than 1,000m². These data centers contain rows upon rows of servers and networking equipment that work together to store and process digital information. Figure 1 shows Google’s data center in Eemshaven, Netherlands, which happens to be only a 30-minute drive away from our office here in Groningen. It opened in 2016 but has undergone two expansions since opening.

A city called Ashburn in Northern Virginia is the data center capital of the world followed by Singapore, Tokyo, London, and Chicago. In the 1990s, major telecom companies built an extensive fiber line network across the region that laid the foundation and provided the infrastructure many data centers still rely on today. Nevertheless, data centers are popping up in the strangest places; even under water, in historical buildings, empty stores in US malls, or under mountains such as the Swiss Fort Knox and old mines. Figure 2 provides an overview of some of the larger data centers around the world.

A data center serves as the physical hub for critical network data and servers, comprising several essential components that work together to store and disseminate information. The core constituents of a data center encompass servers responsible for hosting websites, apps, and databases, typically organized in server racks. Networking equipment, such as switches, routers, and firewalls, establish connections among servers and storage, facilitated by high-speed fiber-optic cables.

The data stored and shared within these data centers includes everything from social media posts and streaming media to business software and financial transactions. Companies like Google, Amazon, Microsoft, and Facebook operate some of the largest data centers in the world to support their massive online platforms and services.

Global data creation is projected to grow to more than 180 zettabytes in 2025. To grasp the sheer size of this number imagine a smartphone with a storage capacity of 256GB. To store 1 zettabyte (ZB), you would need 3.9 billion of these smartphones. So for 180ZB, you would need 702 billion smartphones. Stacked on top of each other, they would reach about 8,250 times the height of Mount Everest. Sadly, a lot of this data is “dark data,” i.e., data collected, processed, and stored for single-use purposes often to never be used again. Think of all the files and photos you have stored on old hard drives or in the cloud only to never look at them again but at a massive scale.

The heat generated by data center operations necessitates precision cooling systems to maintain optimal temperature and humidity levels. Backup power systems, including uninterruptible power supplies, generators, and power distribution units, offer contingency power during outages. Specialized monitoring tools ensure the organization, protection, and oversight of equipment and infrastructure, aiming to prevent disruptions and ensure consistent functionality. 

The environmental cost of the web's infrastructure

Did you know that streaming a two-hour film pollutes as much as a 45-minute car drive or that the 4.6 billion streams of the 2017 hit song Despacito used as much electricity as the combined annual electricity consumption of entire countries? Data centers have come under fire for their environmental impact so let’s have a look at what factors play a role.

The servers, routers and other equipment inside data centers require huge amounts of electricity to run continuously. It's estimated that by 2025, the energy required to power the sprawling internet infrastructure could use 20% of all electricity produced and emit up to 5.5% of the world’s carbon emissions, which is more than the global aviation industry. 

Data centers also require rare earth metals and minerals, such as lithium and copper to produce the miles of cables, sophisticated cooling systems, and electronic components they contain. Extracting and processing these finite resources takes a major toll on the environment. Once the equipment reaches the end of its lifespan, much of it ends up in landfills where the toxic materials it contains can leach into the surrounding soil and groundwater.

The concentrated computing power in data centers produces an enormous amount of excess heat that requires constant cooling. You don’t have to go far to experience it for yourself, we all have felt the heat produced by our laptops. Keeping servers and other equipment from overheating is an energy-intensive process that often relies on water as well as climate-controlled air systems. The heated water used for cooling is frequently just discharged back into local waterways, contaminating the surrounding ecosystems.

Several companies have pledged to make their data centers carbon neutral using power generated through renewable energy sources. A few select projects around the world also show that the excess heat generated by data centers could be used for other purposes as is the case in the English city of Devon, where a data center heats the water for the local public pool. The Butte-aux-Cailles public swimming pool in Paris is also heated by a small data center and several Scandinavian cities use the heat for communal district heating.

The environmental footprint of the internet’s infrastructure is poised to grow rapidly. Significant improvements in efficiency and sustainability will be needed to curb the worst of these effects and prevent lasting damage to the planet. Transitioning to renewable energy, reducing electronic waste, and improving cooling technologies are all steps in the right direction to ensure the web of the future is built on a green foundation.

The scale of the infrastructure behind the digital world is truly mind-bending. As the internet becomes increasingly intertwined with our daily lives, it's important to understand the immense physical components that make it all possible. The internet may feel ephemeral, but it depends entirely on a vast array of very real cables, data centers, and servers.

While the web may seem intangible, the hardware behind it requires an immense amount of resources and coordination to operate and maintain. The next time you open an app or click a link, take a moment to appreciate the hidden systems that make the modern internet possible. 

As you browse websites and stream entertainment, it’s easy to forget about the massive physical infrastructure supporting our digital lives. But deep beneath the ocean and inside often nondescript buildings, an intricate network of fiber optic cables and servers are constantly working to transmit information around the globe. While the web may seem intangible, the hardware behind it requires an immense amount of resources and coordination to operate and maintain. In this blog post, we will delve into the strange world of data centers, exploring what they are, how they work, and why they have come under criticism for their environmental impact.

Guardians of the galaxy: Inside the data centers powering the web

Dataprovider.com

In the 1960s, the first ever networking protocol was made by the United States Department of Defense’s Advanced Research Project Agency (DARPA). It was called ARPHANET. ARPHANET was created in order to make it easier to share long distance information without needing a dedicated phone connection that linked a computer with a network. It started as a basic system that assigned each computer to a unique hostname. 

Afterward, as the internet became more and more popular, this basic system wasn’t enough anymore. Therefore, the Internet Protocol (IP) was implemented in the 1980s. The Internet Protocol is in control of choosing the route of data packets between devices connected to the internet. It makes use of the information from the data packet — including the IP address of the sender and the recipient — to bring the packet to the intended device. The packet will be sent from device to device until it’s received by the device of the recipient. The IP and IP addresses have to work together to ensure that the data is forwarded correctly and reaches the right destination.

IP addresses are also used by servers that host domains, that means domains can be accessed not only by typing the domain name but also by writing the dedicated IP address into a browser. IPv4 is the first version of IP addresses and has 32-bit numbers consisting of four numbers divided by full stops. This is called an octet (e.g., 109.106.241.165). The range of the numbers in an octet is from 0 to 255. Some companies can be identified by their IPv4 address. For example, the IP of Google’s public DNS service is 8.8.8.8, and Quad9, another large DNS provider, has 9.9.9.9, as the very name suggests.

In total, there are over four billion different addresses in IPv4. These are, however, not enough for all internet users, which is why IPv6 was implemented. IPv5 was made, but never used. According to our data, IPv4 addresses account for 85% of all IP addresses, but IPv6 is gaining popularity and has grown by around 3% in the last two years.

The biggest difference between the two is the length of the addresses. IPv6 is a 128-bit hexadecimal address made of 8 parts containing 16 bits separated by a colon (e.g., 2a02:2350:5:10b:3f:3117:ab70:a002). All hexadecimal characters represent four bits, which need to be changed into a hexadecimal number at a time. There is a possibility to have one device with both IPv4 and IPv6 running in parallel. This is called dual stacking. Tunnelling allows IPv6 users to communicate with an IPv4 network to get to IPv6. Network Address Translation (NAT) makes it possible for multiple internal devices with different internal IP addresses to access a host in a different network through a single (public) IP address. The NAT takes care of the mapping between these networks, which means that it associates a particular IP address with a specific device or location within the network.

IP addresses can be categorized into public, private, static and dynamic. A public IP is used outside a network, while a private one is used inside a network. A static IP is always manually made and can’t be changed. On the other hand, the dynamic address does change since it’s set by a Dynamic Host Configuration Protocol (DHCP). The dynamic IP addresses expire over time, but will be given automatically a new lease, or replaced by a new one.

Your publicly known IP address can be found by asking Google ‘’What is my IP Address?’’ If you are looking for your internal IP address, you should click ‘Start > Settings > Network & Internet’ on your Windows computer. On macOS, you go to ‘Menu > System Preferences > Network > Choose your network connection.’ With an Ethernet or USB connection, your internal IP will be shown, and with a Wi-Fi connection you can find the internal IP address under connection status. 

For security reasons, you might want to hide your IP address. You can do this by using a proxy server or Virtual Private Network (VPN). A proxy server makes sure that servers, search engines and other companies can’t see your IP address — they see the IP address of the proxy server instead. 

A proxy server is mostly used as a protective layer against hackers to hide an IP address and to be anonymous on the web. Unfortunately, there are some downsides to using a proxy server: the server might still save private information, such as passwords and usernames. For this reason, you shouldn’t sign into private accounts while using a proxy server, or you should use one that is highly secured. 

A VPN hides your data traffic and safeguards it from other parties. When connecting your device to a VPN before using the internet, your device will be shown under the IP address of the VPN. Via a secured connection, your internet traffic will be sent to the websites you are visiting. A VPN is commonly used to secure data and the IP address. Next to that it is used to secure websites and access content that can’t otherwise be accessed due to geographic reasons.

IP addresses are important for communication on the internet. They give the possibility to recognize the devices connected to a network and send and receive information over the internet. IP addresses are the foundation of the internet, therefore, it is important to have a clear view on the topic.

An Internet Protocol (IP) address is a unique code on the internet linked to a computer or computer network. This unique code grants the device to send and receive information.

Internet Basics #7: A beginner’s guide to IP addresses

Laura Keinänen

Understanding different security measures for networks is crucial whether you are an organization, a company or an individual operating in the digital landscape. In 2021, the US Federal Bureau of Investigation (FBI) recorded nearly 850,000 cybercrime complaints and a total amount of $6.9 billion lost to cybercrime. With cyberthreats on the rise, having a firewall in place is essential for protecting your network and devices. However, there are several types of firewalls alongside various firewall providers, and it can be challenging to determine which is best suited for your needs. In this Internet Basics Series post, we aim to clarify the types of firewalls and perhaps help you assess each of their relevance to your scenario.

The primary objective of a firewall is to permit safe traffic to enter the network while blocking potentially harmful traffic and thus ensure the security of connected devices and systems. Essentially all incoming and outgoing traffic will have to pass through a layer that evaluates each message and blocks those that don’t meet the set security requirements. Without a firewall, a device is more vulnerable to cyberthreats, such as hacking attempts and data theft. Someone could steal, delete or use your data to commit identity theft or financial fraud. 

Firewalls can be divided into three categories based on delivery methods: 

A software firewall, also known as a personal firewall, is installed on a single device, such as a computer, to protect against potential security threats to that device. Software firewalls can distinguish between different programs installed on a computer system. This means that a software firewall can, for example, differentiate between a web browser and a word processing software and apply specific security rules to each program. Most operating systems, such as macOS and Windows, include a built-in firewall. Often the built-in firewall does have to be enabled from system preferences. 

Advantages of a software firewall include ease of maintenance, as well as ease of differentiation in levels of access and permission to users. Disadvantages include slow performance of the system due to using memory and disk space as well as required separate installation on each device. 

A hardware firewall, on the contrary, is a physical device installed between the internet and a network. It is a stand-alone product that can be purchased and typically used on broadband routers. Hardware firewalls can be used as a border device or to segment and protect internal networks in larger enterprises. 

Advantages of hardware firewalls include reduced latency, increased bandwidth and being less prone to cyberattacks due to running independently. Additionally, users can enable VPN (virtual private network) connections for increased security and encryption. Disadvantages include higher costs as well as required expert set-up and monitoring. 

Cloud-based firewalls, also referred to as Firewall-as-a-Service (FWaaS) or Web Application Firewalls (WAF), are offered by many providers, including Amazon and Cloudflare, over the internet. These are software-based, cloud-deployed network devices typically operating within online application environments. 

Cloud-based firewalls are becoming increasingly common due to the rise in remote work. Cloud firewalls can be easily integrated with cloud infrastructure, have high levels of availability, and no need for in-house hardware. The service provider also handles all administrative tasks. Disadvantages of cloud firewalls include privacy concerns and a lack of transparency due to third-party involvement.

A system with a firewall analyzes network traffic based on a predefined set of rules. The firewall will only allow incoming connections that conform to the established security protocols and reject those that don’t. Firewalls can be divided into five types based on their operating technique: packet-filtering firewalls, circuit-level gateways, stateful inspection firewalls, proxy firewalls and next-gen firewalls (NGFW). As the name suggests, NGFWs are a more recent development, while the rest are traditional firewall types that have been in use for many years. Each type of firewall can be deployed as hardware, software or in the cloud.  

Depending on their design, firewalls can typically act on traffic at different layers of the TCP/IP (Transmission Control Protocol/Internet Protocol) protocol stack. Most internet applications, such as the world wide web or email, rely on this stack. In networking, packets are used to send data between computers. Packets have a header that gives information about IP addresses, protocol, source and destination ports. Computer applications are often associated with specific ports to communicate over a network. Incoming packets can thus be efficiently matched to the corresponding application. Packet-filtering firewalls act as a checkpoint to allow or block these packets based on a set of predefined rules. This firewall can, for example, be configured to block or allow certain ports to respectively block or allow specific applications to communicate over the network. More advanced configurations are also possible, such as only blocking a certain application when incoming packets originate from a specific IP address. Circuit-level gateways create a virtual circuit between the sender and receiver. Instead of examining packets, this firewall type verifies the connection (via a TCP handshake) and that the sender has permission to communicate with the receiver. Stateful inspection firewalls inspect packets at multiple levels combining packet inspection with TCP handshake verification. 

However, there may still be ways for malicious traffic to pass through the previously listed firewalls. For example, communications over port 80 (for HTTP) may be allowed, and the connection may be valid, but if a packet arrives with malicious HTTP content it will not be blocked. This is where application-level firewalls come in. Application-level firewalls are able to understand specific application protocols, such as HTTP & HTTPS, and see if an allowed port is abused. 

Proxy firewalls operate as a mediator between internal and external systems, protecting networks by masking client requests before sending them to the host. Finally, next-gen firewalls combine several functions of the aforementioned types of firewalls while adding advanced cybersecurity capabilities. NGFWs are commonly used in heavily regulated industries, such as healthcare or finance. 

Which kind of firewall should you go for, then? Software firewalls are typically adequate for most individual users, as they provide essential protection against common threats. Configuring a firewall for business applications can be much more of a challenge due to the large number of computers on the network. For companies and organizations, the choice of firewall ultimately depends on the specific security needs and amount of network traffic. 

Firewalls serve as the first line of defense in securing a network against potential cyberthreats. The term originated from the physical wall used to prevent the spread of fire within buildings. Similarly, in the digital world, firewalls aim to prevent the spread of harmful or suspicious traffic to devices connected to a network.

Internet Basics #6: Firewalls explained

Jasper van Thuijl

In our Internet Basics series, we discuss the fundamentals of the internet. Today we'll look into the differences between crawling and scraping and explain how we bring structure to the information chaos that is the World Wide Web.

Each month, Dataprovider.com indexes over 750 million domains in over 50 countries. Indexing refers to the process of collecting, parsing and storing data to enable fast information retrieval. To index the web, we first need to discover and visit the web pages from which we can extract our data. This discovery process is called crawling. A crawler (sometimes also called spider or robot) is a software application that automatically visits websites and finds new websites to explore. In most cases, crawlers are operated by search engines. 

A good analogy for a crawler would be a librarian going through a large unorganized pile of books. The librarian reads a few pages from each book and the title to put it in the correct category so that library visitors can quickly find the information they need. However, in the case of the crawler, we can’t know exactly how many web pages we need to browse. Crawlers, therefore, apply a different process.

To discover websites, a crawler does the following: First of all, the crawler starts from an initial list of hyperlinks. The crawler program visits a web page from the list and identifies all hyperlinks on it. The newly discovered hyperlinks are then added to the list and are handled by repeating the previous step. 

Due to the massive size of the internet, crawling is a large-scale operation that requires a huge amount of computing power. As it’s impossible for a crawler to visit every URL on the world wide web, it needs some type of measure to select the most important pages. Often, the number of referrals to a specific website defines its importance. 

If you’re interested in building your own crawler, you don’t have to start from scratch: you can use popular open source web crawler systems such as Pyspider, Apache Nutch or Heritrix by the Internet Archive. These programs are built with scalability in mind and offer integrations with popular databases such as MySQL, MongoDB and Redis.

The process of extracting content from a web page is called scraping. Often, scraping is tailored to extracting information from a specific website or group of websites. For example, a scraper could be explicitly designed to obtain product prices from Amazon for a price comparison website. Other usage examples are automatically retrieving real estate listings for a real estate platform or tracking online presence and reputation for marketing purposes.

As most web pages are designed for humans, and not for automatic processing, it can be difficult to gather the required information from the web page. Modern websites typically use JavaScript frameworks to create a smooth and interactive user experience. As JavaScript is executed in the user’s browser, the effects of these scripts can’t be captured if a scraper is just looking at the HTML code of a website. Moreover, scrapers are highly sensitive to (changes in) the web page layout. Scraping systems, therefore, frequently have specific features to deal with these difficulties. For example, features to allow full JavaScript execution or features to handle pagination and infinitely scrolling web pages.

In most cases, website owners don’t want their content to be scraped since it isn’t beneficial to them. Scraping brings additional traffic, which slows down website load times. More importantly, most website owners don't want their information to be copied and used beyond their control. Frequently scraped targets may therefore actively repel scrapers by taking measures such as rate limiting, detecting unusual traffic and obfuscating website code. As a consequence, scrapers often try to mimic human users to remain anonymous. This is accomplished through actions such as simulating how a human user would see and browse a webpage, faking user agents (pretending to be a regular web browser) and using residential IPs.

In comparison, most crawlers adhere to legal standards and introduce themselves using the User-Agent HTTP request header when visiting a website. For example, the User-Agent is set to “Googlebot” for Google’s crawler, and “Bingbot” for Microsoft Bing’s crawler. Crawlers only index publicly accessible information. If website owners don't want their websites to be (fully) indexed, they can include a robots.txt file in the main directory of their website. This allows them to explicitly specify which parts of the website a crawler may or may not index. 

Alternatively, the indexing of a webpage can be blocked by including the noindex meta tag in the website code. Besides this, many crawlers have a crawl budget, which means they only visit a specified number of URLs from an individual domain in order not to put a burden on the website. Most crawlers visit a website only from time to time to check for updates, while scrapers visit more frequently.

The world wide web is a massively valuable source of information, but also largely unorganized. How do we structure it?

Internet Basics #4: The difference between crawling and scraping

Hylke van der Veen

HTTP works via the client–server model: the client sends a request to the server and based on that request, the server sends a particular response back to the client. This request can be anything from retrieving a web page to creating a user account from a given username and password. Individual requests are independent from each other: for a single request, a single response is sent back.

Apart from HTTP, we often see the HTTPS protocol in web addresses. This stands for Hypertext Transfer Protocol Secure and indicates that the communication has an extra layer of security, so malicious actors can’t easily interrupt it. In a previous post about SSL, we’ve explained what exactly this means and how it works. It’s important to note that everything we discuss in this article applies to HTTPS as well.

HTTP requests can be used for many different purposes, each of which is indicated with its own so-called “method.” Only a select few of these methods are frequently used and are important to know. The most prevalent method is the one you’ve used to get to this very blog: the GET method. This method, as the name suggests, is used for retrieving information from the server. One of these was sent by your browser when you clicked on a link to read this article in order to retrieve the HTML of this web page. 

If, for example, you want to send information to a web server to log into an account or to fill in a form, the POST method is used. Two other often used methods are PUT and DELETE. PUT is very similar to POST but is used to update existing records rather than create new ones, and DELETE, as you might have guessed already, is used to request the deletion of certain data.

But how exactly are these methods used when sending a request?

Figure 1 gives a visual representation of an HTTP request. At first, this might look daunting, but we’ll break it down and look at the individual parts. This is a request to a server which processes the text we give it and, in this case, gives us back an HTML page with our text in a heading. The first part of the request we send to this server is the request line, which is made up of three components: the method, the target and the protocol version. This line tells the server what action we want to take, in which destination that action should be taken and – to avoid confusion – what exact protocol we want to use to communicate. In our case, this is a GET request because we want to retrieve an HTML page. We’re sending this request to the (fictitious) /test endpoint using the HTTP version 1.1 protocol.

Next up are the headers. Request headers can be seen as metadata which informs the server about the data that is requested and about the client sending the request. These headers can be further divided into categories, but we’ll ignore them for now. In our request, we need to pass several headers to tell the server what our request looks like – through the Content headers – and what we expect in return from the response – through the Accept headers. The host header ensures the request is sent to the correct server. 

The User-Agent header tells the server what device we’re on and what browser we use. This header looks different for each device type but commonly starts with Mozilla/5.0 to indicate that the browser is compatible with Mozilla. This is only a small selection of possible headers that can be passed with a request. For a complete overview, check out the Mozilla documentation on HTTP headers.

Finally, separated from the headers with a single empty line, is the request body. This is optional as not every request requires content to be sent to the server, but in our case we want to send some text that will be presented in a heading. This is a very simple example and the request body can be much more complex with even multiple resources.

After processing the request, the server will return a response. If the request is successful, the response might look like this:

The HTTP response is structured very similarly to a request. It starts with a status line that lets the client know which HTTP version is used and how the request has been handled through a status code and a complementing status message. These status codes are divided into five categories and indicate the success or failure of the request. Since our request was successful, we got the status code 200 with the text ‘OK’, indicating that everything has gone as planned. This and other status codes can be found in our proprietary data in which we save the response code and headers of every website we crawl.  

The status line is followed by the response headers that provide information about the server and the returned content. This content is once again separated from the headers with an empty line and, in our case, contains the HTML structure for a web page with the text we’ve sent to the server in a heading.

Figure 3 shows the 20 most occurring response headers from a large sample of available websites from our proprietary data. We can see that some headers occur in almost every response. This makes sense as they contain some very generic information which most responses need, such as the timestamp of when the response was sent, the type of content it sends back or if the connection to the server will be closed or not. 

Hopefully, now you know a bit more about the complex nature of the internet and what goes on behind the scenes when you visit your aunt’s blog or update your LinkedIn profile.

Hypertext Transfer Protocol (HTTP) is a communication protocol that is used for communication between web servers (e.g., a website) and clients (e.g., your browser) on the internet.

Internet Basics #3: HTTP requests explained

You might have noticed that a web address often shows the HTTPS protocol, rather than HTTP and that your browser displays a padlock in front of the URL to indicate that your connection is secure. Although this gives a sense of security, it also raises the question of what exactly this means and how your connection is protected. 

When a website uses the HTTPS protocol, your access to the website is protected by a Secure Sockets Layer (SSL). SSL is an Internet Security Protocol that ensures the connection between a website and a user is secure. SSL protects your connection in three ways. 

First, SSL is a proof of the authenticity of a website: it shows that the website you’re visiting really is what it claims to be and isn’t a fake website set up by attackers who try to obtain private data from visitors.

Secondly, SSL ensures that communication between the two parties is encrypted. With the standard HTTP protocol, data is transmitted in plain text. This means that anyone who can intercept this connection can see the data that is sent in plain text. With HTTPS, however, the data is encrypted before being transmitted, so even if it's intercepted, it looks like a random jumble of characters without the right key to decrypt it. 

Finally, SSL ensures that the transmitted data cannot be tampered with through the use of Message Authentication Codes (MAC). Although, technically, messages sent through SSL can still be intercepted or altered, the MAC ensures that the user will always know if this has happened.

Traditionally, SSL was only used on web pages which process personal information, such as online payments or login screens. These days, however, many websites have an SSL certificate to guarantee the security and authenticity of the website itself. With modern browsers and devices, the time it takes to set up an SSL connection is mostly negligible and, therefore, provides no disadvantages anymore.

Follow us

Questions?