Loura Kruger-Zwart

It's not just people roaming the web anymore, and we’ve known about the new kids on the block for a while: robots. A company or organization’s online robot—also called a bot, crawler, scraper, or spider—is tasked with exploring the internet for specific reasons, like indexing to rank websites or collect data. For most websites, being open to these bots and being indexed by a search engine, for example, is an excellent way to optimize and rank higher in search results. But not every website wants each of its webpages accessed or indexed by a bot—what then? The robots.txt file is this modern problem’s modern solution. 

Essentially, robots.txt is a text file that a website administrator adds to their site to tell bots and crawlers where they can or cannot go, and what they should or shouldn’t index. It's the digital bridge between human and machine, one that helps maintain a symbiotic relationship between the two online.

The bots that traverse the web and collect information are looking to index your site and its webpages, ready to be shown to searchers (in the case of Googlebot) or broken down into technical details, as the Dataprovider.com spider does. SEO is improved when search engine bots can quickly and accurately identify the pages of a website. 

This is where a robots.txt file comes in: it indicates how crawlers can index the web by setting rules and directing to sitemaps. The website owner explicitly allows or disallows (particular) robots to access or index the site, and outlines which bots can access what parts of the site. The owner can direct bots to the unique sitemap, which usually outlines the relevant and appropriate pages to be indexed, and excludes private, irrelevant, or administrative parts of the site. In contrast to LLMs.txt, which gives additional information and specific details, robots.txt is intended to be a more ‘objective’ and straightforward resource for our robotic co-surfers.

However, all of this hinges on an important agreement: adherence to the robots exclusion protocol. First developed by Dutch engineer Martijn Koster in 1994 and used across the web since then, Google sought to upgrade the protocol to an official industry standard in 2019. In 2022, Koster and Google wrote the Proposed Standard together, and it was accepted by the Internet Engineering Task Force (IEFT). As with other kinds of .txt files, this one can also be added to a website root as a markdown file. Website administrators can include incredibly specific instructions to bots, as long or short as needed. Find the original guidelines here and more modern recommendations here.

Robots.txt is the practical outcome of the robots exclusion protocol, but, of course, it is still up to crawlers, bots, and the organization behind them to adhere to the rules. Safe to say that most do—there is a benefit for sites to be indexed, but there is also a benefit for crawlers to have access to these sites and their relevant webpages. Crawlers would rather not be denied access, and so, they adhere to the rules. For example, Wikipedia’s robots.txt file is very explicit about its various instructions: it includes small written explanations (for a human reader’s benefit, no doubt) about which bots are in the ‘naughty corner’ for not following the rules. There are too many bots out there to have specific rules for each, but having some indication on a website about the expectations for robots seems integral to the internet today. Ultimately, however, the protocol remains advisory—a polite suggestion—despite using terms like allow and disallow. 

Curious to see some examples of robots.txt on major websites? Take a look at the notable files at Nike, YouTube, Reddit, LinkedIn, and Wikipedia, or add /robots.txt after any domain name to see that website's bot rules.

At Dataprovider.com, we closely monitor robots.txt files as part of our web crawling process. Since our spider scans the web on a monthly basis, the robots.txt file is our first checkpoint to determine whether we are permitted to access a website: if the file explicitly disallows crawling, we respect those directives and do not access the site. This is part of our privacy by design protocol. While we have always recorded whether a robots.txt file is present, we did not systematically track which bots were specifically allowed or disallowed until recently. This has revealed some impressive numbers: in March 2025, our crawler detected mentions of 112,327 unique user agents specified across all robots.txt files. While some of those mentions may be typos, or for bots that are no longer ‘in service’, the total number gives an indication of just how important it is to provide structure and instructions.

As the web becomes increasingly crowded with more LLM crawlers and other automated agents, understanding who’s out there and what they’re allowed to do is increasingly relevant. Robots.txt provides an interesting glimpse into how website owners manage bot traffic—who they welcome and who they prefer to keep out.

In Figure 1, we break down the share of websites based on whether they have a robots.txt file and whether they allow or disallow crawling. The data reveals that 79% of websites include a robots.txt file, indicating that a significant majority of website owners take bot management seriously.

Next, we examined the bots most frequently mentioned in robots.txt files. One key observation is that among the 70 million robots.txt file analyzed, 24% explicitly allow all bots by using a wildcard directive (*). When looking at specific mentions, various Google-owned bots dominate the list, with one major exception: AhrefsBot. This SEO-focused crawler appears in 4% of robots.txt files, making it the most frequently mentioned bot overall (see Figure 2).

To take the analysis a step further, we focused on bots that are explicitly disallowed from crawling—meaning they are entirely blocked from accessing any part of the website. In Figure 3, we highlight the most frequently disallowed user agents. Among the top bots are PetalBot, Nutch, and MJ12bot.

PetalBot is operated by Petal Search, a search engine developed by Huawei, and is used to index web pages for Huawei’s search ecosystem. Nutch, on the other hand, is an open-source web crawler widely used for research purposes, custom search engines, and large-scale web scraping projects. MJ12bot is the crawler for Majestic, a search engine and SEO tool specializing in backlink analysis and website rankings.

None of these crawlers are inherently malicious; their impact largely depends on how they are configured and used. While concerns about PetalBot stem from Huawei’s ties to the Chinese government (given past allegations of surveillance and data privacy issues), there is no direct evidence that the bot engages in malicious activity. Wikipedia’s robots.txt specifically calls out MJ12bot, saying “Observed spamming large amounts of [...] and ignoring 429 ratelimit responses”, despite MJ12bot’s website saying it obeys robots.txt. 

Creating a robots.txt file on your website isn’t complicated and has plenty of benefits, so why not? Depending on the nature and content of a site, it may be useful to indicate clearly which URLs the crawler can access on your site, to indicate the (ir)relevance of certain pages, or to manage and avoid unwanted visitors. Without it, crawlers can and will access all parts of a site. Website owners put different degrees of detail into their robots.txt instructions: Reddit, for example, disallows all crawlers with two lines of text, while LinkedIn’s robots.txt is over 4,000 lines of specification and rules.

As discussed in our recent article, the rise of generative artificial intelligence (genAI) has introduced another visitor to the digital world. Large Language Models (LLMs) also gather all kinds of information from the web; including LLMs.txt on a website specifies which relevant details the bot should share if asked via ChatGPT, Claude, or another service. However, since this development is still rather recent, many website owners opt to include LLMs instructions in their robots.txt files. 

The internet is an ever-evolving place, as we know, and keeping up with the changes is complicated. Access, security, privacy, and responsibility must be considered when new developments arise, and protocols like robots.txt are one way to bridge the gap between humans and our machines.

Tech

Robots surf the web alongside us, but have a different set of rules for how to traverse the digital world. Whether they’re indexing pages for a search engine, tracking changes for SEO, or collecting information for a structured database, bots need structure and instruction. How do we ensure they’re doing the right thing? Read on to explore the wonderful world of robots.txt.

The digital bridge between human and machine? How robots.txt works

Data privacy

Web trends

Eelco Blaauw

Web3 is among the newest tech lingo, alongside LLM, NFT, crypto-anything, and comparable developments. But before exploring Web3, it is essential to understand its predecessors: Web1 and Web2. The first iteration of the internet, Web1, was characterized by static content and a lack of interactivity. Websites served as digital brochures, where information was provided solely by the website owner, and users could only consume content without contributing or engaging with others.

Web2 introduced a paradigm shift by enabling user-generated content and promoting interactivity. This era brought the rise of social media, blogs, and collaborative platforms, allowing users to create, share, and interact with content dynamically. The widespread adoption of Web2 technologies transformed the internet into an interactive and socially connected space we know and love today.

Now, the internet is undergoing another transformation with the emergence of Web3. This next phase of the web is built on decentralized technologies, aiming to redefine digital ownership, enhance privacy, and reduce reliance on centralized authorities. Web3 is, in part, a response to the ‘walled garden’ ecosystems created by platforms like Facebook, Apple, or YouTube, where user data is harvested, privacy is compromised, and control over personal content is limited. The goal of Web3 is to return power to internet users, and to diminish the dominance of Big Tech.

This shift is made possible by blockchain and related technologies, including cryptocurrencies, decentralized applications (dApps), smart contracts, tokens, and decentralized autonomous organizations (DAOs). As a result, Web3 is often referred to as “the blockchain edition of the internet.”

We were approached by a university researcher from TU Delft, interested in measuring the adoption of Web3 technologies across the web. Our technology detection team took up the challenge, conducting intensive research to develop a new classification field: Web3 Technologies. To accurately identify and track these emerging technologies, we overhauled our detection algorithms, significantly enhancing our insights into the internet's evolving technological landscape. Our enhanced methodology now probes deeper into website code by extending analysis to externally loaded JavaScript files—a crucial adjustment, as Web3 functionalities are often deployed via external scripts rather than being directly embedded in the page.

Among the detected Web3 technologies, some stand out due to their widespread adoption and impact on the ecosystem:

Web3.js (26.4%) – One of the most widely used JavaScript libraries for interacting with Ethereum-based blockchain networks, allowing developers to build decentralized applications (dApps) efficiently.

Ethers (19.4%) – Another popular JavaScript library for Ethereum, known for its simplicity and security, making it a favorite among developers working with smart contracts.

EthereumJS (18.4%) – A collection of JavaScript libraries designed to facilitate Ethereum-based blockchain development, supporting everything from transactions to cryptographic functions.

OpenSea (7.3%) – The leading NFT marketplace, allowing users to trade digital assets on the blockchain.

WalletConnect (4.7%) – A protocol that enables secure connections between dApps and cryptocurrency wallets, allowing users to interact with decentralized platforms seamlessly.

The big question is, how much of the promises, ideals, and functionality of Web3 have been embraced by the masses? Out of the 166 million active websites we currently detect, only about 80,000 (~ 0.05%) have implemented one or more Web3 technologies, reflecting a very modest overall adoption rate. When broken down by country, Figure 2 illustrates that the United States leads far ahead with an adoption rate of 57.8%, while China (3rd) stands at 3.7%, the Netherlands (5th) at 2%, and Japan (11th) at 1.2%.

Let’s dive deeper into the numbers and take a look at the growth of the field over time. Figure 3 shows that since March 2024, the detection of Web3 technologies has been steadily increasing. Notably, during July and August 2024—when our updated technology detection methodology was implemented—the total count jumped from 37,435 in July to 65,528 in August, peaking at 79,076 in September. Following this surge, the numbers remain relatively high, with a gradual rise continuing through January 2025.

In extension of the adoption of Web3 technologies as a whole, Bitcoin as an online payment method has experienced remarkable growth. Figure 4 shows that the number of websites accepting Bitcoin increased from 203,469 in February 2021, to 270,990 in January 2025—an overall growth of approximately 33%. Notably, early 2023 marked a significant surge, with the total peaking at 283,411 in January 2023, representing nearly a 10% increase compared to the end of 2022. This robust adoption is largely driven by the integration of cryptocurrency payment solutions via payment service providers, which inherently support digital currencies without requiring specific Web3 technologies like Web3.js or Ethers.

One of the most well-known aspects of Web3 is undoubtedly cryptocurrency, having unleashed a whole new market and redefined the concept of digital assets. Ever since its inception, it has taken the internet by storm and sparked debate among experts, policymakers, and everyday users alike. From being dismissed as a speculative bubble to being hailed as the future of finance, cryptocurrency remains a focal point of Web3 adoption.

The 2024 U.S. presidential election further amplified its relevance. Since Donald Trump was elected the 47th President of the United States last November, the cryptocurrency market has witnessed a surge in interest and investment. Trump’s vocal support for digital assets, combined with promises to create a more crypto-friendly regulatory environment, has fueled a bullish trend in the industry, as shown by CoinMarketCap data beginning November 6, 2024.

Going against the decentralized spirit of Web3, centralized exchange platforms for cryptocurrencies started to emerge. A few exchanges, shown in Figure 5, highlight this growth.

Despite this momentum, Web3 still faces significant challenges, including regulatory uncertainty, scalability issues, and skepticism from traditional financial institutions. Governments worldwide are debating how to regulate cryptocurrencies without stifling innovation, while companies continue to explore new use cases for blockchain technology.

The post-election surge in cryptocurrency adoption underscores a larger shift: Web3 is no longer a niche concept, but a transformative force shaping the future of the internet and global economies.

Is Web3 truly on the rise, or is it just another buzzword with little real-world traction? Despite the hype and speculation, tangible evidence about its adoption might surprise you. Where does Web3 stand today?

Web3 in the real world: from decentralized tech to global impact

Technology detection

Tech news

The internet is an integral part of our daily lives, and imagining a day without its use is a proper challenge. Our homes and appliances are connected to the internet, our cars, devices, and even sometimes our accessories. Despite being in near-constant contact, defining the internet remains rather tricky. So, what is the internet actually? Let’s dive right in.

The origin of the internet as we know it can be traced back to the Cold War era, conceived to maintain communication between the US government and the military in the hypothetical aftermath of a nuclear attack. Initially, the connected computers were physically linked and decidedly stationary due to their size, so access was restricted to a select group of organizations. In use from 1969 to 1989, the Advanced Research Projects Agency Network, or ARPANET—where the legacy top-level domain .arpa stems from—was the precursor to our internet. This early collection of interconnected computer networks allowed selected scientists and researchers to share information and resources from a distance without paper or traditional mail. It relied on the revolutionary concept of packet switching, a transfer method that remains the foundation of how data is transmitted over the net today.

While ARPANET paved the way for the exchange of digital data, it was the invention of the World Wide Web (WWW) that revolutionized the way we access and consume information. In 1990, Tim Berners-Lee, a British computer scientist, proposed the concept of the World Wide Web as an information management system while working at CERN. According to CERN, the main concepts and terms behind the web came from this document, which described: 

Essentially, WWW is a system of interconnected documents and resources, accessed through the internet using web browsers. It introduced the idea of hyperlinks, allowing users to navigate between different websites, creating an interconnected online space. This marked the beginning of the era of mass information sharing and online communication.

What’s the difference between the internet and the web?

 Although the terms are often used interchangeably, they are not the same thing. Take a look at our dedicated web vs. internet article for all the details, but here’s the rundown.

The internet refers to the vast global network of interconnected computer networks that allows for the transfer of data. It is the infrastructure that enables various online services and applications to communicate with each other.

On the other hand, the web is the system of interconnected documents and resources, accessed through the internet. It’s just one of the many services made possible by the internet, alongside email, file sharing, video conferencing, and more. In simpler terms, the internet is the network, while the World Wide Web is one of the many services that run on top of it.

Unlike phone towers and satellite dishes, both being visible and tangible displays of communication hardware, the internet's connective infrastructure operates largely out of sight. Underwater fiber optic cables, which span vast distances and link continents via major cities, comprise an essential physical component of this entangled network. It’s a global network of interconnected computer systems.

On a level closer to home, our personal devices usually connect to the internet through Wi-Fi routers and modems, which, in turn, are linked to the global network via copper or fiber optic wiring. Once connected, we access the internet wirelessly through radio waves transmitted between our devices and the Wi-Fi router. Despite its convenience, Wi-Fi has its limitations and a couple of downsides too—most notably, it isn’t totally stable at every moment, which means disconnection could interrupt a task or activity that requires constant connection. Think of payment services, livestreams, or online games. This is combatted by a good old-fashioned Ethernet cable, which can be plugged into a computer, console, terminal, or television, straight from a router to provide a measurably more stable internet connection.

WAN, LAN, WLAN, and beyond, the magic of the internet lies in this complex web of connections, made only more intricate and far-reaching the more devices connect to it. Each device connected to the internet becomes a pathway, facilitating the flow of information between computers worldwide. In this way, the internet operates as a dynamic, interconnected ecosystem, enabling seamless communication and sharing of knowledge across the globe.

The internet has undergone significant growth and evolution since its inception, expanding beyond its original purpose of facilitating communication between researchers, and becoming a vital part of modern society. Today, billions of people around the world rely on the internet for work, education, entertainment, communication, and more.

However, once hailed as a space for free expression and democratic engagement, the web has increasingly become a double-edged sword. The proliferation of mis- and disinformation, echo chambers, and cyberattacks orchestrated through social media threats and other online spaces has begun to erode the very foundation of democratic societies. These platforms have been manipulated to influence elections, polarize public opinion, and undermine trust in institutions, thereby posing significant threats to the integrity of democratic processes. 

Moreover, the anonymity and expansive reach of the internet facilitate the spread of extremist ideologies and enable the organization of anti-democratic movements at an unprecedented scale. This and more has been illustrated again just this month, as Meta announces the removal of fact-checkers, and volunteer Wikipedia editors risk being doxxed. As a result, the challenge of maintaining the balance between freedom of expression and the protection of democratic norms in the digital age has become more complex and pressing than ever before.

The internet has come a long way since its humble beginnings as a research project. From its origins as a decentralized communication network, it has transformed into a globally interconnected web of information, connecting people across the globe. The internet and those who use it certainly face challenges, and internet nostalgists yearn for the pre-social media internet, but there’s so much still out there to enjoy. As the internet continues to evolve and expand, it will undoubtedly continue to play a central role in shaping the future of our digital world.

We’ve discussed a lot on this blog—the distinction between the web and the internet, different behind-the-scenes elements of the digital landscape, and the pros and cons of the modern online world, but have we stopped to ask: what is the internet, actually? Read more in our latest blog insight.

What is the internet, actually?

Internet Basics Series

Veronika Vilgis

In 2000, Joost Zuurbier, a Dutch citizen from Amsterdam, visited an island that you can only reach by first flying to Samoa, then boarding a ferry that runs only every two weeks with a variable schedule. The island Tokelau is a remote group of atolls in the South Pacific Ocean. Tokelau was home to just over 1,600 people in 2019, and back in 2000 there were those who had a difficult time grasping what Joost was trying to do. Most could never imagine the future of their nation’s country code top-level domain (ccTLD)—not even the man himself.

Zuurbier’s vision was to identify an unused ccTLD and make it freely available. He discovered that Tokelau’s ccTLD, .tk, was unclaimed and inactive. However, launching and expanding .tk came with three significant challenges. First, he needed to communicate his plan to the people of Tokelau and gain their approval. Second, navigating the bureaucratic process posed another obstacle—he had to secure official approval and delegation of the ccTLD from the Internet Assigned Numbers Authority (IANA). Lastly, due to a requirement from IANA that the servers needed to be built on the island, all the equipment had to be taken to one of the remotest islands in the world. At the time, Tokelau had only three computers for 1,500 people, with only a 9,600 bits per second (bps) connection for e-mail with no attachments. For reference, one million bits make up one megabit (mb), and a good modern download speed is 100mbps.

Fortunately, the final challenge was resolved with the help of an Internet Corporation for Assigned Names and Numbers (ICANN) employee who had visited Tokelau in the 1970s. Understanding that hosting global servers from the island was impractical, the employee recognized that some level of internet activity was still necessary for Tokelau to claim administrative control over its ccTLD. This meant that before anything else, Zuurbier had to introduce internet connectivity to the island.

Once this requirement was met, Zuurbier and the islanders reached an agreement on how the revenue from the ccTLD would be used. With their case prepared, they presented it to IANA. As a result, both initial challenges were also overcome, and IANA officially delegated .tk to Teletok, the company overseeing Tokelau’s telecommunications from 1997 onward, while Taloha, Zuurbier’s organization, managed the technical administration of the domain.

Because .tk domain registrations were free, the extension quickly gained popularity and became one of the most widely registered country code domains, surpassing 25 million registrations. However, the free model also drew the attention of cybercriminals. With the ability to create websites anonymously and in large numbers, scammers exploited .tk domains for various malicious activities, including phishing for passwords and payment details, hosting intrusive pop-up ads, and distributing malware.

Until recently, domain registrations for the .tk ccTLD were managed by OpenTLD B.V., part of Freenom, a registry that manages several freely available TLDs. In November 2023, ICANN terminated OpenTLD's accreditation due to multiple breaches of the Registrar Accreditation Agreement (RAA). Despite this termination, the registration and maintenance of .tk domains remains under Freenom. As a ccTLD, .tk primarily operates under the authority of Tokelau and is not governed by ICANN's RAA.

Given the controversial recent history of the .tk ccTLD, we were curious to look at the domains currently active online. In our database, we detect a total of 6.3 million .tk domains, which includes domains that are no longer active but were at some point within the past twelve years. However, as of January 2025, only 8,862 of those are currently active, 2,430 are redirects, and another 18,231 do not allow crawling. Of the redirects, 53.5% redirect to a .com domain, 10% to another .tk domain and 6.8% redirect to .cf (ccTLD of the Central African Republic).

Among the active domains, just over a third can be traced to Ireland, followed by the Netherlands, and then the US (Figure 3). In terms of security, over 50% have no or an invalid SSL certificate, as shown in Figure 4. 

The rise of the .tk domain was a remarkable story of growth, driven by its unique offering of free registrations that attracted millions of users. At its peak, .tk became one of the most registered country-code top-level domains (ccTLDs) in the world. However, its rapid growth came with a dark side, creating a haven for malicious actors who exploited its anonymity and accessibility. The association with online fraud damaged its reputation, and as a result, it became known for its role in hosting dubious websites. Despite efforts to mitigate abuse, the damage was done, and .tk's image has never fully recovered.

Digital colonialism and the future of .tk

The rise and fall of the .tk domain also highlights the broader issue of digital colonialism. Tokelau, a small, remote territory in the Pacific, found itself at the center of a global internet phenomenon. While Tokelau's local population had limited internet infrastructure, the income generated from .tk registrations—millions of dollars, some years—provided vital financial support to the territory. This arrangement, however, raised ethical concerns about digital colonialism, as a foreign entity (Freenom) controlled the domain's registration process and its profits. The revenue, which was meant to benefit Tokelau, was regularly diverted through intermediaries, leaving the island with limited direct control over the wealth generated by its ccTLD.

With the uncertain future of .tk, Tokelau faces potential financial ramifications. If the domain's usage continues to decline or becomes further restricted, it could impact the income the territory has relied on. Although the digital space has provided an economic lifeline, the loss of .tk’s popularity also exposes Tokelau to the realities of digital dependency. The implications go beyond simple financial loss; they reflect the challenges small nations face in navigating the complexities of the digital age and how they manage valuable assets like ccTLDs, which can be both a source of income and a symbol of sovereignty.

Today, while .tk domains can still be registered through Freenom, the landscape is uncertain. Despite the possibility of registration, the reality is that most domain names appear unavailable, and new registrations have effectively stagnated since October 2024. No new registrar has been announced, and the lack of clarity about the future of .tk raises more questions than answers. It’s clear that while .tk still exists, its glory days may well be behind it. What remains is a small, largely insecure domain extension that continues to attract a few unsavory characters, but for most legitimate users, it’s a relic of the past. The future of .tk remains up in the air, and its tarnished reputation means that it will likely never regain its former prominence in the world of domain registrations.

Domains

Tokelau is a remote island in the pacific. You can reach there only by a ferry that runs every 14 days. But it is the third most-visited island. How so?

From tiny atolls to digital dominance: the .tk story

TLDs

domains

Any time you fill out a survey, contact a business via their website, or submit preferences online, you are essentially filling out a form. These forms are sometimes coded into a website, but for those less technologically inclined, no-code forms can be used on or embedded into a website for visitors to use. The platforms that offer these services are called ‘form builders,’ and their shapes and sizes vary depending on their origin, destination, and intended uses—and with so many uses come so many options. 

At Dataprovider.com, we gather and structure web data like this, tracking the installations and usage of online technologies like form builders. So, to get an inside look at the web’s most used online forms, we turned to the data.

There are two clear winners in the form builder sphere: Contact Form 7 and Medallia. With over 50% of the market share, Contact Form 7 runs the show. This form builder is a WordPress plugin, and a free one at that. This helps clarify the popularity, given that WordPress is the most used CMS globally. Its open-source philosophy means that free plugins are typically well received, and the prevalence of Contact Form 7 across WordPress sites is corroborated by its 10m+ active installations stat on the WordPress Plugin Directory.  

Medallia trails at a close second. This software platform is focused on ‘experience management’—customer, contact center, employee, and digital—utilizing surveys and other feedback collection forms. Following these two front-runners, other WordPress plugins make it high onto the list: WPForms Lite and WPForms Pro rank 3rd and 6th among the most used form builders. Gravity Forms is also a premium WP plugin, as are Formidable Forms and Ninja Forms—ranking 4th, 7th, and 8th most used form builders, respectively.

Beyond the dominance of WordPress websites online today, there are a number of non-plugin form builders that make the top ten. Google Forms is part of Google’s Editors suite and can be used embedded in a site or simply accessed as a link, while Refiner is a ‘microsurvey’ software—these are the little rating boxes that pop up on a website asking for quick, one-click feedback, especially for web and mobile product companies. Microsoft Forms is a free-to-use part of Office 365 for surveys, polls, and quizzes, and finally, Typeform is an interactive high-design software that specializes in making ‘forms worth filling out’. 

We’re sorry to do it, but we had to ask: if we exclude websites that use WordPress as their content management system, what does the top ten look like then?

Unsurprisingly, Medallia dominates this list. As a CMS-agnostic software, it is not limited to WordPress, and seems to be much loved across the web. Contact Form 7 does still make an appearance in second place, as do other non-WP options from Figure 1, but a number of new names enter the race. Getsitecontrol is not a form builder specifically, but offers popups and inline forms as part of its (email) marketing suite. Shopify’s growing popularity is demonstrated by the presence of Hulk Form Builders and Powerful Contact Form Builder on this new list, given that both are apps for Shopify sites. Wufoo and Qualtrics are two builders that make the list but with rather small user bases, with the former being reasonably dated, and the latter focused on internal use for companies and universities.


A core part of web form builders is their purpose as data collection tools—customer data, employee data, and various kinds of personal data are all submitted through online forms.  Businesses need to trust that the plugins are reliable, users need to trust that any personally identifiable information (PII) is stored securely, and the companies behind form builders must ensure this trust is warranted. This kind of valuable data is wanted by hackers, making the software behind online forms targets for cyberattacks, or at least vulnerable to data leaks. 

This means that privacy is absolutely paramount on websites with forms, and digital security cannot be taken lightly when it comes to PII. According to our spider, the security of form-having websites is lacking: we look to the data on SSL certificates, Content Security Policies (CSP), and HTTP headers for details. 

Of websites that use form builders, 93.6% have a valid SSL (Secure Sockets Layer) certificate, meaning the information that travels between a site and its visitors is encrypted (Figure 3). However, this also means that 6.4% of websites that collect information from their visitors have no or an invalid certificate, making it easier for malicious actors to intercept and access potentially sensitive data.

Further, a CSP is specified on only 13.3% of websites that use online forms. CSP helps to mitigate the risk of Cross-Site Scripting (XSS) attacks, which could potentially compromise form submissions or steal personal data. Enforcing a strict policy on which scripts can be loaded and executed reduces the risk of malicious scripts accessing sensitive information. In Figure 4 we see that 86.7% of the websites in question do not protect their visitors in this way.

CSPs are specified via HTTP headers, as are other security measures like X-Frame-Options and X-Content-Type-Options. The X-Frame-Options header prevents clickjacking attacks by ensuring that your website cannot be embedded within a hacked frame. Malicious actors use clickjacking to trick users into submitting sensitive information, and disallowing framing ensures that a website’s forms are not exposed to this risk. Despite its importance, only 11.4% of websites with forms specify this header (Figure 3). X-Content-Type-Options works similarly, allowing websites to specify what kinds of browser actions are disallowed. ‘Sniffing’, for example, has been used for recognizing online file types but opens undue security risks. By setting this HTTP header to “nosniff,” websites can prevent browsers from incorrectly interpreting or executing content types—which could be exploited in drive-by downloads or content injection attacks—and helps protect user data from compromise. That said, only 18.4% of websites that collect information through forms specifically disallow sniffing via the X-Content-Type-Options header. 

Online forms are a modern and efficient way to collect information from various targets, and websites use the form builders that best suit their site and their purposes. The popularity of WordPress plugin form builders matches the general popularity of WordPress, while ‘experience management software’ options are also increasingly used for their broader feedback-collection capabilities. Security remains a crucial point when it comes to the websites using forms, but at least the percentage of valid SSL certificates among them is relatively high. In any case, online forms are here to stay, and it is up to each party to ensure that their data collection processes are secure.  

Name, email, five-star review… we fill out online forms so often we barely notice them anymore. However, form builders and ‘experience management’ software are behind the scenes on all these websites. Which platforms and plugins are most popular across the web? The data reveals all—read on to learn more.

In top form: the 10 most-used form builders across the web

WordPress

Security

What is a Content Management System (CMS)?

A Content Management System (CMS) is a software solution designed to streamline the creation and management of digital content. From personal blogs to large-scale e-commerce platforms, CMSs provide the infrastructure needed to publish and maintain online content efficiently.

With over 100 different CMS platforms detected across 50+ million websites, the market is highly diverse, each solution catering to distinct business needs, technical preferences, and monetization strategies. The evolution of CMS technology has been driven by the demand for more flexible, scalable, and user-friendly solutions. Businesses increasingly prioritize low-code or no-code approaches, security, and integrations with emerging technologies such as AI-driven content automation. As competition in the digital space intensifies, understanding the landscape of CMS offerings is crucial for businesses looking to optimize their content strategy.

Figure 1 shows the most widely used CMSs across the internet. Choosing the right one for your specific needs ensures a strong foundation for managing your content effectively.

Despite the multitude of options, a few key platforms dominate usage across the web. As shown in Figure 1, Wix (9.6%) and Squarespace (8.0%) are among the most popular choices for users seeking all-in-one, hosted website builders. E-commerce-focused platforms such as Shopify (4.5%) and GoDaddy Website Builder (4.8%) also hold significant market share, catering specifically to businesses looking for ready-to-use solutions that simplify website management. Traditional CMS platforms like Joomla (1.4%) and Blogger (1.0%) remain in use but have seen a decline in popularity as newer, more user-friendly solutions gain traction. However, despite the growing competition, WordPress continues to dominate the market, powering nearly 60% of all CMS-based websites. Far surpassing any other platform, this dominance has firmly established WordPress as the industry leader.

The rise (and reign) of WordPress in the CMS landscape

For years, the market leader in CMS solutions has remained unchallenged: WordPress. Originally developed as an open-source blogging platform, WordPress has evolved into a versatile system supporting a wide range of use cases, including forums, learning management systems (LMS), and e-commerce.

As an open-source platform, anyone can contribute to its code, extend its functionality with plugins, or design themes to customize its look and feel. Best of all, it's completely free to use. This has led to a digital ecosystem where countless businesses thrive by offering WordPress-related services, from hosting to theme and plugin development.

No other CMS comes close to matching WordPress in terms of community size, functionality, and years of development. The numbers speak for themselves. Figure 2 illustrates WordPress’s continuous growth over the past few years.

For an even deeper dive into WordPress’s dominance, take a look at our previous article with a detailed breakdown of the WordPress versions on the web.

WordPress.org vs. WordPress.com: what’s the difference?

One thing not covered in the previous article is the difference between WordPress.org and WordPress.com, two separate entities that often cause confusion.

WordPress.org is the home of the WordPress CMS, where the open-source community lives. Here, users can find themes, plugins, and documentation to customize and expand WordPress. This version is self-hosted, meaning users need to install it on their own web server. It is maintained by the WordPress Foundation, a nonprofit organization founded by Matt Mullenweg, the creator of WordPress.

WordPress.com (Managed WordPress Hosting)

WordPress.com, on the other hand, is a managed hosting service that simplifies website creation. As the internet and WordPress gained popularity, more individuals and businesses wanted their own websites, but not everyone had the technical knowledge or time to coordinate hosting, security, and updates. WordPress.com (and managed hosting providers in general) emerged to solve this problem by offering a fully managed experience where users don’t have to worry about servers, security, or maintenance. WordPress.com is owned by Automattic, a privately held company founded and led by Matt Mullenweg.

WordPress.org (Self-Hosted WordPress): You install, customize, and manage everything yourself.

WordPress.com (Managed WordPress Hosting): Hosting, security, and updates are fully handled for you.

The power struggle that could lead to a new CMS

Recent events have sparked controversy around Matt Mullenweg and, by extension, Automattic. As the leader of the WordPress open-source project, Mullenweg plays a central role in shaping its future. Automattic, his company, actively contributes to WordPress’s development to ensure its longevity—but, of course, also has a strong business interest in growing its market share in the managed WordPress hosting scene.

One of the most significant disputes involves a major WordPress hosting provider, WP Engine, and Automattic. The two companies are now locked in legal battles over trademark issues, with Automattic demanding 8% of WP Engine’s revenue. In a blog post, Mullenweg also criticized WP Engine for not contributing to WordPress’s open-source development.

In a controversial move, WordPress.org blocked update requests from sites hosted by WP Engine in September 2024. When a WordPress site—whether managed or self-hosted—checks for plugin updates, it has to contact WordPress.org servers to fetch the latest versions. Blocking this access effectively prevented WP Engine customers from updating their plugins, leading to security risks and operational issues for affected websites. Access to updates has since been restored for WP Engine and its customers.

As a result, after years of steady growth, WP Engine experienced a notable decline. This is clearly illustrated in the figure down below.

This dispute has raised concerns within the open-source community. Among them, Joost de Valk, creator of the widely used Yoast SEO plugin, has publicly voiced his concerns. In a blog post, he called for a leadership change, arguing that no single individual should have full control over the future of an open-source ecosystem.

Mullenweg responded in a follow-up post, suggesting that those dissatisfied with WordPress leadership could create their own alternative, going so far as to propose the name “JKPress.”

WordPress’s overwhelming market share raises important questions about the future of the CMS landscape. With nearly 60% of CMS-based websites relying on WordPress, does its dominance stifle innovation, or does it provide a stable foundation for the web? As businesses continue to demand more specialized and user-friendly solutions, will WordPress adapt to meet these needs, or will emerging platforms carve out new market segments? The rapid rise of platforms like Shopify in e-commerce and the growing popularity of no-code website builders such as Squarespace and Wix suggest a shift in how users approach content management. While the market is currently dominated by a few key players, history has shown that even the most established platforms can be disrupted. Could a new, lightweight CMS gain traction and challenge the status quo? Perhaps we’ll even see something like “JKPress”, rising unexpectedly from the sidelines to become the next big contender.

What happens when one guy has influence over 60% of CMS-based websites? The WordPress.com v.s. WordPress.org v.s. WP Engine case is one component of a larger trend: CMS platforms are increasingly popular, but despite growing competition, a few major players hold the largest market share. Read on to learn more about CMS, WordPress, and the data behind the news.

The CMS market faces uncertainty: the complicated case of WordPress

Hosting

Live chat tools have become indispensable for businesses aiming to enhance customer support, sales, and marketing efforts. Despite their advantages, the adoption of live chat software remains relatively low, with only around 16% of business websites globally implementing such tools. 

What are the top live chat solutions used on the web today?

Among the various live chat solutions available, WhatsApp Business has emerged as the dominant player, capturing 56.2% of the market share among websites utilizing live chat software. WhatsApp Business’s popularity stems from its wide adoption globally (Figure 2), easy integration with existing business processes, and, important for privacy, end-to-end encryption. Owned by Meta Platforms Inc., WhatsApp Business benefits from the global familiarity many users have with the WhatsApp messaging app, which further solidifies its position as a leader in the live chat market.

In the competitive landscape of live chat tools, Tawk.to holds the second position with 6.3% market share, followed closely by Facebook Messenger at 6.0%. Other notable players include giosg (5.5%), Live Chat Inc (5.0%), Zoho SalesIQ (2.6%), Tidio (2.6%), JivoChat (2.1%), and Crisp (2.1%). These solutions, while holding smaller shares, cater to specific business needs with unique features like multilingual support, AI-driven automation, real-time visitor tracking, lead scoring, and seamless CRM integrations. Meanwhile, the tools in the “Other” category accounts for 11.6% of the market and comprises another 56 unique live chat software.

Meta-owned tools dominate on e-commerce websites

When focusing specifically on online stores, the market share of WhatsApp Business remains comparably high, with 55.7% of online stores using it as their preferred live chat tool. Facebook Messenger follows second at 11.3%. Both of the Meta-owned platforms are free to use, reducing barriers to entry for micro, small and medium-sized businesses. Additionally, the familiarity of these platforms among users and business owners alike makes them accessible and easy to adopt, allowing businesses to connect with customers in a way that feels natural and intuitive.

The fastest growing live chat tools based on installation numbers in Q4 of 2024 were giosg, Chatilo, and Alive5 with 87%, 18% and 10% growth, respectively. Giosg’s AI Copilot integrates GPT-3.5 generated answers into their live chat solution and promises high security and data protection, which may be a major selling-point for businesses with complex products where an FAQ page doesn’t cut it. Chatilo is a live chat app that can be integrated with Slack; this integration makes it a useful tool to handle client and customer interactions within a company’s existing chat platform and without the need for separate software. Alive5 combines automation, AI, and human agents to assist website visitors and supports business-class SMS texting and web chat, offering a comprehensive communication solution for businesses.

Geographical adoption of WhatsApp Business

Geographically, WhatsApp Business live chat has seen widespread adoption—particularly in Asia, Europe, and Latin America—with significant usage in emerging markets like Brazil, Russia, India, and Indonesia. This global reach has contributed to its dominance in the live chat market.

Integrating live chat functionality on a website both enhances customer engagement and contributes positively to SEO performance. Features like real-time assistance, proactive engagement, and AI-driven chatbots can lead to longer site visits and improved conversion rates. These factors signal to search engines that a website provides valuable content and a positive user experience, potentially boosting its search engine rankings and, ultimately, a company’s online success.

While the adoption of live chat tools is still limited, platforms like WhatsApp Business have demonstrated significant market penetration due to their robust and accessible offerings. As the digital landscape continues to evolve, the integration of live chat and AI-driven communication tools is poised to play a crucial role in shaping customer service strategies and enhancing online user experiences. 

“Hi! How can we help you today?” Not just limited to physical stores anymore, live chat tools on websites provide quick and helpful customer service to online visitors. But with so many options, who are the leading live chat software providers, and what makes them different? Join us as we dive into the details about Meta’s market share and competitors’ distinctions. 

Top live chat tools revealed: WhatsApp Business takes the crown

Business websites

There are all kinds of behind-the-scenes things going on online, including ‘secret messages’ for different kinds of website visitors. If you’ve had a look around on our blog before, you’ll know all about the humans.txt file message for crediting the humans behind a website’s creation, ads.txt for advertising protocols, and security.txt as the first stop for cybersecurity-related queries. But since the launch of a certain AI helper in November 2022, there’s been a new digital visitor on the online block: Large Language Models (LLMs) are collecting all kinds of information from the internet—about websites, companies, products and services, tips, tricks, and everything in between. But are these LLM-collected pages of information the right details from each website? How do they know which webpages have the most relevant information? 

It makes sense that website creators should be able to communicate with LLMs to make sure they’re retrieving high-quality answers, should future visitors first come in contact with their products, services, or advice via ChatGPT or another GenAI colleague. Introducing the newest website ‘secret message’: LLMs.txt.

As LLMs and crawlers traverse the web and collect information, they’re looking at the first pages of your site — they don’t know if your ‘about us’ page is more important than that obscure blog you wrote five years ago, and it certainly isn’t going to check every single page on your site either. Just as robots.txt indicates how crawlers can index by setting rules and directing to sitemaps, the point of LLMs.txt is to help an LLM find the best information about your website in the most direct way. This could be a list of prioritized pages with relevant details about your site, links to documentation, or a collection of quick points or resources. See a comprehensive example here, on an AI observability platform’s website. 

As with other kinds of .txt files, this one can also be added to your website root as a markdown file. According to llmstxt.org, it should include your website’s title and description, and perhaps some important details about your business’s products or services, if applicable. Next come the links to documentation, core information, or anything relevant you want to be sure an LLM could answer or share if asked. Keeping structured information on hand and easy to access in LLMs.txt files benefits LLMs. This allows them to easily access specific details, like software library features, corporate structures, legal information, personal CV details, product categories, or educational course offerings. Jeremy Howard of llmstxt.org writes: 

Creating a LLMs.txt on your website is not difficult and could bring many benefits, so is there any reason not to have one? On the one hand, this protocol is still very new and more or less still a proposal. Most LLMs likely aren’t scanning for it yet, which means the text might not even be seen. Not even Google has a page for LLMs.txt—but that doesn’t mean LLMs aren’t being considered. For example, Google currently sets rules for tools like ChatGPT, AnthropicAI, PerplexityBot, and Claude in its robots.txt file, as does the Guardian here. Most websites might start this way, including instructions for LLMs on a different page, rather than creating a whole new file. On the other hand, why not get ahead of the curve and do both? 

Keen to see which websites have already embraced LLMs online? Take a look at the growing list of examples here in the LLMs.txt directory.

The way we use the internet is changing: Large Language Models like ChatGPT or Claude can direct human users to websites without a search engine in the middle. But how do LLMs know what site information is actually relevant to the prompt? LLMs.txt seems to be the answer—a way to specify exactly what information LLMs and crawlers should gather and ultimately share if someone asks. Read on to find out more.

Optimizing websites for the future of search? LLMs.txt explained

How to

Typography on the internet has come a long way from the early days of limited fonts and clunky user interfaces. Today, web fonts are an integral part of design, influencing how users interact with websites, how content is consumed, and how brands are perceived. As web design continues to evolve, so does the role of fonts, with new trends, technologies, and practices emerging to meet the growing demand for both aesthetic and functional digital experiences.

In the early days of the internet, web typography was constrained by the availability of “web-safe” fonts—those widely installed across most users' systems—like Arial, Times New Roman, and Courier. This limitation was necessary to ensure consistent rendering across different browsers and operating systems. Using an unrecognized font could mean the text appeared wrong or illegible, or not at all. However, it stifled creativity, as designers were unable to use unique fonts that matched their personal or brand’s identity.

The turning point came in 2010 with the introduction of the Web Open Font Format (WOFF), which allowed fonts to be embedded directly into web pages and meant that all fonts could appear consistently. This was further advanced by the same-year launch of Google Fonts, which provided designers and developers with free access to hundreds of font families. The adoption of these technologies revolutionized web typography, enabling designers to go beyond the basic, “safe” options and experiment with more diverse and expressive typefaces.


With Dataprovider.com’s proprietary tech, we can detect fonts on a website by looking at a CSS property named ‘font-family’. We collect all the values of this property, and the resulting list contains all the fonts specified in the stylesheet(s) on that particular website. This doesn’t necessarily mean all of them are in use, but it gives a good idea of the fonts found most commonly across the web. 

Let’s take a look behind the scenes: for our sample, we exclusively focus on business, e-commerce and content websites, omitting placeholders and parked domains. In Figure 1, we show the top 10 fonts used on websites as of October 2024. Font Awesome is the overall winner, which, for the typography folks, might be disappointing as it isn’t really a letter-based font. Font Awesome is a standalone resource widely used for adding visual elements like social media logos, navigation icons, and user interface elements without relying on traditional image files. This aside, Open Sans is then the actual winner when we just look at letter-based typography. Roboto, Helvetica Neue, and Montserrat rank third, fourth, and fifth in our top 10.

For those interested in the fonts that are less commonly used, you can explore the word cloud in Figure 2—the smaller the word, the rarer the font. 

Next, we take a look at the most commonly used web fonts by country, as seen in Figure 3, and website category, shown in Table 1. There is limited variation overall, with certain fonts showing regional preferences: the Helvetica font is widely used on Canadian websites, Roboto is popular in Russia, and unsurprisingly, Noto Sans KR is the top choice in South Korea.

Our analysis reveals distinct font preferences across different website categories. In Table 1 we see that websites focused on Art & Design and Media content predominantly use Helvetica Neue, more than any other category. This could be because artists, designers, and media professionals are more likely to create their own icons and visual elements than those in other industries who rely on tools like Font Awesome. In contrast, sites with religious content tend to favor Open Sans as their top choice. Meanwhile, Microsoft YaHei emerges as the most commonly used font on adult-themed websites.

Taken together, our analysis shows that even though there is a wide variety of fonts available across the web, there is surprisingly little variation on a global scale. When it comes to the most commonly used fonts, Open Sans, Roboto, and Helvetica are popular choices no matter the content or cultural background. Font Awesome—although not strictly a font—has universally emerged as the go-to tool for displaying visual content on websites. 

We found that some fonts stood out due to their strong association with specific brands, many of which have developed proprietary typefaces. Just as in the physical world, many brands have created digital versions of their custom fonts. For instance, Chanel uses a proprietary font that appears as AB Chanel Couture online. However, custom typography isn't limited to fashion houses: Amazon employs Amazon Ember, DHL uses a font called Delivery, and the financial company Bradesco has its very own Bradesco typeface.

Proprietary fonts are especially popular among car manufacturers, such as Bentley and MINI. Audi takes online branding further, not only by using a custom font, but also providing its dealerships in Germany with ".audi" domain websites, offering a fully branded digital experience.

The Power of Fonts: Visual Hierarchy and Readability

Fonts play a crucial role in establishing a website's visual hierarchy, readability, and overall aesthetic. Typography affects users’ first impressions of a site and influences their behavior as they navigate through content. Font choices can guide the user’s eye to key elements or distract them, potentially leading to higher bounce rates if the text is difficult to read or the layout is cluttered.

One of the most significant challenges in web typography is balancing style with readability. Designers must consider font size, line height, and letter spacing, as well as the font's overall style, to ensure that text is easy to read across different screen sizes. Yet, accurately measuring legibility is difficult as factors such as type size, line spacing, line length, and typestyle, to name a few, interact in complex ways.

Accessibility remains a major issue, with nearly 96% of websites failing basic accessibility standards. The Web Content Accessibility Guidelines (WCAG) recommend readable fonts, adequate contrast ratios, and scalable text sizes.

Responsive typography, using CSS media queries and frameworks like Bootstrap, helps ensure text remains legible across various devices by adjusting dynamically based on screen size.

Some remaining challenges in Web Typography

Despite advances in web fonts, designers still face challenges, particularly with performance and accessibility. For example, large font files can slow down websites. Tools like Glyphanger or the practice of font subsetting can optimize loading times by reducing font file sizes, while variable fonts—combining multiple styles in one file—also speed up load times. From 2020 to 2022, the number of websites making use of variable fonts increased by a whopping 125%. 

The use of fonts on the web has evolved from basic, web-safe options to a diverse landscape of custom and variable fonts that enhance both aesthetics and functionality. The widespread adoption of web fonts, coupled with new technologies like variable fonts and responsive typography, allows designers to create more engaging and accessible online experiences. However, our data shows that despite the abundance of available fonts, most industries and cultures opt for similar font options unless the local script requires a specialized typeface. For brands and websites to stand out, developing their own typeface helps create a unique brand identity offline and online and can take branding to the next level. 

Reading between the lines of web typography reveals more than you expect—for example, the most-used online font has no letters or numbers, and Helvetica Neue has a hold on two industries in particular. The internet and the way we use it has changed over time, and even unassuming elements (like online fonts) are a cornerstone of our digital experience. Dive into web typography history in our latest blog.

From fonts to function: mapping web typography trends and challenges

We've been witnessing a fascinating trend in the digital landscape: the rise of casual mini-games across a wide array of online platforms. This trend isn't just a passing moment: it has truly reshaped how we interact with websites and apps, and it's changing the way companies engage with their users and keep them engaged.

The New York Times: from paper crosswords to mobile Wordle 

One of the most visible and well-known examples of the mini-game ascent comes from the New York Times. NYT has long been known for its iconic crossword puzzles, first published in 1942, which set the stage for daily brain-teasers. Its enduring popularity demonstrates the long-standing appeal of quick, daily mental challenges, and their foray into modern mini-games has catapulted them into the online spotlight. The trend started before the pandemic already—the daily Mini Crossword and Spelling Bee are global favorites—but it was the acquisition of Wordle in 2022 that truly put NYT games on a new scale. This game was unique and the results easy to share, spoiler-free. NYT’s casual game offerings have only continued to grow in popularity since then.

The trend isn't limited to news outlets. Professional networking site LinkedIn has introduced a few casual games that refresh daily, and even e-commerce platforms like bol.com have incorporated games into their apps. The OEC, an economic data platform, has brought in regular visitors with their games like Tradle and Connectrade (homages to NYT’s Wordle and Connections). The spinoffs don’t stop there: Worldle for the geographers-at-heart, Heardle for the more musically inclined, Loldle for the gamers, Waffle for our smartest friends, and Quorlde for the eccentrics among us—just to name a few.

They're quick, tricky enough to feel challenging but not so difficult as to cause frustration, and effortlessly shareable. With usually only one new puzzle per day, the process of completing them provides a sense of downtime without feeling unproductive. Everyone attempts the same puzzle and there are no ‘levels’; the results—your score—can be transformed and emoji-fied to make for spoiler-free comparison among family, friends, or colleagues. This limited availability and global consistency creates a shared experience among players, fostering both a sense of community and a dose of friendly competition.

Of course, there is nothing inherently wrong with playing and enjoying mini-games, but the context of their presence raises some questions. The NYT claims these games drive traffic to their platform and provide levity from a heavy news cycle, but they've also created a separate app for their games, which effectively results in less traffic to their site and main platform. This widespread adoption begs the question: where did this trend come from, and where is it going?

From the early days of flash games to the massive gaming industry we see today, playing games on the internet has always been a core part of the online experience.

However, what we're seeing now is different. These new mini-games provide quick dopamine hits, often taking just 30 seconds or two minutes to play. They're designed for short attention spans and busy lifestyles, often integrated into apps we use daily. The rise of mini-games and its corporate encouragement isn't accidental: it's a response to changing user behaviors and platform needs. With overall engagement declining and users spending less time on individual sites (besides the big social media platforms), these games serve as a way to increase user return, retention, and time spent.

But is this trend a solution or part of a broader problem?

A little deep dive into games online: the case of Indonesia

We took to Dataprovider.com’s Search Engine to see what we could find about how games are mentioned online. We can see a general rise in “games” being referred to across the internet, but there’s one trend line in particular that we couldn’t ignore: Indonesian websites and websites based in Indonesia have seen a meteoric rise in the mentions of games online since 2021. In Figure 1, we can see that Indonesia’s line starts rising at the end of 2021 and continues steadily upwards until the end of 2023. Then, from the beginning of 2024 until now, there’s been an exponential growth in “games” mentions that overtakes the numbers of the UK and Canada—even combined. We tested our data for an increase in English-language Indonesian websites, and discovered that no change was present that would account for the rise. No changes in our internal methodologies could explain the changes either, which sent us to the news to investigate. 

Two unique explanations came to our attention; two answers that seemed on opposite sides of the same coin. On one side of the coin, we have to take a look at gambling—an activity that is illegal for both locals and tourists in Indonesia for religious reasons. However, online gambling has become a problem in the country: in lieu of (legal) traditional casinos, the gambling that happens tends to be in the form of mobile apps and sites, where users can play gambling ‘games’ with ‘real stakes.’ These platforms create considerable political and social concern about gambling gamification, and a governmental task force was set up earlier in 2024 to address the issue. For better or worse, this helps explain the prevalence of “games” on the Indonesian internet. 

The other side of the coin paints a much happier picture, giving context to the data we see in Figure 1. It starts with the “Perpres 19/2024”, a presidential regulation and government initiative for supporting Indonesian mobile/video game developers. The Perpres was announced in 2021—aligning with when the orange trend line starts to rise—and more actively enacted in 2024, essentially when the exponential growth begins. With a large and active online (gaming) community in Indonesia, the initiative is intended to keep more gaming revenue in the country by supporting talent development, funding indie game development studios, and more. Some of the most popular games from Indonesian studios are known around the world (Coffee Talk and Coral Island, especially!). From this angle, the increased mention of “games” on local websites can be clearly attributed to this giant boost to Indonesia’s gaming industry. 

Neither of these explanations are the same kind of mini-game as what the NYT offers, but this case tells us plenty about the way people use the internet today: to find games, to play games, and to discuss games in blogs, articles, and comment sections across the world. At the same time, they all add together as a way to keep people and their activities firmly online. 

It appears that (mini-)games are the newest addition to the attention economy. Just as the printers and publishers before them, online platforms are competing for users' limited time, focus, and screen space. The seemingly harmless venture into casual games—especially from news providers—can spur a sense of service dilution, where platforms expand too far beyond their core offerings, just to keep users engaged. 

It risks becoming a symptom of content collapse, a modern phenomenon where all information flattened and given more or less the same importance. For example, it occurs when heavy news is placed alongside clickbait and quizzes, which can lead to a kind of numbing effect: there’s less emotional room to deal with dark themes, and the joy of puzzling is damped by negative events.

As we look to the future, several questions emerge:

Will mini-games become a standard feature across all types of digital platforms?

How will these games evolve to maintain user interest and generate income for their hosts? 

What impact will this trend have on user behavior and expectations?

How will platforms balance their core services with these additional engagement tools?

The rise of mini-games online represents a significant shift in how we interact with digital platforms. In our article on GenAI Transparency, we explored the impact of the ‘time spent’ metric (and how it interests shareholders): it's not hard to imagine that the interest in keeping users on the platform will take precedence over the reason the users are on the platform—as long as money is being made, and the metrics show growth, quantity will remain above quality. 

But that’s the cynical take, and it could just as well be that mini-games will stay casual, just a fun addition and well-deserved source of levity online. They offer benefits in terms of engagement and user retention on the corporate level, and in terms of a brain-break for users, but of course, casual mini-games also raise important questions about the future of online content and user experience. For those of us on the player side of mini-games, it comes down to being intentional and paying attention to the context in which the games are. It’ll be fascinating to watch how they shape the digital landscape in the years to come.

This article is part of the Modern Internet series, where we dive into today's digital culture and the online landscape. Check out Internet Nostalgia for a head start on finding great spaces to hang out online, or read our piece on GenAI Transparency to learn about maintaining the Living Internet. If you’re curious about the importance of entropy to security, take a look at Low-Tech Internet (and enjoy the lava lamps). 

Crosswords, brain-teasers, puzzles, and more: it’s hardly a surprise to find a mini-game or two, tucked away in news and social media platforms. But what does this mean for how we engage with the modern internet? And why are games so popular on the Indonesian internet, in particular? Take a look at our latest blog for insights about mini-games, modern engagement, and the attention economy.

Mini-games and modern engagement

Culture

The .ai domain has experienced an unprecedented surge in popularity, with the past few years seeing registrations skyrocket. Initially affiliated with the small Caribbean island of Anguilla, the .ai extension has become a highly sought-after digital asset, largely due to its association with artificial intelligence. Tech companies, startups, and even large corporations have rushed to secure their slice of this internet real estate, making it a go-to choice for AI-related branding. We look into the data to see what’s really behind .ai domains and if the rise is just a hype.

Follow us

Questions?

Insights from web data

Industries

Do most online stores have a physical store as well?

Where have all the placeholders gone?

Uncovering networks of websites spreading misinformation about COVID-19

Explore more topics

Recent videos

Jorn talks Trust Grade

Jorn talks Search By Example

Feature Launch | Search By Example