Robots.txt – one of the mechanisms of the Robots Exclusion Protocol. It’s a small file containing instructions for bots that scan the site, allowing or forbidding entry to certain areas of the site. One of the reasons this solution is used is so that the site is not overloaded with requests. By using it, you can block search engines from accessing pages that are not relevant to search results. Robots.txt is necessary, especially on large and complex sites containing a lot of dynamic content.
All phrases in the eCommerce dictionary: Abandoned cart | Anchor Text | API | Audit UX | AuthInfo | Affiliation | Affiliate marketing | B2B | B2C | Black Hat SEO | Bounce Rate | Breadcrumbs | Brand | back-end | branding | Cross-Selling | Cache | Copywriting | CPC (Cost Per Click) | CAC | CRM | CTA (Call to Action) | CTR (Click Through Rate) | CPA | CLV | CSV | Crawl budget | Case Study | Content | CRO (Conversion Rate Optimization) | Conversion Rate | Domain | DNS | Double opt-in | Dropshipping | Duplicate content | Dynamic remarketing | e-Commerce | Expiring domain | ERP | FAQ | FRONT-END | Framework | Favicon | Fanpage | Frame | Google Ads | Google Analytics | Google My Business | Google Search Console | Google Trends | Hashtag | HTML | HTTP | Heatmap | Hosting | Indexing page | Influencer | JavaScript | Conversion | Keyword | KPI (key performance indicator) | KSI (key success indicator) | Landing page | Lead magnet | Sales Funnel | Link | Link building | Layout | Lead | Marketplace | Marketing Strategy | Marketing Automation | m-Commerce | Multichannel | H headers | Networking | Newsletter | Omnichannel | Open Source | Organic results | Organic traffic | Online payment system | Off-Site SEO | On-Site SEO | Page Speed | Page 404 | Pagination | PPC (Pay Per Click) | PPL (Pay Per Lead) | PPS (Pay Per Sale) | PLA (Product Listing Ads) | POP-UP | Redirect 301 | PWA (Progressive Web App) | POS (Point-Of-Sale) | Proxy | Rebranding | Remarketing | ROAS (Return On Advertising Spend) | Robots.txt | ROI | RSS feed | RWD (responsive web design) | Scrum | SEM (Search Engine Marketing) | SEO (Search Engine Optimization) | SEO back-end | SERP | SEO Audit | SPAM | SSL | SXO (Search Experience Optimization) | SaaS (software as a service) | Session | SKU | SXO | A/B Testing | Tagging | Thin content | Trust rank | Targeting | TL;DR | Up-Selling | URL | UX | UI | Usability | Vlog | Watermark | Webinar | WooCommerce | WMS | XML