Introduction to HTTP Header Fields & Response Codes | Lesson 15/34 | Semrush Academy

You will learn about HTTP header fields, which transmit the parameters and arguments important for the file transfer via HTTP protocol.
Watch the full course for free: https://bit.ly/3gNNZdu

0:17 Hypertext Transfer Protocol
0:54 Different header fields
2:06 X-robots headers
2:23 Retry-After
2:33 Caching headers
2:39 HTTP status codes


✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹
You might find it useful:
Tune up your website’s internal linking with the Site Audit tool:
➠ https://bit.ly/2XVxCmL
Understand how Google bots interact with your website by using the Log File Analyzer:
➠ https://bit.ly/3cs0rfC

Learn how to use SEMrush Site Audit in our free course:
➠ https://bit.ly/2Xsb3XT
✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹ ✹

In this chapter, we are talking about HTTP headers in general and respective HTTP header fields which are essential components of the Hypertext Transfer Protocol (HTTP) header. Those fields transmit the parameters and arguments that are important for the files transfer via HTTP, e.g. the desired language, the character set or often information about a client in general. The individual fields of the header are always transferred after the request line or the response line. The lines of the header itself are key-value pairs separated by colons (e.g. Content-type: text/html). The names are specified by different standards.

There are lots of different header fields; many of them are not really relevant to any SEO work, so we'll ignore most of them for now.

What you need to understand and keep in mind is that headers are invisible to regular users, so they will not see them while visiting your website. But from an SEO perspective it is important to understand which headers are present and which headers have been sent back from the server. One of the easiest ways to do this is to use the Chrome Developer Console. Once you've opened the console, you go to the network tab and then select a URL on the left-hand side. On the right-hand side there is the “Headers” sub-tab. You can see the request and the responses respectively, and also which types of key-value pairs have been sent back and forth.

If you don't want to do it in the browser and maybe prefer a website or generally a service that can check multiple URLs at once, then http://httpstatus.io is very helpful.

From an SEO perspective, there are four groups of headers that are really relevant:

X-robots headers: Crawler directives like noindex as well as canonical tags for non-HTML pages. So for example you could do noindex for PDF files only using headers because there is simply not HTML markup available to do so.
Retry-After: The specified time tells Google when it should crawl again (no sooner than that)
Caching header: Cache-Control, ETag, Expires, etc.
Last, but not least, HTTP status code: e.g. 200=success, 3xx=redirects, 4xx=not available (anymore) & site in maintenance etc.
There are two simple ways to review your websites' pages in terms of HTTP status codes by using SEMrush's Site Audit. First, under the Statistics tab you can see a widget with status codes distribution along the webpages, finding out how many pages are returning a certain code. Second, you can get lists of pages returning different status codes by filtering them out in the Crawled Pages tab.

So this chapter is going to mainly cover http status codes and more precisely three different areas:

3xx range - these are all types of redirects.
4xx range - errors on the client's side.
5xx range - errors on the server level.

#TechnicalSEO #TechnicalSEOcourse #HTTPheader #SEMrushAcademy