How RAM Caches, Buses and Virtual Memory Work

In this video from ITFreeTraining, I will look at how RAM caches, buses and virtual memory work in a computer system. Understanding how these work, will give you a better understanding of how to fix performance problems and to understand what sort of performance you may expect to receive.

Download PDF Handout http://itfreetraining.com/handouts/ap...

Memory Hierarchy
0:19 Before I start looking at how caches and buses work in a computer system, I will first look at the memory hierarchy. This will give you an understanding of why these systems are designed the way they are.

At the top of the hierarchy are CPU registers. When a computer wants to perform an operation, for example adding numbers together, the data is stored in the CPU registers. CPU registers are small in number. They are extremely fast and only used to temporarily store data used when the computer is performing calculations or running code.

At the next level is CPU cache. CPU cache is very fast but there is not much of it. CPU cache is measured in megabytes and varies depending upon the CPU.

If the data cannot be found in the CPU cache, the next step is to access the main memory of the computer. This is slower again, but the advantage is that there is more of it.

Lastly there is virtual memory. This is usually stored on a device like a hard disk. This is slower again, but very cheap.

So what does this all mean? Essentially the top of the pyramid is the fastest and as you go down, the speed decreases. The advantage, however, is that it becomes cheaper and, thus, generally there is more of it. It comes down to a trade-off between cost and performance. Now let’s have a look at how this all work together.

Executing Code
1:51 First of all, I will look at how a CPU may execute code. Different CPU’s work in different ways, but what I will do is give a very simplified version of how a CPU may execute code that illustrates the steps that a modern CPU may go through in order to execute code.

First consider that the CPU has some cache. The CPU has multiple levels of cache, but for this example, let’s consider there is only the single cache to make it simple. In the cache is the contents of some memory that I will call memory2.

Let’s consider that the CPU is executing some instructions. Rather than using real computer code, I have simplified the instructions down to make them easy to understand. The first instruction will be to obtain the data in memory1 and store the data in register1. A register holds data temporarily so the data can be used in calculations. The CPU cache however does not have this data. The CPU will thus retrieve the contents of memory1 from the memory modules.

This will take some time. In older CPUs the CPU would need to stop executing instructions until it had the data. When this occurs, the CPU has stalled and is effectively waiting. Waiting means no work is being performed and thus is not ideal.

To help get around this, modern CPUs can execute instructions out of order. While the CPU is waiting for the memory to be retrieved, the CPU will look at the next instruction. In this case, the instruction is to add 1 to regsiter1. The instruction however cannot be executed until the CPU has obtain the data from memory1.

Rather than wait, the CPU can often look at a number of instructions in advance. The CPU will next look at the third instruction. The next instruction will be to get the data from memory2 and place it in register2. The contents of memory2 are already in the cache, so this instruction can be executed immediately.

Looking then at the next instruction, this instruction is adding 2 to register2. Since memory2 was available in the cache and has been placed in register2, this instruction will execute immediately.

The next instruction will add the contents of register1 to register2. In this example, we will consider that the CPU had looked ahead as much as possible and thus was not able to process any more instructions until memory1 is retrieved from the memory module. When this occurs the CPU stalls, that is, it is effectively stopped until the data is obtained.

Description too long for YouTube. Please see the following link for the rest of the description http://itfreetraining.com/ap/3a30

References
https://en.wikipedia.org/wiki/CPU_cache “CPU cache”
https://arstechnica.com/gadgets/2002/... “Understanding CPU caching and performance”
https://gist.github.com/jboner/2841832 “Latency Numbers Every Programmer Should Know”
https://techtalk.pcpitstop.com/2016/1... “Average PC Memory (RAM) Continues to Climb”
https://en.wikipedia.org/wiki/Page_(c...) “Page (computer memory)”

Credits
http://ITFreeTraining.com Trainer: Austin Mason
http://hplewis.com Voice Talent: HP Lewis
http://www.pbb-proofreading.uk Quality Assurance: Brett Batson