Memory System Design and Cache Memory

What is memory Hierarchy?

The Memory hierarchy is a hierarchical representation of computer memory according to some calculations such as access type, capacity, cycle time, latency, bandwidth, cost, etc. In the memory hierarchy, from top to bottom the latency and the bandwidth, capacity increase. but from bottom to top the speed and the cost per bit increase. CPU Register is the most powerful and expensive memory in the hierarchy. for more details see the image attached below.

Parameters of the memory hierarchy

Access type - The action that physically takes place during a read-write operation

Capacity - The storage size that the device has. It is measured in bytes.

Cycle time - The time elapsed from the start of a read operation to the start of a subsequent read.

Latency - The time interval between the request for information and the access to the first bit of that information.

Bandwidth - A measure of the number of bits per second that can be accessed.

Cost - Dollar per megabyte

Locality of Reference - A computer program tends to access memory locations that are near to each other and with a high frequency. It improves the performance of the computer. A computer system can optimize memory management and caching algorithm to reduce the number of cache misses and improve overall performance. There are 2 locality types.

Spatial Locality - The tendency of a program to access memory locations that are close to each other in space. 

( ex: array - Consecutive instruction in a straight line program)

Temporal Locality - The tendency of a program to access the memory location that it has recently accessed. 

( ex: loop - an instruction in a program loop )

The Average memory access time ( tav )

• The probability of finding the requested item in the first level is called the hit ratio, h1
• The probability of not finding the requested item in the first level of the memory hierarchy is called the miss ratio. ( 1 - h1 )
• If it is a miss, it will search the data in the next level of memory hierarchy. 
• In the second level

hit ratio - h2
miss ratio - ( 1 - h2 )

Likewise, the process will do the same until the item is found.

Cache Memory ( Slave memory )

Cache memory is a small high-speed memory that is located near the CPU. When the CPU needs data/instructions to process, then it searches CPU registers first and if the data is not in registers, then it checks data in the cache memory. likewise, it goes to other levels if the data is not there. Also, all the data/instructions in the main memory pass to the CPU registers through the cache memory. Because of this process, the CPU can continue the processes without any delay. Cache memory improves the performance of the computer. So we can say that the cache memory act as a middleman between the processor and the main memory.

• Cache hit - If the requested items are currently in the cache.
• Cache miss - If the requested items are not in the cache.

hc - Cache hit ratio

( 1 - hc ) - Cache miss ratio

If it is a cache miss, a block of requested elements from the main memory will be moved to the cache. It transfers the data as blocks to take advantage of spacial locality.

Bandwidth

The amount of data that can be transferred between the memory and the rest of the system in a given amount of time.

Memory Interleaving

A technique that is used to increase the bandwidth.

main memory access time - tm

cache memory access time - tc

Impact of temporal locality

Assume that instructions in the program loop n times

if n increases, then tav decreases

Impact of spatial locality

Assume that instructions in the program loop n times

if m increases, then tav decreases

Impact of combined temporal and spatial locality

Assume that size of the block transferred from the main memory - m

Assume that tm = mtc. then,

   

Size of the block - m elements

time of the memory access - tm

time of the block access (cache --> processor) = mtc

loop time - n

In conclusion here we provided the complete guide for how the memory hierarchy works and how the cache memory works. Let's explore more about cache memory and data-storing methods in cache memory in the next post. 🍏🐝