SSD

101
SOLID STATE DRIVES 101
Everything You Ever
Wanted to Know
About the Author Index
Steve Larrivee SECTION 1
VP Sales & Marketing Introduction - The Basic NAND Flash Cell 3
Cactus Technologies
SECTION 2
Steve Larrivee has over 30 years’ experience in the data storage Introduction - SLC, MLC and TLC NAND Flash 5
market, including 5 years at Seagate Technology and 10 years at
SanDisk. He joined Cactus Technologies Limited as an equity partner SECTION 3
and Co-Founded Cactus USA in 2007 with partner Tom Aguillon. NAND Architecture - Strings and Arrays 7
SECTION 4

NAND Architecture - Pages and Blocks 9

SECTION 5

NAND Architecture - Planes and Die 11

SECTION 6

NAND Architecture - Component Packaging 13

SECTION 7

SSD Controller Architecture - Basic Overview 15

SECTION 8
SSD Controller Architecture - Channels and Banks 17
SECTION 9

SSD Controller Architecture - Block Diagram 19

SECTION 10

SSD Controller Functions - Wear Leveling 22

SECTION 11

Copyright © 2016 by Cactus Technologies Inc. SSD Controller Functions - Garbage Collection 24
SECTION 12
All rights reserved. No part of this publication text may be uploaded
or posted online without the prior written permission of the publisher. SSD Controller Functions - TRIM Command 26
SECTION 13
For permission requests, write to the publisher, addressed “Attention:
Permissions Request,” to [email protected]. SSD Controller Functions - Over-Provisioning 28
 01 Introduction - The Basic NAND Flash Cell

This section takes a look at the basics of a NAND flash cell, the building The floating gate remains in its charged or uncharged state until it is
block of almost every solid state drive. It is the first of several sections changed by surrounding circuitry. Removing power from the NAND device
describing the basics of Solid State Drives (SSD). does not affect the state of the floating gate which is why it is such a
valuable device for data storage.
In order to store a single bit of data on a solid state drive, you need the
smallest building block - a single NAND flash cell. The simplest NAND cell
can be set to either a 0 or 1 state. It will continue to store that state even How to Read a NAND Cell
after power has been removed.
Control Gate

What does a NAND Cell look like? Control Gate

Insulator
Control Gate
Floating Gate
Control Gate Source (S) Drain (D)
Insulator
Insulator

Floating Gate Silicon Substrate

Source (S) Drain (D)
Insulator

To read a cell, voltage is applied to the control gate and current flow from
Silicon Substrate
the source to drain is attempted.

If there is no current flow, it signifies the floating gate is charged (binary

A simple NAND Flash Cell diagram is shown above. The NAND flash cell 0) - as in the diagram above. If there is current flow, the floating gate is not
is made from a floating gate transistor. Electrical charge is stored on the charged (binary 1) - as in the diagram below.
floating gate which is isolated above and below by oxide insulating layers.
In its simplest form when the floating gate is charged, it is programmed Control Gate
and recognized as a binary 0. When the floating gate has no charge it is
erased and recognized as a binary value of 1. Control Gate

Insulator

FLOATING REFERRED BINARY ASSIGNED Floating Gate

GATE STATE TO AS VALUE
Source (S) Drain (D)
Insulator

Charged Programmed Zero - 0

Silicon Substrate
No Charge Erased One - 1

Section 01: Introduction - The Basic NAND Flash Cell www.cactus-tech.com Page 3
 How to Write a NAND Cell NAND Cell Life
To write a cell, a high voltage is applied to the control gate and electrons The tunneling process described in the Write and Erase functions cause
move from the silicon substrate to the floating gate. This process is called stress on the oxide insulator layer. Over time this stress breaks down the
tunneling since the electrons “tunnel” through the oxide insulator to reach oxide layer and the floating gate becomes unable to maintain a charge. At
the floating gate. See diagram below. some point the cell is no longer usable and must be retired. This is what is
responsible for the finite number of writes/erases per cell of NAND flash.

Control Gate

Control Gate

Insulator

Floating Gate
Source (S) Drain (D)
Insulator

Silicon Substrate

Section 01: The Basic NAND Flash Cell

How to Erase a NAND Cell

To erase a NAND cell, a high voltage is applied to the silicon substrate and
electrons move from the floating gate to the silicon substrate. This uses
the same tunneling process as the writing process. See diagram below.

Control Gate

Control Gate

Insulator

Floating Gate
Source (S) Drain (D)
Insulator

Silicon Substrate

Section 01: Introduction - The Basic NAND Flash Cell www.cactus-tech.com Page 4
 02 Introduction - SLC, MLC and TLC NAND Flash

This section builds on the Basic NAND Flash Cell by showing the advances How MLC and TLC Store More than One Bit Per Cell
in technology from the original SLC to MLC, and TLC NAND Cells.
The previous example shows a SLC (Single Level Cell) NAND Cell. When
For a simple review of the Basic NAND Cell, charges are either stored or any current is detected between the source and drain it can be concluded
not stored on a floating gate which is sandwiched between two layers of the cell is programmed. Since only two states, programmed or erased, are
oxide which act as an insulator. needed to represent one bit, that’s all that is needed.
With MLC (Multi Level Cell) NAND, there is a need to store two bits of data,
Control Gate Control Gate which requires 4 distinct states. In order to accomplish this, the MLC NAND
cell must be able to apply charge to the floating gate at four different levels
Control Gate Control Gate
and later be able to detect which of the four levels is set.
Insulator Insulator
Floating Gate
The diagram below represents the additional electrons in blue on the
Floating Gate floating gate which must be set to precise levels so they can later be read
Source (S) Drain (D) Source (S) Drain (D)
Insulator Insulator accurately. This makes MLC more challenging and slower to write than
their SLC NAND counterpart.
Silicon Substrate Silicon Substrate
Control Gate Control Gate
No Current - Floating Gate Programmed Current Flows - Floating Gate Erased
Control Gate Control Gate

Insulator Insulator
On the original and simplest type of NAND flash, if no current flows between Floating Gate
Floating Gate

the Source and the Drain, it indicates the floating gate has a charge (blue Source (S) Drain (D) Source (S) Drain (D)
Insulator Insulator
dots represent electrons) and therefore is programmed, representing a
binary 0. See diagram above left.
Silicon Substrate Silicon Substrate
If current flow is detected, it indicates the floating gate does not have a
charge and is erased, representing a binary 1. See diagram above right. State 1 - No Charge State 2 - Lightly Charged

Control Gate Control Gate

Control Gate Control Gate

Insulator Insulator
Floating Gate Floating Gate

Source (S) Drain (D) Source (S) Drain (D)

Insulator Insulator

Silicon Substrate Silicon Substrate

State 3 - Medium Charged State 4 - Highly Charged

Section 02: Introduction - SLC, MLC and TLC NAND Flash www.cactus-tech.com Page 5
 TLC (Tri Level Cell and also known as Triple Level Cell) NAND has an even
more complicated mission. It must be able to store and recognize 3 bits
per cell, requiring 8 distinct states.

Voltage Level in SLC, MLC and TLC NAND Cells

The maximum voltage in each cell is about the same. So SLC cells have
plenty of guard band between their states. Because of this, SLC NAND is
able to withstand temperature extremes and other adverse effects much
better than MLC or TLC NAND.

SLC NAND MLC NAND TLC NAND

V 111
Total 11
(Total Voltage 110
Cell Capable
of Storing)
1 101
10
100
011
01
010
0 001
00
000

SLC MLC TLC

1-Bit/Cell 2-Bits/Cell 3-Bits/Cell

Voltage Allocated to each State based

on NAND Flash Technology

The Image above shows the levels of voltage thresholds required to store
the multiple states in each of the memory technologies. Not counting
guard band area, each SLC state is allocated 50% of the voltage range;
MLC 25% and TLC 12.5%.
As you can see, MLC and TLC have much tighter tolerances and will be
more susceptible to external environmental and circuit effects than SLC
NAND. Their principal advantage is cost.

Section 02: Introduction - SLC, MLC and TLC NAND Flash www.cactus-tech.com Page 6
 03 NAND Architecture - Strings and Arrays

The previous two sections focused on the individual NAND Cell, whether Combining Individual NAND Flash Cells into a String
used to store one, two or three bits. This article focuses on the bigger
All by itself, a single flash cell would not be of much value. But combining
picture of how numerous NAND cells are combined into strings and arrays.
many of them is what allows the storage of significant amounts of data.
For a quick review, a single NAND flash cell stores an electrical charge The first step in combining individual NAND cells is the NAND String.
on a floating gate which is isolated by oxide insulating layers above and
below. In its simplest form when there is a charge on the floating gate it is Control Gate Control Gate Control Gate Control Gate Control Gate Control Gate

programmed and recognized as a binary 0. When the floating gate has no Control Gate Control Gate Control Gate Control Gate Control Gate Control Gate

charge it is erased and recognized as a binary value of 1.

Insulator Insulator Insulator Insulator Insulator Insulator

Floating Gate Floating Gate Floating Gate Floating Gate Floating Gate Floating Gate
Source (S) Drain (D)
Insulator Insulator Insulator Insulator Insulator Insulator

Diagram of a Single NAND Flash Cell

Control Gate Control Gate Control Gate Control Gate Control Gate Control Gate

Control Gate
Source (S)
Drain (D)

Control Gate
NAND String (Shown in Diagram and Schematic Versions)
Insulator

Floating Gate
Source (S) Drain (D) The image above shows the NAND String depicted in both a diagram form
Insulator
and in schematic form. Schematic form is typically used to show much
larger arrays.
Silicon Substrate
NAND cells are connected end to end to form a string of cells. Typically 32
or 64 cells are connected together in series with each other, with each cell
representing a bit of data (0 or 1).

Section 03: NAND Architecture - Strings and Arrays www.cactus-tech.com Page 7

 Combining NAND Strings into Arrays While a NAND String can store 32 bits of data, this still only translates into
4 bytes of data or enough for 4 characters. So, strings are combined into
larger arrays to achieve more useful amounts of storage.

Drain (D)

Drain (D)

Drain (D)

Drain (D)

Drain (D)
The image to the left shows the NAND String schematic repeated several
times in an array. Notice the additional connections made to the NAND
strings which serve to tie the array together. The red line connects the
Sources (S) of the individual strings.
The yellow lines connect the Control Gates of the NAND strings. In the
array, the control gates are connected horizontally, but not vertically. In
addition, the Drain (D) lines are not showing connections since they will be
used separately in the array.
Sources (S)

Sources (S)

Sources (S)

Sources (S)

Sources (S)

NAND Flash Array

Section 03: NAND Architecture - Strings and Arrays www.cactus-tech.com Page 8

 04 NAND Architecture - Pages and Blocks

The previous sections focused on the individual NAND Cell, NAND strings
and arrays. This section digs deeper into arrays and builds up to the page Bit Line Bit Line Bit Line Bit Line
and blocks of NAND flash.
String Select Line
For a quick review, single NAND flash cells individually storing a single bit
of 0 or 1 are joined together in strings and arrays to form much larger
data storage structures. These strings are connected to allow storage and Word Line
retrieval of data from selected cells. Very large data storage devices are
made possible by adding more and more NAND cells to the array.
Word Line
The NAND Flash String
The next Image shows the much larger array with control and data lines Word Line
as well. Highlighted in green with a yellow background is the NAND String
discussed in the previous article. There are many strings in between
depicted as dots. Word Line
Strings (shown as columns) are the minimum unit to read and are typically
comprised of 32 or 64 NAND cells. All strings in the array are connected
at one end to a common Source Line (SL) and at the other end to the Bit
Line (BL).
Word Line

Word Line

Ground Select Line

Source Line

NAND Block
Each string also contains two control mechanisms in series with the NAND
cells. String and ground select transistors are connected to the String
Select Line (SSL) and Ground Select Line (GSL).

Section 04: NAND Architecture - Pages and Blocks www.cactus-tech.com Page 9

 The NAND Flash Page The NAND Flash Block

Bit Line Bit Line Bit Line Bit Line Bit Line Bit Line Bit Line Bit Line

String Select Line String Select Line

Word Line Word Line

Word Line Word Line

Word Line Word Line

Word Line Word Line

Word Line Word Line

Word Line Word Line

Ground Select Line Ground Select Line

Source Line Source Line

NAND Block NAND Block

This image above shows the NAND Block with green lines and yellow
The image above shows the NAND Page with green lines and yellow highlighting.
highlighting. A block is a 2-dimensional matrix comprised of pages (rows) and strings
Pages (shown as rows) share the same word line and are the minimum (columns). The total number of bits in a block can be calculated by
unit to program. They are typically comprised of at least 32,768 NAND multiplying the number of strings by the number of pages.
cells, with many of the newer NAND devices have page sizes of 64K or From a Micron presentation at the 2014 Flash Memory Summit, maximum
128K cells. Pages per Block are approaching 512 and Block sizes are reaching up to
Most page sizes are referred to as 2K, 4K, 8K, etc. This signifies the page 8 Mbytes.
size in bytes. So if the page size has 32,768 NAND Cells (bits), this equates
to 4096 bytes or 4K Page size.

Section 04: NAND Architecture - Pages and Blocks www.cactus-tech.com Page 10

 05 NAND Architecture - Planes and Die

Previous sections ranged from the basic NAND cell up to the block level. In The NAND Plane and Die
this section we move up to discuss NAND planes and die.
Bit Line Bit Line Bit Line Bit Line
As discussed in earlier, individual NAND cells are combined on Strings and NAND Plane
Pages which are configured as columns and rows of an array. The overall String Select Line
Block 0
array is called a Block. Some of the latest NAND components have block Block 1
Word Line
sizes as high as 8Mbytes.
Block 2
This section will build from the Block level to show the pieces that make Word Line
Block 3
up a NAND die.
Word Line Block 4

Block 5
The NAND Block Structure Word Line
Block 6

Bit Line Bit Line Bit Line Bit Line

String Select Line Word Line

Block 0

Word Line Block 1

Word Line
Block 2
Word Line Block n - 1
Ground Select Line
Block 3
Source Line
Word Line Block 4
NAND Architecture - Block Structure
Block 5
Word Line

Block 6
This bank of Blocks highlighted in yellow above is referred to as a Plane.
One or many planes are grouped together to form a NAND die highlighted
Word Line
in the illustration on the next page. There are many configurations of die
Word Line
to meet many different design needs of OEMs.

Ground Select Line

Block n - 1
Source Line

NAND Block NAND Architecture - Block Structure

The NAND Block which is made up of a matrix of Strings and Pages is a

building block for larger data structures. A single block is grouped together
in a bank of many other blocks as shown in the illustration above.

Section 05: NAND Architecture - Planes and Die www.cactus-tech.com Page 11

 NAND Die

NAND Plane NAND Plane NAND Plane

Block 0 Block 0 Block 0

Block 1 Block 1 Block 1

Block 2 Block 2 Block 2

Block 3 Block 3 Block 3

Block 4 Block 4 Block 4

Block 5 Block 5 Block 5

Block 6 Block 6 Block 6

Block n - 1 Block n - 1 Block n - 1

NAND Die with Multiple Planes

A single die or multiple die stacked on top of each other are packaged into
a usable form in popular JEDEC standard TSOP, BGA and other packages.
Over time as the requirements for additional storage continue, new ways
of increasing density arise. One of the new technologies for high capacity
SSD making its way to the forefront is 3D Memory.

Section 05: NAND Architecture - Planes and Die www.cactus-tech.com Page 12

 06 NAND Architecture - Component Packaging

Previous sections started at the basic NAND cell and built up to the NAND The illustration below shows a cutaway of a NAND component with
die level. We go up one more level in this section to discuss common NAND multiple layers of die stacked on top of each other to create a single large
component packaging options. capacity NAND memory device.
The NAND die themselves are relatively fragile and require special
equipment for placement and bonding. Typically NAND die are placed
inside a protective component package as opposed to directly on a circuit
board. Most follow an open industry standard defined by JEDEC.
These components allow a manufacturer to place a single or several NAND
die inside one package with a standard pin-out of typically a TSOP or BGA
package. The standard packages are easily handled by the pick and place
systems as they adhere the parts to printed circuit boards (PCBs) prior to
being run through soldering machines.
There is an insulating layer between each NAND die and connections are
made from each NAND die to the substrate using a wire bonding machine.
BGA Packaged NAND The substrate is like a very thin PCB (Printed Circuit Board) which is the
base for the stacked die. As with the single die cutaway shown earlier, the
entire top of the component is encapsulated.
Mold Compound NAND Die Wire Bonding
If the parts share a JEDEC standard form factor, the parts appear physically
identical regardless of the number of die inside.

Substrate Die Attach Solder Balls

A simple single NAND die BGA component package image is shown above.
There is a substrate which has a NAND die attached to it. Wire bonding
machines connect the NAND die to the substrate which has via connections
to the balls on the bottom side. After the wiring is complete, a molding
compound encapsulates the top of the substrate providing for a rugged
physical package.

Section 06: NAND Architecture - Component Packaging www.cactus-tech.com Page 13

 TSOP Packaged NAND

Mold Compound NAND Die Wire Bonding

Tape
Lead
This illustration shows a NAND die in a TSOP package. There are a couple
of difference with the TSOP package from the BGA package. First, there are
leads as opposed to balls that take the connections from the NAND die to
the outside world. Second the entire assembly is encapsulated, not just
the top. The only exiting connection is the end of the lead frame. As with
the BGA package, multiple die can be stacked inside of a TSOP package.
Future sections will look at putting the NAND components and a controller
together to create a Solid State Drive as well as the challenges required of
the SSD controller technology.

Section 06: NAND Architecture - Component Packaging www.cactus-tech.com Page 14

 07 SSD Controller Architecture - Basic Overview

Previous sections described the NAND architecture from the basic NAND As shown to the left, the basic SSD consists of a controller chip which
cell to a packaged component. This section begins to integrate the manages one or more NAND components, each of which could be
controller into the picture. Without a controller, the NAND is a relatively comprised of multiple NAND die. The diagram is generic in the sense that
unintelligent storage device. it doesn’t matter what final host interface is used.
The reason for the controller function is to manage the NAND components
and create a standard interface which communicates well with host
systems. There are many popular interfaces today such as Serial ATA
(SATA), SD, MMC, USB, PCIe as well as Parallel ATA (PATA, aka IDE).
All of these SSD interfaces have a common controller architecture design
in which a controller resides between the NAND memory and the host
system. In future articles we will look at the tasks a controller handles, but
here we focus on the basic architecture of a generic Solid State Drive (SSD).

Examples:
SATA NAND NAND NAND NAND
PATA Controller
Host Host NAND
System Interface
Inter. Mgmt.
SD
USB NAND NAND NAND NAND
etc.

Basic Solid State Drie (SSD) Architecture

As a real life example, see the image of a SATA SSD’s internal circuit board
above. All the rectangular components are NAND chips with the square
component as the single controller for this SSD. The controller does not
necessarily need to be in a square package, it just happens to be in this
case.

Section 07: SSD Controller Architecture - Basic Overview www.cactus-tech.com Page 15

 Something you’ll notice about the previous image is how tightly packed the
NAND components are to each other. The limitation to an SSD’s storage
capacity is how many NAND die can be integrated into the industry
standard package along with a controller’s ability to address (read/write)
each die.

Controller Die
Mold Compound Wire Bonding
NAND Die

Discrete
Contact Pad Substrate Die Attach Components

Cutaway Illustration of microSD Card

For very small packaged SSD such as microSD cards, there is not sufficient
physical space for packaged NAND and controller to be used.
In these cases, the controller die and NAND
die are stacked on top of each other and
connections are made with wire bonding.
The diagram above only shows a single
NAND die in the microSD package, but
multiple NAND die can be stacked with the
controller to make higher density/capacity
parts. The size of the die in the cross
section is not proportional to the actual
size for clarity. With wafer backgrinding techniques as many 16 NAND die
have been successfully integrated into a single microSD card.
The next section will focus more on multiple channels, external RAM and
other features common in today’s Solid State Drive devices.

Section 07: SSD Controller Architecture - Basic Overview www.cactus-tech.com Page 16

 08 SSD Controller Architecture - Channels and Banks

This section focuses on the connection between the SSD controller and
the NAND flash. There are many NAND configurations in SSD design and Bank Control 2
it makes a large difference to the SSD’s overall power, performance and Bank Control 1
cost.
The illustration to the right shows a common 2.5” SATA III SSD NAND Channel 0 Data/Ctrl Bus NAND Data/Ctrl Bus NAND
configuration. In this example, there are 8 Channels connected to the
NAND chips. For each channel there are 2 Banks of NAND components.
Channel 1 Data/Ctrl Bus NAND Data/Ctrl Bus NAND
There is a control line which selects either Bank 1 or Bank 2 to be active on
the Data/Control Bus for a specific channel. This control line is connected
to the Chip Select of each NAND component to enable or disable the Channel 2 Data/Ctrl Bus NAND Data/Ctrl Bus NAND
component.

Channel 3 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

Channel 4 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

Channel 5 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

Channel 6 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

Channel 7 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

BANK 1 BANK 2

NAND Banks and Channels Illustration

Section 08: SSD Controller Architecture - Channels and Banks www.cactus-tech.com Page 17
 NAND Channels Additional SSD Performance Techniques
Channels refer to the number of flash chips the controller can talk to To further increase performance, controllers can take advantage of
simultaneously. Low end SSDs usually have 2 or 4 channels; high end SSDs interleaving. Each NAND flash component can have multiple die in it, this
usually have 8 channels, some have 10 channels. is particularly so for high density parts. 2, 4 and 8 die packs are common.
SSD manufacturers can trade off performance vs power consumption The illustration below shows a cutaway of a TSOP NAND component with
by stuffing less channels at time of manufacture. The limitation on more multiple layers of die stacked on top of each other to create a single large
channels is added die size, pin count and power consumption, which all capacity NAND flash chip.
increase the cost.

NAND Banks
Each flash chip at the same location in a channel together constitutes a
bank. Refer to the diagram on the previous page. Each channel can have
multiple chips. The limitation on maximum number of chips is a result of
pin count, die size and cost considerations.
For a multi-die package, it is possible for each die to carry out a command;
this is refer to as interleaving and can significantly increase device
performance. The ability to interleave is dependent on flash, controller
and firmware support.
Another mechanism to improve performance is multi-plane operation. A
flash chip is internally organized in planes, with low density devices being
single plane and higher density devices with 2, 4 or more planes.
In a multi-plane devices, it is possible for all planes to carry out a command
in parallel (this is like interleaving but for a single die). Multi-plane
operation, when available, can substantially improve device performance.
This section provided a basic understanding of the Channel and Bank
architecture as well as performance enhancement techniques in an SSD.
There are other more advanced techniques, such as copyback writes,
cache reads, etc. not covered in this eBook.

Section 08: SSD Controller Architecture - Channels and Banks www.cactus-tech.com Page 18
 09 SSD Controller Architecture - Block Diagram

This section focuses on the main blocks of a generic SSD controller and its Host Interface
connection to the NAND flash. Controller functionality varies with the type
The host interface of a controller is typically
of product for which it is intended.
designed to one industry standard interface
A simple consumer SD card controller is designed for cost and in some specification. There are several interfaces
cases performance. For this application, it would be an overkill and made to address different system and design
unneeded expense to add an encryption & decryption engine to the silicon. requirements. The most popular are SATA,
In other cases, such as secure military grade SSD, encryption & decryption SD, USB, PATA/IDE and PCIe.
may be an absolute necessity. Other applications rely on SMART data
to predict an imminent failure looming in the future so the SSD can be
replaced prior to an unexpected failure. SMART (Self-Monitoring, Analysis and
Reporting Technology)
The SMART function, available in some
Typical Controller Functions and Blocks controllers, monitors and records data
SMART Buffer/
regarding many attributes of the SSD and
Cache Misc.
I/O
memory. An example is the ability to monitor
the percentage of endurance cycles remaining
Wear NAND NAND NAND NAND
in the SSD since this is a key determining
Host Leveling CPU/ NAND NAND NAND NAND
Interface RISC factor of the life remaining.
Processor
(Examples)
SATA Read & NAND Multiple NAND Channels
PATA Program Memory
SD Interface Wear Leveling
START 1st WRITE 2nd WRITE 3rd WRITE Nth WRITE
Disturb Free LBA #6 Invalid Invalid Invalid
USB Free Free LBA #6 Invalid Invalid

PCle ECC
PAGE Free Free Free LBA #6 Invalid

Wear Leveling is the ability to even out the

Free Free Free Free Invalid

Engine Free
Free
Free
Free
Free
Free
Free
Free
Invalid
Invalid

NAND NAND NAND NAND

Free
Free
Free
Free
Free
Free
Free
Free
Invalid
LBA #6 number of write cycles throughout the
Encrypt
available NAND. Since each NAND block has
DATA BLOCK DATA BLOCK DATA BLOCK DATA BLOCK DATA BLOCK
NAND NAND NAND NAND
& Before Garbage After Garbage
Decrypt
Engine
Write
Collection Collection
a limited number of erase/write cycles, if only
Defect
Invalid Free Free LBA #6

Abort
one physical block is written continuously, it
Invalid Free Free Free

Mgmt. Invalid
Invalid
Free
Free
Free
Free
Free
Free

will quickly deplete its endurance cycles. A

Invalid Free Free Free
Invalid Free Free Free
Invalid Free Free Free

Generic Solid State Drive (SSD) Controller Architecture

LBA #6

DATA BLOCK
Free

SPARE BLOCK NEW

Free

DATA BLOCK
Free

NEW
DATA BLOCK
controller’s Wear Leveling algorithm monitors
and spreads out the writes to different
physical NAND blocks.
The illustration above shows the basic blocks of a SSD. A brief description
of each block follows.

Section 09: SSD Controller Architecture - Block Diagram www.cactus-tech.com Page 19

 Read & Program Disturb CPU/RISC Processor
With the finer and finer trace widths of the The heart of every SSD is the main processing core.
NAND flash, more issues arise to maintain This can be a CPU or RISC processor. The size and
the data contents of the NAND cells. Read performance of the CPU/RISC processor determines
& Program Disturb occur when cells are how capable the controller can be.
read or written causing cross coupling to
adjacent cells and occasionally changing
their values. Controllers need algorithms ECC Engine
and in some cases circuitry to compensate
for this phenomena. Error Checking & Correction are a key part of today’s
Error Check SSD. ECC will correct up to a certain number of bits
Correct per block of data. Without ECC, many of the low cost
consumer flash cards using very inexpensive memory
Encrypt & Decrypt Engine
would not be possible.
For higher security applications, a hardware
encryption and decryption engine is
generally built into the silicon of the Write Abort
controller. The encryption engine is typically
implemented in hardware to ensure speed Write Abort is the when power to the SSD
for encrypting/decrypting on the fly. The is lost during a write to the NAND flash.
most popular encryption method for SDDs Without a battery or SuperCap backed
today is AES256. cache, it is likely this data in transit will be
lost. The more important aspect of this is
to ensure the SSD’s internal metadata and
firmware remain uncorrupted. This is the
Buffer/Cache
function of Write Abort circuitry mainly
Controllers generally have a high speed found in Industrial Grade products.
SRAM/DRAM cache buffer used for buffering
the read and/or write data of the SSD. Since
this cache uses volatile memory, it subjects Miscellaneous I/O
data to loss if power is unexpectedly
removed. It is common to see both internal Simple functions such as chip select pins for the
Misc. Input NAND components are handled with several input/
caches in the controller chip itself as well as
external RAM cache chips.
Output output pins. There are also a number IO functions
required for initial programming and production.

Section 09: SSD Controller Architecture - Block Diagram www.cactus-tech.com Page 20

 Bank Control 2
NAND Memory Interface
The NAND memory interface was covered
Bank Control 1

Channel 0 Data/Ctrl Bus Data/Ctrl Bus

in the previous article on NAND banks and

NAND NAND

channels. Depending on the controller there

Channel 1 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

can be a single NAND channel up to 10 or

Channel 2 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

more. Each channel can have one or more

Channel 3 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

Channel 4 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

NAND chips.
Channel 5 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

Channel 6 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

Channel 7 Data/Ctrl Bus NAND Data/Ctrl Bus NAND

BANK 1 BANK 2

NAND Banks and Channels Illustration

Defect Management
Every controller needs a method to
deal with bad blocks of memory and
new defects. At the point a NAND block
becomes unusable, some action on the
SSD controller’s part must happen. In some
cases a spare sector replaces the failed
block. In a poor controller design, the SSD
fails. Each controller has its method to deal
with defects.

Section 09: SSD Controller Architecture - Block Diagram www.cactus-tech.com Page 21

 10 SSD Controller Functions - Wear Leveling

This section takes a look at how SSD controllers use Wear Leveling The image to the left shows a 256KB NAND block which is comprised
algorithms to compensate for the finite number of erase cycles inherent of 64 Pages containing 4KB of storage capacity each. If the host system
in NAND flash blocks. writes 256KB of data in sequential LBAs and the SSD controller stores this
A SSD controller receives commands from the host system which tell it data in sequential Physical Block Addresses - starting at the first 4K Page
where to read or write a piece of data. For simplicity of this article on and ending at the 64th in the block - then there has effectively been 1
endurance cycle used for the entire block.
Wear Leveling we will make two assumptions: 1) Each piece of data is 4KB
and; 2) The NAND pages are also 4KB. In real world situations, the sizes The chart to the right of the 256KB block shows how sequential data is the
of the data and page sizes can vary depending on host system and NAND ideal method of storing data in a NAND flash device. It evenly distributes
memory used. all of the write cycles to the NAND Pages and Blocks so that one individual
NAND Page/Block is not worn out prior to other Pages/Blocks.
The host system provides the Logical Block Address (LBA) of the data it
would like to read or write. It would be relatively straight forward for the Before continuing a couple clarifications:
SSD controller to simply read or write the LBA to the exact same Physical
• An Endurance cycle only occurs when an erase occurs, so on the first
Block Address. Let’s take a look at what would occur.
write, there is actually not an endurance cycle.
• Reading a NAND cell does not affect the endurance cycles of the NAND
Wear Leveling and Sequential Writing to NAND Memory cell.

256KB Logical Block

4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page

4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page

4K 4K 4K 4K 4K 4K 4K 4K Number
Page Page Page Page Page Page Page Page
of Endurance
4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page Cycles
4K 4K 4K 4K 4K 4K 4K 4K Per Page
Page Page Page Page Page Page Page Page

4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page
Page 60

Page 63
Page 59

Page 62
Page 61
Page 0

Page 3
Page 2

Page 4
Page 1

4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page

4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page

Sequentially Written Flash Block

Section 10: SSD Controller Functions - Wear Leveling www.cactus-tech.com Page 22

 Wear Leveling and Non-Sequential Writing to NAND Memory What Does Wear Leveling Do?
Wear Leveling algorithms in SSD controllers attempt to evenly distribute
256KB Logical Block host system writes throughout the entire SSD. Since NAND flash has a
4K 4K 4K 4K 4K 4K 4K 4K
finite number of writes per block, wear leveling attempts to use every
Page Page Page Page Page Page Page Page
endurance cycle of the SSD prior to the end of its useful life.
4K 4K 4K 4K 4K 4K 4K 4K

There are many different schemes used by the different SSD controller
Page Page Page Page Page Page Page Page

4K 4K 4K Number
designers, but they all share a couple characteristics.
4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page
of Endurance
4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page Cycles
4K 4K 4K 4K 4K 4K 4K 4K Per Page
Page Page Page Page Page Page Page Page 256KB Logical Block 256KB Physical Block
4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page 4K 4K 4K 4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
Translation Table 4K 4K 4K 4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
Page Page Page Page Page Page

Page 60

Page 63
Page 59

Page 62
Page 61
Page 0

Page 3
Page 2

Page 4
Page 1
LOGICAL PHYSICAL
4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page
Page 0 Page 12
4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
4K 4K 4K 4K 4K Page Page Page Page Page Page Page Page Page 1 Page 18 Page Page Page Page Page Page Page Page
Page Page Page Page Page Page Page Page
4K 4K 4K 4K 4K 4K 4K 4K Page 2 Page 3 4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page

Non-Sequentially Written Flash Block 4K

Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page

4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K 4K
Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page Page

Unfortunately, real life usage of the SSDs rarely have the ideal sequential 4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
Page x
Page x
Page y
Page y
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page

usage patterns. There are File system directories which need to be updated 4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page Page x Page y
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page

after any file is altered. There are mismatches in the data sizes from the
host and the Page/Block sizes of the NAND on the SSD. There is the need Logical to Physical Translation
to reclaim previously written blocks and many other factors.
The Non-Sequential image above shows a worst case situation where data As shown above, they store the host data written for a Logical Block
is constantly written to only the first two 4K Pages. If the SSD controller just Address (LBA) to a physical location that has the least amount of endurance
continued writing this data to these physical pages of the NAND, it would cycles used. Host data written to the same LBA is typically not stored in the
quickly exhaust the total number of endurance cycles of these cells. same physical location of NAND. The controller must keep track of the
translation from Logical to Physical block in a table or other method.
Another common occurrence is when static data on the SSD never moves
- such as Operating System and Application data. It’s stored once on the
SSD and thereafter left alone. For these situations most new controllers
will automatically move this static data to other physical NAND locations
so they can take advantage of the endurance cycles of these NAND cells.
All alone, Wear Leveling cannot solve all an SSD’s tasks, but it is an
important part to creating a more reliable SSD which efficiently uses the
limited endurance available in NAND memory.

Section 10: SSD Controller Functions - Wear Leveling www.cactus-tech.com Page 23

 11 SSD Controller Functions - Garbage Collection

This section covers SSD controller’s Garbage Collection algorithms. First Page Writes to Fresh SSD
Garbage Collection is the process to reclaim previously written blocks of
data so they can be rewritten with new data. 128KB Physical Block #1 128KB Physical Block #2 128KB Physical Block #1 128KB Physical Block #2

E E E E E E E E 1 2 3 4 E E E E
The reason there is a need for Garbage Collection is NAND’s requirement E E E E E E E E 5 6 7 8 E E E E
E
to be erased prior to being written. A block is the smallest erasure unit of E E E E E E E E 9 10 E E E E E E
Erased Data

NAND flash and it is typically made up of 32 to 64 Pages. E E E E E E E E E E E E E E E E Valid Data

A page of data can be written as long as it is already erased, but it is very E E E E E E E E E E E E E E E E

D Dirty/Stale Data
inefficient to erase 32 or 64 pages in a block just to reclaim a single Page. E E E E E E E E E E E E E E E E
E E E E E E E E E E E E E E E E
There are many challenges related to Garbage Collection algorithms and
E E E E E E E E E E E E E E E E
efficiency, reliability and performance can be affected by its design.
Fresh SSD with (32) 4KB Pages per Block 10 Pages of Valid Data Written

What Does Wear Leveling Do?

On the first write to a fresh NAND block, things are easy. All pages are
128KB Physical Block #1 128KB Physical Block #2 128KB Physical Block #1 128KB Physical Block #2 erased and it is just a matter of writing sequentially to these erased Pages.
4K 4K 4K 4K 4K 4K 4K 4K
E E E E E E E E
Shown above is how a typical Garbage Collection algorithm would perform
Page Page Page Page Page Page Page Page

4K 4K 4K 4K 4K 4K 4K 4K
E E E E E E E E
the initial task of writing 10 new pages of data on an already erased block.
Page Page Page Page Page Page Page Page

E Erased Data
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page E E E E E E E E
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page E E E E E E E E Valid Data
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page E E E E E E E E
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page E E E E E E E E D Dirty/Stale Data

4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page E E E E E E E E
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page
4K
Page E E E E E E E E

Fresh SSD with (32) 4KB Pages per Block All Pages are Erased

The diagram above shows two blocks of a fresh SSD which contain 32
Pages capable of storing 4KB of data each. This is a typical configuration of
NAND and we will use it for the remainder of this section.
As you can see all of the Pages are erased from the factory and ready to be
written. We will cover the other states of Valid Data and Dirty/Stale Data
as we move forward.

Section 11: SSD Controller Functions - Garbage Collection www.cactus-tech.com Page 24

 Updating Page Data What happens next - Garbage Collection?
128KB Physical Block #1 128KB Physical Block #2 128KB Physical Block #1 128KB Physical Block #2 128KB Physical Block #1 128KB Physical Block #2 128KB Physical Block #1 128KB Physical Block #2

1 2 3 4 E E E E D D D D E E E E D D D D E E E E E E E E 1 2 3 4 E Erased Data

5 6 7 8 E E E E D D D D E E E E E Erased Data
D D D D E E E E E E E E 5 6 7 8
Valid Data
9 10 E E E E E E D D 1 2 E E E E D D 1 2 E E E E E E E E 9 10 11 12
Valid Data
E E E E E E E E 3 4 5 6 E E E E 3 4 5 6 E E E E E E E E 13 14 15 16 D Dirty/Stale Data

E E E E E E E E 7 8 9 10 E E E E 7 8 9 10 E E E E E E E E 17 18 19 20
D Dirty/Stale Data
E E E E E E E E E E E E E E E E 11 12 13 14 E E E E E E E E 21 22 E E

E E E E E E E E E E E E E E E E 15 16 17 18 E E E E E E E E E E E E

E E E E E E E E E E E E E E E E 19 20 21 22 E E E E E E E E E E E E

10 Pages of Valid Data Written Updating 10 Pages of Data Filled NAND Block Reclaiming a NAND Block

We take it a step further in the above diagram by updating the 10 pages Now the moment you’ve all been waiting for. What happens when all of the
we originally wrote to the NAND. Since we would have to erase the entire pages of the NAND block are either occupied with good data or previously
block to erase the pages we want to update, we instead copy and write the written data which is no longer valid?
updated data in the next available erased pages and mark the previous This is where Garbage Collection comes into play. To reclaim the NAND
pages as dirty or stale. pages to an erased state, first any valid data in that block needs to be
At this point, the 10 pages are stored with valid data and we have 10 pages copied and written to the erased pages in a new NAND block. As you can
marked as dirty which cannot be written until the entire NAND Block is see from the latest diagram, 22 valid pages of data were copied from the
erased. full Block #1 and written to the fully erased Block #2.
Once the valid data has been successfully written to the new block, then
Filling the NAND Block the entire Block #1 is erased. This is a one step process and it deletes any
valid or dirty/stale data. Now Block #1 can be written to as if it were a fresh
128KB Physical Block #1 128KB Physical Block #2 128KB Physical Block #1 128KB Physical Block #2 block.
D D D D E E E E D D D D E E E E
E Erased Data The design of the Garbage Collection algorithm has a lot to do with other
D D D D E E E E D D D D E E E E
D D 1 2 E E E E D D 1 2 E E E E
factors such as Write Amplification, so it is an important part of an overall
Valid Data

3 4 5 6 E E E E 3 4 5 6 E E E E
SSD design.
7 8 9 10 E E E E 7 8 9 10 E E E E D Dirty/Stale Data

E E E E E E E E 11 12 13 14 E E E E
E E E E E E E E 15 16 17 18 E E E E
E E E E E E E E 19 20 21 22 E E E E

Mixed Block with Valid, Dirty and Erased Pages Filling the NAND Block

This latest diagram above shows an additional 12 pages of data written

to the NAND block. These new items 11 - 22 were placed in the final 12
erased pages at the end of NAND Block #1.
At this point, the entire NAND Block has been used. There are no longer
erased pages to store data since there is only stored data or dirty data
which needs to be erased to reclaim these pages.

Section 11: SSD Controller Functions - Garbage Collection www.cactus-tech.com Page 25

 12 SSD Controller Functions - TRIM Command

This section covers the TRIM command supported by some SATA, SCSI and Example of Operating System and/or SSD without
other SSD controllers. The TRIM command is related to a SSD’s Garbage TRIM Command
Collection process described in the previous section.
With Garbage Collection, when the Operating System replaces LBA (Logical
128KB Physical Block #1 128KB Physical Block #1
Block Addresses) which already contain data, such as during a file update/ E Deleted Data E Erased Data
overwrite, the SSD stores the updated data on fresh pages and marks the D D D D 1 2 3 4
existing pages as Dirty (or Stale). D D D D 5 6 7 8 Valid Data

At a later time, Garbage Collection reclaims these dirty pages when it D D D D 9 10 11 12

erases an entire block, which is the smallest area of erasure in NAND flash. D Dirty/Stale Data
This freshly erased block is now available to be written with new data. D D D D 13 14 15 16
D D D D 17 18 19 20
So Why the TRIM Command? D D D D 21 22 23 24
When the Operating system overwrites the same LBAs, and the SSD D D D D 25 26 27 28
controller marks them as dirty for future erasure, it is clear there is no D D D D 29 30 31 32
longer valid data in these pages.
But when a file is erased in the operating system, many times only the What the Operating What the
operating system’s directory is updated. In these cases, there is no erase System Knows SSD Controller Thinks
command sent to the SSD for the LBAs no longer considered valid by the
operating system. Without Trim Command
For TRIM to be functional, the command must be supported by both the
In comes the TRIM command. By issuing the TRIM command when a file Operating System and the SSD. The newer versions of the Windows, Mac
is permanently deleted, it notifies the SSD that the associated LBAs no and Linux OS along as well as others support the TRIM command.
longer have valid data and can be marked as dirty for the next round of
The illustration above shows a simplified example of a system without the
data collection.
TRIM command support. In this example, the host system has a file which
is 32 pages in size (128KB), which has been deleted. The operating system
knows this has been deleted and knows these are free areas (LBAs) it can
use to write to.
But since the operating system simply marks these areas as available
in the directory, there is no mechanism to tell the SSD controller these
pages have been deleted. On the physical blocks of the SSD in the image
on the right, you can see the SSD controller still believes there is valid data
in these blocks. Therefore, it does not mark them as dirty and does not
perform garbage collection on them until they are later overwritten by the
operating system.

Section 12: SSD Controller Functions - TRIM Command www.cactus-tech.com Page 26

 128KB Physical Block #1 128KB Physical Block #1
E Deleted Data E Erased Data
D D D D D D D D
D D D D D D D D Valid Data

D D D D D D D D
D D D D D D D D D Dirty/Stale Data

D D D D D D D D
D D D D D D D D
D D D D D D D D
D D D D D D D D

What the Operating What the

System Knows SSD Controller Thinks

With Trim Command

Example: Operating System & SSD with TRIM Command

For systems having TRIM support by both the Operating System and the
SSD, the operating systems and SSD are in sync. After deleting a file, the
operating system issues a TRIM command to the SSD with the deleted
LBAs. The SSD then marks these LBAs as dirty and is able to reclaim them
efficiently during its garbage collection process.
The illustration above shows a simplified example of the TRIM command.
As in the previous example, the host system has a file which is 32 pages in
size (128KB), which has been deleted. The operating system marks these
areas as available in the directory, then issues the TRIM command with
LBAs to the SSD. The SSD controller then marks these pages as dirty for
garbage collection process.

Section 12: SSD Controller Functions - TRIM Command www.cactus-tech.com Page 27

 13 SSD Controller Functions - Over-Provisioning

This section describes Over-Provisioning supported by many Solid State

Drives, whether advertised on not. It is basically data space not available
to the operating system which is used by the SSD to perform internal tasks
such as bad block mapping, wear leveling and garbage collection.
The amount of Over-Provisioned area is generally set at the factory during
the final low level formatting of the SSD. The typical percentages of Over-
Provisioning are 0%, 7% and 28%.
An example of these percentages seen in the market would be a 128GB
SSD which can be marketed as a 128GB with 0% Over-Provisioning, 120GB 7% Difference in Binary to Decimal SSD Capacity
at 7% and 100GB at 28%. It is the same SSD with different amounts of used internally by SSD Controller
“user-space” available.

MARKETED IDEMA DECIMAL BINARY CAPACITY DIFFERENCE

Over-Provisioning Which is Typically Not Counted CAPACITY CAPACITY (BYTES) (BYTES)

One item that is not readily apparent to most users is the fact that NAND
128GB 128,035,676,160 137,438,953,472 7.34%
components are sold in binary capacity points versus SSD which are sold
in decimal capacities. As shown to the right, there is a little over 7% of the
512GB 512,110,190,592 549.755,813,888 7.35%
SSD’s NAND capacity which is not available to the user.

The International Disk Drive Equipment and Materials Association (IDEMA)

has published a standard (LBA 1-03) for drive capacities that most HDD and
SSD manufacturers follow. The 128GB and 512GB capacities are shown in
the table above.
This binary to decimal difference is typically not counted as Over-
Provisioning. The space is used internally by the controller for firmware
storage, spare sectors, bad block mapping, wear leveling and other tasks
similar to Over-Provisioning.

Section 13: SSD Controller Functions - Over-Provisioning www.cactus-tech.com Page 28

 Over-Provisioning at 0%, 7% and 28% What are the Advantages and Disadvantages of
Over-Provisioning?
The advantages of Over-Provisioning are to reduce Write Amplification
while increasing endurance and performance. The key disadvantage of
Over-Provisioning is loss of user capacity.
A user is trading higher reliability and more endurance and in some cases
better performance for less usable space.

7% Difference in Binary to Decimal SSD Capacity

used internally by SSD Controller

0% 7% 28%
OVERPROVISIONING OVERPROVISIONING OVERPROVISIONING

128GB 120GB 100GB

512GB 480GB 400GB

For this discussion, the 7% binary to decimal capacity is already removed.

The table above shows the decimal 128GB and 512GB at the three most
common Over-Provisioning percentages.
Over-Provisioning typically come in 0%, 7% or 28% chunks. Most of the
lower capacity products have 0% Over-Provisioning in addition to many of
the Industrial Grade devices built with the extremely high endurance SLC
NAND.
For many client SSD built with MLC or TLC NAND on the market today, 7%
Over-Provisioning is used and Enterprise SSD typically use 28%.

Section 13: SSD Controller Functions - Over-Provisioning www.cactus-tech.com Page 29

 eBook

SSD 101

Cactus Technologies Limited

Suite C, 15/F, Capital Trade Center
62 Tsun Yip Street, Kwun Tong
Kowloon, Hong Kong
Tel: +852-2797-2277
Email: [email protected]

Cactus USA
3112 Windsor Road
Suite A356
Austin, Texas 78703
Tel: +512-775-0746
Email: [email protected]

Cactus-Tech.com