Physical Disk, Cell Disk, Grid Disk, and ASM Disk in Exadata
Nov 22, 2010 / By Fahd Mirza
Exadata is the Ferrari of the technology world. It’s considered the technology for the select few, but everyone dreams about it. There is very little information available on the Internet right now about the architecture and the administration of Exadata, though it’s a huge subject in itself, which demand numerous high quality books, dedicated blogs, websites and special interest groups.
This blog post just mentions some of very important components of Exadata Storage Server, which are physical disks, cell disks, grid disks and ASM disks and their correspondence.
The Exadata Storage Server contains 12 physical disks.
There is one to one relationship between a physical disk and the cell disk. One physical disk corresponds to a single cell disk.
Then a Grid disk comprises of many cell disks.
And a Grid disk corresponds to one ASM disk.
An ASM diskgroup comprises of many ASM disks.
On the Exadata Storage Server, We can use cellcli command line utility in Exadata to see the information about physical disks, cell disks, grid disks and the cell.
The cellcli utility works from the root, celladmin and cellmonitor (read-only) users. The best practice is actually to run it from the last two less-privileged users, and not from the root user.
Now let’s have look at some of the disk management commands using cellcli utility.
[root@mycell-net0 ~]# cellcli CellCLI: Release 11.2.1.3.1 - Production on Fri Oct 29 07:47:26 GMT 2010 Copyright (c) 2007, 2009, Oracle. All rights reserved. Cell Efficiency Ratio: 140
Just to give you an idea about what cellcli has on offer, look at the output of the help command:
CellCLI> help HELP [topic] Available Topics: ALTER ALTER ALERTHISTORY ALTER CELL ALTER CELLDISK ALTER GRIDDISK ALTER IORMPLAN ALTER LUN ALTER THRESHOLD ASSIGN KEY CALIBRATE CREATE CREATE CELL CREATE CELLDISK CREATE FLASHCACHE CREATE GRIDDISK CREATE KEY CREATE THRESHOLD DESCRIBE DROP DROP ALERTHISTORY DROP CELL DROP CELLDISK DROP FLASHCACHE DROP GRIDDISK DROP THRESHOLD EXPORT CELLDISK IMPORT CELLDISK LIST LIST ACTIVEREQUEST LIST ALERTDEFINITION LIST ALERTHISTORY LIST CELL LIST CELLDISK LIST FLASHCACHE LIST FLASHCACHECONTENT LIST GRIDDISK LIST IORMPLAN LIST KEY LIST LUN LIST METRICCURRENT LIST METRICDEFINITION LIST METRICHISTORY LIST PHYSICALDISK LIST THRESHOLD SET SPOOL START
Let’s see the output of some of the commands listed above:
CellCLI> list physicaldisk detail name: [5:3:2:0] diskType: FlashDisk id: 00000200000000000000 luns: 1_2 makeModel: "MARVELL SD88SA02" physicalFirmware: D20R physicalInsertTime: 2010-10-04T21:05:46+00:00 physicalInterface: sas physicalSerial: 0000000000000000000 physicalSize: 22.8880615234375G slotNumber: "PCI Slot: 1; FDOM: 2" status: normal <pre>
CellCLI> list cell detail name: mycellnet0 bmcType: IPMI cellVersion: OSS_11.2.0.1.0_LINUX.X64_100818.1 cpuCount: 16 fanCount: 12/12 fanStatus: normal id: 0000X00000 interconnectCount: 3 interconnect1: bond0 iormBoost: 0.0 ipaddress1: kernelVersion: 2.6.18-194.3.1.0.2.el5 makeModel: SUN MICROSYSTEMS SUN FIRE X4275 SERVER SAS metricHistoryDays: 7 notificationMethod: snmp notificationPolicy: critical,warning,clear offloadEfficiency: 47,485.1 powerCount: 2/2 powerStatus: normal smtpFrom: smtpFromAddr: smtpPort: smtpServer: smtpToAddr: smtpUseSSL: snmpSubscriber: status: online temperatureReading: 27.0 temperatureStatus: normal upTime: 16 days, 14:01 cellsrvStatus: running msStatus: running rsStatus: running <pre>
CellCLI> list celldisk detail name: FD_15_mycellnet0 comment: creationTime: 2010-07-14T02:05:06+00:00 deviceName: /dev/sdy devicePartition: /dev/sdy diskType: FlashDisk errorCount: 0 freeSpace: 0 id: 00000000-0000-0000-0000-000000000000 interleaving: none lun: 5_3 size: 22.875G status: normal <pre>
CellCLI> list griddisk detail name: CD_15_mycellnet0 availableTo: cellDisk: CD_15_mycellnet0 comment: creationTime: 2010-07-14T02:12:55+00:00 diskType: FlashDisk errorCount: 0 id: 00000000-0000-000-0000-000000000000 offset: 528.734375G size: 29.125G status: active
In the future posts, I will be touching the administration of celldisks, as how to perform operations like (import/export/create/drop/alter) on the celldisks.
Why are there no device file names listed for the griddisk? This is what you make ASM disks out of right?
–R
Hi Rich,
I assume you’re referring to the deviceName and devicePartitions seen for celldisks but not for griddisks. The griddisks are Exadata storage server concepts rather than OS concepts, and can’t be accessed as OS devices. The ASM instances sit on the database servers rather than storage, so it’s a remote request over the network rather than a /dev device. For Exadata disks, the ASM diskstring looks like o/storage server IP/griddisk name
Cheers,
Marc
Pingback: Log Buffer #207, A Carnival of the Vanities for DBAs | The Pythian Blog
Good document, however I differ on one specific line which says a grid disk comprises of many cell disks. I believe a cell disk can be sliced in many grid disks or a single grid disk.
HTH.
Abhijit
>Then a Grid disk comprises of many cell disks.
No. Cell disks are the low-level physical Exadata representation of a device. Early on, we (Exadata development) considered supporting celldisk creation from HW RAID volumes but opted for a 1:1 relationship instead for many reasons. Griddisks are the virtualization of celldisks (the presentation form to ASM). To that end there is a M:M relationship between celldisks and griddisks.
>There is very little information available on the Internet right now about the architecture
…really?
@Fahd
@Kevin
Based on above I made following Assumptions.Correct me if I am wrong..
1.each storage server will have 12 cell disks
2.each storage server can have maximum 12 grid disks or less if any grid disk is made up of more than one cell disk
3.Assuming ASM will be made up of Multiple storage server.each storage server can provide maximum 12 ASM Disks to Exadata server
4.ASM diskgrup can use any number of ASM Disk, comming from any storage server residing in exadata machine
TIA
Anuj
Pingback: Aktuelle Fragen 1Z0-536 Oracle it-pruefungen | www.it-pruefungen.de IT Prüfungsfragen auf Deutsch