Universal Access To All Knowledge
Home OpenLibrary | Software | Wayback Machine | Archive-It | Scanning Services | BookServer | NASA Images | Open Content Alliance | Education | BookMobile | SFLan | Petabox | 301Works.org
Search: Advanced Search
Anonymous User (login or join us)
Upload

Contributors

Antony Antony
Bill "TTK" Moyer
Braam Aangstrom
Brad Tofel
Brewster Kahle
Bruce Baumgart
CR Saikley
Dan Aronson
Diane Barrera
Gawin Dapper
Guy Lauten
Jim Shankland
Joerg Bashir
John Berry
Joshua Coates
Marcel van der Peijl
Matt Laue
Ralf Muehlen
Rop Gonggrip
Sam Mortazavi
Shaun Fogarty
Tom Jones
Tracey Jaquith
Will Mooney

Top Level Contributors

Petabox

Internet Archive Petabox
Petabox 4

A few highlights from the fourth generation Petabox storage system:

  • Density: 650 TeraBytes / rack
  • Power consumption: 6 KW / PetaByte
  • No Air Conditioning, instead use excess heat to help heat the building.
  • Raw Numbers as of December 2010:
    • 4 data centers, 1,300 nodes, 11,000 spinning disks
    • Wayback Machine: 2.4 PetaBytes
    • Books/Music/Video Collections: 1.7 PetaBytes
    • Total used storage: 5.8 PetaBytes
  • October 2012 update: Total used storage: 10 Petabytes






The Previous Version



OVERVIEW

The PetaBox(tm), custom-designed by Internet Archive staff, was originally created to safely store and process one petabyte (a million gigabytes) of information. The goals and design points were:
* Low power: 6kW per rack, 60kW for the entire storage cluster
* High density: 100+ TB/rack
* Local computing to process the data (800 low-end PC's)
* Multi-OS possible, linux standard
* Co-location friendly
* Shipping container friendly: Able to be run in a 20' by 8' by 8' shipping container.
* Easy Maintenance: One system administrator per petabyte
* Software to automate full mirroring
* Easy to scale
* Inexpensive design
* Inexpensive storage
The Internet Archive data center now houses ~3PB of PetaBox storage technology and is expanding steadily.


PETABOX HISTORY 6/2004

* The first 100TB Rack operational in Amsterdam as of June 2004.
* The second 80TB rack is operational in San Francisco
* Internet Archives spins off PetaBox production to newly-formed Capricorn Technologies.
2004 - 2007
* Capricorn replicates the Internet Archive's successful deployment of the PetaBox for major academic institutions, digital preservationists, government agencies, HPC and major research sites, medical imaging providers, digital image repositories, storage outsourcing sites, and other enterprises around the globe.

New PostPetabox Subscribe to or unsubscribe from this forum RSS feed of most recent posts to this forum

Subject Poster Replies Date
Other Poetry is Possible Ethno Indigo Records 0 March 24, 2013 05:33:32am
Design update? ihtoit 3 October 09, 2012 07:24:43pm
   Re: Design update? Coderjoe 0 October 10, 2012 11:10:28pm
   Re: Design update? Coderjoe 0 October 10, 2012 11:18:30pm
   Re: Design update? GridEngine 0 December 24, 2012 10:00:36am
How long does the data last? Seaware 1 October 03, 2012 09:49:17pm
   Re: How long does the data last? Coderjoe 1 October 08, 2012 01:49:19pm
     Re: How long does the data last? Seaware 1 October 09, 2012 12:47:30am
       Re: How long does the data last? Coderjoe 0 October 10, 2012 11:04:18pm
http://www.capricorn-tech.com/ is now dead ... new home? Tyler 1 December 23, 2011 09:24:02am
   Re: http://www.capricorn-tech.com/ is now dead ... new home? Hydriz 1 May 01, 2012 02:01:38am
     Re: http://www.capricorn-tech.com/ is now dead ... new home? Coderjoe 0 June 15, 2012 04:28:19pm
We have 2TB of data to upload Sheikh Naveed 0 November 13, 2011 06:00:58am
6th pair of racks go into service: over 2PB of data space used brewster 3 March 26, 2011 12:48:59pm
   Re: 6th pair of racks go into service: over 2PB of data space used baetis 0 April 20, 2011 01:09:07pm
   Re: 6th pair of racks go into service: over 2PB of data space used Sean_0000001 0 April 23, 2011 12:17:30pm
   Re: 6th pair of racks go into service: over 2PB of data space used Tyler 0 December 02, 2011 02:42:53pm
New pictures of the new petabox brewster 3 March 23, 2011 10:44:09am
   Re: New pictures of the new petabox Tyler 0 March 25, 2011 09:12:40pm
   Re: New pictures of the new petabox Coderjoe 1 October 10, 2011 03:09:15pm
     Re: New pictures of the new petabox GridEngine 1 December 24, 2012 10:19:41am
       Re: New pictures of the new petabox Coderjoe 0 January 07, 2013 11:25:11am
   Re: New pictures of the new petabox emijrp 0 December 06, 2011 03:17:08pm
Current Raw Numbers Sean_0000001 1 March 09, 2011 11:55:07pm
   Re: Current Raw Numbers brewster 0 March 21, 2011 07:34:44pm
could MapReduce be applied? create42000 1 April 20, 2010 07:55:27am
   Re: could MapReduce be applied? Coderjoe 0 April 28, 2010 12:06:23pm
CAD files for Petabox? deerpig 2 March 24, 2010 01:38:29pm
   Re: CAD files for Petabox? thattommyhall 1 April 06, 2010 07:34:18pm
     Re: CAD files for Petabox? deerpig 1 April 06, 2010 08:09:16pm
       Re: CAD files for Petabox? deerpig 0 April 06, 2010 08:14:47pm
   Re: CAD files for Petabox? baetis 0 August 05, 2010 02:26:21pm
Is the Archive site no longer acrhiving maxadds 2 September 12, 2009 09:22:48am
   Re: Is the Archive site no longer acrhiving direwolf0701 0 September 12, 2009 09:27:55am
   Re: Is the Archive site no longer acrhiving brewster 0 October 13, 2009 09:09:57pm
I assume this forum is now obsolete Javik 3 July 10, 2009 10:33:47pm
   Re: I assume this forum is now obsolete Coderjoe 2 July 15, 2009 04:17:24am
     Re: I assume this forum is now obsolete tracey pooh 0 October 16, 2009 08:57:43pm
     Re: I assume this forum is now obsolete jack liu 0 November 24, 2009 01:49:09am
Visiting the installation glenn 0 December 10, 2008 10:32:21pm
Intel Atom based motherboards? TimSmall 1 July 22, 2008 02:29:31am
   Re: Intel Atom based motherboards? NoiseCollector 0 July 22, 2008 08:24:24am
VIA NAS 7800-15LST dunno 0 June 28, 2008 05:26:00pm
SuperMicro Cascaded 128/256/384-drive SAS/SATA chassis Javik 1 November 28, 2006 04:14:51pm
   Understanding all that technical blather :-) Javik 1 November 28, 2006 03:54:45pm
     Re: Understanding all that technical blather :-) elhoim 0 January 11, 2007 06:39:26am
FYI, don't need more PCI slots, just SATA-II or SAS Javik 2 November 23, 2006 09:59:18pm
   24-drives on one SAS channel, from SuperMicro Javik 2 November 25, 2006 06:15:04pm
     Oh look, the first ever 2.5 inch SAS hard drive Javik 0 November 25, 2006 06:14:07pm
     Re: Make that --144-- 2.5" drives on one SAS channel Javik 0 November 29, 2006 12:34:51pm
   Re: FYI, don't need more PCI slots, just SATA-II or SAS Gobba 0 April 14, 2011 04:48:24am
Huge storage = huge challenges? cchipmunks 0 November 14, 2006 03:32:03pm
hi karatekid 1 September 22, 2006 01:53:11pm
   Re: hi cosmicharlie 0 September 22, 2006 02:02:14pm
As many pci slots as possible? AWP 1 July 03, 2006 06:53:49am
   Re: As many pci slots as possible? elhoim 0 July 06, 2006 04:49:52am
all archive storage is on petabox now! tracey pooh 1 June 02, 2006 09:26:50am
   Re: all archive storage is on petabox now! protobadger 0 June 04, 2006 09:22:11pm
powernode rack brewster 1 January 28, 2006 02:18:10pm
   Re: powernode rack Coderjoe 1 January 27, 2006 01:16:15am
     Re: powernode rack brewster 1 January 27, 2006 01:33:36am
       Re: powernode rack Coderjoe 1 January 27, 2006 08:22:10am
         Re: powernode rack brewster 0 January 27, 2006 08:26:52am
current configuration? fgleason 1 January 04, 2006 07:23:32am
   Re: current configuration? fgleason 1 January 12, 2006 12:06:06am
     Re: current configuration? jkrauska 0 January 17, 2006 01:59:12pm
Still sept / oct for LMA > petabox move? Tyler 0 August 30, 2005 02:21:21am
software specs jer0mey 1 August 18, 2005 04:27:55am
   Re: software specs billmoyer 1 July 29, 2006 10:30:18pm
     Re: software specs singod 0 June 11, 2007 10:31:51am
Open Source Specs BeatriceM 0 August 09, 2005 04:08:44am
Hitachi 500GB drives ngngokkiu 1 July 31, 2005 05:20:11am
   Re: Hitachi 500GB drives Nethacker 0 June 03, 2006 11:02:56am
hey. I built something like that sxpert 1 June 21, 2005 07:27:54pm
   Re: hey. I built something like that JohnSebastianBach 0 February 23, 2007 08:47:42pm
1PB and Counting CR Saikley 0 June 21, 2005 02:50:19pm
You guys still on track for LMA transfer to Petabox.. end of summer 05? Tyler 0 May 02, 2005 03:01:29am
like my movie? snipey_2 0 April 24, 2005 02:53:46am
18 PCI slots, 72 SATA drives per computer? Javik 2 March 22, 2005 11:27:14am
   Re: 18 PCI slots, 72 SATA drives per computer? caiser 1 September 03, 2005 08:31:48am
     Re: 18 PCI slots, 72 SATA drives per computer? Javik 1 April 08, 2005 05:42:52pm
       Re: 18 PCI slots, 72 SATA drives per computer? elhoim 0 April 23, 2005 04:00:07am
Drive spin-down? Low-RPM drives? Power saving? Javik 1 March 22, 2005 09:25:21am
   Re: Drive spin-down? Low-RPM drives? Power saving? CR Saikley 1 March 22, 2005 02:58:16pm
     Re: Drive spin-down? Low-RPM drives? Power saving? HDRJapan 0 October 03, 2007 06:14:29pm
Hitachi Case Study Brad Leblanc 0 March 10, 2005 10:45:32am
higher speed node motherboard brewster 0 February 11, 2005 09:21:08am
petabox case design zippy 1 January 20, 2005 04:45:56am
   Re: petabox case design CR Saikley 0 January 20, 2005 06:39:57pm
via board being used brewster 0 January 16, 2005 11:01:08pm
Massive network storage 2 Vladovlado 0 December 03, 2004 06:27:35am
Massive network storage Vladovlado 1 December 03, 2004 06:11:27am
   Re: Massive network storage caiser 0 April 05, 2005 04:16:14am
about 400TB of this design shipped to the Archive brewster 5 October 24, 2004 02:06:39am
   Re: about 400TB of this design shipped to the Archive foundation 0 November 10, 2004 11:58:25pm
   Re: about 400TB of this design shipped to the Archive Curator at the Security Digest Archives 1 February 17, 2005 08:02:09am
     Re: about 400TB of this design shipped to the Archive Brak 1 February 17, 2005 01:23:38pm
   Re: about 400TB of this design shipped to the Archive dunno 1 March 03, 2005 06:43:30am
     Re: about 400TB of this design shipped to the Archive brewster 1 March 03, 2005 01:06:12pm
       Re: about 400TB of this design shipped to the Archive jko 1 March 31, 2005 01:41:11pm
         Re: about 400TB of this design shipped to the Archive James Day 3 April 01, 2005 06:47:18pm
           Re: about 400TB of this design shipped to the Archive dunno 1 June 11, 2005 07:45:20am
             Re: about 400TB of this design shipped to the Archive James Day 0 June 11, 2005 12:27:59pm
           Re: about 400TB of this design shipped to the Archive indianews 0 January 21, 2010 07:27:40am
           Re: about 400TB of this design shipped to the Archive viswiss 0 January 23, 2010 07:55:51am
   Re: about 400TB of this design shipped to the Archive matt-genesi-usa 0 June 23, 2005 10:08:34am
   Re: about 400TB of this design shipped to the Archive indianews 0 January 21, 2010 07:28:57am
Serial console cables? garan 1 October 21, 2004 04:27:39pm
   Re: Serial console cables? brewster 2 October 24, 2004 02:13:48am
     Re: Serial console cables? garan 0 October 24, 2004 02:55:50am
     Re: Serial console cables? Casandro 1 March 03, 2005 09:31:58pm
       Re: Serial console cables? sxpert 0 June 21, 2005 07:54:46pm
Possible solution: Disk Array Rackmounts KirbyMeist 1 August 03, 2004 11:10:02am
   Re: Possible solution: Disk Array Rackmounts brewster 1 October 24, 2004 02:20:05am
     Re: Possible solution: Disk Array Rackmounts foundation 1 November 10, 2004 11:47:11pm
       Re: Possible solution: Disk Array Rackmounts HDRJapan 0 October 03, 2007 09:32:55pm
Filesystem KirbyMeist 2 August 02, 2004 02:08:13pm
   Re: Filesystem brewster 3 August 02, 2004 03:11:42pm
     Re: Filesystem grignak 0 October 04, 2004 12:33:58am
     Re: Filesystem foundation 1 November 10, 2004 11:50:03pm
       Re: Filesystem brewster 1 November 11, 2004 12:39:35am
         Re: Filesystem foundation 1 November 11, 2004 01:18:24am
           Re: Filesystem brewster 1 November 11, 2004 01:40:58am
             Re: Filesystem dunno 1 June 16, 2005 01:55:06pm
               Re: Filesystem foundation 0 July 15, 2005 05:08:30am
my shot at massive network storage. dunno 2 May 31, 2004 10:26:51pm
   Re: my shot at massive network storage. dunno 0 June 02, 2004 10:33:13pm
   Re: my shot at massive network storage. andyj 1 June 13, 2004 10:55:07pm
     Re: my shot at massive network storage. foundation 1 November 10, 2004 11:52:59pm
       Re: my shot at massive network storage. angelbassmuffin 0 December 21, 2005 01:11:01pm
Bootstrapping from USB Youssef Eldakar 1 May 24, 2004 11:19:53pm
   Re: Bootstrapping from USB billmoyer 0 May 28, 2004 09:11:07am
1U power supplies Jeff Anderson-Lee 0 May 20, 2004 02:17:52am
Virtual Disk Software? mbuechler 1 May 12, 2004 03:08:32am
   Re: Virtual Disk Software? bgb 1 May 12, 2004 09:13:57am
     Re: Virtual Disk Software? mbuechler 2 May 12, 2004 09:42:25am
       Re: Virtual Disk Software? billmoyer 1 May 12, 2004 11:25:40am
         Re: Virtual Disk Software? mbuechler 0 May 12, 2004 11:37:26am
       Re: Virtual Disk Software? Brak 0 May 12, 2004 11:39:42am
Selective powering of large petasites Richard BBC Archives 2 May 11, 2004 09:01:06pm
   Re: Selective powering of large petasites illtud 1 May 11, 2004 10:03:13pm
     Re: Selective powering of large petasites brewster 1 May 11, 2004 11:12:14pm
       Re: Selective powering of large petasites JTW 1 May 12, 2004 04:42:26pm
         Re: Selective powering of large petasites brewster 0 May 12, 2004 05:32:24pm
   Re: Selective powering of large petasites Rob TNA 1 May 12, 2004 11:07:33pm
     Re: Selective powering of large petasites Richard BBC Archives 0 May 12, 2004 11:34:43pm
Supplier for the 1u racks? mygdal 2 May 11, 2004 05:37:26pm
   Re: Supplier for the 1u racks? Mericomp 1 May 12, 2004 08:51:11am
     Re: Supplier for the 1u racks? Kz 1 May 19, 2004 03:02:47am
       Re: Supplier for the 1u racks? Jeff Anderson-Lee 0 May 20, 2004 06:01:58am
   Re: Supplier for the 1u racks? CR Saikley 1 May 21, 2004 01:54:15am
     Re: Supplier for the 1u racks? zippy 0 January 04, 2005 03:59:40am
How is this filesystem seen? JTW 1 May 11, 2004 04:35:54pm
   Re: How is this filesystem seen? bgb 1 May 12, 2004 09:07:35am
     Re: How is this filesystem seen? Mahmut Kursun 1 May 15, 2004 10:17:44pm
       Re: How is this filesystem seen? dunno 0 May 05, 2005 02:17:54pm
Two words... lauterm 0 May 11, 2004 11:57:17am
NFS? xtat 1 May 11, 2004 05:12:15am
   Re: NFS? billmoyer 0 May 11, 2004 07:47:02am
TWIKI link to collaboration pages bgb 0 April 02, 2004 12:12:31am
More Thermal Testing CR Saikley 1 March 30, 2004 09:43:20am
   Re: More Thermal Testing JamesDay 0 May 22, 2004 12:27:26pm
Airflow Requirements CR Saikley 0 March 29, 2004 08:07:00am
links to 2003 presentation correct typo bgb 0 March 26, 2004 11:20:27am
links to the 2003 presentations bgb 1 March 26, 2004 09:49:10am
   Re: links to the 2003 presentations bgb 0 March 28, 2004 02:29:49am
Costs, Power & Weight CR Saikley 0 March 24, 2004 05:11:49pm
AC Power Input CR Saikley 0 March 24, 2004 01:08:16am
LCD Display Thoughts Brak 0 March 24, 2004 01:00:38am
PetaBox Rack Specifications CR Saikley 0 March 23, 2004 01:43:18pm
PetaBox Block Diagram CR Saikley 0 March 23, 2004 02:04:25am
Via Motherboard Benchmarking CR Saikley 1 March 22, 2004 08:35:52am
Node Thermal Testing CR Saikley 2 March 22, 2004 03:29:51am
   Re: Node Thermal Testing CR Saikley 0 March 23, 2004 09:17:40am
   Re: Node Thermal Testing CR Saikley 0 March 29, 2004 03:08:52am
via nano-itx board announced (really small motherboard) brewster 0 March 19, 2004 02:55:56pm

View more forum posts

Terms of Use (10 Mar 2001)