ESnet4 IP Network and Science Data Network Configuration and Roll Out Progress and Projected Schedule as of July 23, 2007 For more information contact William E. Johnston (wej@es.net), ESnet Dept. Head, Joe Burrescia (joeb@es.net), ESnet General Manager, or Jim Gagliardi (jvg@es.net), ESnet Operations and Deployment Manager 10/30/2007 10:58 PM This talk is available at www.es.net/ESnet4: ESnet4 IP Network and Science Data Network Configuration and Roll Out Progress and Projected Schedule as of July 23, 2007 For more information contact William E. Johnston (wej@es.net), ESnet Dept. Head, Joe Burrescia (joeb@es.net), ESnet General Manager, or Jim Gagliardi (jvg@es.net), ESnet Operations and Deployment Manager 10/30/2007 10:58 PM This talk is available at www.es.net/ESnet4
ESnet4: ESnet4 Internet2 has partnered with Level 3 Communications Co. and Infinera Corp. for a dedicated optical fiber infrastructure with a national footprint and a rich topology - the “Internet2 Network”
The fiber will be provisioned with Infinera Dense Wave Division Multiplexing equipment that uses an advanced, integrated optical-electrical design
Level 3 will maintain the fiber and the DWDM equipment
The DWDM equipment will initially be provisioned to provide10 optical circuits (lambdas - s) across the entire fiber footprint (80 s is max.)
ESnet has partnered with Internet2 to:
Share the optical infrastructure
Develop new circuit-oriented network services
Explore mechanisms that could be used for the ESnet Network Operations Center (NOC) and the Internet2/Indiana University NOC to back each other up for disaster recovery purposes
ESnet4: ESnet4 ESnet is building its next generation IP network and its new circuit-oriented Science Data Network primarily on the Internet2 circuits (s) that are dedicated to ESnet, together with a few National Lambda Rail and other circuits
ESnet will provision and operate its own routing and switching hardware that is installed in various commercial telecom hubs around the country, as it has done for the past 20 years
ESnet’s peering relationships with the commercial Internet, various US research and education networks, and numerous international networks will continue and evolve as they have for the past 20 years
ESnet4: ESnet4 ESnet4 will also involve an expansion of the multi-10Gb/s Metropolitan Area Rings in the San Francisco Bay Area, Chicago, Long Island, Newport News (VA/Washington, DC area), and Atlanta
provide multiple, independent connections for ESnet sites to the ESnet core network
expandable
Several 10Gb/s links provided by the Labs that will be used to establish multiple, independent connections to the ESnet core
currently PNNL and ORNL
Internet2 / Level3 / Infinera Optical Infrastructure: Denver
Front Range GP
1850 Pearl
Level 3 Kansas City
GPN
1100 Walnut
Level 3 Seattle
1000 Denny Way
Level 3 Portland
1335 NW Northrop
Level 3 Sunnyvale
CENIC
1380 Kifer
Level 3 Pacific Northwest GP
2001 6th Ave
Westin Bldg San Diego
8929 Aero Drive
Kearny Mesa, CA Portland
Oregon GP
707 SW Washington
Qwest San Francisco Oakland Sacramento Los Angeles
CENIC
600 W 7th
Equinix Santa Barbara Los Angeles
818 W 7th
Level 3 Salt Lake
Inter-Mountain GP
572 S DeLong
Level 3 Albuquerque
New Mexico GP
104 Gold Ave SE
Level 3 Tulsa
OneNet
18 W Archer
Level 3 Houston
LEARN
1201 N I-45
Level 3 Dallas San Antonio Baton Rouge
LONI
9987 Burbank
Level 3 New Orleans Chicago
600 W Chicago
Level 3 MC Detroit Chicago
CIC/MREN
MERIT
BOREAS
Internet2
710 N Lakeshore
Starlight Atlanta
180 Peachtree St NE
Level 3 MC Birmingham Charlotte Raleigh
NCREN
5301 Departure Dr
Level 3 Pittsburgh
Pittsburgh GP
143 S 25th
Level 3 Washington
MAX
1755 Old Meadow Lane
McLean, VA
Level 3 Philadelphia
MAGPI
401 N Broad
Level 3 New York
NYSERNET
32 Ave of the Americas New York
111 8th
Level 3 Cambridge
NOX
300 Bent St
Level 3 Tampa Miami
South Florida GP
45 NW 5th
Level 3 Orlando Jacksonville
FLR
4814 Phillips Hwy
Level 3 Indianapolis
1902 S East St
Level 3 Louisville
848 S 8th St
Level 3 El Paso
501 W Overland
Level 3 Nashville
Tennessee GP
2990 Sidco Dr
Level 3 Cleveland
TFN
4000 Chester
Level 3 Albany
316 N Pearl
Level 3 Ogden Rieth Boise
435 W McGregor Drive Tionesta Rawlins Omaha St. Louis Eureka Reno Valentine Sanderson Raton Phoenix San Luis Obispo Mobile Tallahassee Syracuse Rochester Cincinnati Austin Tucson Buffalo Rancho De La Fe
(tentative) Edison Atlanta, SLR
345 Courtland Infinera equipment sites
regen site
Internet2 Core Node
Internet2 network DWDM node (common)
DWDM node (ESnet only) other Level 3 node
Level 3 Fiber
extension from Internet2 network to a RON/Core connector node Internet2 / Level3 / Infinera Optical Infrastructure
ESnet IP + SDN 2007 Configuration and Roll Out Jan 1, 2007: ESnet IP + SDN 2007 Configuration and Roll Out Jan 1, 2007 Denver Seattle Sunnyvale LA San Diego Chicago Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Atlanta Nashville (1(3)) (1(12)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (30) (10) (12) (3) (21) (27) (14) (24) (15) First complement of circuits and PoP co-location become available
NLR CHIC-WASH link operational All circuits are 10Gb/s. Existing site supplied circuit Existing NLR circuits Existing site supplied circuit (0) (1) Corrected Map
ESnet IP + SDN 2007 Configuration and Roll Out Target: Apr. 15, 2007: ESnet IP + SDN 2007 Configuration and Roll Out Target: Apr. 15, 2007 Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Atlanta Nashville (1(3)) (1(12)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (30) (10) (12) (3) (21) (27) (14) (24) (15) ESnet equipment installed in Washington, DC, Jan. 31, 2006. COMPLETED, Feb. 27, 2007
ESnet equipment installed in NYC, Jan. 15, 2007 COMPLETED, Jan. 9
WDC-NYC circuits tested and accepted, Feb. 15, 2007. COMPLETED, Mar. 15
ESnet equipment installed in Chicago, Feb. 28, 2007 COMPLETED
NYC-BOS-CLE-CHI circuits tested and accepted, April 15, 2007 COMPLETED, Mar. 15
WDC-CLE-CHI circuits tested and accepted, April 15, 2007. COMPLETED, Apr. 18 All circuits are 10Gb/s. (0) (1) Corrected Map
ESnet IP + SDN 2007 Configuration and Roll Out Target: May 15, 2007: ESnet IP + SDN 2007 Configuration and Roll Out Target: May 15, 2007 Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Atlanta Nashville (1(3)) (1(12)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (30) (10) (12) (3) (21) (27) (14) (24) (15) ESnet equipment installed in Atlanta, Apr. 30, 2007 COMPLETED, Apr. 23
Atlanta – Wash. circuit tested and accepted, May 15, 2007. COMPLETED, May 16
Build ATL MAN, May 1. COMPLETED
Install ORNL, OC48 backup circuit, May 1. Delayed by Qwest install issues All circuits are 10Gb/s. OC48 (0) (1) Corrected Map
ESnet IP + SDN 2007 Configuration and Roll Out Target: Jun. 15, 2007: ESnet IP + SDN 2007 Configuration and Roll Out Target: Jun. 15, 2007 Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Atlanta Nashville (1(3)) (1(12)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (30) (10) (12) (3) (21) (27) (14) (24) (15) ESnet equipment installed in Albuquerque, May 31, 2007; revised est. June 4. Completed June 5
ESnet equipment installed in Denver, May 31, 2007; revised est. June 11. Completed June 12
ATL-ALB circuits tested and accepted, June 15, 2007. Completed June 9
ALB-DEN-CHI circuits tested and accepted, June 15, 2007. Completed June 26 All circuits are 10Gb/s, unless noted. OC48 (0) (1) Corrected Map
ESnet IP + SDN 2007 Configuration and Roll Out Target: July 15, 2007; revised target: August 1: ESnet IP + SDN 2007 Configuration and Roll Out Target: July 15, 2007; revised target: August 1 Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Atlanta Nashville (1(3)) (1(12)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (30) (10) (12) (3) (21) (27) (14) (24) (15) ESnet equipment installed in Sunnyvale, CA, June, 30, 2007; revised est. July 25.
ALB-ELP-SD-SNV circuit tested and accepted, July 15, 2007; revised est. August 1.
SNV-SLC-DEN circuit tested and accepted; revised est. August 1.
OC48 (0) (1) Corrected Map
ESnet IP + SDN Configuration and Roll Out Target: August 15, 2007: ESnet IP + SDN Configuration and Roll Out Target: August 15, 2007 Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Atlanta Nashville (1(3)) (1(12)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (30) (10) (12) (3) (21) (27) (14) (24) (15) ESnet equipment installed in Seattle, July, 31, 2007
DEN-BOI-SEA-SNV circuits tested and accepted, August 15, 2007. OC48 (0) (1) Corrected Map
ESnet IP + SDN Configuration and Roll Out Target: Sept. 15, 2007 (final configuration for 2007): ESnet IP + SDN Configuration and Roll Out Target: Sept. 15, 2007 (final configuration for 2007) Denver Seattle Sunnyvale Chicago Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Nashville (1(3)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (3) (21) (27) (14) (24) (15) ESnet equipment installed in Nashville, Aug. 31, 2007
ATL-NAS-CHI circuits tested and accepted, Sept. 15, 2007. Atlanta OC48 (0) (1) (30) LA San Diego UCSD Corrected Map
ESnet4 Metro Area Rings, 2007 Configurations: ESnet4 Metro Area Rings, 2007 Configurations Denver Seattle Sunnyvale LA San Diego Chicago Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis Atlanta Nashville All circuits are 10Gb/s. MAX West Chicago MAN Long Island MAN Newport News - Elite San Francisco Bay Area MAN OC48 (1(3)) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (3) (21) (27) (14) (24) (15) (0) (1) (30) Uncorrected Map
ESnet4 IP + SDN, 2008 Configuration (Estimated): ESnet4 IP + SDN, 2008 Configuration (Estimated) Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash. DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis (? ) Atlanta Nashville OC48 (0) (1) Contract for 2008-2011 completed by mid-FY2007.
Last three IP circuits (Kansas City-Houston, Chicago-Nashville-Atlanta, and Cleveland-Washington) and associated ESnet equipment, are installed, tested, and accepted.
First SDN ring is completed (Sunnyvale-Atlanta) and associated ESnet equipment is installed, tested, and accepted.
Second SDN northern tier circuits installed, tested, and accepted. (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (3) (21) (27) (14) (24) (15) (30) Chicago StarLight LA San Diego UCSD Corrected Map
ESnet4 Metro Area Rings, 2008 Configurations (Est.): ESnet4 Metro Area Rings, 2008 Configurations (Est.) Denver Seattle Sunnyvale LA San Diego Chicago Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia (1 ) Indianapolis (? ) Atlanta Nashville (2) (13) (21) All circuits are 10Gb/s, or multiples thereof, as indicated. (E.g. 2 = 20 Gb/s., etc.) San Francisco Bay Area MAN West Chicago MAN Long Island MAN Newport News - Elite OC48 (3) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (21) (27) (14) (24) (15) (1) (30) Uncorrected Map
ESnet4 IP + SDN, 2009 Configuration (Est.): ESnet4 IP + SDN, 2009 Configuration (Est.) Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash. DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis (? ) Atlanta Nashville OC48 (0) (1) (3) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (21) (27) (14) (24) (15) (30) LA San Diego UCSD Chicago StarLight Corrected Map
ESnet4 IP + SDN, 2009 Configuration (Est.): ESnet4 IP + SDN, 2009 Configuration (Est.) Denver Seattle Sunnyvale LA San Diego Chicago Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash. DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis (? ) Atlanta Nashville ESnet SDN switch hubs OC48 (0) (1) 3 2 2 2 2 2 2 3 3 3 1 2 2 3 3 3 3 3 2 (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (27) (14) (24) (15) (30) West Chicago MAN Long Island MAN Newport News - Elite San Francisco Bay Area MAN 2 2 (3) (21) Uncorrected Map
ESnet4 IP + SDN, 2010 Configuration (Est.): ESnet4 IP + SDN, 2010 Configuration (Est.) Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash. DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis (>1 ) Atlanta Nashville OC48 (0) (1) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (27) (14) (24) (15) (30) (3) (21) LA San Diego UCSD Chicago StarLight Corrected Map
ESnet4 IP + SDN, 2011 Configuration (Est.): ESnet4 IP + SDN, 2011 Configuration (Est.) Denver Seattle Sunnyvale Raleigh Jacksonville KC El Paso Albuq. Tulsa Clev. Boise Wash. DC Salt Lake City Portland Baton Rouge Houston Pitts. NYC Boston Philadelphia Indianapolis (>1 ) Atlanta Nashville OC48 (0) (1) (7) (17) (19) (20) (22) (23) (29) (28) (8) (16) (32) (2) (4) (5) (6) (9) (11) (13) (25) (26) (10) (12) (27) (14) (24) (15) (30) (3) (21) LA San Diego UCSD Chicago StarLight Corrected Map
ESnet4: ESnet4 Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits), 500-600 Gbps by 2011-2012 (100 Gb/s circuits) Cleveland Europe
(GEANT) Asia-Pacific New York Chicago Washington DC Atlanta CERN (30+ Gbps) Seattle Albuquerque Australia San Diego LA Denver South America
(AMPATH) South America
(AMPATH) Canada
(CANARIE) CERN (30+ Gbps) Canada
(CANARIE) Europe
(GEANT) Asia-Pacific Asia Pacific GLORIAD (Russia and China) Boise Houston Jacksonville Tulsa Boston Science Data Network Core IP Core Kansas City Australia Core network fiber path is ~ 14,000 miles / 24,000 km Sunnyvale IP core hubs SDN hubs USLHCNet