News

EDI is moving to the cloud!

January 17, 2025

Mark Servilla

Description

The Environmental Data Initiative (EDI) is on a new infrastructure journey, starting in late 2024 and estimated to be completed in early 2025.

Summary:

  • EDI is moving its data repository services from local infrastructure to the Amazon cloud.
  • Going forward, users must sign in to download data from EDI.
  • These changes will lead to improved service and new repository features.

After more than a decade of procuring, deploying, and managing on-premise infrastructure at the University of New Mexico (UNM), we strategically decided to move our compute and storage services to the Amazon Web Services (AWS) cloud last year. Two significant factors led to this decision. First, our vision to remain a leading data repository for environmental data worldwide means that we have to manage data at scale and do this economically. Our cost-benefit analysis indicated that we could move to AWS with a similar investment as would be required for provisioning new on-premise hardware but with the additional perk of allowing us to scale both compute and storage as needed, on-demand and in real-time – a feat not easily accomplished in the institutional setting. Second, AWS provides us with more reliability than our institutions can offer. Because of the built-in redundancy within AWS, EDI infrastructure will be more resilient to data center outages, network congestion, and overall demand for our services, providing a better experience for our community.

Timeline

We are now in the final stages of this transition. Our development and staging environments are fully operational within AWS, and both are using AWS S3 object storage for existing and recently published data. We have also replicated all 15 Terabytes of data from our production environment at UNM to AWS S3 and are currently moving the remaining compute services. We plan to perform our production environment's final "lift and shift" on Wednesday, January 29th. We will bring our systems offline for the entire day to ensure a smooth and complete migration and plan to have all services back online early Thursday, January 30th.

How does this change affect you?

As mentioned above, EDI's infrastructure move to AWS will ultimately result in greater stability and service performance. It will also allow us to adapt more readily to changing demand and open up an environment where we can quickly experiment with new technology to improve our service portfolio. Operationally, this change will be seamless for users, with one exception: EDI will require users to sign in to download data from our Data Portal and REST API. This change will allow EDI to identify the composition of our user community and how they use the data we manage, ensuring the content and our services align with their needs. Additionally, it will provide EDI with a mechanism to track costs related to network data egress from the AWS cloud, as unrestricted access to data by scrapers and robots makes sustainability more difficult. Those who use our REST API to download data will have to use their EDI credentials directly in the HTTP request or provide an EDI authentication token in the request cookie (technical documentation will follow shortly).

This is an exciting time for EDI and our user community. The future holds additional changes, including new features and improved performance, enhancing your experience working with the EDI data repository. We look forward to serving you in 2025 and beyond.