An Arizona State Department Safeguards Oracle E-Business Suite with Tessell

The State Agency

Arizona Department of Environmental Quality, a prominent state agency in the United States is dedicated to preserving and advancing public health and environmental welfare. The agency operates a diverse database infrastructure, including Oracle 19c, SQL Server, MongoDB, and others. The agency Initially managed these resources internally across multiple teams. Notably, the cornerstone Oracle E-Business Suite (EBS), demands constant availability, robust backup measures, and peak performance to uphold critical services and support residents effectively. Alongside database management, the agency’s IT personnel also undertake application development to enhance residence services. 

Facing escalating demands for performance, scalability, and high availability, the agency meticulously evaluated various solutions, including different database-as-a-service (DBaaS) options. Following an exhaustive Proof-of-Concept (POC) phase, the agency strategically opted to transition from self-managed AWS EC2 instances to the Tessell DBaaS Platform on Amazon Web Services (AWS).

The Challenges

Before adopting Tessell’s DBaaS from the AWS Marketplace, the agency managed all of its databases on AWS, including its Oracle E-Business Suite (EBS), an enterprise resource planning (ERP) software application suite reliant on an Oracle database, which serves as the backbone of the agency’s operations. Initially considering AWS RDS as a fully managed DBaaS solution, the agency encountered limitations within the RDS architecture that rendered it unsuitable for EBS. Consequently, the agency sought alternatives to address challenges related to High Availability, Disaster Recovery, Backup, Manual processes, and Security.

High Availability and Disaster Recovery

Lack of resiliency and recoverability posed a significant challenge and unacceptable risk, particularly with EBS. Managing EBS for high availability in a self-managed AWS environment demanded intricate configurations, including separate EC2 instances for the application and Oracle database tiers, deployment across multiple availability zones, auto-scaling mechanisms, Oracle Data Guard replication, AWS Elastic Load Balancing, Elastic File System utilization, and AWS CloudWatch for monitoring. The agency lacked the financial resources to implement automated failover mechanisms, leaving it vulnerable to service disruptions in the event of instance failures. 

In the absence of automated failover, setting up a new recovery instance following a failure was a laborious and manual process. The IT team estimates that in the event of a disaster, the recovery of EBS and production databases could take up to eight hours to complete. While automation scripts existed, the end-to-end process was error-prone and challenging to validate during business continuity simulations.

Backups and Data Protection

The backup process posed significant challenges, particularly for EBS and other large databases within the agency’s infrastructure. Managing backups, especially for critical systems like EBS, proved time-consuming, resource-intensive, and expensive.

To address this, the agency utilized third-party software for backing up EBS and other production databases, typically performing backups once a day, with less frequent backups for downstream environments. However, this approach left the agency vulnerable to potential data loss in the event of system failures or disasters.

In a worst-case scenario, the agency faced the risk of losing up to 3 hours worth of data, highlighting the critical need for a more efficient and reliable backup solution.

Manual Processes

The agency grappled with numerous manual processes, particularly concerning the creation of database copies and the cloning of production servers. These manual tasks not only consumed valuable time and resources but also introduced the potential for human error and inconsistency in system configurations.

Patching the customer-facing portal posed another significant challenge, requiring at least 30 minutes of downtime for maintenance activities. This downtime not only impacted user experience but also highlighted the need for streamlined and automated patching processes to minimize service disruptions.

Moreover, the lack of high availability exacerbated the complexity of recovery processes, further prolonging downtime and impeding the agency’s ability to swiftly restore critical services in the event of system failures.

As the agency’s infrastructure and database count grew, managing users and permissions became increasingly challenging. With manual user management processes in place, maintaining adequate access controls and permissions across various databases and applications proved cumbersome and error-prone. This complexity hindered operational efficiency and heightened security risks within the agency’s IT environment.

Security

The agency’s IT environment encountered challenges related to self-management and infrastructure allocation, prompting a reassessment of security protocols. The  Chief Information Security Officer (CISO) devoted considerable efforts to ensure compliance with state audit requirements, particularly regarding data encryption. However, there remained a need for comprehensive evaluations of Oracle databases to align with industry standards. Additionally, streamlining patch management processes between applications and infrastructure was identified as an area for improvement to bolster the overall security posture.  

The Solution

Tessell provided a comprehensive solution to address the agency’s challenges.

Seamless Zero Downtime Migration

A meticulous migration plan ensured minimal downtime during the transition phase. Despite EBS only supporting Oracle Version 12, Tessell adeptly utilized the latest Oracle versions while maintaining support for version 12 through innovative features. 

Working closely with the agency’s database team, Tessell successfully migrated all Oracle databases, including E-Business Suite, with minimal disruption to operations.

High Availability

Tessell implemented High Availability across Availability Zone deployments, ensuring 24/7 availability for EBS and other production databases. HA was also extended to the UAT instance of EBS, enhancing resilience and minimizing downtime.

Performance

To optimize performance, Tessell migrated EBS from shared server infrastructure to a dedicated EC2 instance, ensuring dedicated resources and improved performance. Tessell also integrates with the agency’s primary monitoring platform.

Operational & Lifecycle Management

The Tessell GUI serves as a centralized platform for monitoring EBS database performance and managing alert logs, streamlining operational management tasks, and enhancing visibility into system health.

Backup

In addition to handling database and OS patching, Tessell provides continuous backup for all databases, including both production and UAT environments, ensuring zero data loss and facilitating rapid recovery in the event of system downtime.

The Results

"I am grateful for Tessell production support availability 24X7 at our terms. Tessell completed the migration ahead of schedule and has even helped us with unrelated issues."
-- Senior DBA

The migration of the Oracle databases (to start) to Tessell DBaaS on AWS produced significant outcomes:

Enhanced Availability

• Achieved 99.99% high availability, minimizing planned downtime.
• Implemented automatic failover to a reserved instance in the event of production server failure, ensuring uninterrupted service and zero data loss.
• Implemented High Availability (HA) for Oracle E-Business Suite. 

Performance

• While specific comparison metrics are pending, the agency DBA staff noted significant performance improvements with Tessell for EBS.

Backup

• Enabled continuous backup for EBS and all production and secondary environment Oracle databases, ensuring zero data loss.
• Facilitated instant data restoration by authorized users down to the second without requiring a DBA intervention.
• Implemented Point-in-Time Recovery (PITR) for any point in time during the last 35 days

Security

• Implemented default encryption for data-at-rest and data-in-transit, enhancing data security.
• Ensured up-to-date patching.
• Enabled automated data masking using scripting.

Manual Processes

• Automated manual processes and streamlined data management.
• Reduced downtime for patching the customer portal to less than one minute through rolling clustered patching.
• Enabled 1-Click patching for infrastructure and applications, with automated roll-back using pre-patch snapshots if needed.

Oracle Best Practices

• Implemented industry best practices integrated into the Tessell platform and applied during migration.
• Consolidated what normally would have required 12 servers down to 6 without service disruption using Tessell functionality.

Customer Satisfaction

• CISO praised Tessell’s security advantages, anticipating smoother state audits.
• DBAs’ appreciation for the time saved on administrative tasks.

Cost Efficiency

• Eliminated backup software requirements, reducing costs.
• Eliminated manual Oracle and OS patching, boosting team morale and freeing up time for strategic initiatives.
• Mitigated downtime risk and costly disaster recovery measures, promoting better sleep for all stakeholders.

Conclusion

Tessell's proficiency in database migration and optimization has proven instrumental in the transformation of the agency’s Oracle database environment, notably in fortifying Oracle E-Business Suite and critical databases.  The migration to Tessell DBaaS on AWS has delivered lasting benefits in availability, security, and cost-efficiency, equipping the agency with enhanced resilience and operational effectiveness to fulfill its mission.  Through Tessell’s solution, the agency is better positioned to navigate evolving challenges and advance its objectives with confidence and agility.

‍