ews
Contact Us

Data Portal

 Government – Opportunity, Spend and Grants

Client Challenge

To resolve these challenges, EWS implemented a scalable data management solution capable of handling millions of daily transactions across diverse formats like databases, APIs, portals, and OData sources. 

  • An extensible ETL solution that ingests and processes data from multiple sources​
  • A one-stop online solution to view calendars, legislation, congress and senate activities, opportunities, spend, and grant data​
  • Solution intended for legislators, news media, traders, and others seeking fast access to public sector data​

Desired Solution

  • Unsupervised operations and minimal human intervention
  • Ability to process multiple datasets in parallel with millions of transactions per run
  • Frequent data updates from sources on regular intervals
  • Unsupervised operations and minimal human intervention​

EWS Solution

EWS built a scalable Data Lake with automated ingestion, transformation, and incremental updates, enabling advanced aggregation and efficient data presentation in Rails.

Data Lake Architecture

Adopted Data Lake design pattern to manage structured and unstructured datasets

Robust Data Ingestion Framework

Data ingestion (including screen-scraping) implemented using Kafka, Python, and Informatica

Multi-Stage Data Transformation

Multiple stages of data transformation and cleansing to ensure consistency and reliability

Layered Data Loading Strategy

Support for Historical, Daily, and Weekly data loads with incremental data updates

Analytical Aggregation Capabilities

Data aggregation designed to support slicing and dicing for deeper analysis

Web-Based Data Presentation

Final data presentation delivered through a Rails-based front-end interface