Overview About Dataforest

Project subject: The US eCommerce company wants to unify all of its architecture in one cloud provider to eliminate legacy approaches and unmanaged servers. Due to technical debt and active development in the last 5 years, logging and notification systems have a lot of gaps that do not allow the Client to identify issues and react to them promptly. Challenges: Absence of technical documentation, and outdated business requirements. Complex IT infrastructure that utilizes multiple cloud providers.  Lack of system monitoring and any issue notification.  Lack of a software architect strategy. Results:  The current architecture was analyzed and raised recommendations for:  Single cloud provider taking into account system spread;  CI/CD process for new code deployment;  Server unification;  Security and vulnerability mitigation actions. Developed a strategy for Infrastructure and application scaling. Developed a monitoring and notification unified approach. Zabbix logging is implemented, hence all servers and DBs are covered with monitoring. PagerDuty was settled on the call system. Creating a monitoring system has improved reaction speed and reliability and had a 200% impact on performance. Technologies: GCP, AWS, UBUNTU, PagerDuty, Zabbix, Grafana, MongoDB, PostgresDB.

Project Subject:  ML startup has encountered infrastructure cost issues during the extensive growth. The main goal was to decrease the monthly operational cost ($75 K) for a large data-driven platform that handles  ~ 240 bln entries monthly ( ~ 30TB), storing raw data for 12 months on AWS. Challenges:  Create AWS infrastructure that performs 2k queries per second and has 99.9 % service availability. Optimize redundancy and 2 times decrease in the cost of infrastructure. Create a failover strategy and possibilities for a larger scale. Results: Cost reduction from $75k to $22k per month with performance 30% over SLA. Removed managed services and set up a self-hosted DB over EC2. Used a cluster of servers for DB sharding, adding Elasticsearch, Kafka, and Redis for different streams of data based on industry standards. Create master/master-slave mirroring in different regions to have a failover strategy. DB architecture was tuned to execute most often queries faster. Updated ETL pipelines to reduce the load on DB.  Technologies: AWS EC2, AWS RDS, PostgreSQL, Python, Kafka, Elasticsearch, Redis.

Client: A transnational company focused on the production of FMCG, products of which are represented in about 150 countries around the world. The company owns more than 400 brands. Challenges: The Reporting department (15 employees) receives daily massive amounts of data (invoices, payments, sales reports, etc) from suppliers, vendors, contractors. The data (PDF files, Excel, Google Sheets, etc from more than 100 systems of suppliers, contractors, vendors) is copied manually by the Reporting department. The Reporting department processes unstructured data analyzes information for trends and generates management reports. Operation process optimization and automation by building integration with all data sources. The reporting system for management and stakeholders development. Results: The developed system unifies unstructured data from all the sources of suppliers, vendors, contractors, and stores it in a structured way. The system monitors the data integration and identifies anomalies. The dashboard with multi-level filtering and dynamic mapping: consolidated dashboard for management; goods delivery reporting; KPIs performance control and prompt notification of the indicators deviations.  access for different groups of employees. 900 hours of manual work reduced monthly. Reports for management and stakeholders. Technologies: Core stack - Python, ReactJS, Django, Pandas; Database  - PostgreSQL; Cloud solution: AWS.

Client: Ukrainian distributor for medical devices and drugs.   Challenges:  Reinvent process workflow from legacy (manual, off-line, paper) approach to new ways of working and thinking using digital, social, mobile, and emerging technologies. Improve customer experience and minimize operational risks. Change the way how suppliers, customers, and contractors interact with each other.  Our solution: CRM; Warehouse management system; Products delivery tracking; Suppliers, Customers, and Contractors working panels and dashboards; Built the whole process paperless by implementing a tailor-made web application that includes: the delivery process by integrating the application with a local delivery company; QR code printing and tracking procedures; data-driven reporting. Technologies: Backend: Python /Django; Frontend: React; Database: PostgreSQL; Cloud solution - AWS.

Client: Law Consulting firm in South America. The client automates, manages, classifies, and stores legal case files, documents, and contracts of all kinds. Challenges:  To obtain data from millions of pages and documents through five different court websites and to avoid overloading them.  To collect data on a daily basis from the legal law case files. To collect not only structured data, but also embedded PDF, Word, JPG files, and other unstructured data. To keep scripts running continuously, to carry new case files, and update old ones if something changed.  Results: Created distributed architecture with Linux nodes and a dynamic pipeline that allows managing high peaks and set priorities. We scrape new cases files immediately during the daytime based on the site traffic and make massive updates during the nighttime. To overcome bot protection and to crawl 14.8 million pages daily we use proxies and special AI technologies. Each day we download about 14 Gb of necessary data. To keep data we use a cloud SQL database with daily dumps to Elasticsearch, meantime files directly upload to Elasticsearch. Technologies: Core stack - Python; Distributed task execution - Celery; Database - PostgreSQL; Search-engine - Elasticsearch; Cloud solution: GCP.

Client: Intellidex - financial services and consulting company with operations in South Africa, the UK, and the USA. Project name: Bank Data Analytics Platform Project subject: Develop a web app for the Client’s subscribers to query analytics about various banks. Challenges: Build an interactive B2B web application with custom dashboards and analytics features. Develop a system that interrogates various sets of financial data, calling up time series. Empower application development with AI functionality. Results: Web-native development from scratch with a design framed within the Client's existing website and UI/UX front-end to facilitate queries, algorithms, and visualizations. High-loaded AI scraping algorithms to generate a real-time database of financial bank data from various open-source websites and financial institutions. The developed system has a calculation capacity that enables users to run algorithms on the data, such as comparing, rebasing, etc. and presenting data graphically, and/or downloading it in PDFs. Technologies: Core stack - Python; Database - PostgreSQL; Web stack - Back-end - Python/Django REST framework / Front-end - ReactJS; Cloud solution: AWS.

Client: The Infrastructure Deal Network - IDN Project subject: Develop a deal origination platform for private equity investments in infrastructure-related sectors. Challenges:  Build from scratch a secure interactive B2B platform with sign-up functionality to connect investment firms to proprietary investment opportunities. Build Internal Security Chat for Advisers and Investors. Empower application development with AI functionality. Results: Built a web-native platform for interacting with Advisors, Investors, and Admins, with registration forms, custom dashboards, and the ability to place and conduct investment transactions. Developed and implemented AI matching algorithms in accordance with specified criteria for the optimal selection of transaction parties. Built a secure system for document exchange and internal chat for transaction parties. Technologies: Web stack - Back-end - Python/Django REST Framework / Front-end - ReactJS; Database - RDS (Postgres);  Cloud solution: Amazon.