(Originally written for SRI Infotech. Republished with permission.)
We hear a lot about Big Data. But how exactly does it help businesses drive profits? When should a big data project be considered?
According to SRI Engineer Eugene Galchenko, big data makes the most sense when there is a significant amount of data to be stored, and when it’s necessary to run quick calculations on that data. “If organization has 100 terabytes of data or less, they don’t need big data.”
“The goal for big data projects” Eugene says, “is primarily to save money. Big data can be used to calculate results from hundreds of terabytes of data in an acceptable time frame to meet business needs. Or, it can be used to facilitate cheap archiving of petabytes of data which can be subsequently retrieved.”
In the most recent Big Data project undertaken by SRI Infotech, Eugene and his team were asked focus on archiving significant amounts of data generated by Oracle previously stored on Teradata systems. “Apps usually running under Oracle, they produce a lot of data,” Eugene explained.
“Teradata is more expensive for storage,” he continued. “So, we needed to provide an ability to upload some amount of data from Teradata platform to Big Data storage”.
In order to integrate the two storage systems, the SRI big data team developed interfaces on both sides Teradata (SQL\PL) and Big data (shell scripts).
Data Warehousing v. Big Data
However, the data wasn’t merely going into cold storage. Permissioned users needed to access the data for day to day business calculations. “It was necessary to develop interfaces to integrate Teradata and Big Data, so that the GUI was available for both.” SRI developers utilized custom Java development to facilitate reporting functions that users expected.
As this project demonstrates, the biggest challenge in big data projects tends to be application design and architecture. “Big Data platforms provide a lot of different tools which can be used in a number of different ways,” Eugene explained. “If the client wants something in between standard tools” – such as reporting capabilities – “that will add complexity.”
The complexity of big data projects depends on the actual problem an organization is trying to solve. As a result, it is essential to have management support for big data initiatives.
Our successful completion of this project stemmed from the strength of our engineers and developers. “At SRI, we have experience with implementing very special cases of integrations between private cloud storage and big data platforms, Teradata and big data, as well our custom UI development experience. As a result, we can use to suggest a successful strategy for our clients.”